Patch 1.26a has an inherent custom map size limit of 8 MB. If you try to load newer modified DotA maps that exceed this size, the game will kick you back to the main menu. To fix this, you must download a community-made game.dll file that removes the 8 MB cap, or play older, optimized map versions.
How to Install Warcraft III: The Frozen Throne & DotA 1 (Patch 1.26a) warcraft iii the frozen throne dota 126 install
Since version 1.26a is no longer the official live version, you must use a manual patcher: Patch 1
Check the bottom-right corner of the main menu. It should read . or play older