[issue13924] Mercurial robots.txt should let robots crawl landing pages.

[Benjamin Peterson]

Benjamin Peterson added the comment:

Unfortunately, I don't think it will be that easy because I don't think robots.txt supports wildcard paths like that. Possibly, we should just whitelist a few important repositories.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13924>
_______________________________________