[issue13924] Mercurial robots.txt should let robots crawl landing pages.

[Ivaylo Popov]

New submission from Ivaylo Popov <popov.ivo at gmail.com>:

http://hg.python.org/robots.txt currently disallows all robots from all paths. This means that the site doesn't show up in Google search results seeking, for instance, browsing access to the python source
https://www.google.com/search?ie=UTF-8&q=python+source+browse
https://www.google.com/search?ie=UTF-8&q=python+repo+browse
https://www.google.com/search?ie=UTF-8&q=hg+python+browse
etc...

Instead, robots.txt should allow access to the landing page, http://hg.python.org/, and the landing pages for hosted projects, e.g. http://hg.python.org/cpython/, while prohibiting access to the */rev/*, */shortlog/*, ..., directories.

This change would be very easy, cost virtually nothing, and let users find the mercurial repository viewer from search engines. Note that http://svn.python.org/ does show up in search results, as an illustration of how convenient this is.

----------
components: None
messages: 152446
nosy: Ivaylo.Popov
priority: normal
severity: normal
status: open
title: Mercurial robots.txt should let robots crawl landing pages.
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13924>
_______________________________________