[Python-Dev] Googlebot and the mail.python.org python-dev archive

[Antoine Pitrou]

Georg Brandl <g.brandl <at> gmx.net> writes:
> 
> Guido van Rossum schrieb:
> > I think the better syntax would be to add site:mail.python.org to the
> > query, but you're right, that doesn't seem to find recent messages.
> > Maybe the absence of a robots.txt file on mail.python.org could be a
> > partial explanation?
> 
> Doesn't the absence of a robots.txt mean "you may index everything"?

It does.
However, pages such as:
    http://mail.python.org/pipermail/python-dev/
(and, it seems, all other pipermail-generated archive pages)
have the following HTML tag in them:
    <META NAME="robots" CONTENT="noindex,follow">
which explicitly instructs Web spiders *not* to index contents nor follow links.

Regards

Antoine.