Feb. 28, 2009
10:37 a.m.
Georg Brandl <g.brandl <at> gmx.net> writes:
Guido van Rossum schrieb:
I think the better syntax would be to add site:mail.python.org to the query, but you're right, that doesn't seem to find recent messages. Maybe the absence of a robots.txt file on mail.python.org could be a partial explanation?
Doesn't the absence of a robots.txt mean "you may index everything"?
It does. However, pages such as: http://mail.python.org/pipermail/python-dev/ (and, it seems, all other pipermail-generated archive pages) have the following HTML tag in them: <META NAME="robots" CONTENT="noindex,follow"> which explicitly instructs Web spiders *not* to index contents nor follow links. Regards Antoine.