[Mailman-Users] How to block spiders?
Skip Montanaro
skip at pobox.com
Wed Jan 21 17:41:56 CET 2004
Paul> Normally with an apache web server you can just place a robots.txt
Paul> file in the root of the documents directory. Not sure how one
Paul> might be able to do this with mailman.
Paul> When you take a look at the archives for a list, one sees
Paul> http://server-name/pipermail/list-name/
Paul> Since there isn't a pipermail directory, one can't just place a
Paul> robots.txt file in that directory.
The robots.txt file only belongs in the server root. Placing it lower in
the hierarchy isn't supported, and I doubt any spiders/crawlers would look
for it lower down. Instead, add a Disallow: rule which references
/pipermail:
User-agent: *
Disallow: /pipermail/
--
Skip Montanaro
Got gigs? http://www.musi-cal.com/submit.html
Got spam? http://spambayes.sf.net/
skip at pobox.com
More information about the Mailman-Users
mailing list