[Catalog-sig] Cheese Shop Meltdown

Rob Cakebread gentoodev at gmail.com
Thu Jun 14 22:35:44 CEST 2007

Until the problems with "dumb spiders"[1] are worked around, couldn't
something simple
be done such as using a spider trap[2]?

Add this to robots.txt:

User-agent: *
Disallow: /pypi?:action=bot_ban

Add a hidden link on the front page, then grab the ip of the offending
spider and block it by adding
it to .htacess

Just a thought, as PyPI has been painfully slow for hours at a time lately.

[1] http://mail.python.org/pipermail/catalog-sig/2007-April/001055.html
[2] http://danielwebb.us/software/bot-trap/

More information about the Catalog-sig mailing list