[Catalog-sig] Cheese Shop Meltdown
gentoodev at gmail.com
Thu Jun 14 22:35:44 CEST 2007
Until the problems with "dumb spiders" are worked around, couldn't
be done such as using a spider trap?
Add this to robots.txt:
Add a hidden link on the front page, then grab the ip of the offending
spider and block it by adding
it to .htacess
Just a thought, as PyPI has been painfully slow for hours at a time lately.
More information about the Catalog-sig