[Catalog-sig] Cheese Shop performance

[Richard Jones]

I've been quiet on the latest happenings because I've have my attention wholly 
consumed by PyWeek and gettng a new job.

The performance issues are related to spiders crawling the wiki and the Cheese 
Shop "browse" interface. The PyPI code attempts to prevent bots from 
accessing that interface, but that requires that we know user-agent strings 
in advance. And people are writing newer, dumber bots all the time :(

There's a very good chance that I'll be able to do some work over the next few 
weeks on the baking ideas that Jim Fulton proposed at PyCon. This would make 
most of the user interface (except the browse) static where it currently goes 
through CGI for all requests. In that situation I'm not sure we'll need to 
have mirrors, though it would certainly help if they were needed.


    Richard