[Catalog-sig] [Distutils] Time for a static cheeseshop mirror for easy_install?

[Phillip J. Eby]

At 10:46 AM 4/9/2007 -0400, A.M. Kuchling wrote:
>On Sat, Apr 07, 2007 at 12:44:56PM -0400, Phillip J. Eby wrote:
> > I don't know whether this will actually solve any problems the cheeseshop
> > itself is having; it may be that ill-behaved web spiders are at fault, or
> > something else altogether.
>
>In this recent case, two different spiders were crawling the wiki very
>quickly, the machine's load average was in the 70s, and the
>out-of-memory killer was killing off PostgreSQL processes.
>
>I don't think the load caused from people running easy_install is
>especially high -- it's certainly not a source of problems -- but
>making static pages would still be good to make mirroring the package
>archive more useful.  Right now people could mirror
>http://cheeseshop.python.org/packages/, but there's nothing there for
>easy_install or for human readers; it's just a tree of package
>directories.

Hm.  Well, actually, if that directory structure were something I could 
code to, easy_install could sure as heck be *made* to use it.  The only 
thing easy_install couldn't get from it was external links to downloads, 
SVN versions, etc.

Notice, for example, that if you use "easy_install -f 
http://cheeseshop.python.org/packages/source/s/simple_json/ simple_json", 
easy_install won't look at the main package index, but just download 
directly.  So an automated form of that calculation could easily be added 
to easy_install.

What I had in mind for an easy_install mirror, however, was a script that 
would just create a /packagename/index.html file with links gathered from 
all versions of a package on the original Cheeseshop, and with packagename 
generated as a setuptools "safe name" in lower case.  This pattern would 
allow easy_install to find every possible relevant link in just one 
(static) web hit.