[Catalog-sig] [Distutils] Time for a static cheeseshop mirror for easy_install?
Phillip J. Eby
pje at telecommunity.com
Mon Apr 9 20:24:02 CEST 2007
At 10:46 AM 4/9/2007 -0400, A.M. Kuchling wrote:
>On Sat, Apr 07, 2007 at 12:44:56PM -0400, Phillip J. Eby wrote:
> > I don't know whether this will actually solve any problems the cheeseshop
> > itself is having; it may be that ill-behaved web spiders are at fault, or
> > something else altogether.
>In this recent case, two different spiders were crawling the wiki very
>quickly, the machine's load average was in the 70s, and the
>out-of-memory killer was killing off PostgreSQL processes.
>I don't think the load caused from people running easy_install is
>especially high -- it's certainly not a source of problems -- but
>making static pages would still be good to make mirroring the package
>archive more useful. Right now people could mirror
>http://cheeseshop.python.org/packages/, but there's nothing there for
>easy_install or for human readers; it's just a tree of package
Hm. Well, actually, if that directory structure were something I could
code to, easy_install could sure as heck be *made* to use it. The only
thing easy_install couldn't get from it was external links to downloads,
SVN versions, etc.
Notice, for example, that if you use "easy_install -f
easy_install won't look at the main package index, but just download
directly. So an automated form of that calculation could easily be added
What I had in mind for an easy_install mirror, however, was a script that
would just create a /packagename/index.html file with links gathered from
all versions of a package on the original Cheeseshop, and with packagename
generated as a setuptools "safe name" in lower case. This pattern would
allow easy_install to find every possible relevant link in just one
(static) web hit.
More information about the Catalog-sig