[Catalog-sig] start on static generation, and caching - apache config.

[Jim Fulton]

On Jul 11, 2007, at 3:03 PM, Phillip J. Eby wrote:

> At 02:25 PM 7/11/2007 -0400, Benji York wrote:
>> Martin v. Löwis wrote:
>>> Benji York schrieb:
>>
>>>> Is your position that PyPI isn't down/very slow on occasion or  
>>>> that when
>>>> it is no one complains?
>>>
>>> Both. I believe it shouldn't be down
>>
>> The cheeseshop has provided its own proof that that believe is  
>> mistaken
>> by being down as I began composing this message. <wink>
>>
>>> Jim Fulton complained that it took 0.3s to
>>> get a single package's page, which I cannot classify as "very slow".
>>
>> During a single run setuptools or zc.buildout may make hundreds of
>> requests to the cheeseshop taking a total time in the minutes.   
>> That's
>> not fast enough.  I can't see a technical reason why these requests
>> couldn't be handled much faster than 3 a second.
>
> An interesting thought for future optimization...  an XML-RPC catalog
> server designed for this use case could in fact do all the
> computation server-side, resolving dependencies and evaluating
> version constraints.  Heck, in theory, it could cache packages'
> external links, and simply hand back to the caller a complete list of
> candidate URLs to choose for downloading.  That way, most activities
> would take only one server round-trip to complete, if the client sent
> a list of everything it expects to need, and the server includes
> everything that the server expects the client to want due to those
> things' dependencies.

That wouldn't help when local (e.g. development) or private  
distributions need to be included in the mix.

I think collecting all of the links for a package that PYPI knows  
about on individual package pages would go a very long way to  
reducing the number of requests.  If these pages were served  
statically (or in similar times), then I think we'd be in very good  
shape.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org