[Catalog-sig] start on static generation, and caching - apache config.

[Jim Fulton]

On Jul 12, 2007, at 5:09 PM, Martin v. Löwis wrote:
...
>> I wonder if it would help much to make multiple HTTP requests in the
>> same connection.  This might be something to look at in setuptools
>> and/or xmlrpclib.
>
> Only for remote connections, due to the round-trips required for
> TCP handshake. Locally, Apache opens a new connection to the FCGI
> servers per requests (using the farmer-worker pattern).

Right, but most connections will be remote, so this is a potential win.

>
>> 1. Since many people will be far away from PyPI, I think our long-
>> term plan should encompass geographic mirrors.  It's good that the
>> server is spending a small amount of time, but it still takes *me* a
>> long time to get data.
>
> Ok. I am, in general, skeptical about mirroring. However, if it
> makes people happy, feel free to implement it.

My goal is to have PyPI provide a simplified version of the data for  
use by setuptools that is easily mirrored using standard mirroring  
tools.  (I may actually prototype this with a kind of mirror.)

> A number of issues should be considered, of course:
> - there should be a way to get authoritative answers somehow,  
> preferably
>   from mirrors, but, if necessary, from the main site

I don't know what you mean.  I envision mirrors as being read-only  
and only used by setuptools. The main site would certainly be  
authoritative.

> - I really wish to collect download counters across mirrors.  
> "Official"
>   mirrors should be obliged to report download statistics once a day
>   or so.

OK.

>
>> 2. It's important to reduce the number of round trips.
>
> A colleague today suggested that the best way to reduce round trips
> is to give each machine a local copy of the index, the same way
> Debian apt works: you do 'apt-get update', and then have a local
> copy of the catalog that you can build against. No roundtrips
> at all (except for the one to update the local catalog), for the
> expense of being out of date if you don't manually update the
> catalog.

Yup. This might be a really nice way to go. It would be especially  
nice if a client could contact PyPI and ask for new data since a  
given time.  I imagine that this request could be as cheap as the  
requests we have now, unless a client was very out of date.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org