[Catalog-sig] start on static generation, and caching - apache config.
Phillip J. Eby
pje at telecommunity.com
Wed Jul 11 21:03:12 CEST 2007
At 02:25 PM 7/11/2007 -0400, Benji York wrote:
>Martin v. Löwis wrote:
> > Benji York schrieb:
> >> Is your position that PyPI isn't down/very slow on occasion or that when
> >> it is no one complains?
> > Both. I believe it shouldn't be down
>The cheeseshop has provided its own proof that that believe is mistaken
>by being down as I began composing this message. <wink>
> > Jim Fulton complained that it took 0.3s to
> > get a single package's page, which I cannot classify as "very slow".
>During a single run setuptools or zc.buildout may make hundreds of
>requests to the cheeseshop taking a total time in the minutes. That's
>not fast enough. I can't see a technical reason why these requests
>couldn't be handled much faster than 3 a second.
An interesting thought for future optimization... an XML-RPC catalog
server designed for this use case could in fact do all the
computation server-side, resolving dependencies and evaluating
version constraints. Heck, in theory, it could cache packages'
external links, and simply hand back to the caller a complete list of
candidate URLs to choose for downloading. That way, most activities
would take only one server round-trip to complete, if the client sent
a list of everything it expects to need, and the server includes
everything that the server expects the client to want due to those
The main obstacle to implementing such a service today, is that it
would have no way of knowing what dependencies to look for, without
sniffing the contents of .egg files. But, as long as a superset of
possible dependencies was listed in PKG-INFO, the server could make
intelligent guesses about what other packages are likely to be
needed, and return their version/download info as well. Returning
information for packages that turn out not to be needed is likely to
be far less expensive than having to make round-trip requests.
An alternative to providing that information from metadata, of
course, would be for the client to include a "referrer" header of
sorts, saying why it is asking for a package. The server could then
simply "learn" the relevant associations.
More information about the Catalog-SIG