[Catalog-sig] start on static generation, and caching - apache config.
Phillip J. Eby
pje at telecommunity.com
Wed Jul 11 21:03:12 CEST 2007
At 02:25 PM 7/11/2007 -0400, Benji York wrote:
>Martin v. Löwis wrote:
> > Benji York schrieb:
>
> >> Is your position that PyPI isn't down/very slow on occasion or that when
> >> it is no one complains?
> >
> > Both. I believe it shouldn't be down
>
>The cheeseshop has provided its own proof that that believe is mistaken
>by being down as I began composing this message. <wink>
>
> > Jim Fulton complained that it took 0.3s to
> > get a single package's page, which I cannot classify as "very slow".
>
>During a single run setuptools or zc.buildout may make hundreds of
>requests to the cheeseshop taking a total time in the minutes. That's
>not fast enough. I can't see a technical reason why these requests
>couldn't be handled much faster than 3 a second.
An interesting thought for future optimization... an XML-RPC catalog
server designed for this use case could in fact do all the
computation server-side, resolving dependencies and evaluating
version constraints. Heck, in theory, it could cache packages'
external links, and simply hand back to the caller a complete list of
candidate URLs to choose for downloading. That way, most activities
would take only one server round-trip to complete, if the client sent
a list of everything it expects to need, and the server includes
everything that the server expects the client to want due to those
things' dependencies.
The main obstacle to implementing such a service today, is that it
would have no way of knowing what dependencies to look for, without
sniffing the contents of .egg files. But, as long as a superset of
possible dependencies was listed in PKG-INFO, the server could make
intelligent guesses about what other packages are likely to be
needed, and return their version/download info as well. Returning
information for packages that turn out not to be needed is likely to
be far less expensive than having to make round-trip requests.
An alternative to providing that information from metadata, of
course, would be for the client to include a "referrer" header of
sorts, saying why it is asking for a package. The server could then
simply "learn" the relevant associations.
More information about the Catalog-SIG
mailing list