[Catalog-sig] start on static generation, and caching - apache config.

[Jim Fulton]

On Jul 12, 2007, at 1:11 AM, Martin v. Löwis wrote:

>> ATM, from my machine, xml-rpc requests to PyPI are taking about .27
>> seconds.  This is only a little less than regular page requests.   
>> With
>> the current API, It would require at best 3 requests to get all of  
>> the
>> distribution URLs.  Presumably, with a change to the API, we could  
>> get
>> this down to one request, but that's still a long time given the  
>> demand
>> I expect on PyPI in the future.
>
> You seem to assume that if you see a round trip time of .27 seconds,
> that then PyPI could only do 3 requests per second. That is not so.

Yeah, it occurred to me on my way home that a substantial part of the  
time might be due to distance.

I wonder what times ab against  http://www.python.org/pypi/ZODB3 from  
inside the python.org network would give.
I wonder if it would help much to make multiple HTTP requests in the  
same connection.  This might be something to look at in setuptools  
and/or xmlrpclib.

....

> So it can currently do 130 XML-RPC requests per second, to
> a single client. Inverting it, a request takes 0.0076s,
> which is a lot less than 0.27s.

Cool. That's much better. Thanks for trying this.

OTOH, this points up a couple things:

1. Since many people will be far away from PyPI, I think our long- 
term plan should encompass geographic mirrors.  It's good that the  
server is spending a small amount of time, but it still takes *me* a  
long time to get data.

2. It's important to reduce the number of round trips.

I'm still opposed to using XML-RPC because:

- It's harder to mirror, and

- It's still slower than static pages.

Note that after our discussion, I'm equally against the current  
approach of parsing a human interface.  I still think it makes a lot  
more sense to have a tailored interface for setuptools.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org