[Catalog-sig] start on static generation, and caching - apache config.

[Jim Fulton]

On Jul 8, 2007, at 12:07 PM, Martin v. Löwis wrote:

>>> So you are saying it's not fast enough already?
>>
>> Uh, yeah.
>
> Can you please be more precise, then? What kind of operation are
> you performing,

I'm using setuptools.  Sertuptools looks at package pages (e.g.  
http://www.python.org/pypi/foobar), it looks at:
http://www.python.org/pypi/ and it doenloads distributions. (AFAICT,  
the later is done dynamically too, which
is especially insane.)

> how long does it take,

Lately, it's has often taken minutes.  This has been the major  
problem.  At the best of times. well, I don't know when those are. :)

ATM, requests for http://www.python.org/pypi/zc.buildout takes about  
1/3 second.  Requests for http://cheeseshop.python.org/packages/2.5/z/ 
zc.buildout/zc.buildout-1.0.0b28-py2.5.egg take about 2.5 seconds.  
Requests for http://www.python.org/pypi/ take about 10 seconds.

I would say that these times are too long.


> and how long should it
> take so that you would consider it fast enough?

IMO, it needs to be much much faster.  If we were serving pages  
staticially, we would be able to serve thousands of requests per  
second.  There's nothing about this application that would make doing  
that hard.

> It's difficult to implement a system if the requirements are
> unknown to those implementing it.

I'm sorry, I've been talking about setuptools all along.  I thought  
the use case was understood.  Also, I thought it was pretty obvious  
that the performance we've been seeing lately is totally  
unacceptable.  It's hard to pinpoint exactly what the acceptable  
performance will be, in part because, we we do better, demand will  
increase.  Note that, as it is now, demand is possibly decreasing  
because people are building their own indexes.

If this was an application that had to be served dynamically (and of  
course, parts of it are), then it would be much more interesting to  
discuss targets for dynamic delivery.  The performance-critical parts  
of this application -- the pages that setuptools uses, can readily be  
served statically, so it makes no sense not to do so.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org