[Catalog-sig] start on static generation, and caching - apache config.

[Jim Fulton]

On Jul 8, 2007, at 1:27 PM, Phillip J. Eby wrote:

> At 01:48 PM 7/7/2007 -0400, Jim Fulton wrote:
>
>> On Jul 7, 2007, at 12:43 PM, Martin v. Löwis wrote:
>> ...
>> > I'm quite skeptical on caching in general (even about the static  
>> page
>> > generation). It *should* be possible to make it fast enough so that
>> > it doesn't need caching.
>>
>> Sure, with more hardware than we want to afford.
>>
>> > I consider caching a work-around, not a
>> > solution - and one with severe drawbacks.
>>
>> The pages we're talking about are static.  They change at well-known
>> times. IMO, It's crazy to serve static content dynamically when it's
>> easy to serve it statically.
>
> If they're effectively static, why can't Apache cache them?   
> Shouldn't we be able to simply add Last-Modified/If-Modified  
> support to the PyPI output, and enable Apache's disk caching for  
> non-logged-in users?

When caching something, you typically specify a age before you start  
checking. That means that content would be stale for that period.   
Sometimes, that is both acceptable and necessary.  In any case,  
dynamic servers typically take just as long to handle an If-Modified  
or Last-Modified request than they do to handle a regular request. It  
would be just as complicated, if not more so, to get the cheeseshop  
software to do this properly than it would to just bake.



> That is, as long as there is a quick last-modified-time query for a  
> package, we can use those to process the If-Modified header.  The  
> modification time could even be memcached, so as not to need a  
> database hit 99% of the time.

No, it can't be cached.  What would you do to make sure that cache  
wasn't stale.

> While that's not necessarily as fast as static page generation,  
> it's a lot less complex to get right, and it saves the main piece  
> of CPU load: i.e., doing SQL queries and actually generating the page.

It is really easy to get static page generation right for an  
application this simple.  YOu know when pages are invalidated.  The  
page relationships are not at all complicated here.


> Pages that pertain to more than one package might be a bit more  
> complex to do this on, but if I understand correctly it's mainly  
> the package-specific pages we're concerned with here, correct?

Yes, and http://www.python.org/pypi/

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org