[Catalog-sig] start on static generation, and caching - apache config.

[Phillip J. Eby]

At 07:37 PM 7/8/2007 +0200, Martin v. Löwis wrote:
> > If they're effectively static, why can't Apache cache them?
>
>That's easy to answer: nobody told Apache to do that
>(and I don't know how to tell it to).
>
>René's approach currently is to generate the files explicitly
>on disk, and then have Apache return them always from disk.
>
> > Shouldn't
> > we be able to simply add Last-Modified/If-Modified support to the PyPI
> > output, and enable Apache's disk caching for non-logged-in users?
>
>How precisely would that work? I.e. what software should put what
>header into what place, and how would the cache then find out that
>the real data have changed?

I was under the impression that when Apache caching is enabled, it 
can add an If-Modified-Since header to incoming requests, and in the 
event that the dynamic content hasn't changed, use its cached version 
of the response.  I am not an expert on this, however.

If it does do this, then PyPI would check for an If-Modified-Since 
header and compare it to the modified date for the page, and return a 
"not changed" response if appropriate.


> > While that's not necessarily as fast as static page generation, it's a
> > lot less complex to get right, and it saves the main piece of CPU load:
> > i.e., doing SQL queries and actually generating the page.
>
>I'm not convinced yet that this is where the time is spent (seeing
>actual profiling data would convince me).

I thought Rene' had done such profiling, as he said it was the 
templates that were taking most of the CPU.


> > Pages that pertain to more than one package might be a bit more complex
> > to do this on, but if I understand correctly it's mainly the
> > package-specific pages we're concerned with here, correct?
>
>I'm not convinced of that, either.

Well, I thought those were the ones we were caching.

It may be that I'm making too many assumptions, but if those 
assumptions are correct, then the whole thing gets a lot easier to 
prove correct, compared to a static cache, due to fewer moving 
parts.  If most CPU time is spent rendering package-specific pages, 
then this approach would fix the problem using the fewest changed 
parts and extra code to maintain.