[Catalog-sig] start on static generation, and caching - apache config.

[René Dudfield]

Hi,

here's the start of the static file generator.  It just works on one
web path, and one fileout at a time so far.  It doesn't figure out the
correct path to put the file, or check to see if there are any
changes.

http://rene.f0o.com/~rene/stuff/pypi/pypi-static-generation.py

# here is like looking at the http://cheeseshop.python.org/pypi/pygame url
python pypi-static-generation.py -create_single /pypi/pygame /tmp/pygame.html

It uses the webui.py code, so that there will not be any repeating of
code.  It does this in a similar manner to how the pypi.py pypi.cgi
and pypi.fcgi codes works.  That is by making its implementation of
the RequestWrapper class.



I thought I'd just keep posting my changes to the mailing list as I
go... so there's some history of changes - and so people can have a
look/review if they want.  If that annoys people I'll stop sending to
the list.



Next up I'm going to put a few functions into store.py.  Ones to check
if a release has changed since a given date.  Also one to see if any
changes at all have happened since a given date.

I'll also add some onChange type functions for releases.  That will be
where all of the code can go for stuff that happens on a change to
releases etc.


cheers,



On 7/8/07, René Dudfield <renesd at gmail.com> wrote:
> Hello,
>
> Cool, ok.  Let's start with event based updating of the static files.
>
>
> I need to make this tool in this way anyway though.  But we can either
> set it up to work with polling, or event based.  We can start with
> event based and switch to polling later if needed.
>
> Since none of the files exists at the moment, the tool will be
> needed to generate them initially.  Also if templates change, or the
> database changes - then the static pages may need regenerating.
>
>
> Polling is just one sql statement to see if something has changed.
> You do this once, no matter how many things have changed.  It's a
> really quick, operation if nothing has changed.
>
> Polling ends up being faster if you are constantly having to do things
> all the time anyway.  It's what network drivers do these days because
> they realise that there are a constant stream of events(interupts)
> anyway - so might as well deal with them at a fixed interval.
>
> Logged in users will not see the static file anyway - since they are
> logged in, they get to see the dynamically generated stuff.
>
> Imagine this case:
> 2-3 users are updating their packages, at a similar time.  The main
> index then gets regenerated 3 times, rather than once.  The more
> people who are changing things the more this method works.  If there
> are 20 people changing things at the same time, then there is still
> only one update of the main index page.  However since the cheeseshop
> only gets updated about 6 times daily, event based is probably better
> for the moment.
>
>
> Anyway... I'm just making the tool which can be used on demand, or at
> regular timings.
>
>
> Cheers,
>
>
>
> On 7/8/07, Jim Fulton <jim at zope.com> wrote:
> >
> > On Jul 7, 2007, at 12:24 AM, René Dudfield wrote:
> > ...
> > > Now I just need to finish off the static file generation code.  It
> > > needs a tool which can run every minute or so, which will look for any
> > > changes.
> >
> > Why not write the files when the underlying packages change?
> >
> > I don't like polling for two reasons:
> >
> > - New pages are out of date for up to the polling interval.  This is
> > especially annoying for someone who uploads a package and wants to be
> > able to access it immediately.
> >
> > - Polling all of the pages to see what's changed doesn't seem
> > scalable to me.
> >
> > ...
> >
> > > I've also updated the http://wiki.python.org/moin/CheeseShopDev page
> > > with some things I noticed when installing the cheeseshop again on my
> > > laptop.  Mainly dependencies, and missing config steps.
> >
> > Thanks!
> >
> > Jim
> >
> > --
> > Jim Fulton                      mailto:jim at zope.com             Python Powered!
> > CTO                             (540) 361-1714                  http://www.python.org
> > Zope Corporation        http://www.zope.com             http://www.zope.org
> >
> >
> >
> >
>