[Catalog-sig] start on static generation, and caching - apache config.
renesd at gmail.com
Sun Jul 8 06:48:45 CEST 2007
here's the start of the static file generator. It just works on one
web path, and one fileout at a time so far. It doesn't figure out the
correct path to put the file, or check to see if there are any
# here is like looking at the http://cheeseshop.python.org/pypi/pygame url
python pypi-static-generation.py -create_single /pypi/pygame /tmp/pygame.html
It uses the webui.py code, so that there will not be any repeating of
code. It does this in a similar manner to how the pypi.py pypi.cgi
and pypi.fcgi codes works. That is by making its implementation of
the RequestWrapper class.
I thought I'd just keep posting my changes to the mailing list as I
go... so there's some history of changes - and so people can have a
look/review if they want. If that annoys people I'll stop sending to
Next up I'm going to put a few functions into store.py. Ones to check
if a release has changed since a given date. Also one to see if any
changes at all have happened since a given date.
I'll also add some onChange type functions for releases. That will be
where all of the code can go for stuff that happens on a change to
On 7/8/07, René Dudfield <renesd at gmail.com> wrote:
> Cool, ok. Let's start with event based updating of the static files.
> I need to make this tool in this way anyway though. But we can either
> set it up to work with polling, or event based. We can start with
> event based and switch to polling later if needed.
> Since none of the files exists at the moment, the tool will be
> needed to generate them initially. Also if templates change, or the
> database changes - then the static pages may need regenerating.
> Polling is just one sql statement to see if something has changed.
> You do this once, no matter how many things have changed. It's a
> really quick, operation if nothing has changed.
> Polling ends up being faster if you are constantly having to do things
> all the time anyway. It's what network drivers do these days because
> they realise that there are a constant stream of events(interupts)
> anyway - so might as well deal with them at a fixed interval.
> Logged in users will not see the static file anyway - since they are
> logged in, they get to see the dynamically generated stuff.
> Imagine this case:
> 2-3 users are updating their packages, at a similar time. The main
> index then gets regenerated 3 times, rather than once. The more
> people who are changing things the more this method works. If there
> are 20 people changing things at the same time, then there is still
> only one update of the main index page. However since the cheeseshop
> only gets updated about 6 times daily, event based is probably better
> for the moment.
> Anyway... I'm just making the tool which can be used on demand, or at
> regular timings.
> On 7/8/07, Jim Fulton <jim at zope.com> wrote:
> > On Jul 7, 2007, at 12:24 AM, René Dudfield wrote:
> > ...
> > > Now I just need to finish off the static file generation code. It
> > > needs a tool which can run every minute or so, which will look for any
> > > changes.
> > Why not write the files when the underlying packages change?
> > I don't like polling for two reasons:
> > - New pages are out of date for up to the polling interval. This is
> > especially annoying for someone who uploads a package and wants to be
> > able to access it immediately.
> > - Polling all of the pages to see what's changed doesn't seem
> > scalable to me.
> > ...
> > > I've also updated the http://wiki.python.org/moin/CheeseShopDev page
> > > with some things I noticed when installing the cheeseshop again on my
> > > laptop. Mainly dependencies, and missing config steps.
> > Thanks!
> > Jim
> > --
> > Jim Fulton mailto:jim at zope.com Python Powered!
> > CTO (540) 361-1714 http://www.python.org
> > Zope Corporation http://www.zope.com http://www.zope.org
More information about the Catalog-SIG