[Moin-user] Very slow page saving caused by notifications
paul at boddie.org.uk
Sun Apr 15 16:18:07 EDT 2012
On Sunday 15 April 2012 21:44:34 Steve McIntyre wrote:
> I've posted a bug at
> Quick summary:
> On a site with a large number of registered users
> (e.g. wiki.debian.org), saving a page taks a very long time. With a
> large number of users, the design of the page subscription system
> doesn't scale well. Saving a page works well, but moin then scans all
> the user data files looking for the subscribed_pages data. With
> thousands of users registered, this can take a very long time; we're
> seeing > 90 seconds on a wiki with more than 10,000 users.
I can see that the offending code is in MoinMoin/Page.py, specifically the
getSubscribers method of the Page class. This looks like a classic case of
needing to "invert" the way the data is stored so that it can be queried more
efficiently - it's a bit like comparing the standard text search
functionality with Xapian-based searching, where the former relies on
scanning pages sequentially (pages yield terms), whereas the latter employs
such "inverted" storage of queryable information (terms yield pages).
> This area needs fixing in some way - maybe add a cache in front of the
> user lookup here, or store the subscribed_pages information
> differently. I might be able to help with coding this, but I'd want to
> see what other people think first in terms of a design.
> What do people think?
I'd be inclined to index the subscription information so that there's a more
efficiently queryable structure (pages yielding subscribers) that can be used
in preference to the existing approach. Having subscriptions amend the index
when created would eliminate any need for periodic reindexing, and I think
you could implement this by having an event handler that can handle the
SubscribedToPageEvent type of event.
There are probably other areas of Moin that could benefit from Xapian-based
indexing, but this certainly looks like a good application of it.
More information about the Moin-user