[Moin-user] Very slow page saving caused by notifications

Paul Boddie paul at boddie.org.uk
Sun Apr 15 16:18:07 EDT 2012


On Sunday 15 April 2012 21:44:34 Steve McIntyre wrote:
>
> I've posted a bug at
>
>   http://moinmo.in/MoinMoinBugs/SubscribedPagesPerformanceProblem
>
> Quick summary:
>
> On a site with a large number of registered users
> (e.g. wiki.debian.org), saving a page taks a very long time. With a
> large number of users, the design of the page subscription system
> doesn't scale well. Saving a page works well, but moin then scans all
> the user data files looking for the subscribed_pages data. With
> thousands of users registered, this can take a very long time; we're
> seeing > 90 seconds on a wiki with more than 10,000 users.

I can see that the offending code is in MoinMoin/Page.py, specifically the 
getSubscribers method of the Page class. This looks like a classic case of 
needing to "invert" the way the data is stored so that it can be queried more 
efficiently - it's a bit like comparing the standard text search 
functionality with Xapian-based searching, where the former relies on 
scanning pages sequentially (pages yield terms), whereas the latter employs 
such "inverted" storage of queryable information (terms yield pages).

> This area needs fixing in some way - maybe add a cache in front of the
> user lookup here, or store the subscribed_pages information
> differently. I might be able to help with coding this, but I'd want to
> see what other people think first in terms of a design.
>
> What do people think?

I'd be inclined to index the subscription information so that there's a more 
efficiently queryable structure (pages yielding subscribers) that can be used 
in preference to the existing approach. Having subscriptions amend the index 
when created would eliminate any need for periodic reindexing, and I think 
you could implement this by having an event handler that can handle the 
SubscribedToPageEvent type of event.

There are probably other areas of Moin that could benefit from Xapian-based 
indexing, but this certainly looks like a good application of it.

Paul




More information about the Moin-user mailing list