[Mailman-Developers] (2.0.6) pipermail takes >1 minute to rebuild indexes on large lists

[Ben Gertzfield]

>>>>> "BAW" == Barry A Warsaw <barry@zope.com> writes:
>>>>> "BG" == Ben Gertzfield <che@debian.org> writes:

    BG> On a massive list (Mailman 2.0.6) I run that regularly gets a
    BG> few hundred or more emails every day, things begin to slow
    BG> down to molasses after a week or two each month, with the
    BG> qrunner process taking literally HUNDREDS of megabytes of RAM,
    BG> and 100% CPU, all the time.

    BAW> FWIW, the combined {zope,python}.org gets many hundreds of
    BAW> messages a day on the python-list and zope mailing lists
    BAW> alone.  I'm not online at the moment, so I can't check, but I
    BAW> know I receive several digests every day from both lists --
    BAW> they're very high traffic.  Needless to say, I haven't
    BAW> noticed any such performance problems...

The problem was when the mbox got up to about 200-300 megs; I can send
you the traces of the function calls with timestamps, and you can see
exactly how slow things get.

    BAW> If your system still can't handle things, then the next step
    BAW> is to set ARCHIVE_TO_MBOX to 1.  This way, Mailman will
    BAW> simply append the message to the .mbox file, which ought to
    BAW> be extremely quick, but it won't attempt to run the Pipermail
    BAW> archiver in real time.  Then you can use whatever archiving
    BAW> scheme you want (e.g. bin/arch nightly, or an external
    BAW> archiver).

Yes, this is probably the right solution.  In fact, I'm actually
leaning towards suggesting that Mailman just come with or depend
upon hypermail for archiving; we're just re-inventing the wheel
by trying to modify pipermail over and over, and it's really not
going to scale.

Ben

-- 
Brought to you by the letters D and Z and the number 19.
"He's like.. some sort of.. non-giving up.. school guy!"
Debian GNU/Linux maintainer of Gimp and GTK+ -- http://www.debian.org/