[Mailman-Developers] big list

[Barry A. Warsaw]

>>>>> "GT" == Graham TerMarsch <mailman@howlingfrog.com> writes:

    GT> The actual size of the "config.db" file was close to 12MB when
    GT> we started having major problems with it, never mind that when
    GT> slurped in QRunner generally ate up 50-70MB of RAM while
    GT> processing the queue.  With the CPU pegged at 95+% usage from
    GT> QRunner, I'd totally expect that most of our CPU time was just
    GT> spent spinning while slurping in and spitting out the list
    GT> after each and every update was made to the list.

I suspect that MM2.1 will perform better here for several reasons.
First, the OutgoingRunner almost never needs to acquire the lock to
get the message out the door.  If all your addresses are non-local and
you're not doing synchronous delivery, then you'll never get failures
during the SMTP dialog, so it'll never acquire the list lock.

Second, you can tune your BounceRunner to only process bounces "once
in a while".  If you really don't need to process them as soon as they
come in, just crank up the sleep interval between sweeps of the bounce
queue.

Third, if you decide to go the VERP route, you'll almost never need to
run through the bounce pipeline, since the bouncing address can be
unambiguously culled from the envelope sender.

MM2.1 inherits MM2.0's data store.  That's not going to change until
MM3.0, but I agree that something more efficient is sorely needed to
avoid list lock contention.  I personally think ZODB is a great way to
go, and may be the default, but the system /will/ be architected so
that you could use something else underneath if you want.

    GT> I'll also note that our install has a number of the Bounce
    GT> handlers removed, to make the processing chain shorter.
    [...]    
    GT> This, however, I expect is related to the performance not of
    GT> Mailman itself, but of Python in general, in the context of
    GT> the regexes and the MIME libraries that Mailman is using.

I'll bet most of that is simply spent importing all the bounce
pipeline modules.  Try strace'ing Python sometime and doing some
imports.  Fear the number of stats going on. :)

Again, MM2.1 will be much better because we've switched to a
long-running daemon design.  The bounce qrunner will warm up after the
initial set of imports, so if that /is/ the culprit, you'll see better
performance there as well.

Cheers,
-Barry