[Mailman-Developers] message data storage

[Donn Cave]

I'm breathing easier this week, now that a new month has started.

At the end of April, one of our new lists was up to 6500 archived
messages for the month.  It takes close to 4 minutes CPU time to
archive a message in a situation like that.  The average delivery
rate to achieve 6500 messages in 30 days is 1 per 6 minutes, so
the system obviously cannot work.  I intervened and threw away 
a couple hundred of them (it's drivel) to break the logjam and
get everyone else going.  We have only a fraction of our expected
population so far, we're just beginning to migrate in earnest from
listproc, so it looks like we're in some trouble.

Most of the work appears to be in the 68Mb database, as in
database/2003-April-article.  It can take up to a minute to get
this through marshal.load(), almost all of it disk I/O.  It
takes nearly 3 minutes CPU time to get all the values through
cPickle.loads(), and I believe the archiver does in fact load
every single one.

Working with the current system (as opposed to the Next Generation
archiver), I'm tempted to think about a semi-permanent per-archive
archiver and queue.  The archiver would exit when its queue has
been empty for some short interval, but any extra messages that
arrive during that time get the database for free.  I'd be interested
in any other ideas.

thanks,
	Donn Cave, donn at u.washington.edu