Re: [Mailman-Developers] message data storage
Hi Donn,
nice to see you on this list!
At the end of April, one of our new lists was up to 6500 archived messages for the month. It takes close to 4 minutes CPU time to archive a message in a situation like that. The average delivery rate to achieve 6500 messages in 30 days is 1 per 6 minutes, so the system obviously cannot work.
wow, that is a lot of traffic and a sluggish system. But I don't see how 6500 mails can generate a 68 MB unless there are a lot of attachments.
Working with the current system (as opposed to the Next Generation archiver), I'm tempted to think about a semi-permanent per-archive archiver and queue. The archiver would exit when its queue has been empty for some short interval, but any extra messages that arrive during that time get the database for free. I'd be interested in any other ideas.
As someone else has noted apart from the performance issue there is also one of usability: searching 6500 messages is a proverbial needle in a haystack. I don't know what the Next Generation archiver is but I guess it's a move towards a more sophisticated persistance system which might allow things like full-text searches. If I understand you correctly you want to archive mails directly as they come in and keep the archiver / db-connection open pretty much all the time. This would seem about the best solution in the short term. Anything else sounds like: database to pass the memory issue to something which is designed to handle it. Our own experience with lists with a large member base is that Mailman isn't that efficient at dealing with them which is why we're using an RDBMS adapter and letting the RDBMS be the muscle while Mailman remains the brains.
Novice question? How easy would it be just to dump to a database rather than using Python storage?
Charlie
participants (1)
-
Charlie Clark