Many filesystems (e.g. the popular ext2) have horrible performance when there are many files in the same directory. The queue system should be modified to avoid this situation. As a test case, try adding 20,000 test address in such a way that Mailman will try to send a welcome message to each of them.
Isn't this optomising for a rather uncommon case.
It may be an uncommon case, but if/when it happens, processing the queue becomes so slow that it's a real problem.
I've just taken some timings:
Adding 20,000 test addresses took about 40 minutes. (They're all addresses at a remote machine where the the whole domain is forwarded to /dev/null ). I consider this to be acceptable.
Now the welcome messages are going out, at a rate of one message every 6.6 seconds. This is on inexpensive hardware (a Cobalt RaQ3) but the server is otherwise idle. At this rate, it would take more than 36 hours to send them all out. I consider this to be unacceptably slow (but it's not catastrophic unless it's normal for your server to put ten or more messages into the virgin queue per minute on the avarage).
What I'm really concerned about is the possiblity of a similar thing happening on the bounces queue. On a busy system, you can easily have bounces coming in with a rate greater than 10 bounces per minute. Temporary network problems can easily result in you getting a large number of bounces at the same time. This could put so many bounces into the queue that processing bounces becomes slower than the rate at which new bounces arrive. Then the bounce-handler is permanently screwed.
Greetings, Norbert.
-- A founder of the http://DotGNU.org project and Steering Committee member Norbert Bollow, Weidlistr.18, CH-8624 Gruet (near Zurich, Switzerland) Tel +41 1 972 20 59 Fax +41 1 972 20 69 http://norbert.ch List hosting with GNU Mailman on your own domain name http://cisto.com