[Mailman-Users] Large site mailman config

[Brad Knowles]

At 12:13 PM +0000 2006-03-03, George Barnett wrote:

>  The processes dont seem to be failing.  New list mail is delivered fine,
>  but what's 'stuck' in the queue doesn't leave until a restart which
>  effectively does a queue run.

	Ahh.  Because Mailman uses the filesystem as a queueing method, 
there is a certain FIFO nature to the messages that are processed, 
and if the queue is deep then new messages that come in won't be 
processed for a while -- the qrunner is going to be at a certain 
point in the inode filestructure for the directory and is not going 
to look at earlier points in the directory even if they are now free 
and would sort earlier in the process.

	I believe that there is some discussion on this in FAQ 6.6.

>  These were increased after some queues were getting more full than
>  others.  Changing these values did help, but I suspect it's fixed
>  the symptoms rather than the problem.

	If you haven't already looked at FAQ 6.6, I'd encourage you to 
read it and compare it with your own experience, and the different 
values recommended as compared to your own.  Using different values 
is fine, but you should understand what the values mean and why they 
are what they are.


	One thing I can tell you is that the filesystem is a critical 
bottleneck for both MTAs and MLMs, and most of the performance tuning 
techniques for MTAs with regard to filesystems will be equally 
applicable to filesystem issues for Mailman and other MLMs.

	Towards that end, I'd recommend you take a look at FAQ 6.3 as 
well, and the slides at 
<http://www.shub-internet.org/brad/papers/sendmail-tuning/> from my 
invited talk "Sendmail Performance Tuning for Large Systems" that I 
gave at SANE'98.  With regards to disk and filesystem performance, 
pretty much everywhere it says "Sendmail", you can substitute 
"Mailman" instead, without loss of generality.

>  I found this information from the faq on python.org.  Our traffic pattern
>  is very bursty.  Lists are small, maybe 25 users per list, but when
>  traffic comes in, it's alot at the same time (monitoring mails).

	Whereas our traffic tends to be more consistent -- receiving 
anywhere from 5000 to 30,000 messages per hour, and around 400,000 
total incoming messages per day, and rejecting about 200,000 messages 
per day (mostly spam).

	By Mailman standards, this is actually only "moderate" size, even 
though a couple of months ago we were rejecting 90-95% of all 
incoming mail as spam, and handling 500,000-600,000 (or more) 
incoming messages per day that were legitimate.


	Oh, and we've got some stuff we're doing with the firewall 
(blocking known abusive IP addresses for short periods of time), 
which doesn't show up in any of our logs.  There's enough of that 
crap that we can't log that information because the impact of doing 
the logging would probably kill the machine.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

  LOPSA member since December 2005.  See <http://www.lopsa.org/>.