[Mailman-Developers] Serious I/O contention issue

[Brad Knowles]

At 1:11 PM +0100 2003/10/14, Mike Bradley wrote:

>  I have a rather large mailing list (35000 members) which is used for
>  announcements only, but I have been experiencing terrible performance
>  problems when the qrunner is active.  The mail server is Postfix,
>  Mailman is version 2.1.3 (previous versions showed the same problems).

	Just checking, but have you seen the following FAQ entries?  See:

		<http://www.python.org/cgi-bin/faqw-mm.py?req=all#4.11>
		<http://www.python.org/cgi-bin/faqw-mm.py?req=all#4.12>
		<http://www.python.org/cgi-bin/faqw-mm.py?req=all#6.4>
		<http://www.python.org/cgi-bin/faqw-mm.py?req=all#6.6>

	I figure you probably have already seen them, but I wanted to be sure.

>  When I send an outgoing mail, the qrunner processes take up all my CPU
>  and the hard disk becomes inaccessible due to constant IO.  Even when
>  there is very little activity in the list and the queues are cleared,
>  the server locks up for several seconds every time an operation is
>  performed.

	What about the machine you're doing this on?  The filesystem? 
How is postfix configured?

	If you read Nick Christenson's book _Sendmail Performance 
Tuning_, please note that many of the criteria are also applicable to 
other MTAs.  See <http://www.jetcafe.org/~npc/book/sendmail/> for 
more info.  Disclaimer: Nick was my co-author for a talk I gave 
previously (see <http://www.shub-internet.org/brad/papers/dihses/>), 
and I was a technical reviewer of this book.


	Clearly, there may well be lots of opportunity here to tune your 
filesystem for maximum performance.

>  I am not familiar with the exact way the application operates, but are
>  the mlist arrays shared/marshalled across the qrunner processes via the
>  _listcache?  If so, then would it be possible to eliminate the
>  mlist.Load() in the OutgoingRunner and implement delayed saving in the
>  mailing list class so that Save() didn't cause a write to disk each
>  time?

	Depending on what your "SMTP_MAX_RCPTS" value is set to, I would 
imagine that this should be loaded and saved each time a message is 
passed from your mailman qrunner to postfix.  The higher 
SMTP_MAX_RCPTS, the less often this process should occur.  Of course, 
others have found that SMTP_MAX_RCPTS should typically be set 
somewhere between 2 and 10 (usually ~5) for best overall performance 
(see the FAQ entries above).

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)