[Mailman-Developers] sleep() after sending a chunk?

[Brad Knowles]

At 3:15 PM +0200 2004/03/30, Fil wrote:

>>  I just don't see the point of this.  You may have stopped a large list
>>  inject being so aggressive on CPU usage, but you have increased the
>>  lifetime of processes by a factor of 30 or so (on your figures), and so
>>  increased the memory pressure and likelihood of swapping etc due to
>>  processes being just as fat but lasting longer.
>
>  The process we're talking about is OutgoingRunner, which is always in
>  memory. I don't mind keeping the whole list in RAM -- I have 3 Gb, and the
>  list (and chunks) probably eat up something like twice the size of the
>  config.pck file, so about 44Mb. The only problem I see is if the list were
>  locked during all that time, but I don't think (from what I understand from
>  the code) that it is.

	Nigel isn't talking about the Mailman processes.  He's talking 
about the MTA processes.  44MB of memory is nothing on a large list, 
and since you have 3GB, that should not even be noticeable to you.

	But the MTA processes sticking around much longer and not being 
able to work as efficiently, that is something that will hurt you 
far, far more.

>  Well that's already the case as postfix (my MTA) has 150 K messages to
>  deliver, isn't it? But right now Mailman injects everything into postfix w/o
>  considering that it might explode the load and system hit. It's brutal, with
>  the load going to 25, and CPU to 70% (and I have a very nice machine, a
>  brand new quadri-processor thing :-] ).

	If it's 150,000 recipients, with SMTP_MAX_RCPTS = 900, then 
that's really just 167 messages, each addressed to multiple 
recipients (up to 900).  Postfix is really good at handling things 
like this in parallel, so that quick sites get more mail dumped on 
them faster, and slower sites get mail delivered to them at a speed 
they are more comfortable with.

	You really need to properly understand just what it is that 
you're mucking about with, before you start making random changes.

>>  You may find you have made things less efficient by having 2 deliveries
>>  to a single list slowed down so that different messages to the same
>>  recipient can no longer be put in the same SMTP session (if your MTA
>>  does that).
>
>  I'm not sure postfix does that, no. And it's not really relevant in this
>  case, as I really only have ONE big list with ONE message to send (and alot
>  of small lists and normal emails to try to sneak through if possible).

	Yes, postfix will coalesce messages together, if it finds 
multiple messages in the queue that are addressed to the same 
recipients.  This is an old time/performance trick that even ancient 
versions of sendmail employed, and all modern MTAs likewise make use 
of.

	And no, you really *DON'T* have just ONE message going out to ONE 
big list.  When a message comes in for that list, it gets broken up 
into chunks (controlled by SMTP_MAX_RCPTS), and each chunk is then 
handed off to the MTA as a separate instance.  This happens in pretty 
much the same way for all lists, regardless of how many recipients 
may be on a particular list, or how many messages may be coming in or 
going out for any given list.

>>  Why not just run the cron jobs under nice instead?
>
>  What cron jobs? Sure I can renice the OutgoingRunner, but it won't change
>  much (except that it will get swapped first if something needs to get
>  swapped).

	I think Nigel was misunderstanding your description of the problem.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)