OK, I've located the SMTP thing in SMTPDirect.py, lines
while chunks:
chunk = chunks.pop()
msgdata['recips'] = chunk
try:
deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
I'd really love to add somme sleep() function just after the deliveryfunc() call, something that would set Mailman to sleep about 1/10th of a second per recipient. I don't know any python, but I guess it would be something like
while chunks:
chunk = chunks.pop()
msgdata['recips'] = chunk
try:
deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
sleep(sizeof(chunk)/10)
Can someone help with the code? I'll test the concept :)
-- Fil
I'd really love to add somme sleep() function just after the deliveryfunc() call, something that would set Mailman to sleep about 1/10th of a second per recipient. I don't know any python, but I guess it would be something like
while chunks: chunk = chunks.pop() msgdata['recips'] = chunk try: deliveryfunc(mlist, msg, msgdata, envsender, refused, conn) sleep(sizeof(chunk)/10)Can someone help with the code? I'll test the concept :)
I did add
sleep(len(chunk)/10)
and it seems to work. I'll test it big time tonight when I send a new batch of 150 K emails, and keep you posted. But I'm not happy with the present algorithm, because I'd like other lists not to be hampered by the big list, so I'd rather have all lists send a chunk and go back to sleep, letting a chacne to any other list to send one of its chunks. I don't think it'll be too much of a rewrite to change that. Any ideas welcome.
-- Fil
On Mon, 2004-03-29 at 17:39, Fil wrote:
I'd really love to add somme sleep() function just after the deliveryfunc() call, something that would set Mailman to sleep about 1/10th of a second per recipient.
I just don't see the point of this. You may have stopped a large list inject being so aggressive on CPU usage, but you have increased the lifetime of processes by a factor of 30 or so (on your figures), and so increased the memory pressure and likelihood of swapping etc due to processes being just as fat but lasting longer.
Your users may notice the big lists getting much slower - and having deliveries smeared over a much longer period.
You may find you have made things less efficient by having 2 deliveries to a single list slowed down so that different messages to the same recipient can no longer be put in the same SMTP session (if your MTA does that). For that matter can multiple deliveries be made against the same list at the same time?
Why not just run the cron jobs under nice instead?
Nigel.
I'd really love to add somme sleep() function just after the deliveryfunc() call, something that would set Mailman to sleep about 1/10th of a second per recipient.
I just don't see the point of this. You may have stopped a large list inject being so aggressive on CPU usage, but you have increased the lifetime of processes by a factor of 30 or so (on your figures), and so increased the memory pressure and likelihood of swapping etc due to processes being just as fat but lasting longer.
The process we're talking about is OutgoingRunner, which is always in memory. I don't mind keeping the whole list in RAM -- I have 3 Gb, and the list (and chunks) probably eat up something like twice the size of the config.pck file, so about 44Mb. The only problem I see is if the list were locked during all that time, but I don't think (from what I understand from the code) that it is.
Your users may notice the big lists getting much slower - and having deliveries smeared over a much longer period.
Well that's already the case as postfix (my MTA) has 150 K messages to deliver, isn't it? But right now Mailman injects everything into postfix w/o considering that it might explode the load and system hit. It's brutal, with the load going to 25, and CPU to 70% (and I have a very nice machine, a brand new quadri-processor thing :-] ).
You may find you have made things less efficient by having 2 deliveries to a single list slowed down so that different messages to the same recipient can no longer be put in the same SMTP session (if your MTA does that).
I'm not sure postfix does that, no. And it's not really relevant in this case, as I really only have ONE big list with ONE message to send (and alot of small lists and normal emails to try to sneak through if possible).
For that matter can multiple deliveries be made against the same list at the same time?
Why not just run the cron jobs under nice instead?
What cron jobs? Sure I can renice the OutgoingRunner, but it won't change much (except that it will get swapped first if something needs to get swapped).
-- Fil
At 3:15 PM +0200 2004/03/30, Fil wrote:
I just don't see the point of this. You may have stopped a large list inject being so aggressive on CPU usage, but you have increased the lifetime of processes by a factor of 30 or so (on your figures), and so increased the memory pressure and likelihood of swapping etc due to processes being just as fat but lasting longer.
The process we're talking about is OutgoingRunner, which is always in memory. I don't mind keeping the whole list in RAM -- I have 3 Gb, and the list (and chunks) probably eat up something like twice the size of the config.pck file, so about 44Mb. The only problem I see is if the list were locked during all that time, but I don't think (from what I understand from the code) that it is.
Nigel isn't talking about the Mailman processes. He's talking
about the MTA processes. 44MB of memory is nothing on a large list, and since you have 3GB, that should not even be noticeable to you.
But the MTA processes sticking around much longer and not being
able to work as efficiently, that is something that will hurt you far, far more.
Well that's already the case as postfix (my MTA) has 150 K messages to deliver, isn't it? But right now Mailman injects everything into postfix w/o considering that it might explode the load and system hit. It's brutal, with the load going to 25, and CPU to 70% (and I have a very nice machine, a brand new quadri-processor thing :-] ).
If it's 150,000 recipients, with SMTP_MAX_RCPTS = 900, then
that's really just 167 messages, each addressed to multiple recipients (up to 900). Postfix is really good at handling things like this in parallel, so that quick sites get more mail dumped on them faster, and slower sites get mail delivered to them at a speed they are more comfortable with.
You really need to properly understand just what it is that
you're mucking about with, before you start making random changes.
You may find you have made things less efficient by having 2 deliveries to a single list slowed down so that different messages to the same recipient can no longer be put in the same SMTP session (if your MTA does that).
I'm not sure postfix does that, no. And it's not really relevant in this case, as I really only have ONE big list with ONE message to send (and alot of small lists and normal emails to try to sneak through if possible).
Yes, postfix will coalesce messages together, if it finds
multiple messages in the queue that are addressed to the same recipients. This is an old time/performance trick that even ancient versions of sendmail employed, and all modern MTAs likewise make use of.
And no, you really *DON'T* have just ONE message going out to ONE
big list. When a message comes in for that list, it gets broken up into chunks (controlled by SMTP_MAX_RCPTS), and each chunk is then handed off to the MTA as a separate instance. This happens in pretty much the same way for all lists, regardless of how many recipients may be on a particular list, or how many messages may be coming in or going out for any given list.
Why not just run the cron jobs under nice instead?
What cron jobs? Sure I can renice the OutgoingRunner, but it won't change much (except that it will get swapped first if something needs to get swapped).
I think Nigel was misunderstanding your description of the problem.
If it's 150,000 recipients, with SMTP_MAX_RCPTS = 900, then that's really just 167 messages, each addressed to multiple recipients (up to 900).
For the record, I must admit it was a huge mistake, because the SMTP connexion is reset after 15 minutes. So Mailman backs off, and waits, and almost nothing comes out. After 5 hours, I had to stop the whole thing, and kill the mail. Now I'm cleaning :(
-- Fil
On Mon, 2004-03-29 at 17:39, Fil wrote:
I'd really love to add somme sleep() function just after the deliveryfunc() call, something that would set Mailman to sleep about 1/10th of a second per recipient.
I just don't see the point of this. You may have stopped a large list inject being so aggressive on CPU usage, but you have increased the lifetime of processes by a factor of 30 or so (on your figures), and so increased the memory pressure and likelihood of swapping etc due to processes being just as fat but lasting longer.
Your users may notice the big lists getting much slower - and having deliveries smeared over a much longer period.
You may find you have made things less efficient by having 2 deliveries to a single list slowed down so that different messages to the same recipient can no longer be put in the same SMTP session (if your MTA does that). For that matter can multiple deliveries be made against the same list at the same time?
Why not just run the cron jobs under nice instead?
Nigel.
-
Brad Knowles -
Fil -
Nigel Metheringham