Problems with multi-machine slicing
After doing some upgrades, I noticed yesterday that my multi-machine setup is no longer properly slicing the queue between machines. I probably missed something, but after going through all my notes on the setup I cannot figure out what the problem in. Hopefully someone else can spot the issue?
I have four mail servers. Three of them are supposed to slice the queue between them, and the fourth machine is set as a backup to process any remaining messages after 2 minutes. On the three slice machines, I have patched mailmanctl as:
def start_all_runners(): kids = {}
Each of these machines has a QRUNNERS section added to mm_cfg.py which defines the slice of each machine -- 3,0,3 / 3,1,3 / 3,2,3 and contains the line: QRUNNER_MESSAGE_IS_OLD_DELAY = None
On the fourth (backup) machine, I have patched Switchboard.py as:
if ext <> extension:
continue
when, digest = filebase.split('+')>>> now = time.time() age = now - float(when) # Only process defined 'old' entries. if not ( hasattr(mm_cfg, 'QRUNNER_MESSAGE_IS_OLD_DELAY') and mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY and age > mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY): continue <<< # Throw out any files which don't match our bitrange. BAW: test # performance and end-cases of this algorithm. MAS: both # comparisons need to be <= to get complete range.
On this fourth machine I have added to mm_cfg.py: QRUNNER_MESSAGE_IS_OLD_DELAY = minutes(2) This machine has NOT had the slices patch added to mailmanctl, so there is no QRUNNERS section in mm_cfg.py.
OK, so if I only have the backuo machine running, mailman will deliver my test message after 2 minutes. That part works fine. However with the three slice machines running, the first machine (3,0,3) sends ALL of the messages out immediately. If I shut down the first machine and leave the other two running, no messages are sent out until after the 2-minute period, then the backup machine sends them. In other words, the queue is not being sliced, and only the first machine is capable of sending out list messages.
I have referenced back to the original article on this subject: https://mail.python.org/pipermail/mailman-users/2008-March/060753.html but it appears I did the correct changes. Has something changed in newer versions of mailman that now prevent this technique from working the same way? Or was there something more to getting slicing to work that was not mentioned in that article?
On 05/24/2014 07:52 AM, Jeff Taylor wrote:
What exactly are the QRUNNERS = assignments in the 3 machines mm_cfg.py files.
If you do
ps -fwA | grep runner=
on the three machines, what are the --runner= arguments for the sliced queues?
You should be seeing one of
--runner=IncomingRunner:0:3 --runner=IncomingRunner:1:3 --runner=IncomingRunner:2:3
on each of the 3 machines.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The entire QRUNNERS section on the machines looks like this:
QRUNNERS = [ ('ArchRunner', 3,0,3), # messages for the archiver ('BounceRunner', 3,0,3), # for processing the qfile/bounces directory ('CommandRunner', 3,0,3), # commands and bounces from the outside world ('IncomingRunner', 3,0,3), # posts from the outside world ('NewsRunner', 3,0,3), # outgoing messages to the nntpd ('OutgoingRunner', 3,0,3), # outgoing messages to the smtpd ('VirginRunner', 3,0,3), # internally crafted (virgin birth) messages ('RetryRunner', 3,0,3), # retry temporarily failed deliveries ]
Per your ps command, on this machine I get:
list 11516 11512 0 08:50 ? 00:00:02 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:3 -s
(plus the runners for each of the other sections as well). On machine
#2 I have 3,1,3 in the cfg and ps shows --runner=IncomingRunner:1:3.
And on machine #3 I have 3,2,3 in the cfg and ps shows
--runner=IncomingRunner:2:3.
I also double-checked that the mailmanctl patch was applied on all three
machines, and that the Switchboard.py was NOT applied to these three.
Any other suggestions?
On 05/24/2014 11:14 AM, Mark Sapiro wrote:
On 05/24/2014 12:14 PM, Jeff Taylor wrote:
OK. That's as it should be.
OK. That's good too.
I'm stumped. It should work. Look in Mailman's qrunner logs for these machines. When a runner exits (unfortunately not when it starts) mailmanctl reports several things including its slice info. Do you see any of these and do they look as expected, e.g., 0/3 on the first machine, 1/3 on the second and 2/3 on the third?.
Also, from what to what did you upgrade?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
After stopping mailman, machine #1 shows: May 24 15:23:34 2014 (11512) Master qrunner detected subprocess exit (pid: 11516, sig: None, sts: 15, class: IncomingRunner, slice: 1/3)
Machine #2: May 24 15:21:56 2014 (12767) Master qrunner detected subprocess exit (pid: 12769, sig: None, sts: 15, class: BounceRunner, slice: 2/3)
Machine #3: May 24 15:22:16 2014 (31849) Master qrunner detected subprocess exit (pid: 31858, sig: None, sts: 15, class: VirginRunner, slice: 3/3)
Now for even more strangeness... After restarting mailman I sent
another test message. Just so you know, my test list has three email
addresses in it, so I would expect the messages to get split up
generally between the three machines (and please confirm my
understanding... if the list has three users on it, each one of the
three machines should forward one message to one user from the list?).
However after restarting and sending 7 more tests, it seems to bounce
between machine #1 and #2 sending the messages. In each case, one
machine sends the message to ALL users. After waiting about 15 minutes
I sent several more test messages. Now it seems to be randomly picking
one of the three machines to send from, but again the copy to all users
is sent from that one machine. I suppose that is better than it was --
at least now all three machines are being used. Is this the way its
supposed to be working?
Regarding the upgrade version, its been too long, I'm afraid I don't know what the old version was. The old machines are running ubuntu oneiric and now have mailman 2.1.14. The newer machines have debian wheezy and mailman 2.1.15. The upgrades happened a few months back, but I only noticed the issue yesterday because I am trying to get rid of the ubuntu machines and replace them with the debian machines. The messages have been getting delivered, but apparently one machine was handling everything.
On 05/24/2014 01:37 PM, Mark Sapiro wrote:
On 05/24/2014 03:05 PM, Jeff Taylor wrote:
OK, that looks good.
No. That's not the way it works. See below.
I think so.
Here's the detail. First the general flow.
- A post arrives and is queued in the in/ queue.
- It is picked up by IncomingRunner and processed through the handler pipeline.
- Assuming it is not held for any reason, it will get queued in the archive/ queue for ArchRunner and in the out/ queue for OutgoingRunner. It will also be added to the list's digest.mbox for eventually being sent to digest members as part of a digest which will be created and queued in the virgin/ queue for VirginRunner which will ultimately queue it in out/ for delivery
- ArchRunner will pick up the message from the archive/ queue and archive it.
- OutgoingRunner will pick up the message from the out/ queue and deliver it to the recipients.
Before we look at slicing, we see that once OutgoingRunner has a message, it will deliver it to all it's recipients, so a single post will always be delivered from the one machine who's OutgoingRunner picked it up from the out/ queue.
Now for slicing. Whenever a message is queued, whether for the in/ queue by mail delivery or some other queue by some handler or other process, it gets a file name of the form tttt+hhhhhhhh.pck. the tttt part is a time stamp so we can ensure fifo processing. The hhhhhhhh part is a hex digest of a sha1 hash of the message, the listname and the current time. Slicing works by dividing that hash space into n equal slices (in your case 3 with slice 0 being the first third, slice 1 the middle third and slice 2 the last third).
So when a runner that is processing slice 0 say, looks at its queue, it will only process those messages in the first third of the hash space.
So bottom line, an incoming message will be queued in the in/ queue and it has an equal chance of being in any slice and will be picked up by the machine processing that slice. Then the message will be later requeued in out/ probably with a different hash. The time is in seconds and may or may not have changed, but the message has likely changed due to subject prefixing, content filtering and/or header refolding. So it will be picked up by the OutgoinRunner processing its slice, and that one runner will deliver to all recipients.
I was curious because it would help me know if there had been relevant changes, but I think it's working as it's supposed to and probably as it wads before.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Its odd, I could have sworn the slicing used to be done per recipient, not per message. I've had to check logs for a client to confirm her messages went out, and generally had check all three machines to verify every user received the message.
The rest of the process works as I expected it to. I think were I flubbed up initially (last time I thought it wasn't working right and went through my process again) was that I had applied the patch to Switchboard.py to all four machines. Its just weird that it took so many restarts (and some reboots) before it started slicing properly again. I haven't made any config changes since I sent my initial plea for help to the list last night.
At least now it looks like I have my notes in order and I can finish getting rid of the other ubuntu machines. Thanks for the help!
On 05/24/2014 04:56 PM, Mark Sapiro wrote:
On 05/24/2014 06:53 PM, Jeff Taylor wrote:
Mailman's outgoing message which includes the addresses of all recipients in the message's metadata is in one out/ queue entry. Whatever slice that's in, it will be picked up by the OutgoingRunner processing that slice of the queue and all recipients will be delivered to that runner's outgoing MTA (default on localhost).
Perhaps there is also some load balancing between MTAs that allows them to share the delivery.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
There is some load-balancing on the incoming side that splits messages between servers, but nothing on the outgoing side. Mailman is run directly on each of the SMTP servers. I started doing the slicing awhile back when I was having trouble with mailman crashing occasionally, and wanted to add more redundancy to ensure the out queue would get processed, even if it took a couple minutes for a backup machine to get to them.
On 05/25/2014 10:52 AM, Mark Sapiro wrote:
I'm honestly not sure about that. Other than the slicing, mailman is pretty much set up with defaults. The only changes I make on the lists is setting them so that replies only come back to the admin, not to the entire list (they are mostly used to send notices to cell phones as text messages).
On 05/25/2014 12:07 PM, Richard Damon wrote:
On 05/25/2014 11:07 AM, Richard Damon wrote:
That wouldn't do it. The VERPing or personalization is done in the OutgoingRunner process which will still be the one process doing the SMTP for all recipients. The only way this won't happen is if some recipients are refused during SMTP and the message is requeued for retry of the failed recipients. The requeued message might then be in a different slice.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/24/2014 07:52 AM, Jeff Taylor wrote:
What exactly are the QRUNNERS = assignments in the 3 machines mm_cfg.py files.
If you do
ps -fwA | grep runner=
on the three machines, what are the --runner= arguments for the sliced queues?
You should be seeing one of
--runner=IncomingRunner:0:3 --runner=IncomingRunner:1:3 --runner=IncomingRunner:2:3
on each of the 3 machines.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The entire QRUNNERS section on the machines looks like this:
QRUNNERS = [ ('ArchRunner', 3,0,3), # messages for the archiver ('BounceRunner', 3,0,3), # for processing the qfile/bounces directory ('CommandRunner', 3,0,3), # commands and bounces from the outside world ('IncomingRunner', 3,0,3), # posts from the outside world ('NewsRunner', 3,0,3), # outgoing messages to the nntpd ('OutgoingRunner', 3,0,3), # outgoing messages to the smtpd ('VirginRunner', 3,0,3), # internally crafted (virgin birth) messages ('RetryRunner', 3,0,3), # retry temporarily failed deliveries ]
Per your ps command, on this machine I get:
list 11516 11512 0 08:50 ? 00:00:02 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:3 -s
(plus the runners for each of the other sections as well). On machine
#2 I have 3,1,3 in the cfg and ps shows --runner=IncomingRunner:1:3.
And on machine #3 I have 3,2,3 in the cfg and ps shows
--runner=IncomingRunner:2:3.
I also double-checked that the mailmanctl patch was applied on all three
machines, and that the Switchboard.py was NOT applied to these three.
Any other suggestions?
On 05/24/2014 11:14 AM, Mark Sapiro wrote:
On 05/24/2014 12:14 PM, Jeff Taylor wrote:
OK. That's as it should be.
OK. That's good too.
I'm stumped. It should work. Look in Mailman's qrunner logs for these machines. When a runner exits (unfortunately not when it starts) mailmanctl reports several things including its slice info. Do you see any of these and do they look as expected, e.g., 0/3 on the first machine, 1/3 on the second and 2/3 on the third?.
Also, from what to what did you upgrade?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
After stopping mailman, machine #1 shows: May 24 15:23:34 2014 (11512) Master qrunner detected subprocess exit (pid: 11516, sig: None, sts: 15, class: IncomingRunner, slice: 1/3)
Machine #2: May 24 15:21:56 2014 (12767) Master qrunner detected subprocess exit (pid: 12769, sig: None, sts: 15, class: BounceRunner, slice: 2/3)
Machine #3: May 24 15:22:16 2014 (31849) Master qrunner detected subprocess exit (pid: 31858, sig: None, sts: 15, class: VirginRunner, slice: 3/3)
Now for even more strangeness... After restarting mailman I sent
another test message. Just so you know, my test list has three email
addresses in it, so I would expect the messages to get split up
generally between the three machines (and please confirm my
understanding... if the list has three users on it, each one of the
three machines should forward one message to one user from the list?).
However after restarting and sending 7 more tests, it seems to bounce
between machine #1 and #2 sending the messages. In each case, one
machine sends the message to ALL users. After waiting about 15 minutes
I sent several more test messages. Now it seems to be randomly picking
one of the three machines to send from, but again the copy to all users
is sent from that one machine. I suppose that is better than it was --
at least now all three machines are being used. Is this the way its
supposed to be working?
Regarding the upgrade version, its been too long, I'm afraid I don't know what the old version was. The old machines are running ubuntu oneiric and now have mailman 2.1.14. The newer machines have debian wheezy and mailman 2.1.15. The upgrades happened a few months back, but I only noticed the issue yesterday because I am trying to get rid of the ubuntu machines and replace them with the debian machines. The messages have been getting delivered, but apparently one machine was handling everything.
On 05/24/2014 01:37 PM, Mark Sapiro wrote:
On 05/24/2014 03:05 PM, Jeff Taylor wrote:
OK, that looks good.
No. That's not the way it works. See below.
I think so.
Here's the detail. First the general flow.
- A post arrives and is queued in the in/ queue.
- It is picked up by IncomingRunner and processed through the handler pipeline.
- Assuming it is not held for any reason, it will get queued in the archive/ queue for ArchRunner and in the out/ queue for OutgoingRunner. It will also be added to the list's digest.mbox for eventually being sent to digest members as part of a digest which will be created and queued in the virgin/ queue for VirginRunner which will ultimately queue it in out/ for delivery
- ArchRunner will pick up the message from the archive/ queue and archive it.
- OutgoingRunner will pick up the message from the out/ queue and deliver it to the recipients.
Before we look at slicing, we see that once OutgoingRunner has a message, it will deliver it to all it's recipients, so a single post will always be delivered from the one machine who's OutgoingRunner picked it up from the out/ queue.
Now for slicing. Whenever a message is queued, whether for the in/ queue by mail delivery or some other queue by some handler or other process, it gets a file name of the form tttt+hhhhhhhh.pck. the tttt part is a time stamp so we can ensure fifo processing. The hhhhhhhh part is a hex digest of a sha1 hash of the message, the listname and the current time. Slicing works by dividing that hash space into n equal slices (in your case 3 with slice 0 being the first third, slice 1 the middle third and slice 2 the last third).
So when a runner that is processing slice 0 say, looks at its queue, it will only process those messages in the first third of the hash space.
So bottom line, an incoming message will be queued in the in/ queue and it has an equal chance of being in any slice and will be picked up by the machine processing that slice. Then the message will be later requeued in out/ probably with a different hash. The time is in seconds and may or may not have changed, but the message has likely changed due to subject prefixing, content filtering and/or header refolding. So it will be picked up by the OutgoinRunner processing its slice, and that one runner will deliver to all recipients.
I was curious because it would help me know if there had been relevant changes, but I think it's working as it's supposed to and probably as it wads before.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Its odd, I could have sworn the slicing used to be done per recipient, not per message. I've had to check logs for a client to confirm her messages went out, and generally had check all three machines to verify every user received the message.
The rest of the process works as I expected it to. I think were I flubbed up initially (last time I thought it wasn't working right and went through my process again) was that I had applied the patch to Switchboard.py to all four machines. Its just weird that it took so many restarts (and some reboots) before it started slicing properly again. I haven't made any config changes since I sent my initial plea for help to the list last night.
At least now it looks like I have my notes in order and I can finish getting rid of the other ubuntu machines. Thanks for the help!
On 05/24/2014 04:56 PM, Mark Sapiro wrote:
On 05/24/2014 06:53 PM, Jeff Taylor wrote:
Mailman's outgoing message which includes the addresses of all recipients in the message's metadata is in one out/ queue entry. Whatever slice that's in, it will be picked up by the OutgoingRunner processing that slice of the queue and all recipients will be delivered to that runner's outgoing MTA (default on localhost).
Perhaps there is also some load balancing between MTAs that allows them to share the delivery.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
There is some load-balancing on the incoming side that splits messages between servers, but nothing on the outgoing side. Mailman is run directly on each of the SMTP servers. I started doing the slicing awhile back when I was having trouble with mailman crashing occasionally, and wanted to add more redundancy to ensure the out queue would get processed, even if it took a couple minutes for a backup machine to get to them.
On 05/25/2014 10:52 AM, Mark Sapiro wrote:
I'm honestly not sure about that. Other than the slicing, mailman is pretty much set up with defaults. The only changes I make on the lists is setting them so that replies only come back to the admin, not to the entire list (they are mostly used to send notices to cell phones as text messages).
On 05/25/2014 12:07 PM, Richard Damon wrote:
On 05/25/2014 11:07 AM, Richard Damon wrote:
That wouldn't do it. The VERPing or personalization is done in the OutgoingRunner process which will still be the one process doing the SMTP for all recipients. The only way this won't happen is if some recipients are refused during SMTP and the message is requeued for retry of the failed recipients. The requeued message might then be in a different slice.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Jeff Taylor -
Mark Sapiro -
Richard Damon