Adding backup to multi-server config
I have been running mailman 2.1 from three servers with NFS sharing for the past couple years, but due to a recent issue I did some digging and found the suggestions for clustering at http://mail.python.org/pipermail/mailman-users/2008-March/060753.html (the post by Mark Sapiro detailing how to set up qrunner slices). It turns out the problem I was having was user error, so apparently mailman *can* run on a low traffic site without issue... however since I now have the proper changes made for qrunner, I have a new question.
My network consists of three primary servers running postfix and mailman. These are each essentially redundant machines, and I am in the process of setting up a fourth machine in a VM for processing SMTP. What I am wondering about is if there would be a way to configure qrunner on this VM so that it only processes messages which have been in the queue for more than X minutes... basically a backup machine so that if the server processes one of the slices goes offline for an extended period, this backup server would make sure the message gets sent out.
From an initial glance, I was thinking of having a configurable Delay variable in mm_cfg.py, specified in seconds, then qrunner could use the timestamp on the queued messages and only process the ones which were older than Delay. This backup machine would *not* have the qrunner slices patch applied to it, as it should be processing any message that doesn't get handled by the primary servers.
Does this sound plausible? Unfortunately I know nothing about python, nor the specifics of mailman's code, so I'm not sure if this would be an easy change that someone could give me the code for... but any help is appreciated.
On 12/18/2012 11:15 AM, Jeff Taylor wrote:
From an initial glance, I was thinking of having a configurable Delay variable in mm_cfg.py, specified in seconds, then qrunner could use the timestamp on the queued messages and only process the ones which were older than Delay. This backup machine would *not* have the qrunner slices patch applied to it, as it should be processing any message that doesn't get handled by the primary servers.
Does this sound plausible?
Yes.
Unfortunately I know nothing about python, nor the specifics of mailman's code, so I'm not sure if this would be an easy change that someone could give me the code for... but any help is appreciated.
Assuming you have something like
QRUNNER_MESSAGE_IS_OLD_DELAY = minutes(2)
in mm_cfg.py for the backup machine and either no mention or something like
QRUNNER_MESSAGE_IS_OLD_DELAY = None
in mm_cfg.py/Defaults.py for the other machines, you could then patch Mailman/Queue/Switchboard.py as follows:
Find the section in the definition of the files() method that looks like
if ext <> extension:
continue
when, digest = filebase.split('+')
# Throw out any files which don't match our bitrange. BAW: test
# performance and end-cases of this algorithm. MAS: both
# comparisons need to be <= to get complete range.
if lower is None or (lower <= long(digest, 16) <= upper):
key = float(when)
while times.has_key(key):
key += DELTA
times[key] = filebase
and add the following
now = time.time()
age = now - float(when)
# Only process defined 'old' entries.
if not (
hasattr(mm_cfg, 'QRUNNER_MESSAGE_IS_OLD_DELAY') and
mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY and
age > mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY):
continue
between
when, digest = filebase.split('+')
and
# Throw out any files which don't match our bitrange. BAW: test
Also, Assuming you have patched mailmanctl as indicated at http://mail.python.org/pipermail/mailman-users/2008-March/060753.html, the mm_cfg.py QRUNNERS entries for the backup machine should look like
QRUNNERS = [ ('ArchRunner', 1,0,1), # messages for the archiver ... ]
i.e., the backup machine should be set up to be machine 0 of 1 machines. The actual number of slices for a given runner on this machine (the first '1' in '1,0,1' above) could be greater than 1, but you probably don't want that.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Python is indentation-sensitive. Do you really mean a dedent there relative to the surrounding stanza? It doesn't seem to be an artifact of TABs or something like that.
Find the section in the definition of the files() method that looks like
if ext <> extension: continue when, digest = filebase.split('+') # Throw out any files which don't match our bitrange. BAW: test # performance and end-cases of this algorithm. MAS: both # comparisons need to be <= to get complete range. if lower is None or (lower <= long(digest, 16) <= upper): key = float(when) while times.has_key(key): key += DELTA times[key] = filebase
and add the following
now = time.time() age = now - float(when) # Only process defined 'old' entries. if not ( hasattr(mm_cfg, 'QRUNNER_MESSAGE_IS_OLD_DELAY') and mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY and age > mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY): continue
between
when, digest = filebase.split('+')
and
# Throw out any files which don't match our bitrange. BAW: test
Stephen J. Turnbull wrote:
Mark Sapiro writes:
Python is indentation-sensitive. Do you really mean a dedent there relative to the surrounding stanza? It doesn't seem to be an artifact of TABs or something like that.
No, it's just lack of careful typing and proof-reading on my part. (Also, a clue that what I wrote was inadequately tested at best).
Find the section in the definition of the files() method that looks like
if ext <> extension: continue when, digest = filebase.split('+') # Throw out any files which don't match our bitrange. BAW: test # performance and end-cases of this algorithm. MAS: both # comparisons need to be <= to get complete range. if lower is None or (lower <= long(digest, 16) <= upper): key = float(when) while times.has_key(key): key += DELTA times[key] = filebase
and add the following
As Stephen notes (thank you), this code block should be indented an additional four spaces.
now = time.time() age = now - float(when) # Only process defined 'old' entries. if not ( hasattr(mm_cfg, 'QRUNNER_MESSAGE_IS_OLD_DELAY') and mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY and age > mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY): continue
So that the first 4 lines line up with the 'between' lines below and the remaining lines are also appropriately indented.
between
when, digest = filebase.split('+')
and
# Throw out any files which don't match our bitrange. BAW: test
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
No problems, I'm at least aware of the indented nature of python. I'll get this code added in later in the week and give it a trial run, see what happens. Thanks!
On 12/23/2012 02:55 PM, Mark Sapiro wrote:
Stephen J. Turnbull wrote:
Mark Sapiro writes:
Python is indentation-sensitive. Do you really mean a dedent there relative to the surrounding stanza? It doesn't seem to be an artifact of TABs or something like that.
No, it's just lack of careful typing and proof-reading on my part. (Also, a clue that what I wrote was inadequately tested at best).
Find the section in the definition of the files() method that looks like
if ext <> extension: continue when, digest = filebase.split('+') # Throw out any files which don't match our bitrange. BAW: test # performance and end-cases of this algorithm. MAS: both # comparisons need to be <= to get complete range. if lower is None or (lower <= long(digest, 16) <= upper): key = float(when) while times.has_key(key): key += DELTA times[key] = filebase
and add the following
As Stephen notes (thank you), this code block should be indented an additional four spaces.
now = time.time() age = now - float(when) # Only process defined 'old' entries. if not ( hasattr(mm_cfg, 'QRUNNER_MESSAGE_IS_OLD_DELAY') and mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY and age > mm_cfg.QRUNNER_MESSAGE_IS_OLD_DELAY): continue
So that the first 4 lines line up with the 'between' lines below and the remaining lines are also appropriately indented.
between
when, digest = filebase.split('+')
and
# Throw out any files which don't match our bitrange. BAW: test
participants (3)
-
Jeff Taylor
-
Mark Sapiro
-
Stephen J. Turnbull