[Mailman-Users] Mailman clustering

[Mark Sapiro]

Luca Villani wrote:
>
>We're starting to move our internal mailman from a single server to a
>multiple servers architecture (starting with two servers).
>
>
>Digging on the list, we find two approaches to this problem:
>
>http://mail.python.org/pipermail/mailman-users/2006-November/054641.html
>http://mail.python.org/pipermail/mailman-users/2007-December/059625.html
>
>
>As far as we prefer to not patch the code, we will try the first
>solution (spanning qfiles slices handling to multiple qrunners, one on
>every server), but we find no examples for this configuration.
>
>Cite:
>
>"IOW, machine 1 could handle the odd slices of qfile/in while machine 2
>could handle the even slices"
>
>
>How can we configure our mailman installation in order to do this?

This still requires some patching or other machinations. First, this
approach assumes that all the mutable Mailman directories are shared
(NFS or ??) across machines.

Defaults.py contains the following:

QRUNNERS = [
    ('ArchRunner',     1), # messages for the archiver
    ('BounceRunner',   1), # for processing the qfile/bounces directory
    ('CommandRunner',  1), # commands and bounces from the outside world
    ('IncomingRunner', 1), # posts from the outside world
    ('NewsRunner',     1), # outgoing messages to the nntpd
    ('OutgoingRunner', 1), # outgoing messages to the smtpd
    ('VirginRunner',   1), # internally crafted (virgin birth) messages
    ('RetryRunner',    1), # retry temporarily failed deliveries
    ]

The '1' in each case is the number of slices/qrunners per queue. You
can copy that to mm_cfg.py and change '1' to '2' so that each queue
will have two slices - 0:2 and 1:2.

But now, mailmanctl on each machine will start runners to process both
slices which is not what you want. You want one machine to have only
the runners processing the 0:2 slices and the other machine to have
only the runners processing the 1:2 slices. This would require
starting the qrunners manually on each machine which is not a good
idea or patching mailmanctl along the lines of changing

def start_all_runners():
    kids = {}
    for qrname, count in mm_cfg.QRUNNERS:
        for slice in range(count):
            # queue runner name, slice, numslices, restart count
            info = (qrname, slice, count, 0)
            pid = start_runner(qrname, slice, count)
            kids[pid] = info
    return kids

to

def start_all_runners():
    kids = {}
    for qrname, count, machine, nummachines in mm_cfg.QRUNNERS:
        for slice in range(machine, count, nummachines):
            # queue runner name, slice, numslices, restart count
            info = (qrname, slice, count, 0)
            pid = start_runner(qrname, slice, count)
            kids[pid] = info
    return kids

where machine and nummachines are the number of this machine (0 or 1 in
this case) and nummachines is the total number of machines (2 in this
case). These would be added to the QRUNNER table in mm_cfg.py as
follows:

The first machine would have

QRUNNERS = [
    ('ArchRunner',     2,0,2), # messages for the archiver
    ('BounceRunner',   2,0,2), # for processing the qfile/bounces
directory
    ('CommandRunner',  2,0,2), # commands and bounces from the outside
world
    ('IncomingRunner', 2,0,2), # posts from the outside world
    ('NewsRunner',     2,0,2), # outgoing messages to the nntpd
    ('OutgoingRunner', 2,0,2), # outgoing messages to the smtpd
    ('VirginRunner',   2,0,2), # internally crafted (virgin birth)
messages
    ('RetryRunner',    2,0,2), # retry temporarily failed deliveries
    ]

and the second would be as above except with 2,1,2 instead of 2,0,2.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan