[Mailman-Users] load-balance 2.1.x

[Mark Sapiro]

David Lee wrote:
>
>(This is in relation to 2.0.9, but I would probably move to 2.1.11.)
>
>I've been reading FAQ 4.75 ("How do I set up ... to load-balance ...") and
>Barry's archived email of 2006/11/29 "Load-balancing mailman between two
>servers".
>
>They talk about setting up multiple "slices" for "qfiles".  But it's not
>clear how this is done.  I suspect I need to copy the "QRUNNERS"
>definition across from Defaults.py to mm_cfg.py, and then do some sort of
>edit.  And that this edit should somehow specify both a power-of-2 number
>of slices (same on both machines), and also some sort of specification of
>which slices this machine should run (different on both machines).  But
>what is that edit?  Is it changing the "1" for one or more of the QRUNNER
>entries (which?) to something like "1:2" (machine1) and "2:2" (machine2)?
>
>Have I missed something in the doucmentation about this particular aspect?


You need to do more than just edit the QRUNNERS list in mm_cfg.py . If
you just change the 1's to 2's, mailmanctl will start two of each
runner on each machine, a 0:2 runner and a 1:2 runner (slices are
numbered from 0).

You also need to modify mailmanctl, so that on one machine it starts
only the 0:2 sliced runners and on the other only the 1:2 sliced
runners.

If you are having separate mm_cfg.py files per machine (see below), it
would be fairly simple to modify the structure of the QRUNNERS list to
specify not only the number of slices, but which slices to start. This
would be done by changing the start_all_runners() function in
mailmanctl.


>Looking more generally, is there any other overall advice on such
>clustering?


You'll find some information in the hits at
<http://www.google.com/search?q=site%3Amail.python.org+inurl%3Amailman+load+balancing>


>I would plan:
>
>o  two front-end Mailman machines, clustered via Linux-HA/heartbeat (of
>which we already have experience).  Ideally these would run live/live,
>maintaining a pair of round-robin service-IP addresses.  (With a machine
>failure, the other machine would adopt the failed machine's service-IP.)
>
>o  a central (NetApp) NFS server would store all the Mailman data (though
>"mm_cfg.py" would presumably be maintained within each machine).  Are
>there any other considerations about what data-storage would be best kept
>per-machine?


Having separate mm_cfg.py files per machine might be tricky. I have no
experience with this, so I'm probably overlooking things, but my
initial thought is it would be easy to just create a symlink from
Mailman/mm_cfg.py to a file outside the shared file system, but this
is problematic. What will happen is the first process that imports
mm_cfg will read it from the symlinked file, but will create
mm_cfg.pyc in the Mailman/ directory and subsequent processes will
just read the shared mm_cfg.pyc. You can work around this by also
having a compiled mm_cfg.pyc symlinked from Mailman/mm_cfg.pyc, but
you have to be careful, because if you edit mm_cfg.py and then just
import it, Python will recompile it and put the new mm_cfg.pyc in
Mailman/mm_cfg.pyc removing the symlink.


>So I'd need to set up that multiple slice/qrunner stuff.  And probably
>(for unattended operation) arrange that failover would reconfigure the
>good machine also to run all slices; similarly that failback (whether
>manual or automatic) after recovery would revert to "only my slices"
>operation.
>
>
>Has this been written up anywhere?  If not, I'd be happy to keep notes of
>the process and submit them back to the Mailman documentation if that
>would help.


I'm not aware of anything beyond what you'll find with the above and
similar google searches.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan