Hi,
I've been going through the archives and I've picked up bits and pieces of the information I need, but in most cases NFS is discussed. I've got 5 Ubuntu servers, 2 postfix mail gateways (MXs for all our domains) and 3 apache web servers. All of those have access to an ISCSI SAN with OCFS2 partitions.
Ideally I'd like to have the mail section running on both gateways (potentially more than just the 2 in the near future) and the web interface available from any of the web servers. This is more for availability than performance. Should separate installations of the web section on each web server be fine or is there shared information needed there? And since I'm using a clustered fs, would I still need to slice up the qfiles directory structure for the mail section?
Or would having the entire mailman installation on shared storage and individual apache/postfix configs per machine be the way to go?
Any other suggestions or advice about this would be greatly appreciated as well.
Thanks Guy
-- Don't just do something...sit there!
Guy wrote:
I've been going through the archives and I've picked up bits and pieces of the information I need, but in most cases NFS is discussed. I've got 5 Ubuntu servers, 2 postfix mail gateways (MXs for all our domains) and 3 apache web servers. All of those have access to an ISCSI SAN with OCFS2 partitions.
Ideally I'd like to have the mail section running on both gateways (potentially more than just the 2 in the near future) and the web interface available from any of the web servers. This is more for availability than performance. Should separate installations of the web section on each web server be fine or is there shared information needed there?
There is shared information in all of the archives, data, lists, locks, logs and qfiles directories. These need to be on shared storage so they are accessible to all web and mail servers.
By the 'mail section', I'm guessing you mean the qrunners. The qrunners need access to all the shared data, and if there are multiple qrunners processing the same queue, they need to be sliced so they don't interfere with each other.
I suppose you could have a separate set of qrunners and separate mail delivery on each mail server, each with its own set of local qfiles, but they would still need access to the shared archives, data, lists, locks and logs directories (well, they could have their own logs too).
And since I'm using a clustered fs, would I still need to slice up the qfiles directory structure for the mail section?
Slicing is not a directory structure thing. Each queue entry is a file in the corresponding queue directory. E.g., qfiles/in/file. Slicing just divides the file name space into sections each of which is processed by its own runner.
Or would having the entire mailman installation on shared storage and individual apache/postfix configs per machine be the way to go?
It's one way to go. Keep in mind that much of the code is common to all processes, so having local code will result in much duplication. OTOH, it will reduce contention on the shared store. This is not a tradeoff that I can give a pat answer for since I don't know what your load is or what the capability of your shared fs is, and I have virtually no experience with this anyway.
Any other suggestions or advice about this would be greatly appreciated as well.
I hope this helps. If it doesn't answer your questions or raises other, ask again.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks for the explanation Mark.
On 22 January 2010 02:28, Mark Sapiro <mark@msapiro.net> wrote:
There is shared information in all of the archives, data, lists, locks, logs and qfiles directories. These need to be on shared storage so they are accessible to all web and mail servers.
By the 'mail section', I'm guessing you mean the qrunners. The qrunners need access to all the shared data, and if there are multiple qrunners processing the same queue, they need to be sliced so they don't interfere with each other.
I've tried installing Mailman on Ubuntu from packages since it includes config for postfix-to-mailman.py. It seems to split a lot of the stuff between /usr, /var and /etc.
/var/lib/mailman looks like this: root@ocelot:/var/lib/mailman# ls -alF total 36 drwxrwsr-x 8 root list 4096 2010-01-28 11:32 ./ drwxr-xr-x 45 root root 4096 2010-01-28 11:32 ../ drwxrwsr-x 4 root list 4096 2010-01-28 11:32 archives/ lrwxrwxrwx 1 root root 20 2010-01-28 11:32 bin -> /usr/lib/mailman/bin/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 cgi-bin -> /usr/lib/cgi-bin/mailman/ lrwxrwxrwx 1 root root 21 2010-01-28 11:32 cron -> /usr/lib/mailman/cron/ drwxrwsr-x 2 root list 4096 2010-01-28 11:32 data/ lrwxrwxrwx 1 root root 25 2010-01-28 11:32 icons -> /usr/share/images/mailman/ drwxrwsr-x 3 root list 4096 2010-01-28 12:13 lists/ lrwxrwxrwx 1 root root 17 2010-01-28 11:32 locks -> /var/lock/mailman/ lrwxrwxrwx 1 root root 16 2010-01-28 11:32 logs -> /var/log/mailman/ lrwxrwxrwx 1 root root 21 2010-01-28 11:32 mail -> /usr/lib/mailman/mail/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 Mailman -> /usr/lib/mailman/Mailman/ drwxrwsr-x 37 root list 4096 2010-01-28 11:32 messages/ drwxrwsr-x 3 list list 4096 2010-01-28 12:13 qfiles/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 scripts -> /usr/lib/mailman/scripts/ drwxrwsr-x 2 root list 4096 2009-06-01 12:35 spam/ lrwxrwxrwx 1 root root 12 2010-01-28 11:32 templates -> /etc/mailman/ -rw-r--r-- 1 root list 11 2009-06-01 12:36 .version
If I put that directory on the SAN and move locks and logs to that location rather than where they pointing now, that should cover everything that needs to be shared right? I'm also uncertain of the crontabs since I'm not totally sure what all of them do. Should I only enable the crontab.in stuff on one of the mail servers? The Ubuntu installation puts that per machine even if /var/lib/mailman is on shared storage but I can't imagine all the machines needing to run all the admin crons? crontab.in looks like this under Ubuntu: # At 8AM every day, mail reminders to admins as to pending requests. # They are less likely to ignore these reminders if they're mailed # early in the morning, but of course, this is local time... ;) 0 8 * * * /usr/bin/python -S /var/lib/mailman/cron/checkdbs # # At 9AM, send notifications to disabled members that are due to be # reminded to re-enable their accounts. 0 9 * * * /usr/bin/python -S /var/lib/mailman/cron/disabled # # Noon, mail digests for lists that do periodic as well as threshhold delivery. 0 12 * * * /usr/bin/python -S /var/lib/mailman/cron/senddigests # # 5 AM on the first of each month, mail out password reminders. 0 5 1 * * /usr/bin/python -S /var/lib/mailman/cron/mailpasswds # # Every 5 mins, try to gate news to mail. You can comment this one out # if you don't want to allow gating, or don't have any going on right now, # or want to exclusively use a callback strategy instead of polling. 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/bin/python -S /var/lib/mailman/cron/gate_news # # At 3:27am every night, regenerate the gzip'd archive file. Only # turn this on if the internal archiver is used and # GZIP_ARCHIVE_TXT_FILES is false in mm_cfg.py 27 3 * * * /usr/bin/python -S /var/lib/mailman/cron/nightly_gzip # # At 4:30AM daily, cull old entries from the 'bad' and 'shunt' queues. 30 4 * * * /usr/bin/python -S /var/lib/mailman/cron/cull_bad_shunt
Slicing is not a directory structure thing. Each queue entry is a file in the corresponding queue directory. E.g., qfiles/in/file. Slicing just divides the file name space into sections each of which is processed by its own runner.
Does slicing still require patching mailmanctl as laid out here - http://mail.python.org/pipermail/mailman-users/2008-March/060753.html? If so, is there an official patch for this? Or is the example shown safe to use?
Thanks Guy
-- Don't just do something...sit there!
On 1/28/2010 7:05 AM, Guy wrote:
I've tried installing Mailman on Ubuntu from packages since it includes config for postfix-to-mailman.py. It seems to split a lot of the stuff between /usr, /var and /etc.
/var/lib/mailman looks like this: root@ocelot:/var/lib/mailman# ls -alF total 36 drwxrwsr-x 8 root list 4096 2010-01-28 11:32 ./ drwxr-xr-x 45 root root 4096 2010-01-28 11:32 ../ drwxrwsr-x 4 root list 4096 2010-01-28 11:32 archives/ lrwxrwxrwx 1 root root 20 2010-01-28 11:32 bin -> /usr/lib/mailman/bin/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 cgi-bin -> /usr/lib/cgi-bin/mailman/ lrwxrwxrwx 1 root root 21 2010-01-28 11:32 cron -> /usr/lib/mailman/cron/ drwxrwsr-x 2 root list 4096 2010-01-28 11:32 data/ lrwxrwxrwx 1 root root 25 2010-01-28 11:32 icons -> /usr/share/images/mailman/ drwxrwsr-x 3 root list 4096 2010-01-28 12:13 lists/ lrwxrwxrwx 1 root root 17 2010-01-28 11:32 locks -> /var/lock/mailman/ lrwxrwxrwx 1 root root 16 2010-01-28 11:32 logs -> /var/log/mailman/ lrwxrwxrwx 1 root root 21 2010-01-28 11:32 mail -> /usr/lib/mailman/mail/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 Mailman -> /usr/lib/mailman/Mailman/ drwxrwsr-x 37 root list 4096 2010-01-28 11:32 messages/ drwxrwsr-x 3 list list 4096 2010-01-28 12:13 qfiles/ lrwxrwxrwx 1 root root 24 2010-01-28 11:32 scripts -> /usr/lib/mailman/scripts/ drwxrwsr-x 2 root list 4096 2009-06-01 12:35 spam/ lrwxrwxrwx 1 root root 12 2010-01-28 11:32 templates -> /etc/mailman/ -rw-r--r-- 1 root list 11 2009-06-01 12:36 .version
If I put that directory on the SAN and move locks and logs to that location rather than where they pointing now, that should cover everything that needs to be shared right?
Yes.
I'm also uncertain of the crontabs since I'm not totally sure what all of them do. Should I only enable the crontab.in stuff on one of the mail servers? The Ubuntu installation puts that per machine even if /var/lib/mailman is on shared storage but I can't imagine all the machines needing to run all the admin crons?
Right. I would suggest installing the crontab on only one machine. There will be problems with duplication of some notices if their respective crons are run more than once. Others can safely run on multiple servers, but there is no need.
crontab.in looks like this under Ubuntu: # At 8AM every day, mail reminders to admins as to pending requests. # They are less likely to ignore these reminders if they're mailed # early in the morning, but of course, this is local time... ;) 0 8 * * * /usr/bin/python -S /var/lib/mailman/cron/checkdbs
checkdbs will produce duplicate notices if run more than once
# # At 9AM, send notifications to disabled members that are due to be # reminded to re-enable their accounts. 0 9 * * * /usr/bin/python -S /var/lib/mailman/cron/disabled # # Noon, mail digests for lists that do periodic as well as threshhold delivery. 0 12 * * * /usr/bin/python -S /var/lib/mailman/cron/senddigests # # 5 AM on the first of each month, mail out password reminders. 0 5 1 * * /usr/bin/python -S /var/lib/mailman/cron/mailpasswds
mailpasswds will produce duplicate notices if run more than once
# # Every 5 mins, try to gate news to mail. You can comment this one out # if you don't want to allow gating, or don't have any going on right now, # or want to exclusively use a callback strategy instead of polling. 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/bin/python -S /var/lib/mailman/cron/gate_news # # At 3:27am every night, regenerate the gzip'd archive file. Only # turn this on if the internal archiver is used and # GZIP_ARCHIVE_TXT_FILES is false in mm_cfg.py 27 3 * * * /usr/bin/python -S /var/lib/mailman/cron/nightly_gzip # # At 4:30AM daily, cull old entries from the 'bad' and 'shunt' queues. 30 4 * * * /usr/bin/python -S /var/lib/mailman/cron/
cull_bad_shunt does no locking so there will be race conditions that could cause (harmless) errors - remove or rename of a non-existent file already removed or renamed by another instance.
The others not noted above as producing duplicates are probably safe because of locking, but there's no need to have that lock contention (and I may be wrong about their being safe).
Slicing is not a directory structure thing. Each queue entry is a file in the corresponding queue directory. E.g., qfiles/in/file. Slicing just divides the file name space into sections each of which is processed by its own runner.
Does slicing still require patching mailmanctl as laid out here - http://mail.python.org/pipermail/mailman-users/2008-March/060753.html? If so, is there an official patch for this? Or is the example shown safe to use?
There is no 'official' patch, but the one you point to is good. I have updated the FAQ at <http://wiki.list.org/x/nYA9> to point to that post and thread.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Guy -
Mark Sapiro