Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Nov 28, 2006, at 6:08 PM, Guy Waugh wrote:
I'm still wondering whether I should be NFS-sharing the qfiles directory. I haven't delved into the Mailman source code to try to figure this out, but...
You should be able to NFS share the qfiles directory, but you want to be careful about how you set up your qrunners. However, this probably won't help you with what I think you really want to do (IIUC), which is load balance the web interface.
First, pending messages are not kept in qfiles -- that's only for messages that are being processed by the mail delivery subsystem. A message that's waiting for moderation will get dequeued until it's approved, at which point it will be re-queued into the appropriate qfile directory.
Access to the "databases" which manage these pending files are all protected by Mailman's lockfile implementation, which has had a long stable history and a high probability of being NFS-safe, modulo bugs in specific NFS implementations of course. So as long as your web requests can be completed within the lock timeouts, you should be able to load-balance admindb management across multiple web servers. Of course, while one server is accessing a list, no other processing for that list will occur on any other machine, as those other machines wait for the first machine's list lock to be released. However, processing involving other lists can still occur, as can outgoing mail delivery, which does not need to acquire a list lock.
The story with qfiles is this: every qfile lives in its own little slice of sha1 hash space and each hash slice is (supposed to be) owned by exactly one qrunner process. This allows the qrunner to process the messages in its hash slice without having to deal with pesky locks which slows things down as contentions are serialized (a good thing when dealing with databases, a bad thing when you're trying to churn out a stream of messages). Thus, if you're looking to load balance qfile directory processing, you can still do that if you assign each qrunner process on each machine a unique slice of the hash space -- it must be unique across all machines. IOW, machine 1 could handle the odd slices of qfile/in while machine 2 could handle the even slices.
Or you could have 8 qrunners on each machine and slice up qfiles/in 16 ways (the implementation requires a factor of 2 in the number of hash slices).Of course, if machine 1 went down, all the messages in its hash slices would sit unprocessed, but it would be a fairly simple matter to reconfigure machine 2 to handle machine 1's slices, or to bring up a fallback machine to handle those slices in the meantime.
That's the intent anyway <wink>. I hope this makes sense and helps you better plan your operational environment.
- -Barry Great, thanks Barry...
So, it sounds like all information about pending posts etc. is held in the databases in the directories I'm already NFS-sharing (i.e. archives, data and lists), and the qfiles directory is specific to the qrunner running on that machine, so I don't need to worry about NFS-sharing any of the other Mailman directories.
Yeehoo!
Thanks again, Guy.
-- Guy Waugh Unix System Administrator IT&TS, Southern Cross University Lismore, NSW, Australia Email: gwaugh@scu.edu.au Ph.: +61 2 6620 3196 Fax: +61 2 6620 3033