[Mailman-Users] Load-balancing mailman between two servers
gwaugh at scu.edu.au
Wed Nov 29 00:59:50 CET 2006
Barry Warsaw wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> On Nov 28, 2006, at 6:08 PM, Guy Waugh wrote:
>> I'm still wondering whether I should be NFS-sharing the qfiles
>> directory. I haven't delved into the Mailman source code to try to
>> figure this out, but...
> You should be able to NFS share the qfiles directory, but you want to
> be careful about how you set up your qrunners. However, this probably
> won't help you with what I think you really want to do (IIUC), which
> is load balance the web interface.
> First, pending messages are not kept in qfiles -- that's only for
> messages that are being processed by the mail delivery subsystem. A
> message that's waiting for moderation will get dequeued until it's
> approved, at which point it will be re-queued into the appropriate
> qfile directory.
> Access to the "databases" which manage these pending files are all
> protected by Mailman's lockfile implementation, which has had a long
> stable history and a high probability of being NFS-safe, modulo bugs
> in specific NFS implementations of course. So as long as your web
> requests can be completed within the lock timeouts, you should be able
> to load-balance admindb management across multiple web servers. Of
> course, while one server is accessing a list, no other processing for
> that list will occur on any other machine, as those other machines
> wait for the first machine's list lock to be released. However,
> processing involving other lists can still occur, as can outgoing mail
> delivery, which does not need to acquire a list lock.
> The story with qfiles is this: every qfile lives in its own little
> slice of sha1 hash space and each hash slice is (supposed to be) owned
> by exactly one qrunner process. This allows the qrunner to process
> the messages in its hash slice without having to deal with pesky locks
> which slows things down as contentions are serialized (a good thing
> when dealing with databases, a bad thing when you're trying to churn
> out a stream of messages). Thus, if you're looking to load balance
> qfile directory processing, you can still do that if you assign each
> qrunner process on each machine a unique slice of the hash space -- it
> must be unique across all machines. IOW, machine 1 could handle the
> odd slices of qfile/in while machine 2 could handle the even slices.
> Or you could have 8 qrunners on each machine and slice up qfiles/in 16
> ways (the implementation requires a factor of 2 in the number of hash
> Of course, if machine 1 went down, all the messages in its hash slices
> would sit unprocessed, but it would be a fairly simple matter to
> reconfigure machine 2 to handle machine 1's slices, or to bring up a
> fallback machine to handle those slices in the meantime.
> That's the intent anyway <wink>. I hope this makes sense and helps
> you better plan your operational environment.
> - -Barry
Great, thanks Barry...
So, it sounds like all information about pending posts etc. is held in
the databases in the directories I'm already NFS-sharing (i.e. archives,
data and lists), and the qfiles directory is specific to the qrunner
running on that machine, so I don't need to worry about NFS-sharing any
of the other Mailman directories.
Unix System Administrator
IT&TS, Southern Cross University
Lismore, NSW, Australia
Email: gwaugh at scu.edu.au
Ph.: +61 2 6620 3196
Fax: +61 2 6620 3033
More information about the Mailman-Users