[Mailman-Users] Mailman on Solaris-based web server

[Richard Barrett]

On 21 Oct 2004, at 20:43, Brad Knowles wrote:

> At 7:08 PM +0100 2004-10-21, Richard Barrett wrote:
>
>>  I have been warned by experts that NFS locking could be a problem 
>> with
>>  this way of working but thus far it has not proven to be a problem.
>
> 	Locking is the bane of any administrator using NFS.  Nick Christenson 
> wrote a seminal paper on how to build a scalable mail system using NFS 
> as the message store (see 
> <http://www.jetcafe.org/~npc/doc/mail_arch.html>).  It's hard to do 
> the same kind of thing in an IMAP environment -- the only IMAP server 
> I know of that uses Maildir (a storage format that is supposedly 
> NFS-friendly) is Courier-IMAP, and it has some serious scalability 
> issues.
>
> 	For large-scale systems, this is a real PITA.
>

Brad

In the light of your comments I looked at what I believe are the 
primary file contention issues for Mailman:

1. handling of files containing messages queued for processing as they 
are moved along the chain of queues from initial delivery by the MTA to 
handoff to the outbound MTA and to the archiver. The code in 
$prefix/Mailman/Queue/Switchboard.py provides the enqueueing and 
dequeueing functions. It appears to me that the operation of this code 
avoids dependence on file system locking during both file creation and 
deletion and is intentionally unaffected by a decision to use NFS 
mounts as the storage for Mailman's qfiles. The residual risk is of 
collisons in generating the file names of queued messages files using 
SHA digests but I think this is fairly small.

2. handling of files containing per list related data structures and 
productions of archiving processes. These operations are protected by a 
list locking scheme implemented using the code in 
$prefix/Mailman/LockFile.py The comments in this module suggest that 
specific efforts have been made to avoid deficiencies associated with 
NFS mounted file systems but there are cautionary notes such as 
ensuring that clocks on the systems concerned are adequately aligned.

My own, non-expert conclusion is that

a. Switchboard.py's code sidesteps any requirement for fine grain 
locking of individual message files when the are being queued and 
dequeued and this avoids deficiencies NFS might have in this respect.

b. The coarse grained locking used for protecting per list related data 
structures and archiving operations specifically addresses known 
deficiencies in NFS.

It seems to me that your reservations are reasonable but it is clear 
that the authors of this part of Mailman (all smarter men than I) were 
conscious of the potential problems and have made efforts to avoid 
them. This may be why, after more than 3 years of operation of a 
moderately active server, with everything of Mailman but the mail and 
cgi wrappers on NFS mounted storage, a major disaster has not struck 
the system yet because of the use of NFS. Indeed, disaster has been 
avoided by being able, courtesy of NFS, to rapidly switch to a backup 
server with all data intact when the primary server died; but other 
people's circumstances may be different and my strategy is only good 
because we have a high reliability NFS server for storing Mailman's 
files.

Richard

>>  The only problems I have had with Linux and NFS is due to what I 
>> believe
>>  to be a kernel lock handling problem on Linux.
>
> 	Cross-platform NFS server/client environments are usually the worst 
> possible thing you can do.  NFS appliance vendors like NetApp work 
> very hard to make their products compatible with the broadest array of 
> client OSes, and even they can't satisfy everyone.
>
> 	In my experience, this is the single most difficult/impossible 
> problem to solve in an NFS environment.
>
> -- 
> Brad Knowles, <brad at stop.mail-abuse.org>
>
> "Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety."
>
>     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
>     Assembly to the Governor, November 11, 1755
>
>   SAGE member since 1995.  See <http://www.sage.org/> for more info.
>