[Mailman-Developers] Maildir delivery

Barry A. Warsaw barry@python.org
Thu, 8 Aug 2002 19:31:24 -0400

I spent a little bit of time today playing with a Qmail-style maildir
delivery mechanism instead of the current mail program approach.  I'm
using a modern Postfix.

The idea is that if we can avoid setting the delivery aliases to a
mail program (i.e. scripts/{post,bounces,confirm,...}) and instead let
the MTA do maildir delivery, we can avoid having to fork a process for
each message coming into the system.  Maildir is nice because we don't
have to worry about locking to avoid race conditions or contention on
the drop files.  Another advantage is that we can essentially do away
with the wrapper program, but see below for caveats.

Costs/risks might include increased disk i/o because of additional
file renames and unlinks, or increased cpu due to inefficiencies in
the code.

I believe I have the whole thing working.  There's a new qrunner
called MaildirRunner, which is fairly different internally than the
other qrunners.  But the idea is essentially the same: every time
through its loop it looks at the files in qfiles/maildir/new and
processes each one in turn.  If it can't atomically rename the file to
qfiles/maildir/cur/<base>:1,P it assumes some other MaildirRunner has
already processed the file and moves on.  So I think we can still have
multiple MaildirRunners and not have to worry about race conditions.

A couple of caveats:

- The mailprog approach actually encodes some useful information in
  the aliases, such as the listname and the "subqueue" the message is
  destined for (i.e. -bounces, -request, etc.).  These aren't
  available when using Maildir so we need to grok that information out
  of some header, preferrably the envelope recipient.

  Postfix puts the envelope recip in Delivery-To: and other MTAs do
  something similar using different headers, but of course there's no
  standard. ;).  If it gets this part wrong, it won't be able to
  deliver the message.  Still, I think it's tractable, and verp bounce
  detection has some similar issues anyway.

  An alternative is a more elaborate and inefficient maildir layout,
  where each list and each subqueue has its own new, cur, and tmp
  subdirectories.  Blech.

- If there are any problems in processing the maildir file,
  MaildirRunner leaves it in qfiles/maildir/cur/<base>:1,X and it will
  never look at it again.  The nice thing is that you need only mv one
  file back to qfiles/maildir/new to get MaildirRunner to look at it
  again.  I still think it's difficult-to-impossible to lose a message
  or deliver it twice.

- Because of the way Postfix handles the permissions on the files and
  directories, you would have to start mailmanctl as root or as
  mailman.  I don't know how many other people use mailmanctl's -u
  option and run the qrunners as their own uid.  I do it all the time
  for debugging, but I may be unique, so this may be a non-issue
  except for testing.

- To switch to Maildir delivery, you'd need to re-run bin/genaliases.
  I'm sure the list auto-detection recipies for Exim and qmail would
  have to change accordingly, and I don't know what special permission
  or filesystem layout issues might exist.

Where should I go from here?  I'm willing to put up a patch set on SF
if folks want to play with it.  I'm also willing to check it into cvs,
but /only/ if it's labeled experimental for MM2.1.  I think this could
improve the efficiency of handling incoming message, but it's equally
likely my code is being too naive.  I don't want to officially support
it for MM2.1, but I /would/ like to get some feedback and experience
with it.

Comments, thoughts?