[Mailman-Developers] Maildir delivery
Barry A. Warsaw
barry@python.org
Thu, 8 Aug 2002 19:31:24 -0400
I spent a little bit of time today playing with a Qmail-style maildir
delivery mechanism instead of the current mail program approach. I'm
using a modern Postfix.
The idea is that if we can avoid setting the delivery aliases to a
mail program (i.e. scripts/{post,bounces,confirm,...}) and instead let
the MTA do maildir delivery, we can avoid having to fork a process for
each message coming into the system. Maildir is nice because we don't
have to worry about locking to avoid race conditions or contention on
the drop files. Another advantage is that we can essentially do away
with the wrapper program, but see below for caveats.
Costs/risks might include increased disk i/o because of additional
file renames and unlinks, or increased cpu due to inefficiencies in
the code.
I believe I have the whole thing working. There's a new qrunner
called MaildirRunner, which is fairly different internally than the
other qrunners. But the idea is essentially the same: every time
through its loop it looks at the files in qfiles/maildir/new and
processes each one in turn. If it can't atomically rename the file to
qfiles/maildir/cur/<base>:1,P it assumes some other MaildirRunner has
already processed the file and moves on. So I think we can still have
multiple MaildirRunners and not have to worry about race conditions.
A couple of caveats:
- The mailprog approach actually encodes some useful information in
the aliases, such as the listname and the "subqueue" the message is
destined for (i.e. -bounces, -request, etc.). These aren't
available when using Maildir so we need to grok that information out
of some header, preferrably the envelope recipient.
Postfix puts the envelope recip in Delivery-To: and other MTAs do
something similar using different headers, but of course there's no
standard. ;). If it gets this part wrong, it won't be able to
deliver the message. Still, I think it's tractable, and verp bounce
detection has some similar issues anyway.
An alternative is a more elaborate and inefficient maildir layout,
where each list and each subqueue has its own new, cur, and tmp
subdirectories. Blech.
- If there are any problems in processing the maildir file,
MaildirRunner leaves it in qfiles/maildir/cur/<base>:1,X and it will
never look at it again. The nice thing is that you need only mv one
file back to qfiles/maildir/new to get MaildirRunner to look at it
again. I still think it's difficult-to-impossible to lose a message
or deliver it twice.
- Because of the way Postfix handles the permissions on the files and
directories, you would have to start mailmanctl as root or as
mailman. I don't know how many other people use mailmanctl's -u
option and run the qrunners as their own uid. I do it all the time
for debugging, but I may be unique, so this may be a non-issue
except for testing.
- To switch to Maildir delivery, you'd need to re-run bin/genaliases.
I'm sure the list auto-detection recipies for Exim and qmail would
have to change accordingly, and I don't know what special permission
or filesystem layout issues might exist.
Where should I go from here? I'm willing to put up a patch set on SF
if folks want to play with it. I'm also willing to check it into cvs,
but /only/ if it's labeled experimental for MM2.1. I think this could
improve the efficiency of handling incoming message, but it's equally
likely my code is being too naive. I don't want to officially support
it for MM2.1, but I /would/ like to get some feedback and experience
with it.
Comments, thoughts?
-Barry