Tokio Kikuchi wrote:

Postfix.py: got duplicate warning while creating virtual-mailman. Shouldn't we include hostname in aliases? Otherwise, true virtual hosting breaks.

the virtual file should indeed contain the hostname. The aliases file should not. These are two separate files, both of which are necessary.

Example:

data/virtual-mailman: mailman@nle.com mailman mailman-admin@nle.com mailman-admin ...etc...

data/aliases: mailman: "|/home/mailman/mail/mailman post mailman" mailman-admin: "|/home/mailman/mail/mailman admin mailman" ...etc...

Both should be created, and both should be deleted when the list is removed. Currently, this is not the case.

Bob

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 27, 2006, at 12:51 AM, Dale Newfield wrote:

This of course begs the questions of how mailman distinguishes between the lists (what's the appropriate argument to the mailman binary, and whether there are any characters besides "." that are disallowed in domain names (what is a good encoding to use as a valid local
address)?

I've just read the Postfix documentation again to see if there are
any new features since the last time we implemented integration with
Postfix virtual domains. Here's the man page:

http://www.postfix.org/VIRTUAL_README.html

What looks interesting is the section entitled:

Postfix virtual MAILBOX example: separate domains, non-UNIX accounts

This appears to allow us to set up true virtual domains without
having to encode destination aliases. The trick though is that we
would use Maildir delivery for all incoming messages, something I'm
keen on switching to for Mailman 2.2 anyway. Maildir is way more
efficient than invoking a mail program per incoming message, Mailman
already supports Maildir (although it isn't the default), and AFAIK
all major MTAs support Maildir.

I'd like to know what you think about updating our Postfix virtual
delivery hooks to use this technique, and about making Maildir
delivery the default. We'd keep the old way around for MTAs that
don't support Maildir (which would be...?) but we'd deprecate that
delivery mechanism.

One thing I'm not sure of is the minimum version of Postfix this
would tie us to. On OS X 10.4 it looks like I've got Postfix 2.1.5
and according to postconf, it supports the necessary variables.

Thoughts?

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRrLK3EjvBPtnXfVAQK4sAQAkkFgWXxxdk1TxLVYxbb3CEO2y59XJtb5 dRcCT8qg0LyxTkFNGEqAQTi9iRIs2/RVuK1KUBVXpSpQbt7zrGfs0s/ukfPVlVbO /h45S+cQ8d9cF2s77PHSqiUgdGckMy7W6v15qJ9zrrMC+M38yqV5oAzpxNc660y8 UtXNleCdeGM= =dgKK -----END PGP SIGNATURE-----

At 3:04 PM -0400 9/27/06, Barry Warsaw wrote:

This appears to allow us to set up true virtual domains without having to encode destination aliases. The trick though is that we would use Maildir delivery for all incoming messages, something I'm keen on switching to for Mailman 2.2 anyway. Maildir is way more efficient than invoking a mail program per incoming message, Mailman already supports Maildir (although it isn't the default), and AFAIK all major MTAs support Maildir.

Sendmail knows nothing of the mailbox delivery method. That is left up to the Local Delivery Agent, which is usually /bin/mail and knows about 7th edition mbox-format, and not much else. You could always substitute a different LDA (e.g., procmail), but that would not be a standard part of sendmail.

Moreover, I'm not keen on Maildir. It makes a lot of trade-offs to try to get something that is NFS-safe, and I'm not convinced those trade-offs are worthwhile, especially not in a non-NFS environment. I think there are other ways that you could get the same benefits (in a mailbox directory solution), without getting the major drawbacks of Maildir per se.

Among other things Maildir creates really hairy long filenames, which can easily blow the iname/inode caching built into most filesystems -- you could get the same benefit by using a better filename naming/hashing scheme with fewer characters. It also does a lot of excessive synchronous meta-data operations (e.g., creating files, renaming files, etc...), and that can place a heavy load on the underlying filesystem.

In his paper regarding what they built for the Earthlinnk mail system, Nick Christenson has clearly proven that you can use the atomic creat() system call in a way that eliminates the need for file locking on NFS, without all the various baggage that Maildir brings to the table.

Mark Crispin also has a lot of good things to say about the weaknesses inherent in Maildir. You should read his comments, too.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

At 9:34 PM -0500 9/27/06, Brad Knowles wrote:

Moreover, I'm not keen on Maildir. It makes a lot of trade-offs to try to get something that is NFS-safe, and I'm not convinced those trade-offs are worthwhile, especially not in a non-NFS environment.

One other problem with Maildir -- it throws all message files into the same directory, and doesn't use a hashed directory scheme. If we're going to do directory hashing, I would think we'd want to do it within our mailbox storage mechanism as well as elsewhere in the queueing system.

In his paper regarding what they built for the Earthlinnk mail system, Nick Christenson has clearly proven that you can use the atomic creat() system call in a way that eliminates the need for file locking on NFS, without all the various baggage that Maildir brings to the table.

If you want to read Nick's paper, go to http://www.jetcafe.org/npc/doc/mail_arch.html. Note that he's also the author of the book _sendmail Performance Tuning_, see http://www.jetcafe.org/npc/book/sendmail/.

Mark Crispin also has a lot of good things to say about the weaknesses inherent in Maildir. You should read his comments, too.

Mark's comments can be read at http://www.washington.edu/imap/documentation/formats.txt.html. Do a "find" on the page for "maildir", and read from there.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 27, 2006, at 10:34 PM, Brad Knowles wrote:

At 3:04 PM -0400 9/27/06, Barry Warsaw wrote:

...

This appears to allow us to set up true virtual domains without having to encode destination aliases. The trick though is that we would use Maildir delivery for all incoming messages, something I'm keen on switching to for Mailman 2.2 anyway. Maildir is way more efficient than invoking a mail program per incoming message, Mailman already supports Maildir (although it isn't the default), and AFAIK all major MTAs support Maildir.

Sendmail knows nothing of the mailbox delivery method. That is left up to the Local Delivery Agent, which is usually /bin/mail and knows about 7th edition mbox-format, and not much else. You could always substitute a different LDA (e.g., procmail), but that would not be a standard part of sendmail.

I'm definitely not proposing to get rid of deliver to program, so at
worst, Sendmail users will continue to use this method. Is there a
better way to get the message from Sendmail into Mailman's incoming
queue?

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRtH6XEjvBPtnXfVAQLnawP9FhQDaHH4TtFlV2oo/FwT1YipNuxl3Kr1 vjswKQtKgQN7QUYuSZwnOSZ3O7PBHjzSNXbuu2GQt2hEYYm0VQkbO4I173EO4HKR xSkPGDrBU6n3NDC3WjV7BSedBSHlPnJXmnEsTobxeUQTCeeJRwsxO3QdPef22kn5 qmVOmMwfkBw= =0R2g -----END PGP SIGNATURE-----

At 11:54 PM -0400 9/27/06, Barry Warsaw wrote:

Looking at Postfix, what other options are readily available? I suppose you could try to hook into the transport maps, but if I understand them correctly, you're still talking about forking a process per message.

Use LMTP instead. This will allow you to completely avoid an intermediate (and unnecessary) queue-to-disk stage.

                   Maybe LMTP to a daemon process is another

option, but there appears to be no documentation on www.postfix.org about LMTP.

Wietse didn't invent LMTP, he's just using the technique that was invented by others. Look in the sendmail source code, it includes an LDA that implements LMTP. For that matter, I think postfix also includes an LDA that implements LMTP.

LMTP is basically just exactly like SMTP, except that it's via the localhost interface only, and simplifies a number of other assumptions as well. So, if you've got code that can do SMTP, then you've already got code that can do LMTP.

Frankly, there's not much to LMTP, which I think is a large part of why you're not likely to see a great deal written about it within the sendmail and postfix codebases.

As for what's best for other MTAs, that's a good question. I think we'll always have to support delivery-to-program since that seems like it's the lowest common method. If there are delivery mechanisms that make more sense for specific MTAs, then I'm all for including them,

LMTP is probably the best and most native method for both sendmail and postfix. I can't speak for other MTAs.

                              but others who are more

familiar with those mailer servers will have to help (read: donate code).

IIRC, sendmail has a Berkeley-style copyright, so the "donation" of code should not be an issue. Converting the code from C to Python, now that may be a bit more of a problem.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 28, 2006, at 12:15 AM, Carson Gaspar wrote:

--On Wednesday, September 27, 2006 11:54 PM -0400 Barry Warsaw barry@python.org wrote:

...

Looking at Postfix, what other options are readily available? I
suppose you could try to hook into the transport maps, but if I
understand them correctly, you're still talking about forking a process per message. Maybe LMTP to a daemon process is another option, but there
appears to be no documentation on www.postfix.org about LMTP.

LMTP is fully supported in postfix. You just set mailbox_transport
(or trhe right hand side of your transport map) to lmtp:... - see postfix's
smtp(8). I've used it to deliver to cyrus imapd for a long time now. See RFC
2033 for the LMTP standard.

Cool, thank for the reference. I'll read up on that.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRtSNXEjvBPtnXfVAQLjfAP9GLgoymsgU+O8oCp+yHn2qR1qE4lkqnId WtSNvUP7cAzL/CtJ6HGdp+Kbyqw/raC9yS797EGBniHu3Y8ACCaInyY0mVS5a8h+ 0oIejbeGCvxJCsvPt653H7MSjgHwdv/xeLZBZx4xKxw3c8fCuYG9eUQrli6eXXIq TkF93BbrtOQ= =Oeh2 -----END PGP SIGNATURE-----

--On Thursday, September 28, 2006 1:07 AM -0400 Barry Warsaw barry@python.org wrote:

Or is there some way I'm missing that would allow us to segregate some domain traffic to Mailman's LMTP server and other traffic to Postfix's standard transports? What about Sendmail?

Shouldn't be an issue with postfix. From the default postfix transport map template:

# TABLE LOOKUP # With lookups from indexed files such as DB or DBM, or from # networked tables such as NIS, LDAP or SQL, patterns are # tried in the order as listed below: # # user+extension@domain transport:nexthop # Mail for user+extension@domain is delivered through # transport to nexthop. # # user@domain transport:nexthop # Mail for user@domain is delivered through transport # to nexthop. # # domain transport:nexthop # Mail for domain is delivered through transport to # nexthop. # # .domain transport:nexthop # Mail for any subdomain of domain is delivered # through transport to nexthop. This applies only # when the string transport_maps is not listed in the # parent_domain_matches_subdomains configuration set- # ting. Otherwise, a domain name matches itself and # its subdomains. # # Note 1: the special pattern * represents any address (i.e. # it functions as the wild-card pattern). # # Note 2: the null recipient address is looked up as # $empty_address_recipient@$myhostname (default: mailer-dae- # mon@hostname).

On Wed, 2006-09-27 at 23:25 -0500, Brad Knowles wrote:

LMTP is probably the best and most native method for both sendmail and postfix. I can't speak for other MTAs.

Exim can do LMTP, over a pipe (ie fork/exec program), a socket or TCP/IP.

Nigel.

-- [ Nigel Metheringham Nigel.Metheringham@InTechnology.co.uk ] [ - Comments in this message are my own and not ITO opinion/policy - ]

At 8:12 AM -0400 9/28/06, Barry Warsaw wrote:

What I find really intriguing about this approach is the ability to reject some messages immediately, presumably allowing the MTA to bounce them.

Yup.

           We could reject the message then before it entered

Mailman's incoming queue.

Indeed, that's a key advantage. IIRC, procmail does this with the system-wide and user-defined rulesets.

I did a quick Google search to see if there were any GPL'd LMTP servers we could piggyback on, the idea being that if we could find a shell of a C program we could embed Python in and talk directly to Mailman during the LMTP protocol.

Does it have to be GPL? Is a Berkeley-type license not okay? Checking the source for sendmail 8.13.8, I find that there is an official part of the package which includes the LDA "mail.local", which is LMTP-capable, among other things. It can also do user mailbox hashing, based on the username. You can either hash directly to a path like /var/mail/u/s/user or use an MD5 hash of the username in a base64 representation (changing "/" to "_"), and you can control how many levels of hashing are to be used.

Seems to me like this would be a pretty obvious candidate.

   Postfix has an lmtp server, but it seems fairly heavyweight

(being tied into the smtp server) and it's not clear to me we could combine our GPL code with Postfix's license.

Please check out the sendmail mail.local stuff and tell me if this is a better alternative. If you need a different license, please let me know -- I've known Eric for many years (since way before the company existed). While I won't make any guarantees, I will say that if we need a different license, I imagine that I can get a more sympathetic ear than you might otherwise be able to find.

ISTM that the trade-off then is rolling our own LMTP server vs. doing maildir delivery. Are we confident that we can implement a high performance enough server that would give us better throughput than maildir would? In Python?

Dunno about doing it in Python, but I will say that going to Maildir as an additional queue-on-disk mechanism on top of everything else we're already doing seems to be a big step backward in terms of potential performance issues and I don't really see any significant positive benefit.

At AOL, we used to use a queue-on-disk method for the Internet mail gateways. Sendmail would take the incoming message, hand it off to a custom LDA, the custom LDA would then dump that in a disk queue asynchronously, then a synchronous queue runner process would come along and pick up the messages and send them over to Stratus. Believe me, this system sucked big time -- we had never ending problems with disk queues building up to the point where the queue runners could never possibly catch up, etc....

And I'm not seeing any real significant operational differences here between what you're talking about doing and what AOL abandoned years ago. Okay, so you're talking about using Maildir instead of a typical "linear" queue-on-disk and you don't have to do file locking to guarantee queue entry creation, but that's still dumping everything into a single directory from which we then have to scan and pull stuff out and you probably do still have to do some sort of file locking in order to make sure that the input and output queue mechanisms don't step on each others toes.

It might be fun to try, but OTOH it /is/ a distraction from other MM 2.2 work that needs to get done. So unless anybody has any leads on existing GPL-compatible code we could use, or feels really motivated to work on a Python version, I'm inclined to go with maildir for MM2.2. It's not like we couldn't add LMTP at some later point.

The single queue directory on disk is already one of our biggest single bottlenecks. I don't see how using Maildir as a delivery mechanism from the MTA to Mailman is going to improve that.

In fact, it seems to me like we're just adding yet another bottleneck of exactly the same sort that we're trying to eliminate elsewhere, but with some additional drawbacks that are unique to Maildir and which will make our overall system performance even worse than it is today.

If we're going to make a big change, it seems to me that LMTP makes much more sense than Maildir. If we can't do LMTP, then I think we'd be much better off working on eliminating other bottlenecks in the system as opposed to adding yet another totally new source of bottlenecks that result from implementing Maildir.

It seems to me that this idea is a case of:

1.  We have to do Something.
2.  This is something.
3.  Therefore, we have to do This.

I think we want to think long and hard about this idea and all it's potential drawbacks and new bottleneck sources, before we take that first step off the cliff.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 28, 2006, at 9:21 AM, Brad Knowles wrote:

At 8:12 AM -0400 9/28/06, Barry Warsaw wrote:

Does it have to be GPL? Is a Berkeley-type license not okay?

GPL would be best, but Berkeley is probably okay. We'd probably want
to get confirmation of that from the FSF. The key thing is that it
has to be compatible with the GPL (and the Python Software LIcense --
see below) so that we can combine the whole kit and kaboodle.

Checking the source for sendmail 8.13.8, I find that there is an
official part of the package which includes the LDA "mail.local",
which is LMTP-capable, among other things. It can also do user
mailbox hashing, based on the username. You can either hash
directly to a path like /var/mail/u/s/user or use an MD5 hash of
the username in a base64 representation (changing "/" to "_"), and
you can control how many levels of hashing are to be used.

Seems to me like this would be a pretty obvious candidate.

So a sketch of the architecture would be to embed Python via its C
API into mail.local, adding callbacks at each point in the LMTP
protocol. From there, we'd write the rules in Python so that they'd
do the message parsing, sanity checking, etc, returning status codes
which mail.local would use to respond to the LMTP command. There has
to be that Python hook, otherwise you can't get to Mailman's data
structures.

...

   Postfix has an lmtp server, but it seems fairly heavyweight

(being tied into the smtp server) and it's not clear to me we could combine our GPL code with Postfix's license.

Please check out the sendmail mail.local stuff and tell me if this
is a better alternative. If you need a different license, please
let me know -- I've known Eric for many years (since way before the
company existed). While I won't make any guarantees, I will say
that if we need a different license, I imagine that I can get a
more sympathetic ear than you might otherwise be able to find.

Cool, that's good to know.

...

ISTM that the trade-off then is rolling our own LMTP server vs.
doing maildir delivery. Are we confident that we can implement a high performance enough server that would give us better throughput than maildir would? In Python?

Dunno about doing it in Python, but I will say that going to
Maildir as an additional queue-on-disk mechanism on top of
everything else we're already doing seems to be a big step backward
in terms of potential performance issues and I don't really see any
significant positive benefit.

I don't think it's an additional queue-on-disk mechanism, certainly
in comparison to what we're doing today. In fact, thinking about it
more, a maildir approach would be better than what we have today not
only because it eliminates the extra process fork/exec, but because
we could segregate queue/in files into per-list subdirectories. That
way, you're not dumping all message destined for Mailman into one
directory. Not as good as directory hashing, but better than what we
have today.

I'll grant you that LMTP delivery has the potential to be the most
efficient mechanism by which messages get from the MTA into Mailman.
But it's certainly more work and more complicated than maildir; will
you grant that maildir is better than what we have today? Think of
it as a waystation on the road to the ultimate uber-performing list
server. :)

Let me just say that ideally, I think LMTP would be a great way to
go. It's not my top priority though. I'm looking for ways to get
more developers involved in the project, and this seems like a
perfect thing for someone seeking Mailman fame and fortune <wink>.
So, anyone care to take the challenge?

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRvXmHEjvBPtnXfVAQLdQAP8DRQdxi/rnHPIB4I+q3xReOpq2yeW7Wsp y+1AerklGjAhy9+NHOAV2h28rj9YBaS9cp0euJuSWTAoceJfeqBvYM6voL/mfs0I elTntMROq5diyzytfTxBz9qMDM+QAfuutcp7nuxlPCuYv7CskeAySZll7+v0P8eD 1unF56C5Dns= =R86i -----END PGP SIGNATURE-----

Brad Knowles wrote:

I think we're better off spending our resources working on trying to resolve the real bottleneck issues that we already know are present in our system as opposed to working on cool stuff that may or may not help but would require more overall changes to more parts of the system and with relatively lower potential payoff.

While I agree with Brad some days more than others, he has a good point here. Presumably every mailman installation is a multiplier for email messages (I.E.: many more go out than come in), so if we're worrying about the bottlenecks at the inbound side, I'm willing to bet they result in much more important bottlenecks on the outbound side...

-Dale

...

...

...

...

"Brad" == Brad Knowles "Re: [Mailman-Developers] LTMP for incoming mail" Thu, 28 Sep 2006 11:06:29 -0500

Brad> Maildir was not designed as an efficient queue-on-disk
Brad> strategy. 

Is in not possible to do Postfix virtual mailbox domains _without_ maildir style delivery? (Considering Postfix virtual mailbox domains is what lead to this, no?) Doesn't the example given in the Postfix VIRTUAL_README show delivery to a mailbox on line 11 and delivery to a maildir on line 12? Why not consider Postfix virtual mailbox domains and their workalikes with other MTAs separately and independently from the choices of delivery format and local delivery agents some of which may also be chosen independently?

Independently of the above and at least for Postfix, would it be worthwhile looking at the Postfix policy daemon plug-in as a way to query Mailman information during the rfc821 conversation and rejecting a lot of messages before DATA? Even more interesting, because of perhaps being portable, might be the milter facility that appeared in Postfix 2.3. I have not yet even looked at the later so it may AFIK be very wide of the mark.

    jam

At 1:29 PM -0400 9/28/06, John A. Martin wrote:

Is in not possible to do Postfix virtual mailbox domains _without_ maildir style delivery?

Probably, but I'm not sure it really buys us much of anything to have separate mailboxes for each list, as opposed to a queue processing mechanism that is generally more robust and capable of easily handling lots of simultaneous transactions.

Independently of the above and at least for Postfix, would it be worthwhile looking at the Postfix policy daemon plug-in as a way to query Mailman information during the rfc821 conversation and rejecting a lot of messages before DATA?

That's going to have the same issues as implementing LMTP, at least as far as it concerns performance and ability to handle these kinds of operations on a large-scale/real-time basis.

                            Even more interesting, because of

perhaps being portable, might be the milter facility that appeared in Postfix 2.3. I have not yet even looked at the later so it may AFIK be very wide of the mark.

Same issues for all. The particular protocol is not particularly relevant to this part of the discussion. Regardless of how you implement the system, it's going to have to do certain types of message scanning and parsing, checking against databases and/or blacklists, etc.... In all cases, so long as you're holding the sender open while you check all these things, you're going to have pretty much the same concerns regarding performance and scalability.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

At 2:40 PM -0400 9/28/06, Barry Warsaw wrote:

I would ask them if their license is GPL compatible. IOW, do they believe we can combine GPL code with theirs? Better yet would be cases where that's actually been done before.

I'll send a note and ask.

Remember, this discussion all started because Postfix virtual host delivery is broken on the trunk. The virtual_mailbox_maps feature is a new one since we last looked at how to integrate Mailman and Postfix.

But this is a pure postfix issue, and now we're talking about making potentially large architectural changes to the system to support this one MTA, and without necessarily giving consideration to whether that buys us anything for any of the other MTAs.

I understand that integration with postfix is sub-optimal today, but I'm not convinced that it makes sense to seriously consider an option that may result in throwing out all the other MTAs, in order to fix things with postfix. Worse yet would be trying to maintain two different systems, one for postfix and one for everyone else. Or making architectural changes to support the new stuff for postfix, which may hurt us for other MTAs.

At the very least, I think it makes sense to look at the overall cost/benefit ratio.

Let's assume that we have two systems that are otherwise identical, with roughly equivalent traffic. System A has a single big list, while system B has a number of smaller lists, but the overall aggregate traffic is equal.

With a Maildir solution, system A will see no benefit to the inbound queueing, because you're going to get the same level of contention within a single inbound directory for the one big list as you would for a single inbound directory for all lists on the system. System B would get a benefit, since each list would not be competing for immediate synchronous meta-data update resources with the other lists, although there would still be some intra-list competition.

With a hashed directory solution, both systems would see the same level of benefit, and intra-list competition would be no worse than inter-list competition. And if the competition were to get too high, you simply increase the level of directory hashing.

With a Maildir solution, you give up your ability to implement a hashed directory solution, because the MTA would no longer know how to write messages to your hashed mailbox-directory-per-list, and to get around that you'd have to have some sort of customized local delivery agent no matter what.

With a hashed directory solution, if necessary or desirable you could still implement a separate directory tree per list within your customized local delivery agent, and that directory tree per list could look however you want.

Moreover, a Maildir mailbox-per-list solution doesn't do anything for outbound queues, whereas a properly implemented hashed directory solution should affect outbound at least as much as inbound, at no additional implementation cost.

virtual_mailbox_maps don't appear to be useful for delivery-to-program, and delivery-to-program with Postfix virtual domains as we're doing them now has important disadvantages, most notably that we have to create both an alias and a virtual recipient, and we have to encode the domain name such that it's a valid alias, without introducing additional collisions. That's icky.

I know that our current solution is sub-optimal, but I'm not convinced that it's the only way to skin this cat. Moreover, I'm also not convinced that Maildir is the only effective way to make use of virtual_mailbox_maps.

I am pretty much convinced that using Maildir will effectively preclude the ability to make use of directory hashing, precisely because you're letting the MTA write directly to a poor standard interface instead of handling the internal issues in a manner that is opaque to the MTA.

I don't think there's anyway to really know without implementing it and doing some measurements. Since we won't be losing delivery-to-program, that would be possible.

True enough, but there is a cost in terms of lost opportunity, and pushing out the delivery schedule by long enough to determine which method is going to work better overall.

Nope, we simply have to implement a MaildirRunner to pull messages out of queue/in using the directory layout format we decide on.

With Maildir, you don't have any choice in what the directory layout will look like. That's standardized within the Maildir implementation, and you can't change that. Otherwise, you wouldn't be using Maildir anymore, you'd be using mailbox-directory-solution-that-looks-kinda-semi-sorta-like-maildir-but-modified-by-Mailman-and-incompatible-with-everything-else.

A directory hashing scheme is orthogonal to a maildir based queue/in scheme.

I'm not convinced of that. In fact, I'm convinced that they are pretty much mutually exclusive. That is, unless you're talking about using Maildir as a second level of queue-on-disk, before you get to the Mailman-internal queue-on-disk mechanism.

Now, if you are talking about two levels of queue-on-disk so that we can get both Maildir and queue directory hashing, I think that's going to be much, much worse than sticking with the existing postfix virtual domain solution.

Or someone running a huge site that would really benefit from LMTP could funnel a portion of their profits into paying us to add it <wink>.

I don't think we're doing enough traffic on python.org for them to justify paying for it. I don't think that Apple is doing enough traffic with Mailman for them to justify paying for it -- not with what we've heard about how the new(er) MacOS X hardware is performing, and especially not with the total lack of any support (or even acknowledgement) that we get from the corporate types.

I don't think that any of the open-source projects (like FreeBSD) are going to be in a position to pay for something like this, or to develop & contribute the necessary code, although they might be doing enough traffic that they could certainly use these features if they were available.

I think that only leaves us with a site like SourceForge, and I think you've probably got better contacts there than any of the rest of us.

Absolutely. But getting LMTP support into Mailman will still require a developer to step up and write code.

I'm not that concerned about LMTP. I think that's a big enough issue that we can leave that alone for now.

Maybe Tokio or Mark can be convinced, or maybe there's another developer lurking out there who would be interested. I just want to unbreak Postfix virtual domains and then fry our bigger fish.

I would like to see them unbroken, but I also don't want to see anything done that would preclude the use of hashed queue directories, or that would add a second level of queue-on-disk and yet another source of potential bottlenecks.

I think the thing you're missing is that we need to get the messages from the MTA into Mailman's incoming queue /somehow/, and we're basically limited by what the various MTAs have to offer.

I certainly was not understanding your point that you wanted to use this as a way to unbreak postfix virtual domains, no.

No, I didn't get that at all.

I'm still not convinced that this is the best way to unbreak postfix virtual domains, however.

Fixing Postfix virtual domain integration is a real problem that needs solving, which is how this whole thread started.

Agreed, this is a real problem that needs to be resolved.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 28, 2006, at 4:59 PM, Brad Knowles wrote:

I know that our current solution is sub-optimal, but I'm not
convinced that it's the only way to skin this cat. Moreover, I'm
also not convinced that Maildir is the only effective way to make
use of virtual_mailbox_maps.

It may not be. And maybe there's some way other than
virtual_mailbox_maps to better skin the Postfix virtual domain cat.
I haven't heard it yet though.

I am pretty much convinced that using Maildir will effectively
preclude the ability to make use of directory hashing, precisely
because you're letting the MTA write directly to a poor standard
interface instead of handling the internal issues in a manner that
is opaque to the MTA.

Not at all. If we implement directory hashing, all the other queues
will gain from them equally, regardless of what MTA is being used,
because after the incoming queue, the MTA doesn't figure into it.
The incoming queue has always been a bit special anyway.

With Maildir, you don't have any choice in what the directory
layout will look like. That's standardized within the Maildir
implementation, and you can't change that. Otherwise, you wouldn't
be using Maildir anymore, you'd be using mailbox-directory-solution- that-looks-kinda-semi-sorta-like-maildir-but-modified-by-Mailman- and-incompatible-with-everything-else.

Except that above the maildirs, you could partition the directories
by domain/list/recipient. At least, that's the way I read the docs
for virtual_mailbox_maps. I haven't yet tried it though, so maybe it
doesn't work the way I expect it to.

...

A directory hashing scheme is orthogonal to a maildir based queue/in scheme.

I'm not convinced of that. In fact, I'm convinced that they are
pretty much mutually exclusive. That is, unless you're talking
about using Maildir as a second level of queue-on-disk, before you
get to the Mailman-internal queue-on-disk mechanism.

Now, if you are talking about two levels of queue-on-disk so that
we can get both Maildir and queue directory hashing, I think that's
going to be much, much worse than sticking with the existing
postfix virtual domain solution.

This maildir proposal /only affects the incoming queue/ which has
always been special anyway. None of the other queues will be
maildir, and they can all be hashed directories. The two are
completely independent.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRRxdh3EjvBPtnXfVAQKkcwP/aJYv25I8jt2sh6YizKTB7iz8VKEenLRY 04DsuGv4RwZZqxsMO1a385XyjXGo7229mrdU14PHhoNMQ1MGoYPYAKVWc8B05n8X aBWUqNdnwobAnOTWMlK1TNY85wb5nWQ4O9KIgi/HmEz+9r/ujPtKUgeCLL3YyDvJ XEH9zOL7GOU= =rqoL -----END PGP SIGNATURE-----

At 11:19 PM -0400 9/28/06, emf wrote:

I can't find a filesystem that has a filename dependency for inode caching, so I suspect I'm completely misunderstanding this. Could you expand on that a bit?

Some filesystems implement an in-memory hash of recently accessed files, but the filenames are typically truncated to fourteen characters, and the paths to the files may likewise be truncated.

Maybe; but there are at least two filesystems (XFS, reiserfs) and likely more that handle file renaming/creating really cheaply, and have their own ninja ways of dealing with really large directories that are the product of a rather large amount of coding hours.

XFS and ReiserFS do not comprise the entire universe of all filesystems in the world in which Mailman will be operated.

There will be plenty of BSD, Solaris, HP-UX, MacOS X, and other OSes where Mailman will be used, and even on Linux you're much more likely to run into ext2fs or ext3fs than either XFS or ReiserFS on most of the several hundred distributions that are available.

Maildir has the advantage of being bog standard and readily comprehended. While I'm all in favor of some lmtp delivery mechanism, I don't see why we should continue inventing our own queue-on-disk approach merely to cater to poorly designed filesystems.

While XFS and ReiserFS may have their advantages (and XFS on SGI Irix is much better than XFS on Linux), we can't assume that any portion of the Mailman community will be using these kinds of filesystems. We must be more conservative in our estimates of what filesystem features will be available, and code accordingly.

If we were to assume that everyone had XFS, then let's assume they all have XFS on Irix, or even Veritas VxFS.

It seems to me like anyone likely to end up with a huge enough incoming mailman queue to care about Maildir's inefficiencies would also be able to put a sensible filesystem underneath it.

That may simply not be possible. Moreover, I have some real operational problems with both XFS on Linux and ReiserFS, and I would not run a production mail system using them. Maybe IBM's JFS, if I were forced to run a production mail system on Linux at all, but certainly not XFS or ReiserFS.

To be honest, I wouldn't run a real production mail system on anything less than Veritas VxFS, and I'd be real choosy about my underlying hardware, too -- think Hitachi, not EMC.

So your assumptions about what kinds of filesystems may or may not be appropriate are not necessarily going to coincide with the decisions that other people make, or the kinds of hardware and OS they may be forced to live with.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 29, 2006, at 9:52 AM, Tokio Kikuchi wrote:

Barry Warsaw wrote:

...

Other than that, we'd need a reliable standards-compliant LMTP server written in Python (and no, smtpd.py- or Twisted-based versions are not acceptable ;).

Why no smtpd.py ? There is a MailmanProxy Object in the code which
was written by you, Barry. Any SMTP server can become a LMTP server IIRC.

http://docs.python.org/lib/node624.html

Only because smtpd.py is asyncore based and I'm not convinced it can
be made high performance enough. An smtpd.py based LTMP server could
provide an interesting proof of concept though.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRR0sCHEjvBPtnXfVAQJPTwP/XONZhqcXwU29qfROOOJ29+ITWUl0uuu4 Eeafwq+15ndxOItEB5xXeEIc7/ctGDGAARuXsYME+ylvTk9bG3fWhN+kb+VdYuP1 GbAEPaNjJ90+3zjhyDGkKRjru4gH01KKEh5YJygYvIEyor8FcEr+fkrPpMK2ZTNX Mfw+IJBc73o= =VHNo -----END PGP SIGNATURE-----

At 12:23 PM -0400 9/29/06, emf wrote:

Furthermore, many MTAs *do* understand Maildir,

MTAs should be sticking to the job of transmitting e-mail between themselves. If they're spending any time mucking about with local mailbox formats, they're making a mistake. That's a job for a Local Delivery Agent, and not an MTA.

Now, granted many mail system packages also include one or more sample LDAs in the tarball, either as an official part of the system or in some sort of contrib/ directory somewhere, but let's make sure that we're using the proper terms for the proper objects.

Therefore, by definition, MTA != LDA.

                                             and most admins

do as well;

If this discussion has taught me anything, it's that after all these years we still have virtually no one in this business that really does understand Maildir or any other mailbox format, although many claim to do so.

         using our own queue-on-disk format means MTAs must

access Mailman via LTMP, pipe invocation, or the like, and if there are issues with the queue the administrator likely must learn our queue-on-disk format.

Our queue-on-disk format is already much simpler than Maildir, at least when it comes to directory structure, and the directory hashing schemes that I've been talking about have been around for many years. No new thought needs to be put into implementing them.

I even convinced Wietse that he should implement a lot of the same concepts, back when I first got involved in postfix in '98, when it was still being called VMailer.

Now, if you want to get outside of directory structure, our queue-on-disk format includes a lot of things that are Mailman-specific, such as creating message pickles, and I don't think that anyone is talking about getting rid of those aspects.

Most of the maildir phenomena you have an issue with wouldn't even arise in the use case under discussion; a mail would enter maildir/new , mailman would suck it out, and that would be that; renaming wouldn't occur and the number of elements in the queue is unlikely to become large enough to pressure filesystem indexing schemes.

You really need to go back and review exactly how messages are created using Maildir.

With Maildir, when a message comes in, a temporary file is opened with truncate (with certain measures taken to try to ensure that the selected filename will be unique), and if that system call succeeds, then the system appends the incoming message and renames the file, before it ever closes it.

If that creat() system call fails, then there is already a file by that name, and the LDA has to try again. This is how they "safely" create files on NFS, with an operation that is supposed to be atomic, and allows them to avoid file locking.

I'd have to check, but I think there are some more synchronous meta-data operations in here, too. Certainly, every time you look to see if more messages have come in, you have to scan the entire directory, and you have to stat() each and every file, and if you want to pick up the message and move that somewhere else, you're going to have to do further synchronous meta-data operations that involve locking the entire directory structure while they are taking place.

Now, your application may not see those locking, scanning, and stat operations, you may simply see "move this file to another location", but the underlying filesystem code has to do a lot of work to support that. And regardless of whether you're using locking or not, you still have race conditions that you have to code for -- or at least be aware of, because they are potential areas where you may have problems in the future.

If you don't read the comments that Mark Crispin has written about the weaknesses in Maildir, and you haven't read the code to see what it's actually doing, then I don't see how you can participate in this discussion.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

On Fri, 29 Sep 2006, Brad Knowles wrote:

If you don't read the comments that Mark Crispin has written about the weaknesses in Maildir, and you haven't read the code to see what it's actually doing, then I don't see how you can participate in this discussion.

I did read through both of the references you provided, more out of curiosity than any particular interest in whether or not Maildir is used to address the issues under discussion. To be honest, I don't see how either reference is particularly relevant. The Christenson paper is a nine year old architecture document that doesn't appear to make any mention of Maildir. At best it seems to imply that there are perhaps some better ways to solve some of the problems Maildir was designed to address. Crispin's comments appear to specifically address Maildir inefficiencies associated with management by an IMAP server. These inefficiencies are mostly (entirely?) unrelated to the delivery and one-time read scenario that is under discussion for Mailman (assuming I correctly understand the proposed feature). And to be fair, even the nature and severity of these Maildir inefficiencies is open to debate. See for example http://www.courier-mta.org/mbox-vs-maildir/#theend.

And since when is reading all source code for all programs, system calls, etc. associated with a Mailman function a requirement for making a comment or expressing an opinion on this list? If by chance you feel that you have such insight into the source code, share it (as you have) and leave it at that. Don't try to exclude people out of hand and scare off others who might have useful input. Among other things a list is a place for learning and gaining a deeper understanding of the issues associated with the topic.

Jim

At 3:37 PM -0700 9/29/06, Carson Gaspar wrote:

Brad, if your _incoming_ queue is so big that you have to worry, your servers are woefully underspec'd.

That may or may not be true, but that doesn't make the problem magically go away.

                                                      If you can

provide a detailed use case where it matters for the _incoming_ queue, please do so.

Basically, it's any site that has one or more lists that have relatively high levels of incoming traffic, and which run into synchronous meta-data bottleneck issues. This could be a lower-end machine with a filesystem that does not perform as well as could be, and a more moderately sized list. Or, this could be a site where they've already thrown the biggest/best configured machine at the problem that they can, and yet they're still seeing problems.

On the high end, from the reports we got after Apple upgraded the system for lists.apple.com, I don't think they're too likely to have these kinds of problems. But the FreeBSD folks might already be there, and I imagine that the SourceForge people are definitely there.

But I'm sure there are more relatively smaller lists running on relatively smaller/less well configured hardware, and which are running into the same kinds of problems.

All this aside, it's clear that I'm not going to convince anyone here of anything, so I'm just going to unsubscribe from the list and I'll ask everyone to make sure that they do not include me on any further messages on this subject.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

At 10:36 AM +0900 10/1/06, Tokio Kikuchi wrote:

...

                               An smtpd.py based LTMP server could

provide an interesting proof of concept though.

I've almost finished writing this primitive LMTP server.

I am no longer on the list. Please do not include me in any future discussions on subjects relating to the list.

-- Brad Knowles, brad@stop.mail-abuse.org

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

Founding Individual Sponsor of LOPSA. See http://www.lopsa.org/.

Hi,

Barry Warsaw wrote:

When you feel confident about your lmtp implementation, go ahead and check it in (probably in Mailman/bin/lmtp.py with hooks for mmshell

• -- or I can do the latter). I think at this early date we should make both LMTP and Maildir delivery possible, then we'll try to get real-world feedback from users as to which we should ultimately recommend.

I've commited it as LMTPRunner.py because it should be restarted by mailmanctl.

                                   How much work is your LMTP  

implementation doing when it receives the message?

It does the same thing as MaildirRunner does, although it doesn't do saving the misdirected messages but only to report them in error log.

If that's the case, there's some common constant amount of work being
done and I'm just wondering how efficient the LMTP part will be.

My impression is that it is almost the same as Maildir delivery.

-- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/