- Barry Warsaw barry@list.org wrote:
On May 12, 2008, at 6:57 PM, Mark Sapiro wrote:
This predates my experience with Mailman. It is based on the statistics provided by Chuq and outlined in the FAQ. It's true that these statistics may only be applicable to lists with primarily US members, and may be outdated in any case, but I can't provide any more information on why it's done that way. Perhaps it's an idea that's outlived its usefulness.
It may have. I see from the NEWS file that SMTP_MAX_RCPTS was introduced on the last day of 1998, and these chunking tricks were essential back then given the state of the art in MTAs. A decade is a long time in this field, so it makes sense to re-evaluate Mailman's MTA hand-off machinery.
As Ian Eiloart already pointed out, Mailman MUST comply with RFC 2821. I'd even say lower the default setting of SMTP_MAX_RCPTS to 100 (mark this change in the NEWS file or changelog file, and add MTA specific instructions in the README.<MTA> files).
We clearly need to support personalization[1] in Mailman, although it would be nice if there were extensions we could rely on in the MTA to push even that out farther. I'd like to hear from any Sendmail, Postfix, or Exim experts on this list to see if the chunking even makes sense any more.
My inquiry on postfix-users on this matter hasn't yielded any satisfying answers yet, and I'm by no means a Postfix "expert", but I have several years of experience and I can read the documentation, available at http://www.postfix.org/SCHEDULER_README.html - especially the part talking about how the preemptive schedueler picks delivery jobs. Fro, what's written there, it seems Postfix's queue manager - qmgr - is able to group recipients on a roughly "per destination" base even if different mails are assigned to those recipients. That means, if the third batch contains an entry for "foo@aol.com.invalid" and the sixth batch has a recipient "bar@aol.com.invalid", by the time the qmgr begins to deliver to "@aol.com.invalid" it will correctly group delivery to those two recipients, applying concurrency feedback (see first part of SCHEDULER_README.html) as needed. Therefore, in an environment without latency, a zero message delivery time, an unlimited amount of delivery agents and an unlimited number of memory for recipient entries (well, last two don't need to be "unlimited", jst large enough so that every recipient of every incoming message can be processed instantly), we would lose performance by NOT grouping those "@aol.com.invalid" recipients.
In reality, the number of delivery agents, the maximum number of concurrent connections to a destination and the number of recipients which can be kept in Postfix's active_queue are limited, queue file generation, setting up a TCP connection and transmitting of a message do take more than zero time (the last two steps may even fail!). If in the above example batch six enters the active_queue before all delivery to "@aol.com.invalid" is done, the only drawback of NOT grouping recipients would then be the overhead to either keep the actual message contents in memory twice or to do two "read file" operations to get those contents to memory for transmission.
OTOH, if by grouping we increase the overall number if "DATA" commands (and therefore messages) transmitted to Postfix by Mailman, we might have even higher overhead.
Please don't take that statement for granted. I didn't write "qmgr". And to be honest, I didn't even start to think about network bandwith usage. I strongly believe chunking the way it's done doesn't make sense anymore, if you are not running some old qmail ;)
I will note that we still get requests to batch deliveries to the outgoing MTA. People want to be able to throttle Mailman's outgoing bandwidth to X number of messages per hour or day. Ideally, this shouldn't be Mailman's task, but it's a valid request and one we should keep in mind.
I would be happy if we could determine the two or three strategies that address 80% of Mailman's users, package them up as simple-to- configure defaults, and then provide a robust plugin architecture that people can use to build all kinds of wacky strategies for the other 20%
Ok, from my point of view, I'd say 80% of users are made up of two groups:
The first group doesn't care at all. They want their MTA to do the work, they fine tune it (or don't), but basically, as long as their machines don't crash, they couldn't care less about the order in which messages are delivered to the MTA. For those people, Mailman needs a way to get messages out to the downstream MTA quickly, which implies two constraints:
- The recipient list must be stored in a way that allows quick generation of the necessary SMTP commands.
- Either the MTA or Mailman must have an option similar to SMTP_MAX_RCPTS to avoid unnecessary delays and another option to not exhaust the maximum number of concurrent connections the MTA will acceppt.
Advantages of this approach:
- It is never Mailman's fault.
- All "tuning" can be done in the MTA's configuration, if "tuining" is needed.
- Recipient ordering is only donw once, saving a (tiny) amount of CPU time.
Drawbacks of ths approach:
- Mailman has absolutely zero control about the order or time messages are delivered to their final recipients.
- If, for any reason, delivery to the MTA fails, we (actually, you *g*) need to have code in place to handle a queue of outgoing messages anyways.
The second group of those 80% want a finer control, and from what you describe, they all want a "per timeslice" behaviour. Delivering only a certain amount of messages, delivering messages only during off-hours (don't know if this is real word, I mean evening and early morning), delivering only a certain volume of messages, all those are basically timeslice constraints. To handle those, Mailman needs to:
- Know what time it is. (obviously!)
- Considering that all delivery strategies are <N>/time_slice,
Mailman obviously need to know <N>, which can be:
- the total number of messages, as in "multiply every outgoing message whith the number of list subscribers and sum that up"
- for volume based delivery, the total number of messages and at least an educated guess on their size
- number of recipients for a single destination
- ...
So if Mailman would now the concept of "per timeslice" and the characteristics of outgoing mail, you already had a very flexible interface in place which allowed for refined control of message flow. OTOH, aside from some very special cases (UUCP, ETRN-only), Mailman doesn't know the internal state of the MTA it delivers to. If there were, for example, a network outage and the MTA would queue the contents of 10 timeslices until the network is restored, all that refined flow control was for nothing.
Honestly, I don't know what kind of "wacky" the remaining 20% of mailman users could want, but I can think of some things which are normally done in the MTA, for example concurrency control to certain destinations. I have no clue how that could be achieved, because, again, Mailman does NOT know about the state of the MTA it delivers to, and implementing an ESMTP engine and queue manager in Mailman seems like doing the job twice.
Cheers Stefan
P.S: While writing this message, I didn't have access to a dictionary. I apologize in advance.