[Mailman-Developers] Regarding Handlers/SMTPDirect.py and "chunkify"

Wed May 14 14:16:28 CEST 2008

* Barry Warsaw <barry at list.org> wrote:
> On May 12, 2008, at 6:57 PM, Mark Sapiro wrote:
> 
>> This predates my experience with Mailman. It is based on the
>> statistics
>> provided by Chuq and outlined in the FAQ. It's true that these
>> statistics may only be applicable to lists with primarily US members,
>> and may be outdated in any case, but I can't provide any more
>> information on why it's done that way. Perhaps it's an idea that's
>> outlived its usefulness.
> 
> It may have.  I see from the NEWS file that SMTP_MAX_RCPTS was
> introduced on the last day of 1998, and these chunking tricks were
> essential back then given the state of the art in MTAs.  A decade is a
> long time in this field, so it makes sense to re-evaluate Mailman's
> MTA hand-off machinery.

As Ian Eiloart already pointed out, Mailman MUST comply with RFC 2821.
I'd even say lower the default setting of SMTP_MAX_RCPTS to 100 (mark
this change in the NEWS file or changelog file, and add MTA specific
instructions in the README.<MTA> files).

> We clearly need to support personalization[1] in Mailman, although it
> would be nice if there were extensions we could rely on in the MTA to
> push even that out farther.  I'd like to hear from any Sendmail,
> Postfix, or Exim experts on this list to see if the chunking even
> makes sense any more.

My inquiry on postfix-users on this matter hasn't yielded any
satisfying answers yet, and I'm by no means a Postfix "expert", but I
have several years of experience and I can read the documentation,
available at http://www.postfix.org/SCHEDULER_README.html - especially
the part talking about how the preemptive schedueler picks delivery
jobs. Fro, what's written there, it seems Postfix's queue manager -
qmgr - is able to group recipients on a roughly "per destination" base
even if different mails are assigned to those recipients. That means,
if the third batch contains an entry for "foo at aol.com.invalid" and the
sixth batch has a recipient "bar at aol.com.invalid", by the time the
qmgr begins to deliver to "@aol.com.invalid" it will correctly group
delivery to those two recipients, applying concurrency feedback (see
first part of SCHEDULER_README.html) as needed. Therefore, in an
environment without latency, a zero message delivery time, an
unlimited amount of delivery agents and an unlimited number of memory
for recipient entries (well, last two don't need to be "unlimited",
jst large enough so that every recipient of every incoming message can
be processed instantly), we would lose performance by NOT grouping
those "@aol.com.invalid" recipients.

In reality, the number of delivery agents, the maximum number of
concurrent connections to a destination and the number of recipients
which can be kept in Postfix's active_queue are limited, queue file
generation, setting up a TCP connection and transmitting of a message
do take more than zero time (the last two steps may even fail!). If in
the above example batch six enters the active_queue before all
delivery to "@aol.com.invalid" is done, the only drawback of NOT
grouping recipients would then be the overhead to either keep the
actual message contents in memory twice or to do two "read file"
operations to get those contents to memory for transmission.

OTOH, if by grouping we increase the overall number if "DATA" commands
(and therefore messages) transmitted to Postfix by Mailman, we might
have even higher overhead.

Please don't take that statement for granted. I didn't write "qmgr".
And to be honest, I didn't even start to think about network bandwith
usage. I strongly believe chunking the way it's done doesn't make
sense anymore, if you are not running some old qmail ;)

> I will note that we still get requests to batch deliveries to the
> outgoing MTA.  People want to be able to throttle Mailman's outgoing
> bandwidth to X number of messages per hour or day.  Ideally, this
> shouldn't be Mailman's task, but it's a valid request and one we
> should keep in mind.

> I would be happy if we could determine the two or three strategies
> that address 80% of Mailman's users, package them up as simple-to-
> configure defaults, and then provide a robust plugin architecture that
> people can use to build all kinds of wacky strategies for the other 20%

Ok, from my point of view, I'd say 80% of users are made up of two
groups:

The first group doesn't care at all. They want their MTA to do the
work, they fine tune it (or don't), but basically, as long as their
machines don't crash, they couldn't care less about the order in which
messages are delivered to the MTA. For those people, Mailman needs a
way to get messages out to the downstream MTA quickly, which implies
two constraints:

1. The recipient list must be stored in a way that allows quick
   generation of the necessary SMTP commands.
2. Either the MTA or Mailman must have an option similar to
   SMTP_MAX_RCPTS to avoid unnecessary delays and another option to
   not exhaust the maximum number of concurrent connections the MTA
   will acceppt.

Advantages of this approach:
1. It is never Mailman's fault.
2. All "tuning" can be done in the MTA's configuration, if "tuining"
   is needed.
3. Recipient ordering is only donw once, saving a (tiny) amount of CPU
   time.

Drawbacks of ths approach:
1. Mailman has absolutely zero control about the order or time
   messages are delivered to their final recipients.
2. If, for any reason, delivery to the MTA fails, we (actually, you
   *g*) need to have code in place to handle a queue of outgoing
   messages anyways.

The second group of those 80% want a finer control, and from what you
describe, they all want a "per timeslice" behaviour. Delivering only a
certain amount of messages, delivering messages only during off-hours
(don't know if this is real word, I mean evening and early morning),
delivering only a certain volume of messages, all those are basically
timeslice constraints. To handle those, Mailman needs to:

1. Know what time it is. (obviously!)
2. Considering that all delivery strategies are <N>/time_slice,
   Mailman obviously need to know <N>, which can be:
   - the total number of messages, as in "multiply every outgoing
     message whith the number of list subscribers and sum that up"
   - for volume based delivery, the total number of messages and at
     least an educated guess on their size
   - number of recipients for a single destination
   - ...

So if Mailman would now the concept of "per timeslice" and the
characteristics of outgoing mail, you already had a very flexible
interface in place which allowed for refined control of message flow.
OTOH, aside from some very special cases (UUCP, ETRN-only), Mailman
doesn't know the internal state of the MTA it delivers to. If there
were, for example, a network outage and the MTA would queue the
contents of 10 timeslices until the network is restored, all that
refined flow control was for nothing.

Honestly, I don't know what kind of "wacky" the remaining 20% of
mailman users could want, but I can think of some things which are
normally done in the MTA, for example concurrency control to certain
destinations. I have no clue how that could be achieved, because,
again, Mailman does NOT know about the state of the MTA it delivers
to, and implementing an ESMTP engine and queue manager in Mailman
seems like doing the job twice.

Cheers
Stefan

P.S: While writing this message, I didn't have access to a dictionary.
I apologize in advance.
-- 
Stefan Förster     http://www.incertum.net/     Public Key: 0xBBE2A9E9
218 DSA/RSA host keys to go...
Worst. Nightmare. Ever.