[Mailman-Developers] Regarding Handlers/SMTPDirect.py and "chunkify"

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On May 12, 2008, at 6:57 PM, Mark Sapiro wrote:

> This predates my experience with Mailman. It is based on the  
> statistics
> provided by Chuq and outlined in the FAQ. It's true that these
> statistics may only be applicable to lists with primarily US members,
> and may be outdated in any case, but I can't provide any more
> information on why it's done that way. Perhaps it's an idea that's
> outlived its usefulness.

It may have.  I see from the NEWS file that SMTP_MAX_RCPTS was  
introduced on the last day of 1998, and these chunking tricks were  
essential back then given the state of the art in MTAs.  A decade is a  
long time in this field, so it makes sense to re-evaluate Mailman's  
MTA hand-off machinery.

What makes the most sense for us to do today?  In principle, I would  
like to simplify this code rather than complicate it with yet another  
configuration option.  So what use cases do we need to support?  Can  
we get closer interaction with the upstream MTA to hand it the  
outgoing message and recipients more efficiently?  The more we can  
push these decisions into the MTA, the better off Mailman will be.

As an example of an efficiency I'm likely to add to MM3: I believe I  
can avoid calculating the full recipient list, requiring storing it in  
memory or in a queue file.  I think I can build it so that we only  
iterate through the recipients one at a time, as we're talking to the  
upstream MTA.  That would be a big I/O and memory footprint savings on  
the Mailman side.  Are there other thing we can do to push most of the  
processing, chunking, and efficiency calculations into the MTA, where  
it belongs (IMO)?  To gain this benefit, we may have to ditch  
chunking, or only support what grouping is available in SQL.

We clearly need to support personalization[1] in Mailman, although it  
would be nice if there were extensions we could rely on in the MTA to  
push even that out farther.  I'd like to hear from any Sendmail,  
Postfix, or Exim experts on this list to see if the chunking even  
makes sense any more.

I will note that we still get requests to batch deliveries to the  
outgoing MTA.  People want to be able to throttle Mailman's outgoing  
bandwidth to X number of messages per hour or day.  Ideally, this  
shouldn't be Mailman's task, but it's a valid request and one we  
should keep in mind.

I would be happy if we could determine the two or three strategies  
that address 80% of Mailman's users, package them up as simple-to- 
configure defaults, and then provide a robust plugin architecture that  
people can use to build all kinds of wacky strategies for the other 20%.

- -Barry

[1] Technically, not VERP, but close enough for our purposes.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkgpGGAACgkQ2YZpQepbvXHNwACgpBxMYwh136zybvpKWJtEKol4
Ge0An1vLzJWz9mRGOSwd/nR6bM+OkdEo
=DNsQ
-----END PGP SIGNATURE-----