[Mailman-Developers] Adding headers to mailman generated mails

[Brad Knowles]

At 5:50 PM -0500 2004/01/22, Barry Warsaw wrote:

>  Nigel can provide details, but I think the same embedding feature could
>  be used to have the MTA do the final stitching of content template and
>  personalized data.  It would be A Project to hack together, but I think
>  it could be a neat idea to play with, although I'm not sure how much it
>  would help.

	This is similar to what Eric Allman (at that time, before 
Sendmail Inc. existed), Bryan Costales (at the time, working for 
InfoBeat/Mercury Mail), and I (working at AOL) were discussing back 
in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) 
server, based on a modified version of sendmail along with a standard 
language for transmitting that content.  With MMTP servers on both 
ends, it would not matter how many thousands or millions of 
recipients you might have, only one copy of the message body would be 
transmitted, and all the rest would be filled in on the remote end.

	We ultimately gave up on this idea because we realized that it 
would make the spam problem much, much worse.  The same things that 
help regular MTAs transmit millions of customized messages per hour 
to their paying customers would probably allow spammers to transmit 
billions of messages per hour to everyone in the universe.


	Certainly, before any serious discussion of creating something 
like an MMTP server, and trying to make that a standard which you 
would expect programs like sendmail, postfix, and Exim to implement, 
I believe that the spam issue needs to be addressed.  You need to be 
able to prove how this cannot be abused to generate spam instead.

>  If the MTA could do what Mailman does here -- not creating a disk image
>  for each instance of the message, but stitching it together in member as
>  it's going out on the wire -- I think you'd greatly improve disk
>  contention.

	I'm not sure that the MTA could safely do that in memory.  At 
least, it would be difficult to ensure that the MTA gets this done 
right.  This would be akin to handling the entire message queue in 
memory for all messages, something which can't really be done safely 
except under very strict circumstances.

	The only MTA I know of that is capable of doing things like this 
is the latest release of version 8 sendmail, and even then it 
defaults to handling messages in memory that are no larger than 4KB.


	Yes, filesystem I/O is the number one killer, specifically 
synchronous meta-data updates.

	But then people on this list have heard me harping on this 
subject for a long time, and should know by now that I will refer 
them to Nick Christenson's book _Sendmail Performance Tuning_ (see 
<http://www.jetcafe.org/~npc/book/sendmail/>), or my own slides from 
an invited talk entitled "Sendmail Performance Tuning for Large 
Systems" at 
<http://www.shub-internet.org/brad/papers/sendmail-tuning/>.

>  I don't think Postfix has the same embedding capabilities, although I
>  haven't looked at what Postfix 2.1 may provide.

	I'm not aware of anything like this, but I'd have to check.

>  In a sense, that's what we've talked about before.  If there were a
>  standard language that the mail server and list manager could agree on
>  for both defining the template, and defining the per-recipient data
>  source, we could have a more efficient mechanism, with perhaps a hope of
>  mta agnosticism.

	That would be nice.  However, I fear that we have much more basic 
problems that are much more serious, and which need to be resolved 
before we can expect to start worrying about such subjects as 
increasing efficiency in the interfaces between MLMs and MTAs.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)