At 5:50 PM -0500 2004/01/22, Barry Warsaw wrote:
Nigel can provide details, but I think the same embedding feature could be used to have the MTA do the final stitching of content template and personalized data. It would be A Project to hack together, but I think it could be a neat idea to play with, although I'm not sure how much it would help.
This is similar to what Eric Allman (at that time, before Sendmail Inc. existed), Bryan Costales (at the time, working for InfoBeat/Mercury Mail), and I (working at AOL) were discussing back in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) server, based on a modified version of sendmail along with a standard language for transmitting that content. With MMTP servers on both ends, it would not matter how many thousands or millions of recipients you might have, only one copy of the message body would be transmitted, and all the rest would be filled in on the remote end.
We ultimately gave up on this idea because we realized that it would make the spam problem much, much worse. The same things that help regular MTAs transmit millions of customized messages per hour to their paying customers would probably allow spammers to transmit billions of messages per hour to everyone in the universe.
Certainly, before any serious discussion of creating something like an MMTP server, and trying to make that a standard which you would expect programs like sendmail, postfix, and Exim to implement, I believe that the spam issue needs to be addressed. You need to be able to prove how this cannot be abused to generate spam instead.
If the MTA could do what Mailman does here -- not creating a disk image for each instance of the message, but stitching it together in member as it's going out on the wire -- I think you'd greatly improve disk contention.
I'm not sure that the MTA could safely do that in memory. At least, it would be difficult to ensure that the MTA gets this done right. This would be akin to handling the entire message queue in memory for all messages, something which can't really be done safely except under very strict circumstances.
The only MTA I know of that is capable of doing things like this is the latest release of version 8 sendmail, and even then it defaults to handling messages in memory that are no larger than 4KB.
Yes, filesystem I/O is the number one killer, specifically synchronous meta-data updates.
But then people on this list have heard me harping on this subject for a long time, and should know by now that I will refer them to Nick Christenson's book _Sendmail Performance Tuning_ (see <http://www.jetcafe.org/~npc/book/sendmail/>), or my own slides from an invited talk entitled "Sendmail Performance Tuning for Large Systems" at <http://www.shub-internet.org/brad/papers/sendmail-tuning/>.
I don't think Postfix has the same embedding capabilities, although I haven't looked at what Postfix 2.1 may provide.
I'm not aware of anything like this, but I'd have to check.In a sense, that's what we've talked about before. If there were a standard language that the mail server and list manager could agree on for both defining the template, and defining the per-recipient data source, we could have a more efficient mechanism, with perhaps a hope of mta agnosticism.
That would be nice. However, I fear that we have much more basic problems that are much more serious, and which need to be resolved before we can expect to start worrying about such subjects as increasing efficiency in the interfaces between MLMs and MTAs.
-- Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+ !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++) tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)