--On 1 April 2008 03:11:17 +0200 Mads Kiilerich <mads@kiilerich.com> wrote:
Mark Sapiro wrote, On 03/31/2008 06:26 PM:
So what do I do practically in this case. Calling out to verify the recipient won't help because the recipient is valid.
I think that we - admins and developers - have to realize that times are changing.
agreed.
Once upon a time when the RFCs were written then mail was asynchronous store-and-forward. MTAs accepted mails before being 100% certain that they could and would deliver them, and as soon as the mail had hit the disk then the receiving MTA took over delivery responsibility from the sending MTA.
true.
In such a scenario Mailman "is just" a MTA with bouncing/holding filtering and dynamic alias expansion of envelope-rcpt and replacement of envelope-from with a robot address.
yes.
But now in modern times all mail handling has to be synchronous.
that would be nice.
Spam and virus and backscatter does that a MTA should never take responsibility for anything from an untrusted sender before the mail has actually been taken fully care of by someone - perhaps just by making it the problem of the next MTA/MDA. Yes, doing this all the way to the final mailboxes might be more expensive than the old way of doing it - but at the same time the same amount of work (or less?) has to be done, so it can't be that bad ... except for the number of live connections. I think that the RFCs allows fully synchronous mail handling, they just have to be used another way than they usually are today.
No, they don't allow fully synchronous mail handling. The problem is that much spam filtering depends on message content (headers and body). And, different recipients have different views on what is acceptable content. The RFCs don't allow us handle differences of opinion, because SMTP has no way of saying "foo@example.com" will accept this message, but "bar@example.net" will not.
In theory, this could be dealt with by adopting LMTP instead of SMTP. LMTP does allow us to reject or defer for individual recipients after we've seen the message content. But that's always going to be a slow process, and it'll be a brave sysadmin who first says "you can only deliver to us using LMTP".
The first step to this would be an extension to SMTP which allows a server to say "you may switch to LTMP" if you wish. MTA's might be configured to switch to single recipient deliveries when LMTP wasn't available. But, what spammer would use LMTP if SMTP were available?
It's nice to know that Mailman 3 is likely to have an LMTP engine, because it will be possible for lists to use different content-based rules, and reject (not bounce) messages to multiple lists depending on individual list rules. For example, one list might reject a message with attachments. Of course, this will only push the problem back to the local MTA, but that is at least some progress.
In Marks specific case in this scenario: The verification phase has been replaced by actually doing the delivery all the way to the final recipients, so the recipient ending up being invalid will be handled properly.
In this scenario Mailman is a MTA which decouples the synchronous chain. Mails are accepted and enqueued by Mailman iff responsibility is taken, either for immediate delivery or manual review - all other mails are rejected. All filtering thus has to be done synchronously. Delivery to list members is a separate process where target mail hosts probably will provide synchronous delivery status feedback (perhaps as DSN bounces from a trusting "real" MTA) (and if other mail hosts chooses to accept and then bounce later then they have a problem, but Mailman can still handle receiving bounces).
Lots of details are missing in this description. But do you agree that this is the direction we have to take as developers and admins?
That would depend on whether we thought converting the world to LMTP is worth the effort. Perhaps it is, but not in my view simply so that we can fight spam. The real problem that we have to solve is trusting return-paths. At the moment, we can't trust or distrust anyone because it is so trivial to forge return-paths. SPF and DKIM offer solutions to authenticate senders, and hold domain owners accountable for email that's sent.
Once they're widely deployed, we will be able to make reasonable decisions about whether to trust the email sender based on the sending domain, or even on the individual sender.
At that point, we'll be able to synchronously chain callouts, without requiring to look at any message content. Sites will reliably be able to blacklist or whitelist based on Mail From. We'll be able to build trust networks (like RBL lists, but with domains and email addresses, not IP addresses), and we won't need to scan content at all.
/Mads
ps: This might be a bad description of trivial observation, but it suddenly made sense to me and I wanted to share it, even though I'm mostly a lurker on this list. Take it or leave it ;-)
-- Ian Eiloart IT Services, University of Sussex x3148