Adding DMARC support for Mailman 3
Greetings,
I am writing on behalf of a group of companies and single persons, who would like to see a limited feature set of the DMARC¹ standard supported by Mailman 3.
Since I know we're all eager to get MM3 out as soon as possible and any additional new feature request stands against that I've contacted Barry offlist and asked if he'd agree that the companies involved pay us, sys4², to implement the feature. He did and we also agreed to dedicate a significant part of the payment to mailman's FSF donation account.
Before we take out to write code, I would like to ask mailman-developers how it should be done to fit best into Mailman's architecture. Here are the DMARC features that should go into Mailman 3:
- don't allow email that comes from a domain with a DMRAC record of p=reject
- take ownership of the email and send it with a From: using the domain of the mailing list. (There's a patch for this for Mailman 2.1, which might might be helpful for Mailman 3.)
- find the authentication-results header and rewrite it as an Original-Authentication-header: http://tools.ietf.org/html/draft-kucherawy-original-authres-00.html
Speaking of an RFC written by Murray Kucherawy. I've contacted Murray in advance and asked him to assist in case we had any questions regarding his RFC(s). He subscribed and ready to help.
I hope I was able to bring all parties required together to make a Mailman DMARC implementation come true and I am curious to hear what you have to say.
p@rick
¹) DMARC http://www.dmarc.org/ adds policies and more to DKIM, which gives digitally signed identity to sender domains and has become a cornerstone to reputation based mail management. Having DMARC support in Mailman 3 is - in my eyes - a reason for postmasters (not end-users) to upgrade their MM 2.x installation as soon as possible.
²) In case you ask yourself, who "sys4" is: We've accompanied Mailman 3 development since Pycon in Chicago (in 2008 ?). Florian Fuchs, whom I don't have to introduce, works with us as well Ralf Hildebrandt and I. We, Ralf and I, are on the python.org postmaster team. Also with sys4 are Antoine Nguyen and Christian Roesser - both are exerienced Python programmers and both have spent significant time developing email applications.
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Mon, Jul 1, 2013 at 3:44 PM, Patrick Ben Koetter p@sys4.de wrote:
Before we take out to write code, I would like to ask mailman-developers how it should be done to fit best into Mailman's architecture. Here are the DMARC features that should go into Mailman 3:
- don't allow email that comes from a domain with a DMRAC record of p=reject
- take ownership of the email and send it with a From: using the domain of the mailing list. (There's a patch for this for Mailman 2.1, which might might be helpful for Mailman 3.)
- find the authentication-results header and rewrite it as an Original-Authentication-header: http://tools.ietf.org/html/draft-kucherawy-original-authres-00.html
I'm all for experimentation and being adaptive as new things come along, and I'm obviously supportive of the DMARC effort. That said, I hope that these are going to be configurable options, defaulted "off" for backward compatibility. This is because:
(a) the second bullet above is a significant departure from current use (as I understand it), and fails the test of least surprise if we were going to suddenly see that MM3 does things quite differently than previous versions or, indeed, other packages; and
(b) I'm uneasy about Original-Authentication-Results. As far as I'm aware there's only a single, proprietary implementation. Its proponents have explained the logic to me several times, but I remain unconvinced. I'm all for experimentation in order to provide data for future efforts, so I don't really object, but this shouldn't be taken as a well-vetted proposal just because there's an (expired) draft about it.
-MSK
On Jul 02, 2013, at 12:44 AM, Patrick Ben Koetter wrote:
Before we take out to write code, I would like to ask mailman-developers how it should be done to fit best into Mailman's architecture. Here are the DMARC features that should go into Mailman 3:
- don't allow email that comes from a domain with a DMRAC record of p=reject
- take ownership of the email and send it with a From: using the domain of the mailing list. (There's a patch for this for Mailman 2.1, which might might be helpful for Mailman 3.)
- find the authentication-results header and rewrite it as an Original-Authentication-header: http://tools.ietf.org/html/draft-kucherawy-original-authres-00.html
In this message, I'll talk about implementation strategies.
In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution.
I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have.
In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins.
For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal.
Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages.
-Barry
Barry Warsaw writes:
For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results,
This is somewhat problematic. DMARC results are potentially trivalent. If action is "reject" and pct is less than 100, some hits are "rejects" and some are "quarantine". Misses are misses. So I guess you do this with a chain of two rules, the first one verifying the message and if that hits (ie, verification fails) the second one rolls the dice for pct.
and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it.
Silent discards without content analysis make me queasy. I guess we can work around that by doing DMARC checks after the content checks, although the draft implies the DMARC checks should be done early. Or we could reject, but unfortunately we can't reject in the SMTP transaction, so we need to issue a DSN. That makes me really queasy, because DSNs for illegitimate mail suck all around.
In case of a quarantine, maybe this should go into a separate queue that silently waits for a moderator to look at the messages, and discards them after a reasonable period of time (maybe two weeks?) So they'd be there if somebody asks for a lost message, but otherwise no bother.
Steve
On Jul 11, 2013, at 03:23 AM, Stephen J. Turnbull wrote:
Barry Warsaw writes:
For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results,
This is somewhat problematic. DMARC results are potentially trivalent. If action is "reject" and pct is less than 100, some hits are "rejects" and some are "quarantine". Misses are misses. So I guess you do this with a chain of two rules, the first one verifying the message and if that hits (ie, verification fails) the second one rolls the dice for pct.
While ugly, that might be the best we can do for now. I have thought about adding an action to links for when the rule misses, the default being 'Defer' (i.e the next link in the chain executes as normal). That would at least give you more control over each step in the chain. But handling more than two cases quickly gets into ugliness.
Another possibility is to collapse the reject/quarantine "hit" into a single boolean result. Rules can add key/values to the metadata dictionary, so you could imagine that a hit wouldn't jump directly to the Reject or Hold chains. Instead it would jump to a custom (terminal) chain that made the more specific determination of whether to reject or hold the message.
and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it.
Silent discards without content analysis make me queasy.
Of course, we'd likely log and fire an event, so at least it wouldn't happen completely silently.
I guess we can work around that by doing DMARC checks after the content checks, although the draft implies the DMARC checks should be done early. Or we could reject, but unfortunately we can't reject in the SMTP transaction, so we need to issue a DSN. That makes me really queasy, because DSNs for illegitimate mail suck all around.
Yep. There is some limited ability to do additional checking at LMTP time, but this isn't pluggable currently.
In case of a quarantine, maybe this should go into a separate queue that silently waits for a moderator to look at the messages, and discards them after a reasonable period of time (maybe two weeks?) So they'd be there if somebody asks for a lost message, but otherwise no bother.
Currently there's only one moderation queue, but it can be set up to auto-discard held requests after a period of time.
-Barry
Barry Warsaw writes:
On Jul 11, 2013, at 03:23 AM, Stephen J. Turnbull wrote:
This is somewhat problematic. DMARC results are potentially trivalent. If action is "reject" and pct is less than 100, some hits are "rejects" and some are "quarantine". Misses are misses. So I guess you do this with a chain of two rules, the first one verifying the message and if that hits (ie, verification fails) the second one rolls the dice for pct.
While ugly, that might be the best we can do for now.
Verbose, yes. Is it really ugly, though? I don't know how much you were directly influenced by iptables and SIEVE, but the idea of rule chains as a way to very flexibly configure filters has been implemented many times. The model is very simple and completely flexible.
Instead it would jump to a custom (terminal) chain that made the more specific determination of whether to reject or hold the message.
This is pretty much what I was suggesting.
Silent discards without content analysis make me queasy.
Of course, we'd likely log and fire an event, so at least it wouldn't happen completely silently.
No, but it might be many days before the originator gets around to asking why their message hasn't appeared.
Yep. There is some limited ability to do additional checking at LMTP time, but this isn't pluggable currently.
Does LMTP provide the necessary ability to reject?
Steve
On Jul 12, 2013, at 11:56 AM, Stephen J. Turnbull wrote:
Yep. There is some limited ability to do additional checking at LMTP time, but this isn't pluggable currently.
Does LMTP provide the necessary ability to reject?
Not reject in the Mailman sense of sending a bounce message (possibly DSN formatted), but it could refuse to accept the message from the LMTP client.
-Barry
Barry Warsaw writes:
On Jul 12, 2013, at 11:56 AM, Stephen J. Turnbull wrote:
Yep. There is some limited ability to do additional checking at LMTP time, but this isn't pluggable currently.
Does LMTP provide the necessary ability to reject?
Not reject in the Mailman sense of sending a bounce message (possibly DSN formatted), but it could refuse to accept the message from the LMTP client.
That's what I meant. With luck, then, the LMTP rejection could happen during the MTA's SMTP transaction.
participants (4)
-
Barry Warsaw -
Murray S. Kucherawy -
Patrick Ben Koetter -
Stephen J. Turnbull