Hi,
There is a thread about ARC sealing in bind-users[*]. They're applying ARC signatures, although they run Mailman 2. The last message hypothesizes a user option like so:
*From munging*:
Set this option to /Disabled/ to receive messages with the original From:
line intact. Keep in mind that disabling this option will fail DMARC, so
keep it enabled unless your MTA either doesn't check DMARC or accepts ARC
overrides.It doesn't seem difficult to implement. It requires trusting the users, though. Would Mailman implement something like it? Why? Why not?
Best Ale
[*] https://lists.isc.org/pipermail/bind-users/2022-September/106612.html
Alessandro Vesely writes:
There is a thread about ARC sealing in bind-users[*].
Not sure what you mean by "sealing". Do you mean they're not implementing the rest of the protocol?
They're applying ARC signatures, although they run Mailman 2. It doesn't seem difficult to implement.
It's not. But
It's a bad idea to do it in Mailman.
It was implemented in Mailman 3 three or four years ago as a proof of concept during the development of ARC.
There is a milter available for Postfix and Sendmail from the Trusted Domain Project https://github.com/trusteddomainproject/OpenARC as is the basic implementation which I presume is adaptable to Exim, qmail, and other MTAs.
This is the preferred approach, as matter of conformance because it should be implemented by the edge MTA(s), and as a practical matter because Mailman *can't* do SPF since it is never an edge MTA. There is also a pure Python implementation on PyPI, I believe (this is the basis for the Mailman 3 plugin, or maybe it was dkimpy).
It requires trusting the users, though.
I don't think so, not any more than any other sign-and-send protocol. What it requires is implementation by recipient domains who trust your host, because if they don't it's 2014 all over again for your subscribers if you have any DMARC p=reject posters.
Would Mailman implement something like it?
Yes for Mailman 3, it's already done (but you are recommended to configure it in the MTA). No for Mailman 2, it's EOL.
Steve
Hi Steve!
On Sun 04/Sep/2022 13:38:39 +0200 Stephen J. Turnbull wrote:
Alessandro Vesely writes:
There is a thread about ARC sealing in bind-users[*].
Not sure what you mean by "sealing". Do you mean they're not implementing the rest of the protocol?
They add a complete ARC set. Actually, they add three ARC sets to each message, one at reception (with a full ARC-Authentication-Results), followed by intermediate and final sets.
They're applying ARC signatures, although they run Mailman 2. It doesn't seem difficult to implement.
It's not. But
It's a bad idea to do it in Mailman.
It was implemented in Mailman 3 three or four years ago as a proof of concept during the development of ARC.
There is a milter available for Postfix and Sendmail from the Trusted Domain Project https://github.com/trusteddomainproject/OpenARC as is the basic implementation which I presume is adaptable to Exim, qmail, and other MTAs.
This is the preferred approach, as matter of conformance because it should be implemented by the edge MTA(s), and as a practical matter because Mailman *can't* do SPF since it is never an edge MTA. There is also a pure Python implementation on PyPI, I believe (this is the basis for the Mailman 3 plugin, or maybe it was dkimpy).
Thank you for that much needed clarification. I heard someone saying that the IETF was waiting to implement Mailman 3 to use ARC in mailing list...
BTW, there is also a Perl implementation of ARC included in Mail::DKIM.
It requires trusting the users, though.
I don't think so, not any more than any other sign-and-send protocol.
I think you misunderstood my question here.
What it requires is implementation by recipient domains who trust your host, because if they don't it's 2014 all over again for your subscribers if you have any DMARC p=reject posters.
Exactly. I asked bind-users if anyone verifying ARC saw any difference after trusting isc.org. Besides adding ARC sets, bind-users do From: munging, obviously. Nobody saw any difference.
The point is how does Mailman know whether a recipient's MX trusts this particular list. And what does it do when it knows. Some people babble something about DNS records, which looks difficult. Another possibility could be an SMTP extension, difficult to implement as it involves multiple levels.
An easy way would be to ask the subscriber whether to do From: munging or not. I repeat a possible wording for that option, which would be enabled by default:
*From munging*:
Set this option to /Disabled/ to receive messages with the original From:
line intact. Keep in mind that disabling this option will fail DMARC, so
keep it enabled unless your MX either doesn't check DMARC or accepts
overrides trusting our ARC sets.Then, a user can disable From: munging for the messages destined to her. That's easy for those who run their own MTA. People using Gmail, say, would have to figure out, presumably by trial and error.
If such an option is not given, a mailing list could add the Author: header field defined by RFC9057. Receivers could restore From: after DMARC filtering.
That assuming that someone is willing to do something to avoid munged From:'s, which I'm beginning to doubt.
Best Ale
Alessandro Vesely writes:
On Sun 04/Sep/2022 13:38:39 +0200 Stephen J. Turnbull wrote:
Not sure what you mean by "sealing". Do you mean they're not implementing the rest of the protocol?
They add a complete ARC set. Actually, they add three ARC sets to each message, one at reception (with a full ARC-Authentication-Results), followed by intermediate and final sets.
? That's not how this works. You add (ARC-)Authentication-Results immediately after the message is received from a remote, and the ARC-Message-Signature and ARC-Seal immediately before transmitting it to a remote. I guess if you do the full set including full authentication at each stage, it doesn't harm the protocol, but it does impose a fair amount of unnecessary work on each recipient.
Thank you for that much needed clarification. I heard someone saying that the IETF was waiting to implement Mailman 3 to use ARC in mailing list...
That seems unlikely. Unless you're running a single-host system, it's much preferable to do it on border MTAs, rather than in Mailman. I suspect it's more that they plan to do a big revamp of infrastructure including Mailman 3 and they would do it at that time.
BTW, there is also a Perl implementation of ARC included in Mail::DKIM.
Thanks, I'll keep that in mind.
Exactly. I asked bind-users if anyone verifying ARC saw any difference after trusting isc.org. Besides adding ARC sets, bind-users do From: munging, obviously. Nobody saw any difference.
As far as I know ARC adds nothing if you're also doing From munging. The point of ARC is entirely to get rid of From munging. Once you've munged From, your DKIM will be valid and you have DMARC from alignment.
The point is how does Mailman know whether a recipient's MX trusts this particular list.
It doesn't. Recipients can do any damned thing they want. In the case of AT&T and Verizon, pretty much everything they do is damned. They frequently block all traffic from well-behaved lists because there's a spammer in the same netblock, or just randomly.
And what does it do when it knows. Some people babble something about DNS records, which looks difficult. Another possibility could be an SMTP extension, difficult to implement as it involves multiple levels.
I really don't see the bad actors in this space (by which I mean the ISP-based freemail providers) doing either, since they don't even send "we hate you spammer!" DSNs, they just black hole your list traffic and any attempt to complain about it. If services sent honest DSNs, we could discount bogus 550s and not count them as bounces.
An easy way would be to ask the subscriber whether to do From: munging or not. Then, a user can disable From: munging for the messages destined to her. That's easy for those who run their own MTA. People using Gmail, say, would have to figure out, presumably by trial and error.
I'm dubious. People are going to get their subscriptions disabled by experimenting. I don't know how many people still want non-personalized lists, but implementing that would require a bit of effort since two messages would need to be prepared depending on whether don't-munge-for-me was set.
If such an option is not given, a mailing list could add the Author: header field defined by RFC9057. Receivers could restore From: after DMARC filtering.
This can't work. DMARC v2 will quickly be forced to check Author alignment. From the RFC:
In that regard, it would be reasonable for an MUA that would
normally organize, filter, or display information based on the
From: field to give the Author: header field preference.Translated into Scum-of-the-Earth-Spammer:
You can use this header to send "referred by someone in victim's
contact list" (that you stole from Yahoo) spam and it will bypass
DMARC v1 because you can use an aligned From. All-Hail-Author!> That assuming that someone is willing to do something to avoid
munged From:'s, which I'm beginning to doubt.
If ISPs cared at all about their customers, they'd implement ARC and be fine. It completely solves the problem by putting it on mailing lists and other mediators to filter spam before sealing. The problem, as always, is recipient domains that are unwilling to publish consistent policies or respond to non-delivery issues.
On Tue 06/Sep/2022 06:41:36 +0200 Stephen J. Turnbull wrote:
Alessandro Vesely writes:
On Sun 04/Sep/2022 13:38:39 +0200 Stephen J. Turnbull wrote:
I asked bind-users if anyone verifying ARC saw any difference after trusting isc.org. Besides adding ARC sets, bind-users do From: munging, obviously. Nobody saw any difference.> As far as I know ARC adds nothing if you're also doing From munging. The point of ARC is entirely to get rid of From munging. Once you've munged From, your DKIM will be valid and you have DMARC from alignment.
The tale goes that large mailbox providers want ARC as a tool to filter mail streams from lists that don't do a good filtering themselves. ARC, they say, allows to attribute reputation correctly. However, I don't think they can tell who a message is from, after it has been munged. In that respect, From: munging hampers ARC.
The point is how does Mailman know whether a recipient's MX trusts this particular list.
It doesn't. Recipients can do any damned thing they want. In the case of AT&T and Verizon, pretty much everything they do is damned. They frequently block all traffic from well-behaved lists because there's a spammer in the same netblock, or just randomly.
Heck, that sounds similar to a blog I just read: https://cfenollosa.com/blog/after-self-hosting-my-email-for-twenty-three-yea...
And what does it do when it knows. Some people babble something about DNS records, which looks difficult. Another possibility could be an SMTP extension, difficult to implement as it involves multiple levels.
I really don't see the bad actors in this space (by which I mean the ISP-based freemail providers) doing either, since they don't even send "we hate you spammer!" DSNs, they just black hole your list traffic and any attempt to complain about it. If services sent honest DSNs, we could discount bogus 550s and not count them as bounces.
A MLM would continue From: munging unless a receiver is able to tell it to not do it.
An easy way would be to ask the subscriber whether to do From: munging or not. Then, a user can disable From: munging for the messages destined to her. That's easy for those who run their own MTA. People using Gmail, say, would have to figure out, presumably by trial and error.
I'm dubious. People are going to get their subscriptions disabled by experimenting. I don't know how many people still want non-personalized lists, but implementing that would require a bit of effort since two messages would need to be prepared depending on whether don't-munge-for-me was set.
I see that archived messages are not munged. How come? Isn't the archive a regular subscriber?
At worst, one could set up two lists, fed by the same stream, one with munging enabled and the other not, letting users subscribe to the one they prefer.
If such an option is not given, a mailing list could add the Author: header field defined by RFC9057. Receivers could restore From: after DMARC filtering.
This can't work. DMARC v2 will quickly be forced to check Author alignment.
A list can set the Author: header field by copying From:. In the rare case when Author: is already set and differs from From:, it has to be checked. I guess that case is so rare that the list can handle it by sending such messages to the moderator queue. Alternatively, apply DMARC to the Author: domain. That's the "simple" de-munging method in my draft: https://datatracker.ietf.org/doc/html/draft-vesely-dmarc-mlm-transform
From the RFC:
In that regard, it would be reasonable for an MUA that would normally organize, filter, or display information based on the From: field to give the Author: header field preference.
MUAs won't notice Author: fields any time soon.
Translated into Scum-of-the-Earth-Spammer:
You can use this header to send "referred by someone in victim's contact list" (that you stole from Yahoo) spam and it will bypass DMARC v1 because you can use an aligned From. All-Hail-Author!
They're doing that with From:, and it works fine. It is very hard for MUAs to tell spoofed From:'s, and munging doesn't help. Then, some people hold that even writing THIS IS PHISHING loud and clear won't prevent users from opening and clicking that link. Darwin will tell...
That assuming that someone is willing to do something to avoid munged From:'s, which I'm beginning to doubt.
If ISPs cared at all about their customers, they'd implement ARC and be fine. It completely solves the problem by putting it on mailing lists and other mediators to filter spam before sealing.
Implementing and deploying ARC is not enough, you also need to trust the ARC signer/ sealer.
ARC won't be effective until it has been deployed at more than 60% of SMTP servers and that's not a problem. :-) https://www.rhyolite.com/anti-spam/you-might-be.html#senior-IETF-member-5
Best Ale
Alessandro Vesely writes:
The tale goes that large mailbox providers want ARC as a tool to filter mail streams from lists that don't do a good filtering themselves. ARC, they say, allows to attribute reputation correctly. However, I don't think they can tell who a message is from, after it has been munged.
I don't see why they would care who the message is from. They know who added the ARC seal who claim to have validated the original From. If any of them are highly trusted, or the whole chain is unbroken and none is known untrustworthy, that should be good enough.
In that respect, From: munging hampers ARC.
Sure, a little, especially if the recipient doesn't understand the theory. On the other hand, there are only a few MLMs that munge, they should be able to extract the original address in most cases.
A MLM would continue From: munging unless a receiver is able to tell it to not do it.
Sure, but as you say you need to trust the users, in this case, the receivers. If it's all I know about them, I wouldn't trust a user who uses a "p=reject" provider to be clueful enough to make that decision safely.
I see that archived messages are not munged. How come? Isn't the archive a regular subscriber?
In general, no, they're not. Messages are sent to archive services before the munging process takes place. There are services like mail-archive.com where you interface with the archiver as a regular subscriber, but Pipermail (Mailman 2) and HyperKitty (Mailman 3) both short-circuit the pipeline.
At worst, one could set up two lists, fed by the same stream, one with munging enabled and the other not, letting users subscribe to the one they prefer.
To be honest, while I'm at best 50% willing to implement the user option, I could easily be persuaded by a few experiments with the dual list proposal. This could be implemented almost transparently for the subscriber (except at subscription time) by using an umbrella list.
A list can set the Author: header field by copying From:.
This isn't a problem. I would be perfectly happy to accept a patch to handle Author today as long as it's RFC-ly correct, which looks straightforward.
MUAs won't notice Author: fields any time soon.
Then they're useless.
- They need to display them, or most users won't be able to use them in any way.
- It's important that they be included in automatically formatted replies, and the munged From ignored.
- You'd hope From would either not be displayed or the content of the Author field substituted into From.
Translated into Scum-of-the-Earth-Spammer:
You can use this header to send "referred by someone in victim's contact list" (that you stole from Yahoo) spam and it will bypass DMARC v1 because you can use an aligned From. All-Hail-Author!They're doing that with From:, and it works fine. It is very hard for MUAs to tell spoofed From:'s,
True, but DMARC should give essentially zero false negatives. If it says from is aligned, and the sending domain is reliable, it's not spoofed.
Then, some people hold that even writing THIS IS PHISHING loud and clear won't prevent users from opening and clicking that link. Darwin will tell...
See the recent Twitter thread by @FatManTerra, who performed a very cruel experiment where he advertised an obvious scam "for the purpose of making people scammed by Terra whole" and amassed > $100,000 in BitCoin in 24 hours. (He returned it all, but it's unethical and must make his victims feel like idiots. I don't disagree with their self-assessment, but it wasn't nice to rub their noses in it.)
Implementing and deploying ARC is not enough, you also need to trust the ARC signer/ sealer.
I see it the other way around: once people have implemented ARC, you have the opportunity to serve your users by trusting the ARC sealers. We didn't really have that before, because of the ease of creating fake mediators.
ARC won't be effective until it has been deployed at more than 60% of SMTP servers and that's not a problem. :-) https://www.rhyolite.com/anti-spam/you-might-be.html#senior-IETF-member-5
I doubt 60% of servers will implement within the decade, but I guess a lot more than 60% of legit mail is handled somewhere by an ARC- conforming server. Google has long been a powerful advocate of ARC. I think there's hope.
On Tue 06/Sep/2022 15:28:10 +0200 Stephen J. Turnbull wrote:
At worst, one could set up two lists, fed by the same stream, one with munging enabled and the other not, letting users subscribe to the one they prefer.
To be honest, while I'm at best 50% willing to implement the user option, I could easily be persuaded by a few experiments with the dual list proposal. This could be implemented almost transparently for the subscriber (except at subscription time) by using an umbrella list.
That's right. I'm going to add this as another method to get rid of munged From:'s in my draft. Would you give it a review for me, please?
Internet-Draft MLM Transformations September 2022
The ARC method
For each list with From: munging enabled, a participating MLM MUST configure a twin list with From: munging disabled. Both lists have the same posting address, but separate subscriber lists. Subscribers who think that their mailbox provider runs a suitably configured DMARC filter can subscribe to the twin list. Users subscribed to both lists receive double messages until they unsubscribe or suspend delivery from one of them.
DMARC local policy overrides is one of the use cases that [RFC8617] provides for ARC. It requires that a DMARC filter be configured to accept the authentication assessments provided by a verified ARC chain when all of the domains involved are marked as trusted. In that case, the filter overrides DMARC policy and acts as if the current Authentication-Results: were the ARC-Authentication-Results: (AAR:) of the first ARC set (i=1). Normally, a MLM SHOULD apply DMARC policy on message arrival, so the first AAR: is expected to have dmarc=pass.
MLMs which in turn trust third party domains, such that they override DMARC failures in posted messages, MUST communicate the list of trusted domains to subscribers when they announce the creation of the twin list, and on subsequent modifications. How users can query the list of domains trusted by their mailbox providers is beyond the scope of this document. Anyway, all the domains possibly trusted by the list MUST be trusted by the user's MTA as well, and by any subsequent MTA in case of forwarding.
TIA Ale
Alessandro Vesely writes:
Internet-Draft MLM Transformations September 2022
- The ARC method
This twin-list proposal doesn't depend specifically on ARC. Any subscriber who thinks they know better than the DMARC protocol can choose to ignore the DMARC policy, and they would benefit from the twin-list proposal. For example, if the un-authenticated received chain indicates that the MLM received the post directly from an author not in the MLM's domain, and I trust the MLM, and its DKIM signature is valid, I would accept that mail and ignore DMARC. Of course if the MLM host provides a sealed AAR field, that's much better, but the one-hop-to-MLM scenario is very strong evidence of authenticity if the MLM is very trustworthy.
For each list with From: munging enabled, a participating MLM MUST
To be honest, I don't think that Mailman would be willing to conform to this. I haven't seen the rest of your draft, but this section is really more of an Informative/BCP RFC than Standards Track. That is, this is an option that list creators can already exercise with the cooperation of the list owners, so there's no need to impose requirements on MLMs.
If this option gets traction (ie, some vocal fans, I don't ask for a huge number), then I think Mailman would be willing, and would prefer, to go all the way to recipient-controlled From-munging as you originally suggested. Maintaining synchronization of configurations of two lists will be tedious for the admin, or involve relatively complicated coding if we arrange to automatically mirror configuration changes. A per-subscriber option for From munging would be simpler to develop and maintain.
configure a twin list with From: munging disabled. Both lists have the same posting address, but separate subscriber lists.
I don't think having the same same posting address is likely to work. Most MLMs probably won't implement at all, because list creators can do it for themselves by having an umbrella list with two subscribers, the twin lists. The twin lists would be configured to refuse subscriptions and posts from non-members. On the other hand, doing all this cleanly is more complicated than implementing recipient- controlled From munging.
Subscribers who think that their mailbox provider runs a suitably configured DMARC filter can subscribe to the twin list. Users subscribed to both lists receive double messages until they unsubscribe or suspend delivery from one of them.
Recipient-controlled From-munging would also prevent unnecessary double messages. It's just cleaner.
This next paragraph seems unrelated to recipient controls on From munging. It seems pretty obvious, and is ARC-specific. https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09 would be the natural home but it's expired, so it doesn't do any harm to have it in your draft. Have you considered reviving that document?
DMARC local policy overrides is one of the use cases that [RFC8617] provides for ARC. It requires that a DMARC filter be configured to accept the authentication assessments provided by a verified ARC chain when all of the domains involved are marked as trusted. In that case, the filter overrides DMARC policy and acts as if the current Authentication-Results: were the ARC-Authentication-Results: (AAR:) of the first ARC set (i=1). Normally, a MLM SHOULD apply DMARC policy on message arrival, so the first AAR: is expected to have dmarc=pass.
This paragraph also seems unrelated to recipient controls on From munging, and is not ARC-specific.
MLMs which in turn trust third party domains, such that they override DMARC failures in posted messages, MUST
This "MUST" isn't going to work. The MLM software can't ensure this, and MLM operators will do as they please.
communicate the list of trusted domains to subscribers
Currently most sites (MTAs) operate content filters and blacklists, but not whitelists, and the MLMs trust their sites. I don't see why that needs to change with ARC. Either way, I'm pretty sure those most sites would be unwilling to publish those lists and keep them up-to- date, and the MLMs don't have them.
Note that it's quite rare for the MLM to trust anybody under current conditions. Most mail flows to MLMs are direct, and for MLMs that host open subscription lists, they can be from anywhere. So MLMs expect their sites to provide content filtering and sender authentication, not trust (reputation) algorithms, and the MLMs don't participate in maintaining any of these facilities.
when they announce the creation of the twin list, and on subsequent modifications. How users can query the list of domains trusted by their mailbox providers is beyond the scope of this document. Anyway, all the domains possibly trusted by the list MUST be trusted by the user's MTA as well, and by any subsequent MTA in case of forwarding.
The recipient MTA has the whole chain, it can decide whether it wants to trust any individual signer in the chain. If it doesn't, it can choose to consider the whole chain invalid. As a subscriber, checking your own whitelist or blacklist is very cheap compared to the process of validating the ARC chain.
Anyway, I think individual subscribers aren't going to be making those decisions, and won't want to unless they're the kind of person who wants to run their own MTA, I think. This issue it discussed from several angles in the I-D linked above, and their conclusion is always the same: reputation maintenance is hard, and very site-specific, so no recommendations in the I-D. I think you should provide rationales for these recommendations, and for a MUST they have to be very persuasive, I think.
Sincere regards, Steve
Hi Steve,
thanks for your precious insights.
Some comments inline and a new version:
On Sat 10/Sep/2022 10:41:21 +0200 Stephen J. Turnbull wrote:
Alessandro Vesely writes:
- The ARC method
This twin-list proposal doesn't depend specifically on ARC.
Right.
For each list with From: munging enabled, a participating MLM MUSTTo be honest, I don't think that Mailman would be willing to conform to this.
It is the MLM as a whole which has to conform, if it wishes to participate. Not the mailing list software.
I haven't seen the rest of your draft, but this section is really more of an Informative/BCP RFC than Standards Track.
Experimental, actually.
The old version is here: https://datatracker.ietf.org/doc/draft-vesely-dmarc-mlm-transform/
If this option gets traction (ie, some vocal fans, I don't ask for a huge number), then I think Mailman would be willing, and would prefer, to go all the way to recipient-controlled From-munging as you originally suggested.
Should find a list wishing to experiment with it. I'm going to ask again to IETF's Dispatch list when the new method is published in the draft.
I push ARC as the authentication method because that was the major objection to using Author: (the "simple" method in the old version.)
Maintaining synchronization of configurations of two lists will be tedious for the admin, or involve relatively complicated coding if we arrange to automatically mirror configuration changes.
Couldn't symlink most stuff?
A per-subscriber option for From munging would be simpler to develop and maintain.
Sure. That was the first idea.
configure a twin list with From: munging disabled. Both lists have the same posting address, but separate subscriber lists.
I don't think having the same same posting address is likely to work. Most MLMs probably won't implement at all, because list creators can do it for themselves by having an umbrella list with two subscribers, the twin lists. The twin lists would be configured to refuse subscriptions and posts from non-members.
I'm not clear how that would work. Would you expand?
Recipient-controlled From-munging would also prevent unnecessary double messages. It's just cleaner.
Yes.
https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09 would be the natural home but it's expired, so it doesn't do any harm to have it in your draft.
What I dislike of that document is its considering the availability of a global reputation system as a widespread feature of all mail servers, while only the known giants actually have one. In that respect, ARC is a centripetal protocol, which is why I've been opposing it until this attempt.
Have you considered reviving that document?
No.
DMARC local policy overrides is one of the use cases that [RFC8617] provides for ARC. It requires that a DMARC filter be configured to accept the authentication assessments provided by a verified ARC chain when all of the domains involved are marked as trusted. In that case, the filter overrides DMARC policy and acts as if the current Authentication-Results: were the ARC-Authentication-Results: (AAR:) of the first ARC set (i=1). Normally, a MLM SHOULD apply DMARC policy on message arrival, so the first AAR: is expected to have dmarc=pass.This paragraph also seems unrelated to recipient controls on From munging, and is not ARC-specific.
It describes the mechanism by which a receiver accepts dmarc=fail using ARC. (W.r.t. other papers, I add that the domains have to be marked as trusted, which would act as a simplified reputation system.)
MLMs which in turn trust third party domains, such that they override DMARC failures in posted messages, MUSTThis "MUST" isn't going to work. The MLM software can't ensure this, and MLM operators will do as they please.
communicate the list of trusted domains to subscribersCurrently most sites (MTAs) operate content filters and blacklists, but not whitelists, and the MLMs trust their sites.
Right.
Anyway, I think individual subscribers aren't going to be making those decisions, and won't want to unless they're the kind of person who wants to run their own MTA, I think. This issue it discussed from several angles in the I-D linked above, and their conclusion is always the same: reputation maintenance is hard, and very site-specific, so no recommendations in the I-D.
Should I add that it's out of scope to speculate how users can convince their mailbox provider to trust/ whitelist a given MLM?
I think you should provide rationales for these recommendations, and for a MUST they have to be very persuasive, I think.
Right.
And here's the new text:
The no-munging method
For each list with From: munging enabled, a participating MLM MUST configure the possibility to have From: munging disabled. Depending on the mailing list software used, a MLM can devise various methods to accomplish the task:
Create a twin list by symbolic links of most configuration files, except the subscriber lists. Both lists thus have the same posting address. Subscribers who think that their mailbox provider accepts dmarc=fail from the MLM domain can subscribe to the non-munging list. Users subscribed to both lists receive double messages until they unsubscribe or suspend delivery from one of them.
Have an umbrella list with two subscribers, the twin lists. The twin lists would be configured to refuse subscriptions and posts from non-members.
Have a per-subscriber option for From: munging. This is the simplest and cleanest possibility, but requires a mailing list software with this specific feature.
Before allowing subscription to a non-munging list, a MLM MAY test that a recipient effectively receives its messages by sending a test message with a broken signature from a domain having p=reject.
DMARC local policy override is one of the use cases that [RFC8617] provides for ARC. It says that a DMARC filter can be configured to accept the authentication assessments provided by a verified ARC chain when all of the domains involved are marked as trusted. Accepting the assessments means that the filter acts as if the current Authentication-Results: were the ARC-Authentication-Results: certified by the first ARC set, the one with i=1. The first ARC set SHOULD be by the MLM itself, since indirect posts can be problematic when final receivers have not marked the preceding intermediate domains as trusted.
To produce an ARC set, a MLM's MTA performs SPF, DKIM and DMARC checks upon receiving a message from the author's domain. The results are saved in Authentication-Results: fields marked with the MLM's domain, while making sure that no spoofs exist. The ARC filter uses those fields to produce ARC-Authentication-Results: at the time when it seals the message, which is the last step before the message leaves the MLM domain.
ARC is not the only means by which a receiver can accept messages which fail DMARC. Simply whitelisting the MLM domain, authenticated by SPF or DKIM would suffice. The extra information that ARC brings can serve for final receivers to evaluate MLM's filtering and compute author's reputation. However, even MTAs that lack such sophisticated reputation mechanisms can find ARC filters more convenient to set up, as that is exactly their function.
Setting the Author: header field is still useful to quickly check whether From: munging took place.
Best Ale
First let me make clear that (1) I do have influence on Mailman's position here but (2) I am not authoritative and (3) Mailman has no position yet. I'm discussing this and that and we'll see where my position and eventually Mailman's come out. So anything I say may be wrong (always check my logic ;-) and I may change my mind. :-)
Alessandro Vesely writes:
It is the MLM as a whole which has to conform, if it wishes to participate. Not the mailing list software.
If you mean the decision is list by list, conformance doesn't mean much -- the subscribers still need to learn the rules list by list, most of them won't know what "RFC 9999" conformance means, and other sites interacting with such a site will need to check the conformance of lists individually.
On the other hand, if you mean site-wide, if it were implemented in Mailman (and other software), conformance would be much more likely and much more likely to be site-wide.
I push ARC as the authentication method because that was the major objection to using Author: (the "simple" method in the old version.)
Yes, I agree, authentication is important, and ARC provides validation of the right data for some purposes. I'm not sure it "does what you want", but I do "want what it does".
Maintaining synchronization of configurations of two lists will be tedious for the admin, or involve relatively complicated coding if we arrange to automatically mirror configuration changes.
Couldn't symlink most stuff?
I don't think there's anything to symlink. In Mailman 3 all of this configuration information is in an RDBMS like PostgreSQL, and routing of posts and modification of messages (both bodies and headers).
I'm not clear how that would work. Would you expand?
- List-A@example.com has two subscribers: List-A-munge@example.com List-A-nomunge@example.com List-A-[no]munge accepts subscriptions according to site and list policy.
- List-A is configured not to allow other subscribers under any circumstances. List-A-[no]munge accept subscribers under the site and list policy.
- List-A-[no]munge refuse all posts, and advertise List-A as the destination for posts. List-A accepts posts according to site and list policy.
- List-A-munge gets From munging for all posts, List-A and List-A-nomunge never get From munging. (In theory List-A-munge could do munging only for p=reject posters, but always doing it probably makes it easier for subscribers to maintain their filters.)
https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09 would be the natural home but it's expired, so it doesn't do any harm to have it in your draft.
What I dislike of that document is its considering the availability of a global reputation system as a widespread feature of all mail servers,
90% of the email users on the Internet are served by organizations that can afford comprehensive and reasonably accurate reputation databases and update algorithms. (Whether they do bother with accuracy is another question.) So I think it's reasonable to ask "how does a reputation database affect this feature" several times. For the rest of us, there are less sophisticated but still useful shared reputation databases (ie, the RBLs), and local databases such as SpamBayes can be useful.
while only the known giants actually have one. In that respect, ARC is a centripetal protocol, which is why I've been opposing it until this attempt.
Everything is centripetal, because the only way we really know how to scale networks while maintaining discoverability is hierarchically. All reasonably decentralized networks have a (usually very expensive) centralized system at their foundation. I don't see ARC as being particularly biased toward centralization, just because powerful reputation systems are expensive. If anything, it's the opposite for the mailing list community, because it makes it easier for an independent list host construct and maintain its reputation, and should get it better treatment from those with reputation systems.
Should I add that it's out of scope to speculate how users can convince their mailbox provider to trust/ whitelist a given MLM?
I think that's always out of scope. It doesn't hurt to add it, but technically its out of scope for an RFC.
- The no-munging method [...]
- Have an umbrella list with two subscribers, the twin lists. The twin lists would be configured to refuse subscriptions and posts from non-members.
Whether to refuse posts from non-members is independent of the no-munging protocol.
Before allowing subscription to a non-munging list, a MLM MAY test that a recipient effectively receives its messages by sending a test message with a broken signature from a domain having p=reject.
This would require the MLM site to maintain a separate site. Otherwise you're spoofing a third party. Not good.
Accepting the assessments means that the filter acts as if the current Authentication-Results: were the ARC-Authentication-Results: certified by the first ARC set, the one with i=1. The first ARC set SHOULD be by the MLM itself,
This SHOULD is inappropriate, because the order of the chain is completely out of control of the ARC sealers. What matters is (1) an unbroken AR chain with all AAR showing a pass for authentication of the previous ARC set, (2) a valid DKIM signature on the received message from a trusted source, and (3) some trusted system (not necessarily i=1!) validates From alignment. (More details below.)
since indirect posts can be problematic when final receivers have not marked the preceding intermediate domains as trusted.
I don't think this is a practical problem in the case of mailing lists. In almost all cases[1], there is *one* DKIM break, and that is at the mailing list. If the MLM system finds a valid DKIM from the originating system on the incoming post, it can check From alignment and include the DMARC pass in its AAR. If at the final receiver, the MLM system's DKIM and ARC set validate and the MLM's ARC set says DMARC passed, then the only question about DKIM and DMARC is "do I trust the MLM?"
Note that from the point of view of ARC, the question of trust is only about the authentication of the post, that is, "did the mailbox in From send essentially the same content?" It doesn't help us decide whether the post is appropriate to distribute to the list. Either the sender or the MLM could be a spammer, or it could just be off-topic. The decision whether to trust the MLM's spam/virus filters is a different question, and I don't think it's affected by ARC, except to the extent that the MLM can be authenticated to the extent that the ARC chain is trusted, so if the MLM sends malcontent, they can't say, oh, that was spoofed.
ARC is not the only means by which a receiver can accept messages which fail DMARC. Simply whitelisting the MLM domain, authenticated by SPF or DKIM would suffice. The extra information that ARC brings can serve for final receivers to evaluate MLM's filtering and compute author's reputation. However, even MTAs that lack such sophisticated reputation mechanisms can find ARC filters more convenient to set up, as that is exactly their function.
I don't understand this last sentence. You still have to mark the trusted hosts somehow. You can maintain a whitelist easily with
echo mailman-developers@python.org >> /etc/MTA/trusted-mailing-listsand use that list equally easily with no authentication or with ARC. ARC means you can trust the results more, though.
Setting the Author: header field is still useful to quickly check whether From: munging took place.
Agreed. I don't object to having Author for this purpose, I just don't think it helps with authentication.
Regards, Steve
Footnotes: [1] I have never heard of problems with DKIM breaks before or after the mailing list, only at the mailing list. Since From munging has been around in a couple of different forms since April 2014, I'm pretty sure that signatures broken before or after the mailing list would have been raised on one of the Mailman lists. I'm not saying it never happens, just that I'm sure it's very rare.
On Tue 13/Sep/2022 10:14:12 +0200 Stephen J. Turnbull wrote:
Alessandro Vesely writes:
Maintaining synchronization of configurations of two lists will be tedious for the admin, or involve relatively complicated coding if we arrange to automatically mirror configuration changes.
Couldn't symlink most stuff?
I don't think there's anything to symlink. In Mailman 3 all of this configuration information is in an RDBMS like PostgreSQL, and routing of posts and modification of messages (both bodies and headers).
It would also be possible to link DB tables, or to define triggers that replicate insert/ update/ delete on a number of tables/ fields. The question is how much more insight than average list keeping would be needed to do it.
Would this approach make sense with Mailman 2?
I'm not clear how that would work. Would you expand?
- List-A@example.com has two subscribers: List-A-munge@example.com List-A-nomunge@example.com List-A-[no]munge accepts subscriptions according to site and list policy.
- List-A is configured not to allow other subscribers under any circumstances. List-A-[no]munge accept subscribers under the site and list policy.
- List-A-[no]munge refuse all posts, and advertise List-A as the destination for posts. List-A accepts posts according to site and list policy.
If the site policy is to accept posts from subscribers, it needs to inspect the union of sub-lists subscriber sets. How could that be accomplished?
- List-A-munge gets From munging for all posts, List-A and List-A-nomunge never get From munging. (In theory List-A-munge could do munging only for p=reject posters, but always doing it probably makes it easier for subscribers to maintain their filters.)
How difficult is that to set up?
I saw some lists deploying a home-brewed From: munging tool. In that case they can control it directly.
https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09 would be the natural home but it's expired, so it doesn't do any harm to have it in your draft.
What I dislike of that document is its considering the availability of a global reputation system as a widespread feature of all mail servers,
90% of the email users on the Internet are served by organizations that can afford comprehensive and reasonably accurate reputation databases and update algorithms. (Whether they do bother with accuracy is another question.) So I think it's reasonable to ask "how does a reputation database affect this feature" several times.
IMHO, it becomes overly complicated. Domain-based reputation is already fuzzy, and for giant organizations it becomes unmeaningful —they're just too big to block. Now, start gaming all possible combinations of domains. Replaying a modified message is all too easy, and ARC can be ambiguous about who modified what in a message. Yes, you could feed a neural network with that. Will it be reliable?
For the rest of us, there are less sophisticated but still useful shared reputation databases (ie, the RBLs), and local databases such as SpamBayes can be useful.
DMARC was thought so that From: bank.example can hardly be faked. Allowing fuzzy overrides is much like getting back to content analysis. I'd mark as trusted only a few domains, based on personal knowledge.
while only the known giants actually have one. In that respect, ARC is a centripetal protocol, which is why I've been opposing it until this attempt.
Everything is centripetal, because the only way we really know how to scale networks while maintaining discoverability is hierarchically. All reasonably decentralized networks have a (usually very expensive) centralized system at their foundation. I don't see ARC as being particularly biased toward centralization, just because powerful reputation systems are expensive.
It is not the cost. To have a global knowledge of the Internet you need to have a user base that is statistically relevant with respect to the global population. That is, you have to be Google, or Microsoft, or Yahoo!, ...
If anything, it's the opposite for the mailing list community, because it makes it easier for an independent list host construct and maintain its reputation, and should get it better treatment from those with reputation systems.
Yes. However, I think that a list that experiments non-munging will be whitelisted sooner by small, personal sites who trust it than by large orgs computing its reputation.
The no-munging method [...]
Before allowing subscription to a non-munging list, a MLM MAY test that a recipient effectively receives its messages by sending a test message with a broken signature from a domain having p=reject.
This would require the MLM site to maintain a separate site.
It can be a dummy subdomain, a few lines in a zone file. I'll change that line to "from a (sub)domain having p=reject", to have it more apparent.
Otherwise you're spoofing a third party. Not good.
Yeah, you'd have to ask permission to do that.
Accepting the assessments means that the filter acts as if the current Authentication-Results: were the ARC-Authentication-Results: certified by the first ARC set, the one with i=1. The first ARC set SHOULD be by the MLM itself,This SHOULD is inappropriate, because the order of the chain is completely out of control of the ARC sealers.
Hm... a list SHOULD reject posts arriving with an ARC chain, valid or not. Shouldn't it? I see no reasons to post indirectly (except for internal list-to-list flows which don't need ARC seals).
What matters is (1) an unbroken AR chain with all AAR showing a pass for authentication of the previous ARC set, (2) a valid DKIM signature on the received message from a trusted source, and (3) some trusted system (not necessarily i=1!) validates From alignment. (More details below.)
since indirect posts can be problematic when final receivers have not marked the preceding intermediate domains as trusted.I don't think this is a practical problem in the case of mailing lists. In almost all cases[1], there is *one* DKIM break, and that is at the mailing list. If the MLM system finds a valid DKIM from the originating system on the incoming post, it can check From alignment and include the DMARC pass in its AAR. If at the final receiver, the MLM system's DKIM and ARC set validate and the MLM's ARC set says DMARC passed, then the only question about DKIM and DMARC is "do I trust the MLM?"
You're right, I simplified too much. Changed that paragraph as follows:
DMARC local policy override is one of the use cases that [RFC8617]
provides for ARC. It says that a DMARC filter can be configured to
accept the authentication assessments provided by a verified ARC
chain when the domains involved are marked as trusted. Accepting the
assessments means that the filter acts as if the current
Authentication-Results: were the ARC-Authentication-Results:
certified by the first ARC set, the one with i=1. In the unusual
case where the first ARC set is not by the MLM itself, it is REQUIRED
that an aligned DKIM signature be reported as pass by the MLM's ARC-
Authentication-Results:. That certifies that the message was
essentially intact when it reached the MLM. So the result can be
overridden when the MLM domain and any successive domain in the ARC
chain are marked as trusted.Note that from the point of view of ARC, the question of trust is only about the authentication of the post, that is, "did the mailbox in From send essentially the same content?" It doesn't help us decide whether the post is appropriate to distribute to the list. Either the sender or the MLM could be a spammer, or it could just be off-topic. The decision whether to trust the MLM's spam/virus filters is a different question, and I don't think it's affected by ARC, except to the extent that the MLM can be authenticated to the extent that the ARC chain is trusted, so if the MLM sends malcontent, they can't say, oh, that was spoofed.
ARC is not the only means by which a receiver can accept messages which fail DMARC. Simply whitelisting the MLM domain, authenticated by SPF or DKIM would suffice. The extra information that ARC brings can serve for final receivers to evaluate MLM's filtering and compute author's reputation. However, even MTAs that lack such sophisticated reputation mechanisms can find ARC filters more convenient to set up, as that is exactly their function.I don't understand this last sentence. You still have to mark the trusted hosts somehow. You can maintain a whitelist easily with
echo mailman-developers@python.org >> /etc/MTA/trusted-mailing-lists
(Cannot authenticate the local part.)
and use that list equally easily with no authentication or with ARC.
In theory, ARC was developed as a tool for overriding DMARC failures. Does it make sense to have trust marks for ARC distinguished from generic whitelisting flags?
ARC means you can trust the results more, though.
At least, more specifically.
Best Ale
Alessandro Vesely writes:
It would also be possible to link DB tables,
No, it's not. It's all one row (IIRC).
or to define triggers that replicate insert/ update/ delete on a number of tables/ fields.
Which is exactly the complexity I don't want in Mailman if we can avoid it. Keep it to flat tables in approximately normal form.
The question is how much more insight than average list keeping would be needed to do it.
"Too much." :-) The simple "message per subscriber + per-subscription 'munge' flag" is just so much better. For example, there are at least three conceivable values: no munge, munge all, munge p=reject (and I think Mailman actually implements munge p>=quarantine as well!) This gets very tedious in the twin-list implementation.
Would this approach make sense with Mailman 2?
The umbrella + twin lists approach is perfectly possible with Mailman 2, but the admin has to implement it themselves. We are not going to implement it or release it.
If the site policy is to accept posts from subscribers, it needs to inspect the union of sub-lists subscriber sets. How could that be accomplished?
There's a "sibling list" feature for exactly this purpose.
- List-A-munge gets From munging for all posts, List-A and List-A-nomunge never get From munging. (In theory List-A-munge could do munging only for p=reject posters, but always doing it probably makes it easier for subscribers to maintain their filters.)
How difficult is that to set up?
cost of 1 list X 3.
IMHO, it becomes overly complicated.
So don't do it. Others will, however -- isn't that what "decentralization" is all about? -- and some of them are quite good at it. Why not take advantage of that to make at least some mail flows cleaner and more useful?
DMARC was thought so that From: bank.example can hardly be faked.
Yes, and that's still true.
Allowing fuzzy overrides is much like getting back to content analysis.
Fuzzy overrides are not "allowed". They *happened*. Gmail did it from the get-go. RFC 8617 is recognition that it does happen, and a protocol that purports to improve the accuracy of overrides.
I'd mark as trusted only a few domains, based on personal knowledge.
Have you analyzed your mail flows to see if there seem to be frequent messages with multiple DKIM breaks? AFAICS, in practice *you* as an individual will need to trust your mailing lists because that's the only place signatures are going to be invalidated, and you can demand everybody else has to pass DMARC. This is no different from before, except replay attacks via mailing lists are going to be harder. For large sites with many users with diverse mail flows, the benefits of both ARC and reputation systems are much larger.
It is not the cost. To have a global knowledge of the Internet you need to have a user base that is statistically relevant with respect to the global population. That is, you have to be Google, or Microsoft, or Yahoo!, ...
That's not true. Bayesian filters work well for almost everyone.
If anything, it's the opposite for the mailing list community, because it makes it easier for an independent list host construct and maintain its reputation, and should get it better treatment from those with reputation systems.
Yes. However, I think that a list that experiments non-munging will be whitelisted sooner by small, personal sites who trust it than by large orgs computing its reputation.
They're *already* whitelisted by the small personal sites who trust it. The critical question is how fast are the large orgs going to learn to trust small ARC participants, because it's exactly those large orgs that are the root of all evil^Wer, most nondelivery problems that we small sites experience.
ARC and DMARC are *not* targered at *us*, though if the large orgs use them effectively we will benefit. They're for large sites with hundreds of thousands (and sometimes billions) of users who are targeted by ransomeware hackers and national espionage agencies.
This would require the MLM site to maintain a separate site.
It can be a dummy subdomain, a few lines in a zone file. I'll change that line to "from a (sub)domain having p=reject", to have it more apparent.
Yeah, I was in a hurry. Thing is, there are a lot of inexperienced folks out there who would just send mail from "biden@whitehouse.gov" or something like that. :-)
Hm... a list SHOULD reject posts arriving with an ARC chain, valid or not. Shouldn't it? I see no reasons to post indirectly (except for internal list-to-list flows which don't need ARC seals).
This is *Internet mail*. "NO REASON" is its slogan! But here's my personal use case: I use a Japanese telco as my home ISP but use my server at my employer as smarthost. My employer (research university) doesn't know or care, but they do intercept that mail at the firewall, and could reasonable ARC seal it. Then it goes to the MLM, which breaks the DKIM signature.
So (1) as long as my DKIM signature is intact, which it is, the MLM should accept my post and add an AAR with dkim=passed; dmarc=passed, and (2) you don't need to trust my employer, you only need to trust the MLM (and of course since both my DKIM and my employer's DKIM break at the MLM, if you don't trust it, it doesn't matter if you trust my employer).
You're right, I simplified too much. Changed that paragraph as follows:
DMARC local policy override is one of the use cases that [RFC8617] provides for ARC. It says that a DMARC filter can be configured to accept the authentication assessments provided by a verified ARC chain when the domains involved are marked as trusted. Accepting the assessments means that the filter acts as if the current Authentication-Results: were the ARC-Authentication-Results: certified by the first ARC set, the one with i=1. In the unusual case where the first ARC set is not by the MLM itself, it is REQUIRED that an aligned DKIM signature be reported as pass by the MLM's ARC- Authentication-Results:. That certifies that the message was essentially intact when it reached the MLM. So the result can be overridden when the MLM domain and any successive domain in the ARC chain are marked as trusted.
I still think you're going to get pushback on the requirements language, but the analysis is complete and accurate.
and use that list equally easily with no authentication or with ARC.
In theory, ARC was developed as a tool for overriding DMARC failures. Does it make sense to have trust marks for ARC distinguished from generic whitelisting flags?
Yes and no. Yes, the semantics are different, so having different attributes could provide more accurate assessments. But no, doing that means you're on the way to a full reputation system! ;-)
On Wed 14/Sep/2022 06:35:50 +0200 Stephen J. Turnbull wrote:
I still think you're going to get pushback on the requirements language, but the analysis is complete and accurate.
That sounds good to me. I've posted the new draft version[*]. As all drafts, it is still open to discussions. Meanwhile let me thank you for your essential support.
Best Ale
[*] https://datatracker.ietf.org/doc/draft-vesely-dmarc-mlm-transform/
participants (2)
-
Alessandro Vesely -
Stephen J. Turnbull