@mark:

The Hotmail error message does make it sufficiently clear that these bounces are due to DMARC. I will probably file an RFE to catch these against Mailman 3. Would you like me to do that for Mailman 2, or is this "obvously not worth it" in your opinion? (I intend to supply code "eventually". ;-)

I should also fix the FAQ to mention the names of the new options, no?

@paul:

Mark Sapiro writes:

On October 9, 2017 11:56:02 PM PDT, paul@tokyoprogressive.org wrote:

...

Hi and hope the answer(s) to my question are relatively simple. On one of two lists I manage, some people are getting deleted due to too many bounces. And the bounces seem to be related to their mail provider not allowing the messages. As far as I can tell, the main culprits are gmail, yahoo, and hotmail.

The messages that *cause* the problem are from Yahoo! and Hotmail users (I just ban posting from those addresses, but I can do that because using them for university business is against Monkeyshow = Japanese government policy ;-). Many large providers will bounce these messages when resent via Mailman (and most featureful list managers).

Gmail does "quarantine" these messages by putting them in "spam". (I think this is the appropriate action, both according to the DMARC standard and according the the "finger in eye" theory of corporate rivalry :-).

I think this is a DMARC issue. See https://wiki.list.org/x/17891458.

The Hotmail errors explicitly state that "on behalf of" messages violate Yahoo!'s policy, so yes, this is a DMARC issue in at least some cases.

I suspect the reason that this "suddenly" came up after changing the list's Reply-To policy is that some Hotmail and Yahoo! users started sending to list when they hadn't been doing so before.

Note that the relevant option name has changed in recent Mailman; it is now Privacy Options -> Sender Filters -> DMARC Moderation Action. You almost certainly want that set to "Munge From", and the following option DMARC Quarantine Moderation Action set to "Yes". (Note that "quarantine" is a setting on the mail sender's side; Mailman neither knows nor cares about Gmail's quarantine policy.)

If subscribers complain that Yahoo! and Hotmail users' mail now comes "From: LIST on behalf of USER (EMAIL) LIST@YOURDOMAIN.TLD" but others come "From: USER <EMAIL>", you may also want to set the third option DMARC None Moderation Action to "Yes".

Let me repost that. Somehow my signature appeared mid post and made it look like that was my entire message. I should not post from my phone. Easier to edit at the computer as I am now.

Thanks, Mark and Stephen, Anyway, to confirm my version is 2.1.23

It seems that the names that both of you, Mark and Stephen, are referring to the same functions with slightly different names because of the version?

I want to make sure that I don’t make the wrong setting.

Mark mentioned list admin UI on the Privacy

options... -> Sender filters page set

dmarc_moderation_action = Munge From dmarc_quarantine_moderation_action = Yes

and if it exists

dmarc_none_moderation_action = No

WELL I HAVE ONE THAT SAYS Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=quarantine as well as p=reject and Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=none as well as p=quarantine and p=reject

Are these the same, and is that a YES and NO?

On the other hand Stephen said:

Note that the relevant option name has changed in recent Mailman; it is now Privacy Options -> Sender Filters -> DMARC Moderation Action. You almost certainly want that set to "Munge From", and the following option DMARC Quarantine Moderation Action set to "Yes". (Note that "quarantine" is a setting on the mail sender's side; Mailman neither knows nor cares about Gmail's quarantine policy.)

and

If subscribers complain that Yahoo! and Hotmail users' mail now comes "From: LIST on behalf of USER (EMAIL) LIST@YOURDOMAIN.TLD" but others come "From: USER <EMAIL>", you may also want to set the third option DMARC None Moderation Action to "Yes”.

OK, there are three: Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy. MUNGE FROM Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=quarantine as well as p=reject YES? Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=none as well as p=quarantine and p=rejectn NO??

Have I got the gist of what you both are saying? Are these the agreed upon settings?

Finally, I am understanding (hopefully correctly) that Yahoo and Hotmail are the trouble makers. And it especially makes trouble for the others (AOL and GMAIL) Good reason to suggest at least my YAHOO and HOTMAIL users switch to another provider. I found the free and secure provider disroot that offers a large amount of space. Maybe I will suggest that one.

Thanks,

Paul

Thanks Mark. Have made the changes (and told people they can refrain from changing providers for the time being.

I appreciate your help as well as that of the other contributors.

On Oct 12, 2017, at 5:13, Mark Sapiro mark@msapiro.net wrote:

On 10/11/2017 03:31 AM, mailbox.org wrote:

...

OK, there are three: Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy. MUNGE FROM Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=quarantine as well as p=reject YES? Shall the above dmarc_moderation_action apply to messages From: domains with DMARC p=none as well as p=quarantine and p=rejectn NO??

Have I got the gist of what you both are saying? Are these the agreed upon settings?

Yes.

Thank you Steve!

Now I understand it is not all bad. Just the way that AOIL and YAHOO went about it (or something like that).

paul

On Oct 15, 2017, at 5:07, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:

tl;dr I miswrote when I wrote that Hotmail is a problem sending domain. It never was and currently is not. I was thinking of AOL which was and is a problem.

The rest of this post explains why DMARC is a mostly good thing, including a *very* high-level view of what it is good *for*.

Mark Sapiro writes:

...

On 10/11/2017 03:31 AM, mailbox.org wrote:

...

...

Finally, I am understanding (hopefully correctly) that Yahoo and Hotmail are the trouble makers.

I miswrote here. Hotmail doesn't publish a DMARC p=reject policy; it's AOL that does. There are many other domains that do publish p=reject but they also have internal policies against posting to mailing lists. Yahoo! and AOL are the only posting addresses you're likely to run into that cause problems.

...

...

And it especially makes trouble for the others (AOL and GMAIL) Good reason to suggest at least my YAHOO and HOTMAIL users switch to another provider. I found the free and secure provider disroot that offers a large amount of space. Maybe I will suggest that one.

I'm with Mark here. Unless you "own" your users in some way (most of mine are my students, for example), it's way more trouble than it's worth to ask people to change. They'll need to move archived mail, for example. Also, AFAIK it's not possible to disable a Yahoo! address unless you delete it. BUT IT MIGHT NOT STAY DELETED: Yahoo! recycles unused addresses after a few months. It turns out that such reused account names will have access to any resources that authenticate using that Yahoo! address. So in practice users probably should keep their Yahoo! accounts anyway, even if they don't use them.

DMARC itself is a good thing, and you should encourage users to use email providers who participate in the protocol. Specifically, Gmail has an excellent policy: if it believes that a message that violates a sending provider's p=reject policy is a mailing list post, it will "quarantine" the mail in the "spam" folder. This means that there are no bounces for the *receiver* (which is why your subscribers were getting disabled or unsubscribed), and the receiver can easily find the mail at minor inconvenience (if they know to look, which is something of a problem, of course). I don't know of other providers with this policy but I expect it is in use at others and will probably spread.

How is DMARC a good thing? DMARC does the following

(1) Provide a way for email providers to get reports about usage of their mailboxes by third parties from recipients of such mail.

This helps them to learn about spam campaigns, especially "spear-spamming" where the bad guys know your correspondents and send you spam (or phishing) email that appears to be from an acquaintance. The DMARC consortium claimed in late 2015 that over 80% of all email was covered by DMARC reporting, so providers who have the skills to take advantage of this data have extremely precise knowledge of usage.

(2) Provide a way to notify recipients that all mail from the provider's domain is authenticated, and mail whose credentials do not verify must be presumed to be malicious. This is the "p=reject" policy that several of us have mentioned.

For (2) to make sense, the email provider should have a policy that prohibits use of its mailboxes to post to mailing lists, and it must not provide "on behalf of" services such as sending photographs or newspaper articles using your address in From. This makes sense for banks and other financial institutions, and use of DMARC "p=reject" has pretty much eliminated phishing using mail with real bank addresses in From.

This is how Yahoo! and AOL met trouble. Both leaked N x 100,000,000 contact lists to spammers, who used them for spear-spamming, much of it phishing of various sorts. Turning on p=reject is said to have reduced those spam campaigns from MILLIONS OF SPAMS PER MINUTE (!!) to a trickle. The business argument for doing this despite collateral damage to lists and various on-behalf-of businesses and their clients is obvious, and given how dangerous spear-phishing is, there's even a plausible ethical argument for it. (You can say "they shouldn't have leaked", but they did -- now what?)

Note that there is a new protocol in the works called ARC which will mitigate the problem for mailing lists as it's adopted. Unfortunately it is no help for "on behalf of" services, but as a mailing list developer and admin, I'll take it! Gmail and I think Yahoo! are already using it experimentally, although I don't know how they evaluate the "transitive trust" that is involved. (Ie, ARC involves the mailing list testifying that they verified the credentials of the poster.)

HTH

Steve

Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/paul%40tokyoprogressiv...

On 10/14/2017 02:07 PM, Stephen J. Turnbull wrote:

For (2) to make sense, the email provider should have a policy that prohibits use of its mailboxes to post to mailing lists, and it must not provide "on behalf of" services such as sending photographs or newspaper articles using your address in From. This makes sense for banks and other financial institutions, and use of DMARC "p=reject" has pretty much eliminated phishing using mail with real bank addresses in From.

Some drive by comments:

• IMHO, "on behalf of" services (I like that description) should be sent with a From: address that reflects the service -and- utilize a Reply-To: that reflects the email address of the purported sender. (Further, the service's From: address /should/ be legitimate and not bounce. But that's more pedantic.)

• I feel like DMARC is perfectly compatible with mailing lists as long as the mailing list is set up to modify the message as it passes through the list:

1. Change the From: header to reflect the mailing list.
2. Send the message with an SMTP from reflecting the mailing list. (VERP is suggested.)
3. Remove any / all DKIM headers.

• I *STRONGLY* feel that mailing lists / forwarders / etc are email endpoints. Many of them generate new messages with content based on the incoming content. - Thus it is perfectly acceptable to do all of the above /because/ it is a /new/ and /different/ message.

I know that I am not personally sending this message to anyone other than the single address that is the mailman-users mailing list. - The mailman-users mailing list is what is sending message to all the subscribers, *NOT* me. Both my mail server and the mail list server's MTA logs will corroborate this. - I think pretending that I am /personally/ (thus my MTA is) sending messages to all the subscribers is a farce. Further I believe that said farce is part of (if not the crux of) the perceived problems with SPF / DKIM / DMARC on conjunction with mailing lists.

Think about it this way. If Alice sends a message to Bob, who has his email configured to forward to Charlie who also forwards to Dave, and so on until we reach Mike, I will *STRONGLY* argue that I never sent a message to Mike if asked.

Sure, /someone's/ server sent a message to Mike, possibly claiming to be from me. But it was *NOT* /from/ me or my server. Thus, the message is bogus and /should/ be treated as such.

• I recently compared forwarders / mailing lists to be like phone messages. The person taking the phone message does not pretend to be the caller when passing the message along. Instead the message taker typically says something to the effect of "$SoandSo called and left a message for you." The phone message is a /new/ message based on the contents of the original call, *NOT* a (replay) of the original call.

-- Grant. . . . unix || die

On 10/17/2017 10:55 AM, Christian F Buser via Mailman-Users wrote:

I can perfectly follow your thoughts and arguments, they appear to be justified and reasonable.

Thank you. I tried to make them so that people could understand, even if they choose to disagree.

However, could you please elaborate whether Mailman (version 2.x or 3.x) or any other mailing list software really follows your ideas?

Yes!!! Mailman (and other MLMs) /can/ be configured to be SPF / DKIM / DMARC compliant!

If it is just a question of the setup for the mailing list, I would expect your instructions on how to set it up properly.

I don't have the exact step by step details. - I'm sure others (Mark...) on this list can give specifics on /how/ to configure Mailman.

The high level as I understand it is to do the following:

1. Set dmarc_moderation_action to munge From header.
2. Set REMOVE_DKIM_HEADERS to Yes (1) or 2 or 3.
3. Send messages from the list address. I recommend VERP.

Doing those three things ensures that messages leaving the mailing list do not violate the original sending domain's SPF / DKIM / DMARC security settings.

I would suggest that you also consider adding SPF / DKIM / DMARC for the domain of the mailing list to apply similar protections to outgoing messages. However that is not necessary to avoid undesired bounces.

Thank you, Christian You're welcome.

-- Grant. . . . unix || die

On 10/17/2017 03:22 PM, Mark Sapiro wrote:

Agreed, but the above imply NOT RFC 5322 compliant.

Please elaborate, if you're referring to more than From: vs Sent-By:.

In other words, an invalid DKIM signature SHOULD be treated no differently from no signature.

Fair enough. - I suspect DKIM by itself can tolerate that, like you are referencing.

I believe the problem is when DMARC is added to the mix, particularly with a policy of reject.

-- Grant. . . . unix || die

On Tue, 2017-10-17 at 14:54 -0700, Mark Sapiro wrote:

In the spirit of DMARC mitigation, we all agree that it is a necessary evil, at least in some cases, but that doesn't change the fact that it is an 'evil'.

Just as an aside here, my understanding is that validation of an email by DMARC requires ONE of two things: EITHER the DKIM signature in the email must validate, OR the domain of the From body header must resolve to the IP address of the Sender system (list server or mail reflector). Is this correct? Where's a reference on this?

-- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson

On Tue, 2017-10-17 at 16:28 -0600, Grant Taylor via Mailman-Users wrote:

That is a per domain setting left up to the DMARC publisher.

The DMARC publisher is not the system refusing delivery. The publisher advertises a policy. The receiving system honors it, or not.

At least my understanding is that you can specify any of the following=20 conditions to cause DMARC to pass / fail.

1. /Only/ SPF
2. /Only/ DKIM
3. SPF /or/ DKIM
4. SPF /and/ DKIM

Any system which REQUIRES DKIM validation to pass is out of compliance with RFCs, as I understand it. A DKIM signature which doesn't validate MUST be treated the same as no DKIM signature at all.

And I don't believe we're dealing with SPF here, just alignment between the domain of an email's author (From) and the IP address of the system communicating SMTP server from which the recipient's SMTP server received the mail. Correct me if I'm wrong on this, but I don't believe a SPF record in DNS is required.

-- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson

On 10/17/2017 03:28 PM, Grant Taylor via Mailman-Users wrote:

On 10/17/2017 04:15 PM, Lindsay Haisley wrote:

...

Just as an aside here, my understanding is that validation of an email by DMARC requires ONE of two things: EITHER the DKIM signature in the email must validate, OR the domain of the From body header must resolve to the IP address of the Sender system (list server or mail reflector).  Is this correct? Where's a reference on this?

That is a per domain setting left up to the DMARC publisher.

Not quite.

At least my understanding is that you can specify any of the following conditions to cause DMARC to pass / fail.

1. /Only/ SPF
2. /Only/ DKIM
3. SPF /or/ DKIM
4. SPF /and/ DKIM

No. The standard says DMARC passes if either SPF or DKIM passes with an aligned domain. The only thing the DMARC publisher controls for one or the other or both is whether alignment is strict or relaxed.

See my post that I was still typing when this was sent https://mail.python.org/pipermail/mailman-users/2017-October/082621.html.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/17/2017 03:15 PM, Lindsay Haisley wrote:

Just as an aside here, my understanding is that validation of an email by DMARC requires ONE of two things: EITHER the DKIM signature in the email must validate, OR the domain of the From body header must resolve to the IP address of the Sender system (list server or mail reflector). Is this correct? Where's a reference on this?

The reference is the DMARC standard RFC 7489 https://www.rfc-editor.org/rfc/rfc7489.txt.

It's more complicated than the above. There is a concept of domain alignment. Alignment is satisfied in either "strict" or relaxed "mode". A dmarc policy record may optionally specify either mode for DKIM alignment or SPF alignment or both with the default being "relaxed.

For a message to pass DMARC it must meet 1 of 2 requirements.

1. It must possess a valid DKIM signature from a domain aligned with the From: domain. In strict mode aligned means equal. In relaxed mode aligned means the corresponding organizational domains are equal.

or

2. It must pass SPF. SPF works on the domain of the SMTP envelope from. Thus for SPF to pass, that domain must publish an SPF record specifying the IP of the sending server as a permitted sender. Further, for DMARC the envelope from (SPF) domain must align with the From: domain. Again, in strict mode aligned means equal. In relaxed mode aligned means the corresponding organizational domains are equal.

Note that if you are relaying mail, SPF probably will pass for your server if the envelope from domain is your server, but it won't align with an unmunged From: domain and if it does align because you didn't rewrite it, SPF will fail unless the original sending domain publishes SPF that permits your server as a sender.

So the bottom line is as an "unaffiliated" relay without munging From:, SPF will never pass for DMARC and DKIM will only pass if you don't transform the message in ways that break the From: domain's DKIM signature.

There is a remote possibility that the originating domain that publishes a DMARC policy relies on SPF and doesn't DKIM sign the message in which case, unmumged, relayed mail will almost certainly fail DMARC.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/17/2017 05:04 PM, Grant Taylor via Mailman-Users wrote:

On 10/17/2017 05:07 PM, Mark Sapiro wrote:

My brain is failing to translate "corresponding organizational domains" to "sub-domains" properly and what that means for strict vs relaxed.

In another thread on mailman-developers, I discussed organizational domains with Lindsay, so I assumed he knew.

In summary, every domain has a corresponding organizational domains which may be the same or a "super" domain. In short, the organizational domain is the domain that might be found in whois. For "common" tlds like .com, .org, net, .edu, etc. the organizational domain is the top two levels. E.g. the organizational domain for some.sub.domain.example.com is example.com, but it's much more complicated than that. See https://publicsuffix.org/list/ if you want to know more.

...

So the bottom line is as an "unaffiliated" relay without munging From:, SPF will never pass for DMARC and DKIM will only pass if you don't transform the message in ways that break the From: domain's DKIM signature.

I assume that you're talking about the SMTP envelope from and not the From: header.

No. I could have slipped, but when I write From: domain, I mean the domain of the address in the From: header (That's what DMARC is all about - verifying that the message actually came from the domain of the address in the From: header). If I mean the domain of the envelope from, I try to use that phrase, but in the context of DMARC that domain is only relevant for SPF and only if it "aligns" with the From: domain.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/17/2017 06:28 PM, Lindsay Haisley wrote:

Yes, technically I know, but this kind of stuff makes my head hurt and my hats to change colors, so I fall back on "If it works, don't fix it".

I hear that and I feel your pain. Somehow it was all simpler when I was younger, and I don't think it was just because I was younger.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On Wed, 2017-10-18 at 12:38 +0900, Stephen J. Turnbull wrote:

This whole thread reminds me of an evangelical arguing with a Jesuit. 2000 years of Bible study does make for strong debating!

Welll....  Spending much time reading RFCs can certainly put one in a biblical frame of mind ;)  Lots of SHOULD, MUST and MAY therein.

-- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 |
http://www.fmp.com | -- Hiram W Johnson

On 10/17/2017 04:46 PM, Grant Taylor via Mailman-Users wrote:

I decided to see if there was an update to RFC 5322, and lo and behold there is.  RFC 6854, which specifically updates RFC 5322 section 3.6.2 and allows group address syntax exists.

TL;DR:  From: can now contain a Group address / name, which can zero or one or more mailbox addresses.

I feel like RFC 6854 provides some light at the end of the tunnel and allows mailing list managers to modify the From: to be the group, including the Group's address.

Group address syntax is something else. it is a specific syntax which is a name followed literally by a colon followed by a list of zero or more mailboxes (email addresses) and terminated by a semicolon.

E.g., from the RFC

Second, consider an email message that is meant to be "from" the two managing partners of a business, Ben and Carol, and that is sent by their assistant, Dave. This message could always have been presented this way:

  From: ben@example.com,carol@example.com
  Sender: dave@example.com

This change allows it to be represented this way:

  From: Managing Partners:ben@example.com,carol@example.com;
  Sender: dave@example.com

The group syntax has always been allowed in To: and some other headers. RFC 6854 just extends it to From:

This is most commonly seen with some MUAs when all the recipients are Bccs, the message is

To: undisclosed recipients:;

Sender: is not needed because it would be the same as the Group's from address.

There's no such thing as a group's address unless the addresses are listed along with the group name.

Anyway, using a group name alone as From: avoids DMARC as there is no From: address domain for a DMARC lookup.

Similarly, I found wording in RFC 5322 that indicates that a user agent forwarding a message, is actually a new message.  Section 3.6.6 has the following copy:

...

    Note: Reintroducing a message into the transport system and using resent fields is a different operation from "forwarding". "Forwarding" has two meanings: One sense of forwarding is that a mail reading program can be told by a user to forward a copy of a message to another person, making the forwarded message the body of the new message.  A forwarded message in this sense does not appear to have come from the original sender, but is an entirely new message from the forwarder of the message.  Forwarding may also mean that a mail transport program gets a message and forwards it on to a different destination for final delivery.  Resent header fields are not intended for use with either type of forwarding.

That type of forwarding is exactly what is done by Mailman's DMARC Wrap Message action and that is the reason that action exists. Because in that case the list message is RFC 5322 compliant. However many MUAs, particularly mobile apps, have difficulty rendering such a message in a good way, so Wrap Message isn't always the best option.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/18/2017 09:31 AM, Grant Taylor via Mailman-Users wrote:

Um....  My interpretation of 6854 § 1 and § 4 makes me think that an empty group list is perfectly acceptable.  Further, the group list can be non-empty and contain the lists posting address.

True, but in either case it still does not represent the "author" of the message.

I would rather do something like the following so that users could reply to the message.  (It would also avoid potential MUA issues as indicated by RFC 6854.)

I would think that it would be acceptable to use a From "group address" that is the mailing list.  I.e.

   From: Mailman Users:mailman-users@python.org;

Possibly even something like the following:

   From:  Grant via Mailman Users:mailman-users@python.org;

Arguably, this is conceptually very similar to what has become the defacto method to deal with DMARC today by munging the From:

   From:  Grant via Mailman Users mailman-users@python.org

The difference is that RFC 6854 codifies that there are times to alter the from.  -  At least that's how I'm interpreting this.

That's where you are wrong. All RFC 6854 does is allow the "group" syntax to be used as the content of the From: header. It does not change the RFC 5322 at al requirement that the From: header represent the author(s) of the message.

Further, if you believe the fact that the outbound message is indeed a completely new message (as I do) then it's completely legit to set the from to what ever you want.  ():-)

This is the crux of our disagreement. The outbound message is still the original author's message, albeit slightly altered by subject prefixing, content filtering and/or other transformations to conform with list policies. I don't agree that it is a completely new message. I think it is still the original message with only technical and formatting changes.

...

That type of forwarding is exactly what is done by Mailman's DMARC Wrap Message action and that is the reason that action exists. Because in that case the list message is RFC 5322 compliant. However many MUAs, particularly mobile apps, have difficulty rendering such a message in a good way, so Wrap Message isn't always the best option. It sounds like you're talking about message/rfc822 message attachments. -  That is a viable option.

However I see no reason that you can't take the body copy from the incoming email and use it directly in the new outgoing email.  No need to message/rfc822 wrap (or other digest like raping) the outgoing message.

The difference is wrapping the message preserves the original message's headers (particularly From:) and makes it the content of another message which says essentially "here's the message the list received". That outer message can be From: the list and still be standards compliant.

However, if you are just sending the body of the original message From: the list, according to RFC 5322 et al, you are saying the list is the author of that message body. This is not true and is why I say the message is not compliant with RFC 5322 et al.

Granted, all things considered, this is what most of us choose to do. I'm not saying this shouldn't be done. It is something we are forced to do because certain freemail providers choose to publish DMARC p=reject policies contrary to the original intent of DMARC, but all I'm saying is we should not forget that when we do this, we are sending messages that are not strictly standards compliant.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/18/2017 11:14 AM, Grant Taylor via Mailman-Users wrote:

I think it will be interesting to see what happens as more and more domains adopt DMARC, including those that use p=reject.  Especially with some of governmental institutions purportedly being mandated to use DMARC.  -  IMHO, DMARC is going to eventually become the new norm.

DMARC is not the problem. It is perfectly reasonable for say, irs.gov to publish DMARC p=reject as long as mail From: irs.gov is not an employees personal post to an email list. Presumably the IRS would have rules against that.

The problem is when general ESPs that provide addresses in their domain for anyone to use for any personal purpose publish DMARC p=reject.

I also wonder what ARC is going to do to this paradigm.

ARC has the potential to help. When say a yahoo.com user posts to a list on my server and the list sends the post to a hotmail.com user, ARC allows me to certify that Yahoo's DKIM signature was valid when I received the mail, then I broke the sig but resigned the mail with my domain's sig and sent it on to Hotmail. Now there is a chain by which Hotmail can verify my sig and the fact that I certify Yahoo's sig. The crux however is Hotmail has to trust me. Now if I'm GoogleGroups, Hotmail will probably trust me but if I'm mail.python.org there might be a mechanism by which I can ask Hotmail and every other ISP to trust me, but is that going to work in practice. I think that remains to be seen.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/18/2017 11:35 AM, Mark Sapiro wrote:

DMARC is not the problem. It is perfectly reasonable for say, irs.gov to publish DMARC p=reject as long as mail From: irs.gov is not an employees personal post to an email list. Presumably the IRS would have rules against that.

Would they?  Shouldn't IRS sysadmins who use Mailman in the course of their jobs send messages to this mailing list using their @irs.gov addresses?

Not all submissions to public mailing lists are personal use.

Jordan Brown writes:

Would they?  Shouldn't IRS sysadmins who use Mailman in the course of their jobs send messages to this mailing list using their @irs.gov addresses?

As Mark says, they should use an @sysadmins.irs.gov address or something like that, which would have its own p=none policy. Note that this has been already standard practice at Yahoo! (!), AOL (!!), LinkedIn, and several banks that participate in IETF discussions. Since 2013 for Yahoo! and LinkedIn IIRC.

This need not be burdensome. Many MUAs support automatic "personality" switching based on addressee.

Steve

On 10/19/2017 04:07 AM, William Bagwell wrote:

So if enough users of Yahoo and AOL requested something such as user@list.aol.com to not be DMARC p=reject they /might/ listen?

I think that won't happen. The use of p=none subdomains by various entities that publish p=reject for their primary domain is intended for addresses for their own staff to use in communicating via mailing lists and perhaps other channels. If a freemail provider such as Yahoo would be willing to create a lists.yahoo.com domain with p=none for use by their freemail users, that domain would be subject to the same abuses that caused them to publish p=reject in the first place.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Grant Taylor via Mailman-Users writes:

IMHO, DMARC is going to eventually become the new norm.

It has been so since late 2015, according to the DMARC Consortium. At that time they claimed that 80% of legitimate email was originated at domains that participate in DMARC reporting protocols. I don't think p=reject will ever be the norm for freemail providers.

I also wonder what ARC is going to do to this paradigm.

It may or may not help mailing lists. It depends on whether the spammers successfully jump on it to obfuscate themselves, which they could do, in which case you might end up in the current situation where you need to apply for whitelisting at some of the large providers. On the other hand, the large providers are getting better at identifying responsible lists for themselves, and ARC would definitely make authenticating those lists easier.

Steve

-- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN

On 10/19/2017 12:37 AM, Stephen J. Turnbull wrote:

The IETF has NO position on WHEN this should be done because it's not relevant to interoperability. My personal reasoning with respect to mailing list managers like Mailman which normally pass through all text/plain, and perhaps add some tags to Subject and prefix or suffix the body, is that users (including posters) would be quite annoyed if de-duping didn't work. And those of us who deal with mail in sophisticated ways would be quite upset if the Message-ID we give it doesn't correspond to the Message-ID distributed by the list and in the archive.

I believe RFC 6377 makes it fairly clear if a message is new or not. TL;DR: If anything other than the SMTP envelope is modified, then the MLM is a resending MLM, which necessitates a new message with a new author and Message-ID.

I can respect your concern about the Message-ID changing, especially with deduplication. However, I counter that the new message from the resending MLM is in fact a different message than the one that the original author sent to the resending MLM. So, if you were in the To / CC / BCC of the message from the original author, you /should/ receive two copies of the message.

Fortunately nicer MLMs, like Mailman, can detect that a list subscriber was included in the To or CC and act on the subscriber's configured option if they want to receive a copy of the message from the MLM that they received directly.

-- Grant. . . . unix || die

On 10/18/2017 11:50 AM, Mark Sapiro wrote:

This is the crux of our disagreement. The outbound message is still the original author's message, albeit slightly altered by subject prefixing, content filtering and/or other transformations to conform with list policies. I don't agree that it is a completely new message. I think it is still the original message with only technical and formatting changes.

RFC 6377 - DomainKeys Identified Mail (DKIM) and Mailing Lists, disagrees with you. (RFC 6377 is also currently known as BCP 167.)

§ 3.2 calls this out specifically:

resending: A resending MLM (see Sections 5.2 and 5.3 of [EMAIL-ARCH]) is one that may make changes to a message. The output of such an MLM is considered to be a *new message*; *delivery of the original has been completed* prior to distribution of the reposted message. Such messages are often reformatted, such as with list-specific header fields or other properties, to facilitate discussion among list subscribers.

/The output of a resending MLM is/ *a new message*.

MLM Output: *MLM* (sending its reconstructed copy of the originating user's message) *is Author*; MLM's ADMD is Originator and Signer; the ADMD of each subscriber of the list is a Verifier; each subscriber is a Receiver.

*The resending MLM is the author* /of the new message/.

The dissection of the overall MLM operation into these two distinct phases allows the DKIM-specific issues with respect to MLMs to be isolated and handled in a logical way. The main issue is that the repackaging and reposting of a message by an MLM is actually the construction of a completely new message, and as such, the MLM is introducing new content into the email ecosystem, consuming the Author's copy of the message, and creating its own. When considered in this way, the dual role of the MLM and its ADMD becomes clear.

Since we have been talking about modifying more than /just/ the SMTP envelope, we are indeed talking about a resending MLM and not an alias MLM.

-- Grant. . . . unix || die

On 10/19/2017 09:14 PM, Grant Taylor via Mailman-Users wrote:

RFC 6377 - DomainKeys Identified Mail (DKIM) and Mailing Lists, disagrees with you.  (RFC 6377 is also currently known as BCP 167.)

I am too tired at the moment to respond to your posts more completely. I may do so tomorrow. But I suggest that if you are going to quote RFCs that you understand the differences between Best Current Practice and Standards Track categories.

Also, I don't disagree that there are issues between DKIM, DMARC and Mailing Lists that make seamless integration of these impossible without changing long standing norms and expectations for Mailing Lists. I also think Mailman (both 2.1 and 3) give you tools to do pretty much whatever you want in this vein except for changing the Message-ID: of the original post. Note that one of the biggest reasons for that is if the list copy has a different Message-ID: and some people receive and reply to a list copy and some receive a direct To: or Cc: and reply to that and people use MUAs that produce threaded views based on Message-ID:, References: and In-Reply-To: headers, threading can get pretty messed up.

Finally, I think all we disagree on (as Steve implied in a post a day or two ago) is very arcane, small technical details, and while we may never come to agreement on these, I think we do agree that Mailman can operate in this environment in ways we think are satisfactory.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/23/2017 12:57 AM, Stephen J. Turnbull wrote:

Grant Taylor via Mailman-Users writes:

...

RFC 6377 - DomainKeys Identified Mail (DKIM) and Mailing Lists, disagrees with you. (RFC 6377 is also currently known as BCP 167.)

tl;dr version: RFC 5598 (non-normative but authoritative) disagrees with you. In practice, the mailing list *decides* whether it is producing new messages or not, and adjusts Message-ID if "new". The RFC recommends that a message undergoing only Mailman-style changes should be considered the *same* message.

I appreciate you going to the source here, but you shouldn't read RFCs the way you read Wikipedia. As I joked before, you need an education comparable to a Jesuit's Bible study to parse RFCs with facility. Anybody can read RFCs, of course (I have no training in this!), but be careful: context matters.

RFCs are a record of a process. Unless you were directly involved that that process, RFCs are about as useless as garbage. They are not only without clear explanation, but they are often just plain wrong and contradictory.

People who suggest reading them need to have their meds adjusted.

-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013

On 10/17/2017 04:40 PM, Grant Taylor via Mailman-Users wrote:

On 10/17/2017 03:22 PM, Mark Sapiro wrote:

...

Agreed, but the above imply NOT RFC 5322 compliant.

Please elaborate, if you're referring to more than From: vs Sent-By:.

...

In other words, an invalid DKIM signature SHOULD be treated no differently from no signature.

Fair enough.

Why? If this message doesn't match its signature, then it has been altered in transit for sure. If were not signed, like when I post from home (because I can't be arsed to set gpg up on winderz), then there's no telling if it was or wasn't. One of those things is quite a bit not like the other.

-- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

On 10/17/2017 03:28 PM, Dimitri Maziuk wrote:

On 10/17/2017 04:40 PM, Grant Taylor via Mailman-Users wrote:

...

On 10/17/2017 03:22 PM, Mark Sapiro wrote:

...

In other words, an invalid DKIM signature SHOULD be treated no differently from no signature.

Fair enough.

Why? If this message doesn't match its signature, then it has been altered in transit for sure. If were not signed, like when I post from home (because I can't be arsed to set gpg up on winderz), then there's no telling if it was or wasn't. One of those things is quite a bit not like the other.

Why? Because that's what the DKIM standard, RFC 4871, says.

You have a point, but to be safe you should assume that unsigned mail has been altered and if it's important, insist on some kind of cryptographic verification.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 10/17/2017 11:45 AM, Dimitri Maziuk wrote:

If these actually exist, my spamassassin has been delivering to /dev/null for quite some time now. My impression is they largely died off, possibly thanks to adoption of SPF.

If these actually exist? - I'm talking about someone configuring their old email address to forward to their new email address. - I just happened to extrapolate out further. I.e. old college email forwards to Yahoo, which forwards to Gmail, etc. - I suspect the single level forwarding is quite common.

Are we talking about the same thing? I.e. .forward files? Or are you thinking something more nefarious?

Now it is much easier and cheaper to send spam from botnets of perfectly legitimate pwn3d peecees. Or to anonymously register a perfectly valid domain (e.g. tnеtсоnsulting.net -- there's 3 "language-specific script" chars in there), add all the DMARC embellishments, and send perfectly compliant spam as gtaylor from there.

I scowl at you sir. I dislike being the example. But I think what you did is quite neat and perfectly valid example. Nicely played sir.

I actually have no idea how to defend against such attacks, save for registering all such permutations.

I wonder how some such language-specific script characters would show up in logs. Especially ASCII without UTF support.

For bonus points, pay with stolen credit card number and have your spam campaign all done by the time visa fraud department calls you domain registar.

/me wonders what color Dimitri's hat is. ;-) #knowtheyenemy

-- Grant. . . . unix || die

On 10/17/2017 06:00 PM, Dimitri Maziuk wrote:

I've a "tactical foliage green" kufiah, best five bucks I ever spent on an article of clothing.

I like it.

The point was that SPF will flag messages with ineptly spoofed From addresses, and I don't seem to see any of those anymore.

;-)

As for DKIM, say you proved that the message was altered after the postmaster@yourdomain was done with it. Now what? Depending on how you look at it, the standard says either

• now pretend you don't know if it was altered (in your interpretation: "maliciously") or not, or
• (in Mark's version) assume anything not signed is malicious and invalid. I strongly dislike either alternative.

I personally work under the assumption that:

If DKIM signature validates, then I consider the message good.

If DKIM signature fails, then there is something wrong with the message, and treat it suspiciously. Read: I increment the spam score. (If the spam score is high enough I reject the message at SMTP time.)

If there is no DKIM signature, I continue processing normally.

-- Grant. . . . unix || die

On 10/18/2017 09:18 AM, Dimitri Maziuk wrote:

Then you seem to misunderstand what crypto signatures actually do.

I believe I understand what the crypto signatures actually do.

We are each entitled to decide what to actually do based on the result of the crypto signature (in)validity.

If signature check fails, then the message is not what its author actually wrote. IRL it's mainly SorceForge and the like injecting its ads into signed parts, (and the real reason google is pushing https and dkim so hard is it's messing with their ad revenue,) but in principle if the check fails the message *content* is *invalid*. Whoever the author and whatever the content.

I believe I remember (but can't point to) something in the DKIM spec that referenced the possibility that the DKIM signature could be broken by things as benign as an MTA doing a content transfer encoding conversion. - I have personally seen this.

As such, you can't be 100% positive that the message content's meaning / copy has actually changed, just that something about the message has changed. - Thus it is advised to only treat valid signatures as a good thing and be cautious of treating invalid signatures as a bad thing.

I use DKIM validity as a signal that I then make decisions based on. - Hence why I have chosen to alter spam score on my mail server based on the DKIM result.

-- Grant. . . . unix || die

On 10/18/2017 11:51 AM, Dimitri Maziuk wrote:

Like tnеtсоnsulting.nеt being a benign minor encoding change in a couple of characters?

No. That is not a simple content encoding change. Content (re)encoding changes the representation of the same encoded data.

<е> 1077, Hex 0435, Octal 2065 != <e> 101, Hex 65, Octal 145 <с> 1089, Hex 0441, Octal 2101 != <c> 99, Hex 63, Octal 143 <о> 1086, Hex 043e, Octal 2076 != <o> 111, Hex 6f, Octal 157

An MTA changing the encoding method of data to / from: base 64 / quoted-printable / 8-bit, is distinctly different than what you have done, which is changing actual encoded data.

The (decimal) number 17 can be encoded multiple ways:

10001 = binary base 2 25 = hex base 6 21 = octal base 8 17 = decimal base 10 11 = hexadecimal base 16

All five encoded numbers represent the same value (decimal) 17.

What you have done (in the spirit of a white hat) is actually a homograph attack. Something quite different from simple encoding differences.

Quite similar to a computer seeing a the following three characters as quite distinctly different things, each with different computational meanings.

0 O o

Just because the authors of the RFC have also chosen to stick the square peg in the round hole doesn't make the hole any less round, nor the peg any less square.

Fair.

Somewhere I've a 10-year old e-mail from Whit Diffie explaining how SSL was a PR solution to a marketing problem. So this kind of problem-finding and problem-solving has made to SMTP RFCs now, colour me shocked.

I'd be curious to read said email, if it's convenient to dig up.

-- Grant. . . . unix || die

On 10/18/2017 01:07 PM, Dimitri Maziuk wrote:

17 == 0x11. "17" != "0x11". Which was precisely the point: if your MTA, say, does unicodedata.normalize( 'NFKD' ... ), and turns u-umlaut into a regular "u", you may consider it benign. Many won't.

I would not consider that benign at all.

I'm referring to the difference between:

• ü - ASCII (?)
• =C3=BC - quoted-printable
• w7w= - base 64
• ü - HTML

All four representations are for the *same* letter / character / glyph / byte(s).

I consider those to be (effectively) benign content encoding changes. - Note the content is the same, with the only difference being how it's encoded.

Most importantly, crypto signature will change, and DKIM check will fail.

DKIM, by design will fail if anything that is signed changes. (See the ROPEMAKER attack for a better explanation about anything signed.)

Benign is in the eye of the beholder.

~eh~ ... Okay.

We're inserting this stuff into a database where a search for "Wutrich" will find neither "Wütrich" nor "W\u0308trich" so I wouldn't consider it benign at all.

I do not consider "Wutrich" and "Wütrich" to be the same string. The former may be considered a poor representation of the latter.

I'm not sure which Unicode code point 308 is, but I doubt that it is the same as <ü> 252, Hex 00fc, Octal 374. (I would have to look it up to know for sure.)

I would hope that data would be normalized to the same encoding in the database. I.e. "=C3=BC" (quoted-printable) would be normalized to "ü" and stored in the database as such.

I would further hope that any search of the database would be able to do something like a character class (type) search so that it could match on "W[üu]trich". (Adjust as necessary.)

-- Grant. . . . unix || die

On 10/18/2017 02:10 PM, Dimitri Maziuk wrote:

They are different ASCII representations of the same byte, yes. They are not the same text.

Hum. I wonder if we have been talking about slightly different things.

I've been referring to "ü" being displayed the same in MUAs which is interpreting the different underlying text in the various content transfer encodings.

Sign the text, re-encode text and signature together, anyone who cares about it can decode it back to where the signature will match.

Do I understand you correctly to mean to create the signature before applying transport encoding?

Only, you can't do that on the MX, it has to be done on the client.

Why can't you do it at the MX?

Or do you mean that it's inefficient to do so at the MX?

DKIM is designed to produce false positives. Which means DKIM-based tests will have low specificity (https://en.wikipedia.org/wiki/Sensitivity_and_specificity).

My experience ~> opinion, save for mailing lists, differs. In fact, most of the email that I receive passes DKIM.

Which makes them bad for detecting spam. But that's OK, DMARC in general is for *fraudulent* e-mail, not *unsolicited* e-mail.

I don't think DKIM (or SPF or DMARC) have /anything/ to do with spam detection. SPF is for envelope sender authorization. DKIM is for message integrity. DMARC is for policy and reporting. None of that has anything to do with spam detection / filtering.

In fact, I've found that spammers (worth their salt) tend to be early adopters of email technology. Thus they are quite likely to send spam that passes SPF and DKIM and DMARC.

I'm sure once I'm plagued by *fraudulent* e-mail, I'll start caring about RFC 7489 and the rest of them.

I started caring about SPF / DKIM / DMARC for a couple of reasons:

1. I'm pedantic and want to have the best filtering / security that I possibly can on my personal domain.

2. I was seeing blow back from mailing lists about DKIM and / or DMARC. Thus I dug in more and learned more.

To each his / her own motivation (or lack there of.)

When those e-mail are from mailman I'll start caring about what mailman does with DMARC headers. But at this point I'd just strip them all off.

I suspect that when (if) you care will be after you implement filtering (Chicken / Egg?) that possibly rejects messages from mailing lists. Or possibly if your messages with enhanced security cause others to have a problem. (Again with the chicken & egg.)

(And since I'm tripping down the memory lane: https://catless.ncl.ac.uk/Risks/23/21#subj9.1)

:-P

-- Grant. . . . unix || die

On 10/18/2017 03:42 PM, Dimitri Maziuk wrote:

Because the very first $relayhost may apply transport encoding. You have to compute the hash before that happens.

It's my understanding that DKIM is usually applied by the egress MSA / MTA.

I guess an MSA could apply DKIM itself. It would need to publish it's public key / selector in DNS. So that's probably a reason not to have every MUA apply DKIM itself. It is probably much more economical to apply DKIM at the MSA / 1st MTA.

Ideally intermediary MTAs / receiving MTA would not need to apply content transfer encoding.

It's my understanding that MTAs prefer to avoid changing the message unless there is a requirement to do so. I.e. downstream MTA won't accept the message as it currently is.

My "why can't you..." question was more why can't an MX do an operation that an MUA can do. - I was thinking you were saying that a receiving MTA couldn't validate before accepting a message.

That does not contradict what I said. Low specificity means low probability of detection of "bad stuff". I.e. it doesn't mean much that most of it passes.

Ohkay, so what exactly am I the end user is supposed to need it for?

I don't know that DKIM is really targeting end users. I think DKIM is more targeting postmasters to configure on their MTAs.

I'm using a Thunderbird add-on that allows me to see / validate DKIM in my receiving MUA. (My MSA applies DKIM for me.)

I, as a postmaster, want DKIM for a couple of reasons, 1) I want to be able to filter incoming messages based on DKIM (for better or worse) and 2) outgoing DKIM signing for use in conjunction with DMARC.

You (/me waves hands around the room) may not care enough to bother with DKIM. That's your prerogative. Just like we are all free to run our mail servers that way that we want to.

-- Grant. . . . unix || die

Grant Taylor via Mailman-Users writes:

I use DKIM validity as a signal that I then make decisions based on. - Hence why I have chosen to alter spam score on my mail server based on the DKIM result.

You can do that. But call it what it is: a deliberate decision NOT to conform to a standards-track RFC.

The fact of the matter is that the spammers are laughing at you. THEY have perfectly valid DKIM signatures, or if they're going to try a replay attack, they remove the DKIM signature they're about to break. Broken DKIM signatures principally mean somebody added a footer to the body, a DMARC mitigation in From, or a tag to the Subject. So this rule primarily targets perfectly legitimate mail posted to mailing lists.

(I don't understand Dimitri's claim about SourceForge ads; all the mail I get from SourceForge is originated there and AFAIK the DKIM validates. If it doesn't, their system is pretty brain-damaged.)

Steve

-- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN

On 10/17/2017 09:10 AM, Grant Taylor via Mailman-Users wrote:

I know that I am not personally sending this message to anyone other than the single address that is the mailman-users mailing list.  -  The mailman-users mailing list is what is sending message to all the subscribers, *NOT* me.

That is quite true, but in this example, the mailing list is the 'sender' of the message I receive. It is not the 'author' of the message. You are still the 'author'. RFC 5322, sec 3.6.2 and predecessors are clear:

The "From:" field specifies the author(s) of the message, that is, the mailbox(es) of the person(s) or system(s) responsible for the writing of the message. The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan