On 5/14/2014 2:49 PM, Stephen J. Turnbull wrote:

Gary Algier writes:

...

I have been following the discussion of the DMARC issues and Mailman's attempts to live with it. I was wondering if anyone has an "Executive Summary" of the DMARC issue in a general sense.

How about the following:

DMARC is a set of protocols for Internet mail that are used to by participating hosts to exchange information about mail traffic. This information is primarily useful for fighting email fraud, including spam and phishing. The basic procedure is that recipient hosts which participate in the protocol check whether the apparent sending domain (the domain that appears in the address in "From" header field) participates, using a query to the DNS defined by the DMARC protocol. If so, the recipient host will send to that domain some information about mail which purports to be from the sending host, but fails to validate.

The validation refers specifically to, and only to, the domain in the "From" header field. The DMARC protocol itself does not validate the mailbox (user), although some originating hosts may do so. Nor can it guarantee that the user's account hasn't been used fraudulently.

For participants in the DMARC protocol, valid mail provides a strong guarantee that the domain of the address in the "From" header field is authentic, with some strong restrictions required to ensure success of validation. These restrictions include:

(1) When authentic mail is delivered directly from the sending host to the recipient, validation will succeed.

(2) When authentic mail is forwarded verbatim by all intervening Internet hosts (including mailing lists and user mailbox forwarding), validation will succeed.

Authentic mail may fail to validate in many common circumstances, where a forwarding host alters certain headers or any part of the body of the message. This may occur for forwarding hosts which add advertisements to the footer or the like, or if a forwarding host reformats the body (it shouldn't do that, but some do). There are also restrictions on some parts of the mail. For example, the "From" header field must contain exactly one address (multiple author addresses are permitted by the mail standards and are occasionally useful, though rarely seen nowadays).

Failure to validate is also frequent on mailing lists. Common transformations that *will* cause mail distributed by a mailing list to fail validation include (but are not limited to)

• Adding a list tag or sequence number to the Subject header field.

• Adding a footer, typically containing unsubscribe information for public discussion lists or legal disclaimers for corporate lists.

• Removal of banned content, such as executable file attachments or excessively large attachments.

Because many common practices with Internet mail can cause authentic mail to fail DMARC validation, although mail which successfully validates may be trusted to be from the domain in the address in the "From" field, failure to validate in general *does not* imply lack of authenticity.

Under some circumstances, a sending domain may be willing to severely restrict its users' use of their addresses. For example, they may not be allowed to post to mailing lists, and recipients may be advised to *always* keep their addresses updated to the domain where they actually read their mail. This is common practice with banks and other financial institutions. Such organizations can be reasonably sure that mail which fails to validate is "fraudulent" (although the intent may be merely mischievious rather than truly felonious). They may take advantage of a further feature of the DMARC protocol, the "policy", which is the action that the sending domain suggests that recipients take with mail that appears to be from that domain but fails validation.

The normal policy is "none", that is, the recipient should deliver the mail as though the apparent sending domain did not participate in DMARC. (Of course, if the mail fails spam or virus checks, it would be quarantined or discarded as usual.) The second level is "quarantine", which means that the mail should not be delivered directly to the user's mailbox in the ordinary way, for fear of fraudulent use of the sending address (typically phishing). The third level is "reject", which means that the sender recommends that the mail not be delivered at all, to completely prevent fraudulent misuse of their domain.

Unfortunately, unless the sending domain has truly draconian policies about the use of addresses in that domain, use of "quarantine" or "reject" makes it virtually certain that some authentic mail will not be read in a timely fashion or (in the case of "reject") never delivered at all.

The "reject" policy may also result in denial of services to third parties (one common case is having delivery disabled from a mailing list, or even being unsubscribed), due to rejected mail being "bounced" back to the forwarding host. At present we know of *no* hosts which distinguish DMARC bounces from other delivery failures. Therefore they are treated as a permanent delivery failure to the recipient due to problems at the recipient's host, rather than due to a policy of the sender. As a consequence, those unrelated recipients may be dropped from distribution lists.

Editorial comments:

In some cases (where the mail validates according the DKIM, one of the protocols used in DMARC validation), the mail can be trusted not to have been altered in any way by a "man in the middle".

It is entirely up to the recipient whether to honor the published policy or not. (GMail, for example, does not -- it treats the policy as advisory and makes case by case decisions, allowing most list mail to be delivered despite failing DMARC validation.) However, because the original use of policy was envisioned to be banks vulnerable to phishing (not large freemail providers incapable of protecting their users' address books), most recipients do honor the policy published by sending domains.

The action of Yahoo! and AOL in publishing policies of "reject" basically amounts to deciding that they do not care if their users' mail can be reliably delivered. Nevertheless, Mailman is committed to providing mitigation options for mailing list owners that improve the reliability of mail delivery for posters from those domains, and protect other subscribers from the (unintentional) denial of service attacks being conducted by Yahoo! and AOL.

I believe the word "attack" is justified because Yahoo! and AOL are now aware that their policies have the effect of denying service to third parties (even if not intended by them), but they continue to implement them. It seems likely to me that as mailing lists and others implement workarounds that bypass DMARC validation, these workarounds will be mimicked by spam and phishing mails to bypass DMARC validation, and in response AOL and Yahoo! will take further action detrimental to the reliability of everyone's mail service, not just their own users'.

...

Of course if someone says that the current MS365 implementation has addressed this, then that's a different (unfortunate) story.

Seems unlikely to me (this isn't a new variant of a traditional problem to be addressed by database updates like virus checking), although they probably will roll out something within a couple months (counting from April 22). Nevertheless I doubt it will be as flexible as Mailman already is (and most likely will be tuned to protect Hotmail users if Hotmail should decide to use the "reject" policy).

...

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

How (sadly) apropos!

HTH

Steve

Is this also true?

Users from DMARC-reject domains send mail to mailing lists, and the resulting mail from the mailing list is rejected. Enough rejections can cause the mailing list possibly to be blacklisted for sending lots of "spam" mail.

--Barry Finkel

At Wed, 14 May 2014 15:24:32 -0500 "Barry S. Finkel" <bsfinkel@att.net> wrote:

On 5/14/2014 2:49 PM, Stephen J. Turnbull wrote:

...

Gary Algier writes:

...

I have been following the discussion of the DMARC issues and Mailman's attempts to live with it. I was wondering if anyone has an "Executive Summary" of the DMARC issue in a general sense.

How about the following:

DMARC is a set of protocols for Internet mail that are used to by participating hosts to exchange information about mail traffic. This information is primarily useful for fighting email fraud, including spam and phishing. The basic procedure is that recipient hosts which participate in the protocol check whether the apparent sending domain (the domain that appears in the address in "From" header field) participates, using a query to the DNS defined by the DMARC protocol. If so, the recipient host will send to that domain some information about mail which purports to be from the sending host, but fails to validate.

The validation refers specifically to, and only to, the domain in the "From" header field. The DMARC protocol itself does not validate the mailbox (user), although some originating hosts may do so. Nor can it guarantee that the user's account hasn't been used fraudulently.

For participants in the DMARC protocol, valid mail provides a strong guarantee that the domain of the address in the "From" header field is authentic, with some strong restrictions required to ensure success of validation. These restrictions include:

(1) When authentic mail is delivered directly from the sending host to the recipient, validation will succeed.

(2) When authentic mail is forwarded verbatim by all intervening Internet hosts (including mailing lists and user mailbox forwarding), validation will succeed.

Authentic mail may fail to validate in many common circumstances, where a forwarding host alters certain headers or any part of the body of the message. This may occur for forwarding hosts which add advertisements to the footer or the like, or if a forwarding host reformats the body (it shouldn't do that, but some do). There are also restrictions on some parts of the mail. For example, the "From" header field must contain exactly one address (multiple author addresses are permitted by the mail standards and are occasionally useful, though rarely seen nowadays).

Failure to validate is also frequent on mailing lists. Common transformations that *will* cause mail distributed by a mailing list to fail validation include (but are not limited to)

• Adding a list tag or sequence number to the Subject header field.

• Adding a footer, typically containing unsubscribe information for public discussion lists or legal disclaimers for corporate lists.

• Removal of banned content, such as executable file attachments or excessively large attachments.

Because many common practices with Internet mail can cause authentic mail to fail DMARC validation, although mail which successfully validates may be trusted to be from the domain in the address in the "From" field, failure to validate in general *does not* imply lack of authenticity.

Under some circumstances, a sending domain may be willing to severely restrict its users' use of their addresses. For example, they may not be allowed to post to mailing lists, and recipients may be advised to *always* keep their addresses updated to the domain where they actually read their mail. This is common practice with banks and other financial institutions. Such organizations can be reasonably sure that mail which fails to validate is "fraudulent" (although the intent may be merely mischievious rather than truly felonious). They may take advantage of a further feature of the DMARC protocol, the "policy", which is the action that the sending domain suggests that recipients take with mail that appears to be from that domain but fails validation.

The normal policy is "none", that is, the recipient should deliver the mail as though the apparent sending domain did not participate in DMARC. (Of course, if the mail fails spam or virus checks, it would be quarantined or discarded as usual.) The second level is "quarantine", which means that the mail should not be delivered directly to the user's mailbox in the ordinary way, for fear of fraudulent use of the sending address (typically phishing). The third level is "reject", which means that the sender recommends that the mail not be delivered at all, to completely prevent fraudulent misuse of their domain.

Unfortunately, unless the sending domain has truly draconian policies about the use of addresses in that domain, use of "quarantine" or "reject" makes it virtually certain that some authentic mail will not be read in a timely fashion or (in the case of "reject") never delivered at all.

The "reject" policy may also result in denial of services to third parties (one common case is having delivery disabled from a mailing list, or even being unsubscribed), due to rejected mail being "bounced" back to the forwarding host. At present we know of *no* hosts which distinguish DMARC bounces from other delivery failures. Therefore they are treated as a permanent delivery failure to the recipient due to problems at the recipient's host, rather than due to a policy of the sender. As a consequence, those unrelated recipients may be dropped from distribution lists.

Editorial comments:

In some cases (where the mail validates according the DKIM, one of the protocols used in DMARC validation), the mail can be trusted not to have been altered in any way by a "man in the middle".

It is entirely up to the recipient whether to honor the published policy or not. (GMail, for example, does not -- it treats the policy as advisory and makes case by case decisions, allowing most list mail to be delivered despite failing DMARC validation.) However, because the original use of policy was envisioned to be banks vulnerable to phishing (not large freemail providers incapable of protecting their users' address books), most recipients do honor the policy published by sending domains.

The action of Yahoo! and AOL in publishing policies of "reject" basically amounts to deciding that they do not care if their users' mail can be reliably delivered. Nevertheless, Mailman is committed to providing mitigation options for mailing list owners that improve the reliability of mail delivery for posters from those domains, and protect other subscribers from the (unintentional) denial of service attacks being conducted by Yahoo! and AOL.

I believe the word "attack" is justified because Yahoo! and AOL are now aware that their policies have the effect of denying service to third parties (even if not intended by them), but they continue to implement them. It seems likely to me that as mailing lists and others implement workarounds that bypass DMARC validation, these workarounds will be mimicked by spam and phishing mails to bypass DMARC validation, and in response AOL and Yahoo! will take further action detrimental to the reliability of everyone's mail service, not just their own users'.

...

Of course if someone says that the current MS365 implementation has addressed this, then that's a different (unfortunate) story.

Seems unlikely to me (this isn't a new variant of a traditional problem to be addressed by database updates like virus checking), although they probably will roll out something within a couple months (counting from April 22). Nevertheless I doubt it will be as flexible as Mailman already is (and most likely will be tuned to protect Hotmail users if Hotmail should decide to use the "reject" policy).

...

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

How (sadly) apropos!

HTH

Steve

Is this also true?

Users from DMARC-reject domains send mail to mailing lists, and the resulting mail from the mailing list is rejected. Enough rejections can cause the mailing list possibly to be blacklisted for sending lots of "spam" mail.

Mailman treats the DMARC-reject messages as typical mail delivery failure bounces. After enough of them, Mailman disables the receiver's subscription (stops trying to send that member's e-mail address). Typically, the disabling threshold is less than the spam blacklisting threshold. Unless the list admin keeps 'blindly' reseting the bounce flag (or otherwise forces Mailman to keep delivering mail to problem addresses), the list is not likely to blacklisted. But in any case, yes, the domains which strickly enforce DMARC policys of 'reject' have effectively created a denial-of-service attack on the list. It is in essence a mis-use of the DMARC standard.

--Barry Finkel

---

Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/heller%40deepsoft.com

-- Robert Heller -- 978-544-6933 / heller@deepsoft.com Deepwoods Software -- http://www.deepsoft.com/ () ascii ribbon campaign -- against html e-mail /\ www.asciiribbon.org -- against proprietary attachments

Actually, From: domains can request reports even if DMARC p=none. It is unclear what might be done with these reports, but given what some domains have done with DMARC already, I for one would not be surprised if this information was used to color the reputation of the sending server.

Note that currently, Yahoo.com only requests aggregate reports which *I think* do not identify the sending server, but AOL.com requests failure reports as well which are intended to identify servers and actual senders.

I've been collecting DMARC aggregate reports for over two years and have over 40,000 of them. I use some scripts that decompress and parse the reports and put the interesting bits into a mysql database. I also have 22,000 failure reports (fewer providers send them.)

The aggregate reports do indeed identify the sending server and are pretty interesting. For some of the larger mail systems, it's clear from the tags in the reports that they have a pretty good idea where the mailing lists are, which makes me wonder why they don't use that info to whitelist around the DMARC damage. I don't see any evidence that DMARC failures alone are likely to get a server blacklisted, although I wouldn't be surprised if it were a factor along with user complaints and spam filter statistics.

R's, John

PS: The scripts are at http://www.taugh.com/rddmarc/ if you want to play along on your own system. You can (and should) collect DMARC stats without publishing any DMARC policies.

On May 14, 2014, at 11:47 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Peter Shute writes:

...

When MS365 forwards the mails sent to the distribution list, should that make the DMARC authentication fail? I thought that only happened if you made changes like adding a prefix to the subject line like Mailman does.

If it forwards verbatim *and* the sending domain signs the mail with DKIM (the common case), DMARC validation will succeed. Without DKIM, DMARC validation is guaranteed to fail. However, even in the sender uses DKIM, *any* change *whatsoever* to the body will cause validation to fail, and there are several changes to the header that could cause it to fail. Furthermore, which parts of the header are protected by the DKIM signature are determined by the sender, not by DMARC AFAIK.

If distribution lists are pure forwards, MS365 will be OK. But I find it hard to believe that that level of functionality is popular with users -- there's a reason why all popular MLMs implement subject prefixes, body headers and body footers, and it isn't "because it's the Microsoft way".

Especially as legally mailing lists are required to add unsubscribe instructions in the footer.

best regards, Larry

-- Larry Finch finches@portadmiral.org

On 05/14/14 23:47, Stephen J. Turnbull wrote:

Peter Shute writes:

...

When MS365 forwards the mails sent to the distribution list, should that make the DMARC authentication fail? I thought that only happened if you made changes like adding a prefix to the subject line like Mailman does.

If it forwards verbatim *and* the sending domain signs the mail with DKIM (the common case), DMARC validation will succeed. Without DKIM, DMARC validation is guaranteed to fail. However, even in the sender uses DKIM, *any* change *whatsoever* to the body will cause validation to fail, and there are several changes to the header that could cause it to fail. Furthermore, which parts of the header are protected by the DKIM signature are determined by the sender, not by DMARC AFAIK.

If distribution lists are pure forwards, MS365 will be OK. But I find it hard to believe that that level of functionality is popular with users -- there's a reason why all popular MLMs implement subject prefixes, body headers and body footers, and it isn't "because it's the Microsoft way".

I ran some tests this morning. I created an Exchange distribution list here and added myself five ways on the list:

1. On our Exchange server (how I receive internal emails)
2. On a local Linux server running sendmail and dovecot (how I receive "real mail")
3. A Yahoo address.
4. A Gmail address.
5. An iCloud address.

I then sent an email to the list and to my work sendmail address. It was delivered to both work addresses and the iCloud address.

Gmail put it in my Spam folder with the warning:

Be careful with this message. Our systems couldn't verify that this message was really sent by yahoo.com. You might want to avoid clicking links or replying with personal information.

There is also a link to their "Why messages are marked as Spam" page.

On Yahoo I found the bounce in my Spam folder with the following:

This is an automated message from the Extensible Content Security at host wg.ulticom.com.

The message returned below could not be delivered to its intended destinations.

For further assistance, please send mail to <XXXXXXXXXXX@wg.ulticom.com>.

If you do so, please include this problem report. You can delete your own text from the message returned below.

Reason: <XXXXXXXX@yahoo.com>: host mta7.am0.yahoodns.net[98.138.112.34] said: 554 5.7.9 Message not accepted for policy reasons. See http://postmaster.yahoo.com/errors/postmaster-28.html (in reply to end of DATA command)

The server wg.ulticom.com is our WatchGuard Anti-Spam appliance. It had no trouble accepting it when it came in the first time (it does not do DMARC checks), but when it tried to delivery the email to Yahoo, they rejected it. Of course, the reject went to Yahoo anyway, but with a MAILER-DAEMON sender address.

I saved the two copies from my sendmail address and compared them: % h=$(sed -n -e 'y/:/|/' -e '/DKIM-Signature|/s/.* h=\([^;]*\).*/\1/p' direct.eml) % diff -s -u <(egrep -i "^($h):" direct.eml) <(egrep -i "^($h)" list.eml) Files /dev/fd/4 and /dev/fd/5 are identical % diff -s -u <(sed '1,/^$/d' direct.eml) <(sed '1,/^$/d' list.eml) Files /dev/fd/4 and /dev/fd/5 are identical

The first diff compares only the headers in the DKIM Signature. The second diff compares the body. The DKIM checks seem to be good. So, it seems that nothing has changed in the content or checked header. It must be SPf.

% dig +short TXT _spf.mail.yahoo.com "v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:206.108.40.0/27 ip4:199.16.139.0/26 ?all"

It seems that in the case of a simple Exchange distribution list, the Yahoo members will fail (into their Spam folder!), some others may fail depending upon their service's SPF fussiness, and others may have to root around in their Spam folders for the content.

-- Gary Algier, WB2FWZ gaa@ulticom.com +1 856 787 2758 Ulticom Inc., 1020 Briggs Rd, Mt. Laurel, NJ 08054 Fax:+1 856 866 2033

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

On 05/15/14 11:15, Larry Finch wrote:

On May 15, 2014, at 10:53 AM, Gary Algier <gaa@ulticom.com> wrote:

...

On 05/14/14 23:47, Stephen J. Turnbull wrote:

I then sent an email to the list and to my work sendmail address. It was delivered to both work addresses and the iCloud address.

Gmail put it in my Spam folder with the warning:

Be careful with this message. Our systems couldn't verify that this message was really sent by yahoo.com. You might want to avoid clicking links or replying with personal information.

There is also a link to their "Why messages are marked as Spam" page.

On Yahoo I found the bounce in my Spam folder with the following:

This is an automated message from the Extensible Content Security at host wg.ulticom.com.

The message returned below could not be delivered to its intended destinations.

It seems that in the case of a simple Exchange distribution list, the Yahoo members will fail (into their Spam folder!), some others may fail depending upon their service's SPF fussiness, and others may have to root around in their Spam folders for the content.

On the lists that I manage on listserv I’ve discovered that many ISPs honor Yahoo and AOL’s p=reject, and will not even put the message in the spam folder. Among them are: Comcast, SBCGlobal, AT&T, AOL, Rogers, Earthlink, Hotmail and a few others I don’t recall. So essentially half of my list members will not get posts from Yahoo or AOL.

best regards, Larry

-- Larry Finch finches@portadmiral.org

Apparently, simple Exchange distribution lists do not rewrite headers or touch the body so DKIM passes. However, the distribution lists also do not change the envelope sender so the SPF fails. In order to get through DKIM, the internal author address ("From: ") and a bunch of other headers must stay the same, which Exchange does. Most mailing list software rewrites something, which makes DKIM fail. However, the mailing list software will use an envelope address from the list so SPF should not fail.

Summary: Can't use Exchange distribution lists: SPF will fail. Can't use mailing list software without changing the author, etc.: DKIM will fail.

Time for sendmail aliases? Or perhaps, SPF will fail?

-- Gary Algier, WB2FWZ gaa@ulticom.com +1 856 787 2758 Ulticom Inc., 1020 Briggs Rd, Mt. Laurel, NJ 08054 Fax:+1 856 866 2033

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

Gary Algier writes:

I ran some tests this morning. I created an Exchange distribution list here and added myself five ways on the list:

1. On our Exchange server (how I receive internal emails)
2. On a local Linux server running sendmail and dovecot (how I receive "real mail")
3. A Yahoo address.
4. A Gmail address.
5. An iCloud address.

I then sent an email to the list and to my work sendmail address.

Where did you send the mail from, what address was in "From", and what host did the DKIM signing? Does the domain listed in "From" have a DMARC record?

The DKIM checks seem to be good. So, it seems that nothing has changed in the content or checked header. It must be SPf.

It could be SPF, but if it is it has nothing to do with DMARC. DMARC accepts either SPF or DKIM as evidence of authenticity. That is, either may fail as long as at least one succeeds.

If it is indeed SPF, then it doesn't matter what you use. The problem is that the host where the distribution list or mailing list is hosted is not SPF-authorized, and almost certainly not which MTA or MLM you use.

I'm not sure if you care about DMARC, or just whether it gets through. But if the latter, I'm not at all clear on exactly what you're trying to test.

% dig +short TXT _spf.mail.yahoo.com "v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:206.108.40.0/27 ip4:199.16.139.0/26 ?all"

This is mostly unrelated to Yahoo's behavior when receiving messages.

Amusingly enough, RFC 7208 deprecates the "ptr" mechanism strongly.

Around a year ago we started using Exchange Online. In my experience, the undesired responses are correct.

On May 16, 2014, at 09:13 AM, mail.ulticom.com <gaa@ulticom.com> wrote:

My management wants to replace our in-house email with an Exchange Online (aka MS365) solution. Several high priced consultants have told them that Exchange can do it all. I am trying to prove otherwise.

Actually to be fare, two consulting firms have said:

1. Exchange can't do what Mailman can.

It has fancy forwards, but besides rudimentary moderation, very little in the way of traditional mailing list features.

2. MS365 can't work with Mailman.

I migrated my lists to a dedicated subdomain to work around this, and some of the details surrounding that are documented in the list archives.

3. MS365 will cost more, don't do it.

This may or may not be true. It’s a subscription service, so costs will change. It seems like these services are trending to go down in price as competition goes up and more users sign on, but not always. (We qualified for nonprofit pricing, which has gone from $2/user/mo to FREE, which is nice. However, we’ve been told that we no longer qualify for the nonprofit program.)

Feel free to contact me directly for more details.

--

Matthew Needham mneedham@hdfgroup.org 217-531-6110

The HDF Group 1800 South Oak Street, Suite 203 Champaign, IL 61820

On 5/14/2014 2:49 PM, Stephen J. Turnbull wrote:

Gary Algier writes:

...

I have been following the discussion of the DMARC issues and Mailman's attempts to live with it. I was wondering if anyone has an "Executive Summary" of the DMARC issue in a general sense.

How about the following:

DMARC is a set of protocols for Internet mail that are used to by participating hosts to exchange information about mail traffic. This information is primarily useful for fighting email fraud, including spam and phishing. The basic procedure is that recipient hosts which participate in the protocol check whether the apparent sending domain (the domain that appears in the address in "From" header field) participates, using a query to the DNS defined by the DMARC protocol. If so, the recipient host will send to that domain some information about mail which purports to be from the sending host, but fails to validate.

The validation refers specifically to, and only to, the domain in the "From" header field. The DMARC protocol itself does not validate the mailbox (user), although some originating hosts may do so. Nor can it guarantee that the user's account hasn't been used fraudulently.

For participants in the DMARC protocol, valid mail provides a strong guarantee that the domain of the address in the "From" header field is authentic, with some strong restrictions required to ensure success of validation. These restrictions include:

(1) When authentic mail is delivered directly from the sending host to the recipient, validation will succeed.

(2) When authentic mail is forwarded verbatim by all intervening Internet hosts (including mailing lists and user mailbox forwarding), validation will succeed.

Authentic mail may fail to validate in many common circumstances, where a forwarding host alters certain headers or any part of the body of the message. This may occur for forwarding hosts which add advertisements to the footer or the like, or if a forwarding host reformats the body (it shouldn't do that, but some do). There are also restrictions on some parts of the mail. For example, the "From" header field must contain exactly one address (multiple author addresses are permitted by the mail standards and are occasionally useful, though rarely seen nowadays).

Failure to validate is also frequent on mailing lists. Common transformations that *will* cause mail distributed by a mailing list to fail validation include (but are not limited to)

• Adding a list tag or sequence number to the Subject header field.

• Adding a footer, typically containing unsubscribe information for public discussion lists or legal disclaimers for corporate lists.

• Removal of banned content, such as executable file attachments or excessively large attachments.

Because many common practices with Internet mail can cause authentic mail to fail DMARC validation, although mail which successfully validates may be trusted to be from the domain in the address in the "From" field, failure to validate in general *does not* imply lack of authenticity.

Under some circumstances, a sending domain may be willing to severely restrict its users' use of their addresses. For example, they may not be allowed to post to mailing lists, and recipients may be advised to *always* keep their addresses updated to the domain where they actually read their mail. This is common practice with banks and other financial institutions. Such organizations can be reasonably sure that mail which fails to validate is "fraudulent" (although the intent may be merely mischievious rather than truly felonious). They may take advantage of a further feature of the DMARC protocol, the "policy", which is the action that the sending domain suggests that recipients take with mail that appears to be from that domain but fails validation.

The normal policy is "none", that is, the recipient should deliver the mail as though the apparent sending domain did not participate in DMARC. (Of course, if the mail fails spam or virus checks, it would be quarantined or discarded as usual.) The second level is "quarantine", which means that the mail should not be delivered directly to the user's mailbox in the ordinary way, for fear of fraudulent use of the sending address (typically phishing). The third level is "reject", which means that the sender recommends that the mail not be delivered at all, to completely prevent fraudulent misuse of their domain.

Unfortunately, unless the sending domain has truly draconian policies about the use of addresses in that domain, use of "quarantine" or "reject" makes it virtually certain that some authentic mail will not be read in a timely fashion or (in the case of "reject") never delivered at all.

The "reject" policy may also result in denial of services to third parties (one common case is having delivery disabled from a mailing list, or even being unsubscribed), due to rejected mail being "bounced" back to the forwarding host. At present we know of *no* hosts which distinguish DMARC bounces from other delivery failures. Therefore they are treated as a permanent delivery failure to the recipient due to problems at the recipient's host, rather than due to a policy of the sender. As a consequence, those unrelated recipients may be dropped from distribution lists.

Editorial comments:

In some cases (where the mail validates according the DKIM, one of the protocols used in DMARC validation), the mail can be trusted not to have been altered in any way by a "man in the middle".

It is entirely up to the recipient whether to honor the published policy or not. (GMail, for example, does not -- it treats the policy as advisory and makes case by case decisions, allowing most list mail to be delivered despite failing DMARC validation.) However, because the original use of policy was envisioned to be banks vulnerable to phishing (not large freemail providers incapable of protecting their users' address books), most recipients do honor the policy published by sending domains.

The action of Yahoo! and AOL in publishing policies of "reject" basically amounts to deciding that they do not care if their users' mail can be reliably delivered. Nevertheless, Mailman is committed to providing mitigation options for mailing list owners that improve the reliability of mail delivery for posters from those domains, and protect other subscribers from the (unintentional) denial of service attacks being conducted by Yahoo! and AOL.

I believe the word "attack" is justified because Yahoo! and AOL are now aware that their policies have the effect of denying service to third parties (even if not intended by them), but they continue to implement them. It seems likely to me that as mailing lists and others implement workarounds that bypass DMARC validation, these workarounds will be mimicked by spam and phishing mails to bypass DMARC validation, and in response AOL and Yahoo! will take further action detrimental to the reliability of everyone's mail service, not just their own users'.

...

Of course if someone says that the current MS365 implementation has addressed this, then that's a different (unfortunate) story.

Seems unlikely to me (this isn't a new variant of a traditional problem to be addressed by database updates like virus checking), although they probably will roll out something within a couple months (counting from April 22). Nevertheless I doubt it will be as flexible as Mailman already is (and most likely will be tuned to protect Hotmail users if Hotmail should decide to use the "reject" policy).

...

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

How (sadly) apropos!

HTH

Steve

Is this also true?

Users from DMARC-reject domains send mail to mailing lists, and the resulting mail from the mailing list is rejected. Enough rejections can cause the mailing list possibly to be blacklisted for sending lots of "spam" mail.

--Barry Finkel

At Wed, 14 May 2014 15:24:32 -0500 "Barry S. Finkel" <bsfinkel@att.net> wrote:

On 5/14/2014 2:49 PM, Stephen J. Turnbull wrote:

...

Gary Algier writes:

...

I have been following the discussion of the DMARC issues and Mailman's attempts to live with it. I was wondering if anyone has an "Executive Summary" of the DMARC issue in a general sense.

How about the following:

DMARC is a set of protocols for Internet mail that are used to by participating hosts to exchange information about mail traffic. This information is primarily useful for fighting email fraud, including spam and phishing. The basic procedure is that recipient hosts which participate in the protocol check whether the apparent sending domain (the domain that appears in the address in "From" header field) participates, using a query to the DNS defined by the DMARC protocol. If so, the recipient host will send to that domain some information about mail which purports to be from the sending host, but fails to validate.

The validation refers specifically to, and only to, the domain in the "From" header field. The DMARC protocol itself does not validate the mailbox (user), although some originating hosts may do so. Nor can it guarantee that the user's account hasn't been used fraudulently.

For participants in the DMARC protocol, valid mail provides a strong guarantee that the domain of the address in the "From" header field is authentic, with some strong restrictions required to ensure success of validation. These restrictions include:

(1) When authentic mail is delivered directly from the sending host to the recipient, validation will succeed.

(2) When authentic mail is forwarded verbatim by all intervening Internet hosts (including mailing lists and user mailbox forwarding), validation will succeed.

Authentic mail may fail to validate in many common circumstances, where a forwarding host alters certain headers or any part of the body of the message. This may occur for forwarding hosts which add advertisements to the footer or the like, or if a forwarding host reformats the body (it shouldn't do that, but some do). There are also restrictions on some parts of the mail. For example, the "From" header field must contain exactly one address (multiple author addresses are permitted by the mail standards and are occasionally useful, though rarely seen nowadays).

Failure to validate is also frequent on mailing lists. Common transformations that *will* cause mail distributed by a mailing list to fail validation include (but are not limited to)

• Adding a list tag or sequence number to the Subject header field.

• Adding a footer, typically containing unsubscribe information for public discussion lists or legal disclaimers for corporate lists.

• Removal of banned content, such as executable file attachments or excessively large attachments.

Because many common practices with Internet mail can cause authentic mail to fail DMARC validation, although mail which successfully validates may be trusted to be from the domain in the address in the "From" field, failure to validate in general *does not* imply lack of authenticity.

Under some circumstances, a sending domain may be willing to severely restrict its users' use of their addresses. For example, they may not be allowed to post to mailing lists, and recipients may be advised to *always* keep their addresses updated to the domain where they actually read their mail. This is common practice with banks and other financial institutions. Such organizations can be reasonably sure that mail which fails to validate is "fraudulent" (although the intent may be merely mischievious rather than truly felonious). They may take advantage of a further feature of the DMARC protocol, the "policy", which is the action that the sending domain suggests that recipients take with mail that appears to be from that domain but fails validation.

The normal policy is "none", that is, the recipient should deliver the mail as though the apparent sending domain did not participate in DMARC. (Of course, if the mail fails spam or virus checks, it would be quarantined or discarded as usual.) The second level is "quarantine", which means that the mail should not be delivered directly to the user's mailbox in the ordinary way, for fear of fraudulent use of the sending address (typically phishing). The third level is "reject", which means that the sender recommends that the mail not be delivered at all, to completely prevent fraudulent misuse of their domain.

Unfortunately, unless the sending domain has truly draconian policies about the use of addresses in that domain, use of "quarantine" or "reject" makes it virtually certain that some authentic mail will not be read in a timely fashion or (in the case of "reject") never delivered at all.

The "reject" policy may also result in denial of services to third parties (one common case is having delivery disabled from a mailing list, or even being unsubscribed), due to rejected mail being "bounced" back to the forwarding host. At present we know of *no* hosts which distinguish DMARC bounces from other delivery failures. Therefore they are treated as a permanent delivery failure to the recipient due to problems at the recipient's host, rather than due to a policy of the sender. As a consequence, those unrelated recipients may be dropped from distribution lists.

Editorial comments:

In some cases (where the mail validates according the DKIM, one of the protocols used in DMARC validation), the mail can be trusted not to have been altered in any way by a "man in the middle".

It is entirely up to the recipient whether to honor the published policy or not. (GMail, for example, does not -- it treats the policy as advisory and makes case by case decisions, allowing most list mail to be delivered despite failing DMARC validation.) However, because the original use of policy was envisioned to be banks vulnerable to phishing (not large freemail providers incapable of protecting their users' address books), most recipients do honor the policy published by sending domains.

The action of Yahoo! and AOL in publishing policies of "reject" basically amounts to deciding that they do not care if their users' mail can be reliably delivered. Nevertheless, Mailman is committed to providing mitigation options for mailing list owners that improve the reliability of mail delivery for posters from those domains, and protect other subscribers from the (unintentional) denial of service attacks being conducted by Yahoo! and AOL.

I believe the word "attack" is justified because Yahoo! and AOL are now aware that their policies have the effect of denying service to third parties (even if not intended by them), but they continue to implement them. It seems likely to me that as mailing lists and others implement workarounds that bypass DMARC validation, these workarounds will be mimicked by spam and phishing mails to bypass DMARC validation, and in response AOL and Yahoo! will take further action detrimental to the reliability of everyone's mail service, not just their own users'.

...

Of course if someone says that the current MS365 implementation has addressed this, then that's a different (unfortunate) story.

Seems unlikely to me (this isn't a new variant of a traditional problem to be addressed by database updates like virus checking), although they probably will roll out something within a couple months (counting from April 22). Nevertheless I doubt it will be as flexible as Mailman already is (and most likely will be tuned to protect Hotmail users if Hotmail should decide to use the "reject" policy).

...

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

How (sadly) apropos!

HTH

Steve

Is this also true?

Users from DMARC-reject domains send mail to mailing lists, and the resulting mail from the mailing list is rejected. Enough rejections can cause the mailing list possibly to be blacklisted for sending lots of "spam" mail.

Mailman treats the DMARC-reject messages as typical mail delivery failure bounces. After enough of them, Mailman disables the receiver's subscription (stops trying to send that member's e-mail address). Typically, the disabling threshold is less than the spam blacklisting threshold. Unless the list admin keeps 'blindly' reseting the bounce flag (or otherwise forces Mailman to keep delivering mail to problem addresses), the list is not likely to blacklisted. But in any case, yes, the domains which strickly enforce DMARC policys of 'reject' have effectively created a denial-of-service attack on the list. It is in essence a mis-use of the DMARC standard.

--Barry Finkel

---

Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/heller%40deepsoft.com

-- Robert Heller -- 978-544-6933 / heller@deepsoft.com Deepwoods Software -- http://www.deepsoft.com/ () ascii ribbon campaign -- against html e-mail /\ www.asciiribbon.org -- against proprietary attachments

Actually, From: domains can request reports even if DMARC p=none. It is unclear what might be done with these reports, but given what some domains have done with DMARC already, I for one would not be surprised if this information was used to color the reputation of the sending server.

Note that currently, Yahoo.com only requests aggregate reports which *I think* do not identify the sending server, but AOL.com requests failure reports as well which are intended to identify servers and actual senders.

I've been collecting DMARC aggregate reports for over two years and have over 40,000 of them. I use some scripts that decompress and parse the reports and put the interesting bits into a mysql database. I also have 22,000 failure reports (fewer providers send them.)

The aggregate reports do indeed identify the sending server and are pretty interesting. For some of the larger mail systems, it's clear from the tags in the reports that they have a pretty good idea where the mailing lists are, which makes me wonder why they don't use that info to whitelist around the DMARC damage. I don't see any evidence that DMARC failures alone are likely to get a server blacklisted, although I wouldn't be surprised if it were a factor along with user complaints and spam filter statistics.

R's, John

PS: The scripts are at http://www.taugh.com/rddmarc/ if you want to play along on your own system. You can (and should) collect DMARC stats without publishing any DMARC policies.

On May 14, 2014, at 11:47 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Peter Shute writes:

...

When MS365 forwards the mails sent to the distribution list, should that make the DMARC authentication fail? I thought that only happened if you made changes like adding a prefix to the subject line like Mailman does.

If it forwards verbatim *and* the sending domain signs the mail with DKIM (the common case), DMARC validation will succeed. Without DKIM, DMARC validation is guaranteed to fail. However, even in the sender uses DKIM, *any* change *whatsoever* to the body will cause validation to fail, and there are several changes to the header that could cause it to fail. Furthermore, which parts of the header are protected by the DKIM signature are determined by the sender, not by DMARC AFAIK.

If distribution lists are pure forwards, MS365 will be OK. But I find it hard to believe that that level of functionality is popular with users -- there's a reason why all popular MLMs implement subject prefixes, body headers and body footers, and it isn't "because it's the Microsoft way".

Especially as legally mailing lists are required to add unsubscribe instructions in the footer.

best regards, Larry

-- Larry Finch finches@portadmiral.org

On 05/14/14 23:47, Stephen J. Turnbull wrote:

Peter Shute writes:

...

When MS365 forwards the mails sent to the distribution list, should that make the DMARC authentication fail? I thought that only happened if you made changes like adding a prefix to the subject line like Mailman does.

If it forwards verbatim *and* the sending domain signs the mail with DKIM (the common case), DMARC validation will succeed. Without DKIM, DMARC validation is guaranteed to fail. However, even in the sender uses DKIM, *any* change *whatsoever* to the body will cause validation to fail, and there are several changes to the header that could cause it to fail. Furthermore, which parts of the header are protected by the DKIM signature are determined by the sender, not by DMARC AFAIK.

If distribution lists are pure forwards, MS365 will be OK. But I find it hard to believe that that level of functionality is popular with users -- there's a reason why all popular MLMs implement subject prefixes, body headers and body footers, and it isn't "because it's the Microsoft way".

I ran some tests this morning. I created an Exchange distribution list here and added myself five ways on the list:

1. On our Exchange server (how I receive internal emails)
2. On a local Linux server running sendmail and dovecot (how I receive "real mail")
3. A Yahoo address.
4. A Gmail address.
5. An iCloud address.

I then sent an email to the list and to my work sendmail address. It was delivered to both work addresses and the iCloud address.

Gmail put it in my Spam folder with the warning:

Be careful with this message. Our systems couldn't verify that this message was really sent by yahoo.com. You might want to avoid clicking links or replying with personal information.

There is also a link to their "Why messages are marked as Spam" page.

On Yahoo I found the bounce in my Spam folder with the following:

This is an automated message from the Extensible Content Security at host wg.ulticom.com.

The message returned below could not be delivered to its intended destinations.

For further assistance, please send mail to <XXXXXXXXXXX@wg.ulticom.com>.

If you do so, please include this problem report. You can delete your own text from the message returned below.

Reason: <XXXXXXXX@yahoo.com>: host mta7.am0.yahoodns.net[98.138.112.34] said: 554 5.7.9 Message not accepted for policy reasons. See http://postmaster.yahoo.com/errors/postmaster-28.html (in reply to end of DATA command)

The server wg.ulticom.com is our WatchGuard Anti-Spam appliance. It had no trouble accepting it when it came in the first time (it does not do DMARC checks), but when it tried to delivery the email to Yahoo, they rejected it. Of course, the reject went to Yahoo anyway, but with a MAILER-DAEMON sender address.

I saved the two copies from my sendmail address and compared them: % h=$(sed -n -e 'y/:/|/' -e '/DKIM-Signature|/s/.* h=\([^;]*\).*/\1/p' direct.eml) % diff -s -u <(egrep -i "^($h):" direct.eml) <(egrep -i "^($h)" list.eml) Files /dev/fd/4 and /dev/fd/5 are identical % diff -s -u <(sed '1,/^$/d' direct.eml) <(sed '1,/^$/d' list.eml) Files /dev/fd/4 and /dev/fd/5 are identical

The first diff compares only the headers in the DKIM Signature. The second diff compares the body. The DKIM checks seem to be good. So, it seems that nothing has changed in the content or checked header. It must be SPf.

% dig +short TXT _spf.mail.yahoo.com "v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:206.108.40.0/27 ip4:199.16.139.0/26 ?all"

It seems that in the case of a simple Exchange distribution list, the Yahoo members will fail (into their Spam folder!), some others may fail depending upon their service's SPF fussiness, and others may have to root around in their Spam folders for the content.

-- Gary Algier, WB2FWZ gaa@ulticom.com +1 856 787 2758 Ulticom Inc., 1020 Briggs Rd, Mt. Laurel, NJ 08054 Fax:+1 856 866 2033

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

On 05/15/14 11:15, Larry Finch wrote:

On May 15, 2014, at 10:53 AM, Gary Algier <gaa@ulticom.com> wrote:

...

On 05/14/14 23:47, Stephen J. Turnbull wrote:

I then sent an email to the list and to my work sendmail address. It was delivered to both work addresses and the iCloud address.

Gmail put it in my Spam folder with the warning:

Be careful with this message. Our systems couldn't verify that this message was really sent by yahoo.com. You might want to avoid clicking links or replying with personal information.

There is also a link to their "Why messages are marked as Spam" page.

On Yahoo I found the bounce in my Spam folder with the following:

This is an automated message from the Extensible Content Security at host wg.ulticom.com.

The message returned below could not be delivered to its intended destinations.

It seems that in the case of a simple Exchange distribution list, the Yahoo members will fail (into their Spam folder!), some others may fail depending upon their service's SPF fussiness, and others may have to root around in their Spam folders for the content.

On the lists that I manage on listserv I’ve discovered that many ISPs honor Yahoo and AOL’s p=reject, and will not even put the message in the spam folder. Among them are: Comcast, SBCGlobal, AT&T, AOL, Rogers, Earthlink, Hotmail and a few others I don’t recall. So essentially half of my list members will not get posts from Yahoo or AOL.

best regards, Larry

-- Larry Finch finches@portadmiral.org

Apparently, simple Exchange distribution lists do not rewrite headers or touch the body so DKIM passes. However, the distribution lists also do not change the envelope sender so the SPF fails. In order to get through DKIM, the internal author address ("From: ") and a bunch of other headers must stay the same, which Exchange does. Most mailing list software rewrites something, which makes DKIM fail. However, the mailing list software will use an envelope address from the list so SPF should not fail.

Summary: Can't use Exchange distribution lists: SPF will fail. Can't use mailing list software without changing the author, etc.: DKIM will fail.

Time for sendmail aliases? Or perhaps, SPF will fail?

-- Gary Algier, WB2FWZ gaa@ulticom.com +1 856 787 2758 Ulticom Inc., 1020 Briggs Rd, Mt. Laurel, NJ 08054 Fax:+1 856 866 2033

Nielsen's First Law of Computer Manuals: People don't read documentation voluntarily.

Gary Algier writes:

I ran some tests this morning. I created an Exchange distribution list here and added myself five ways on the list:

1. On our Exchange server (how I receive internal emails)
2. On a local Linux server running sendmail and dovecot (how I receive "real mail")
3. A Yahoo address.
4. A Gmail address.
5. An iCloud address.

I then sent an email to the list and to my work sendmail address.

Where did you send the mail from, what address was in "From", and what host did the DKIM signing? Does the domain listed in "From" have a DMARC record?

The DKIM checks seem to be good. So, it seems that nothing has changed in the content or checked header. It must be SPf.

It could be SPF, but if it is it has nothing to do with DMARC. DMARC accepts either SPF or DKIM as evidence of authenticity. That is, either may fail as long as at least one succeeds.

If it is indeed SPF, then it doesn't matter what you use. The problem is that the host where the distribution list or mailing list is hosted is not SPF-authorized, and almost certainly not which MTA or MLM you use.

I'm not sure if you care about DMARC, or just whether it gets through. But if the latter, I'm not at all clear on exactly what you're trying to test.

% dig +short TXT _spf.mail.yahoo.com "v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:206.108.40.0/27 ip4:199.16.139.0/26 ?all"

This is mostly unrelated to Yahoo's behavior when receiving messages.

Amusingly enough, RFC 7208 deprecates the "ptr" mechanism strongly.

Around a year ago we started using Exchange Online. In my experience, the undesired responses are correct.

On May 16, 2014, at 09:13 AM, mail.ulticom.com <gaa@ulticom.com> wrote:

My management wants to replace our in-house email with an Exchange Online (aka MS365) solution. Several high priced consultants have told them that Exchange can do it all. I am trying to prove otherwise.

Actually to be fare, two consulting firms have said:

1. Exchange can't do what Mailman can.

It has fancy forwards, but besides rudimentary moderation, very little in the way of traditional mailing list features.

2. MS365 can't work with Mailman.

I migrated my lists to a dedicated subdomain to work around this, and some of the details surrounding that are documented in the list archives.

3. MS365 will cost more, don't do it.

This may or may not be true. It’s a subscription service, so costs will change. It seems like these services are trending to go down in price as competition goes up and more users sign on, but not always. (We qualified for nonprofit pricing, which has gone from $2/user/mo to FREE, which is nice. However, we’ve been told that we no longer qualify for the nonprofit program.)

Feel free to contact me directly for more details.

--

Matthew Needham mneedham@hdfgroup.org 217-531-6110

The HDF Group 1800 South Oak Street, Suite 203 Champaign, IL 61820