On 05/23/2015 02:45 PM, Allan Hansen wrote:

So against my better judgement, I included this hack in Cleanse.py;

22c22 < from email.Utils import formataddr, parseaddr —

...

from email.Utils import formataddr

69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid' < del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of filez-

It would help me understand what you did if you would post

diff -u Cleanse.py.original Cleanse.py.new

So that your patch is not reversed and I can see context, but it looks like you replaced

del msg['x-pmrqc']

at the end of Cleanse.py with your added code. If that is what you did, why delete the

del msg['x-pmrqc']

line?

In any case, I will refrain from discussing the merits of adding .invalid to the domain, but why do it for all domains and not just yahoo.com and aol.com or actually look up the From: domain's DMARC policy and only do it for domains with DMARC p=reject.

I don’t get compile errors, but Cleanse.pyc is not being updated. I have stopped and restarted Mailman and I have also rebooted, but same non-action. I have not tried ‘compileall’ and am not eager to, either (permissions, where to invoke, etc). Any suggestions?

Does your hack do what you expect?

Cleanse.pyc will not be updated until Mailman processes at least one post and then only if IncomingRunner can write to Cleanse.pyc. It may be that Cleanse.pyc is in Mailman's group but not owned by the Mailman user and is not group writable.

None of this matters. when Python imports the module, it will notice that Cleanse.py is newer than Cleanse.pyc and will load and compile Cleanse.py. Then it will write the byte-compiled result to Cleanse.pyc if it can, but in any case, it will be using the code in Cleanse.py.

If there is a problem with permissions, you can compile Cleanse.py with compileall as a user that can write the file, but it won't change the results.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Stephen,

Much appreciated. Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.

If I do this and add the bit about the Reply-To, what would the code look like?

Yours,

Allan

On May 24, 2015, at 6:10 , Stephen J. Turnbull <stephen@xemacs.org> wrote:

Allan Hansen writes:

...

69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid'

This is known to be a bad idea, as it increases the spam score at many sites (because the author's mail domain doesn't resolve). Subscribers at such sites may have trouble receiving mail, and your list(s) may be tagged as suspicious.

I would recommend the From-munging approach:

name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from'] msg['from'] = formataddr((name, addr))

Mark (or you) probably have better code, and in some cases you may want to add the addr to the Reply-To field.

...

< del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of file

Hi Stephen,

Yes, there is a good reason. I’m using Mailman as it came with the OS X Server and am not prepared to replace it. Also, Mailman no longer comes pre-installed on the Apple platform, so I’m basically stuck. This is why I tried the simplest hack I could find. I have 44 busy lists and I’m weary of messing anything up, as I have basically no time or background to fix it.

Yours,

Allan

On May 24, 2015, at 11:06 , Andrew Hodgson <andrew@hodgsonfamily.org> wrote:

Allan Hansen wrote:

...

Stephen,

...

Much appreciated. Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.

Probably a good reason why you can't do this, but is there any way you can upgrade to the latest 2.1.20? It means the code for doing this is already there for you and will work by looking up the relevant domain's DMARC policy in DNS. I use it on all the lists here by default now by munging the From: header and it works when it needs to.

Thanks. Andrew.

On 05/24/2015 09:02 AM, Allan Hansen wrote:

Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.

You only have to mung those addresses for which the domain publishes a DMARC p=reject policy. This almost certainly does not include aol and yahoo provided 'other' domains.

The code in Mailman 2.1.18+ actually does a DNS lookup of the DMARC policy. In all the myriad @python.org list posts, the only domains I see with p=reject are aol.com, yahoo.com and paypal.com.

If I do this and add the bit about the Reply-To, what would the code look like?

I'm currently between flights in an airport. When I get a chance, I'll post some code.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 05/24/2015 03:19 PM, Allan Hansen wrote:

$ host -t TXT _dmarc.btopenworld.com _dmarc.btopenworld.com descriptive text "v=DMARC1\; p=none\; fo=1\; rua=mailto:dmarcagg@btinternet.com, mailto:dmarc_agg@auth.returnpath.net\;"

The domain publishes DMARC p=none. Thus, no ISP should treat a message From: someone@btopenworld.com any differently than the same message From: someone@elsewhere.com.

Here is the reject notice:

Final-Recipient: rfc822; subscriber@aol.com Original-Recipient: rfc822;subscriber@aol.com Action: failed Status: 5.2.1 Remote-MTA: dns; mailin-04.mx.aol.com Diagnostic-Code: smtp; 521 5.2.1 : AOL will not accept delivery of this message.

I see this exact rejection reliably from AOL. When an AOL user posts to a list, the list post sent back to that user is rejected in this way, even though AOL accepts the same post for delivery to other AOL users.

I have experimented with this using my own AOL address to send and reflecting various versions of the message back. I munged a lot of headers including I think Message-Id:, and I always got rejected. I gave up trying to figure out what AOL is looking at, but this reject occurs to list posts from aol.com, even though the From: is munged to the list address.

In any case, that's not the reject reason uses for a reject due to DMARC policy.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Hi Stephen,

You’re right. AOL does not accept these messages with ‘invalid’ at the end.

You’re recommending this:

name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from’] msg['from'] = formataddr((name, addrs))

Can I copy this code directly into the file? Is <list-post address> valid syntax? (I have 40+ lists)

Yours,

Allan

On May 24, 2015, at 6:10 , Stephen J. Turnbull <stephen@xemacs.org> wrote:

Allan Hansen writes:

...

69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid'

This is known to be a bad idea, as it increases the spam score at many sites (because the author's mail domain doesn't resolve). Subscribers at such sites may have trouble receiving mail, and your list(s) may be tagged as suspicious.

I would recommend the From-munging approach:

name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from'] msg['from'] = formataddr((name, addr))

Mark (or you) probably have better code, and in some cases you may want to add the addr to the Reply-To field.

...

< del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of file

On 23/05/15 22:45, Allan Hansen wrote:

I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy.

Me too. Sadly, Yahoo has recently (28 March) compounded their mess, probably necessitating an update to workarounds on some Mailman installations. Initially they said the policy would just involve "low-volume Yahoo international domains" http://comments.gmane.org/gmane.mail.spam.dmarc/2411 but when the deadline came it also included yahoo.co.uk, yahoo.fr and all Yahoo user domains I know of: http://comments.gmane.org/gmane.mail.spam.dmarc/2414

Background for anyone who doesn't know it: https://mail.python.org/pipermail/mailman-users/2014-April/ http://wiki.list.org/x/17891458 http://wiki.list.org/DEV/DMARC

[snippets] On 24/05/15 00:39, Mark Sapiro wrote:

In any case, I will refrain from discussing the merits of adding .invalid to the domain, but why do it for all domains and not just yahoo.com and aol.com or actually look up the From: domain's DMARC policy and only do it for domains with DMARC p=reject.

Some workarounds may look up _dmarc TXT record, others may maintain static lists of affected domains, some may choose to break RFC 5322 consistently because of some ISPs wrongly using p=reject for user email that is sent to discussion lists. In the case of static lists, these may need to be extended to include the above Yahoo domains.

On 21/08/15 19:26, Stephen J. Turnbull wrote:

DMARC p=reject gives list admins an unpleasant choice: (1) violate the mail standards and suffer various degradations of service because others in the mail system assume conformance (eg, your "wrong duplicate" problem), (2) tell your p=reject users that their posts are going to be rejected or discarded by many subscribers, or (3) stop decorating posts with [List] tags or material prefixed and affixed to the message body (so that the originator's DKIM signature will remain valid and the DMARC checks will pass).

N.B. The tech staff from Yahoo! and AOL have acknowledged (on the ietf-dmarc mailing list) that their employers are knowingly breaking mailing lists (and other services) to address their security fiascos. The designers of DMARC have always maintained that the Yahoo!/AOL use case is abusive -- DMARC was designed to protect official mail to customers sent on behalf of corporations by their employees, not the general use mail of users with addresses at freemail providers. In other words, mailing lists just shouldn't receive mail from p=reject domains, ever. No problem -- until Yahoo! and AOL decided to *create* one.

IMO, given those facts, posting from a Yahoo! or AOL address is just plain rude. (I can and do get away with banning their posts. I wish everybody could do that.)

Yes, someone really should explain to Marissa Mayer that every new anti-forgery acronym isn't appropriate or useful for user freemail and it's making Yahoo look incompetent and/or antisocial.

CK

Cedric Knight writes:

On 23/05/15 22:45, Allan Hansen wrote:

...

I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy.

Me too.

For some good news: people working with DMARC have come up with a protocol that may help lists with good reputations, and Mailman will implement it this summer.

Now the bad news: they're not going to revert to p=none. From management's point of view, p=reject is a rather good solution to a nasty problem. The massive leaks of address books that made "referral from a friend spam" possible means they're committed to this indefinitely, unless they do away with their traditional email addresses (ie, @aol.com and @yahoo.com). But that could cost them hundreds of millions of users.

It's certainly true that Yahoo! admins have stated that their little April Fool's joke didn't cost them any users to speak of, which is all that management really worried about, in view of the huge costs (both technical -- a spike in mail flows to Yahoo! of 6X the normal level -- and reputational -- the huge amount of directed spam that was being sent to correspondents of Yahoo users everywhere) involved in doing nothing.

A year and a bit later Ms. Zwicky (who arguably is doing her best for both Yahoo! users and Yahoo!'s bottom line, if lacking a little on the corporate social responsibility side) said that they were still getting probes that indicated that the spammers were ready to restart their "campaigns" if p=reject were ever relaxed. So they aren't going to do that.

Sadly, Yahoo has recently (28 March) compounded their mess,

I guess their take on the current situation, two years later, is that any indirect mailflows that they haven't already killed outright are prepared to deal with this extension.

Anyway, I would say that any large email provider that keeps user "friend" data on their servers (rather than on the user's machine) needs to be prepared to publish p=reject in the event they get cracked. You don't have to like the situation, but don't waste neurons hoping it will go away.

Steve