Hi,
I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy. My AOL and Yahoo subscribers are pretty upset at me because I won’t let them post. A number now have two subscriptions, one for posting (from GMail) and another for receiving the messages.
So against my better judgement, I included this hack in Cleanse.py;
22c22 < from email.Utils import formataddr, parseaddr —
from email.Utils import formataddr
69,74d68
<
< # Added to deal with DMARC issuej
< name, addrs = parseaddr(msg.get('from'))
< addrs += '.invalid'
< del msg['from']
< msg['from'] = formataddr((name, addrs))
\ No newline at end of file
I found it in the discussion list.
I don’t get compile errors, but Cleanse.pyc is not being updated. I have stopped and restarted Mailman and I have also rebooted, but same non-action. I have not tried ‘compileall’ and am not eager to, either (permissions, where to invoke, etc). Any suggestions?
The host OS is Mac OS X Server 10.5.8 with Mailman 2.1.14
Yours,
Allan
On 05/23/2015 02:45 PM, Allan Hansen wrote:
So against my better judgement, I included this hack in Cleanse.py;
22c22 < from email.Utils import formataddr, parseaddr —
from email.Utils import formataddr
69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid' < del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of filez-
It would help me understand what you did if you would post
diff -u Cleanse.py.original Cleanse.py.new
So that your patch is not reversed and I can see context, but it looks like you replaced
del msg['x-pmrqc']at the end of Cleanse.py with your added code. If that is what you did, why delete the
del msg['x-pmrqc']line?
In any case, I will refrain from discussing the merits of adding .invalid to the domain, but why do it for all domains and not just yahoo.com and aol.com or actually look up the From: domain's DMARC policy and only do it for domains with DMARC p=reject.
I don’t get compile errors, but Cleanse.pyc is not being updated. I have stopped and restarted Mailman and I have also rebooted, but same non-action. I have not tried ‘compileall’ and am not eager to, either (permissions, where to invoke, etc). Any suggestions?
Does your hack do what you expect?
Cleanse.pyc will not be updated until Mailman processes at least one post and then only if IncomingRunner can write to Cleanse.pyc. It may be that Cleanse.pyc is in Mailman's group but not owned by the Mailman user and is not group writable.
None of this matters. when Python imports the module, it will notice that Cleanse.py is newer than Cleanse.pyc and will load and compile Cleanse.py. Then it will write the byte-compiled result to Cleanse.pyc if it can, but in any case, it will be using the code in Cleanse.py.
If there is a problem with permissions, you can compile Cleanse.py with compileall as a user that can write the file, but it won't change the results.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 23/05/15 22:45, Allan Hansen wrote:
I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy.
Me too. Sadly, Yahoo has recently (28 March) compounded their mess, probably necessitating an update to workarounds on some Mailman installations. Initially they said the policy would just involve "low-volume Yahoo international domains" http://comments.gmane.org/gmane.mail.spam.dmarc/2411 but when the deadline came it also included yahoo.co.uk, yahoo.fr and all Yahoo user domains I know of: http://comments.gmane.org/gmane.mail.spam.dmarc/2414
Background for anyone who doesn't know it: https://mail.python.org/pipermail/mailman-users/2014-April/ http://wiki.list.org/x/17891458 http://wiki.list.org/DEV/DMARC
[snippets] On 24/05/15 00:39, Mark Sapiro wrote:
In any case, I will refrain from discussing the merits of adding .invalid to the domain, but why do it for all domains and not just yahoo.com and aol.com or actually look up the From: domain's DMARC policy and only do it for domains with DMARC p=reject.
Some workarounds may look up _dmarc TXT record, others may maintain static lists of affected domains, some may choose to break RFC 5322 consistently because of some ISPs wrongly using p=reject for user email that is sent to discussion lists. In the case of static lists, these may need to be extended to include the above Yahoo domains.
On 21/08/15 19:26, Stephen J. Turnbull wrote:
DMARC p=reject gives list admins an unpleasant choice: (1) violate the mail standards and suffer various degradations of service because others in the mail system assume conformance (eg, your "wrong duplicate" problem), (2) tell your p=reject users that their posts are going to be rejected or discarded by many subscribers, or (3) stop decorating posts with [List] tags or material prefixed and affixed to the message body (so that the originator's DKIM signature will remain valid and the DMARC checks will pass).
N.B. The tech staff from Yahoo! and AOL have acknowledged (on the ietf-dmarc mailing list) that their employers are knowingly breaking mailing lists (and other services) to address their security fiascos. The designers of DMARC have always maintained that the Yahoo!/AOL use case is abusive -- DMARC was designed to protect official mail to customers sent on behalf of corporations by their employees, not the general use mail of users with addresses at freemail providers. In other words, mailing lists just shouldn't receive mail from p=reject domains, ever. No problem -- until Yahoo! and AOL decided to *create* one.
IMO, given those facts, posting from a Yahoo! or AOL address is just plain rude. (I can and do get away with banning their posts. I wish everybody could do that.)
Yes, someone really should explain to Marissa Mayer that every new anti-forgery acronym isn't appropriate or useful for user freemail and it's making Yahoo look incompetent and/or antisocial.
CK
On 04/09/2016 11:56 AM, Cedric Knight wrote:
On 23/05/15 22:45, Allan Hansen wrote:
I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy.
Me too. Sadly, Yahoo has recently (28 March) compounded their mess, probably necessitating an update to workarounds on some Mailman installations.
This should have no effect on Mailman 2.1.18+ installations since the deprecated 'from_is_list' mitigations apply to all list mail and the recommended 'dmarc_moderation_action' mitigations already check the From: domain's DMARC policy.
There is one bug at <https://bugs.launchpad.net/mailman/+bug/1549420> that is fixed in 2.1.21. Prior to that, if a message was From: a subdomain of an 'organizational domain' and the subdomain didn't publish a DMARC policy, the organizational domain wasn't checked for dmarc_moderation_action. Beginning in 2.1.21 it is checked.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Cedric Knight writes:
On 23/05/15 22:45, Allan Hansen wrote:
I have waited almost a year for AOL and Yahoo to admit that they messed up and to remove their DMARC policy.
Me too.
For some good news: people working with DMARC have come up with a protocol that may help lists with good reputations, and Mailman will implement it this summer.
Now the bad news: they're not going to revert to p=none. From management's point of view, p=reject is a rather good solution to a nasty problem. The massive leaks of address books that made "referral from a friend spam" possible means they're committed to this indefinitely, unless they do away with their traditional email addresses (ie, @aol.com and @yahoo.com). But that could cost them hundreds of millions of users.
It's certainly true that Yahoo! admins have stated that their little April Fool's joke didn't cost them any users to speak of, which is all that management really worried about, in view of the huge costs (both technical -- a spike in mail flows to Yahoo! of 6X the normal level -- and reputational -- the huge amount of directed spam that was being sent to correspondents of Yahoo users everywhere) involved in doing nothing.
A year and a bit later Ms. Zwicky (who arguably is doing her best for both Yahoo! users and Yahoo!'s bottom line, if lacking a little on the corporate social responsibility side) said that they were still getting probes that indicated that the spammers were ready to restart their "campaigns" if p=reject were ever relaxed. So they aren't going to do that.
Sadly, Yahoo has recently (28 March) compounded their mess,
I guess their take on the current situation, two years later, is that any indirect mailflows that they haven't already killed outright are prepared to deal with this extension.
Anyway, I would say that any large email provider that keeps user "friend" data on their servers (rather than on the user's machine) needs to be prepared to publish p=reject in the event they get cracked. You don't have to like the situation, but don't waste neurons hoping it will go away.
Steve
Allan Hansen writes:
69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid'
This is known to be a bad idea, as it increases the spam score at many sites (because the author's mail domain doesn't resolve). Subscribers at such sites may have trouble receiving mail, and your list(s) may be tagged as suspicious.
I would recommend the From-munging approach:
name, addr = parseadder(msg.get('from'))
if addr.endswith('aol.com') or addr.endswith('yahoo.com'):
# I forget what happens if it's a bare address
name = "%s (%s) via list" % (name if name else "Anonymous", addr)
addr = <list-post address>
del msg['from']
msg['from'] = formataddr((name, addr))Mark (or you) probably have better code, and in some cases you may want to add the addr to the Reply-To field.
< del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of file
Stephen,
Much appreciated. Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.
If I do this and add the bit about the Reply-To, what would the code look like?
Yours,
AllanOn May 24, 2015, at 6:10 , Stephen J. Turnbull <stephen@xemacs.org> wrote:
Allan Hansen writes:
69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid'This is known to be a bad idea, as it increases the spam score at many sites (because the author's mail domain doesn't resolve). Subscribers at such sites may have trouble receiving mail, and your list(s) may be tagged as suspicious.
I would recommend the From-munging approach:
name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from'] msg['from'] = formataddr((name, addr))
Mark (or you) probably have better code, and in some cases you may want to add the addr to the Reply-To field.
< del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of file
Allan Hansen wrote:
Stephen,
Much appreciated. Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.
Probably a good reason why you can't do this, but is there any way you can upgrade to the latest 2.1.20? It means the code for doing this is already there for you and will work by looking up the relevant domain's DMARC policy in DNS. I use it on all the lists here by default now by munging the From: header and it works when it needs to.
Thanks. Andrew.
Hi Stephen,
Yes, there is a good reason. I’m using Mailman as it came with the OS X Server and am not prepared to replace it. Also, Mailman no longer comes pre-installed on the Apple platform, so I’m basically stuck. This is why I tried the simplest hack I could find. I have 44 busy lists and I’m weary of messing anything up, as I have basically no time or background to fix it.
Yours,
AllanOn May 24, 2015, at 11:06 , Andrew Hodgson <andrew@hodgsonfamily.org> wrote:
Allan Hansen wrote:
Stephen,
Much appreciated. Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.
Probably a good reason why you can't do this, but is there any way you can upgrade to the latest 2.1.20? It means the code for doing this is already there for you and will work by looking up the relevant domain's DMARC policy in DNS. I use it on all the lists here by default now by munging the From: header and it works when it needs to.
Thanks. Andrew.
Allan Hansen writes:
Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form.
Oddly enough, it turns out that they only use DMARC p=reject at their principal domain (aol.com and yahoo.com). You can check for any given domain by prepending _dmarc. and checking the TXT record. For example, for aol.com it would be "host -t TXT _dmarc.aol.com" if you have the host utility for doing DNS lookups.
I would have to do this for all addresses, to be safe.
If you're worried about safety and care about conforming to standards, you really should upgrade to at least Mailman 2.1.18-1. That allows you to be nonconformant only for authors whose addresses are in troublesome domains, and handles the reply-to issue as well as possible (making everybody happy isn't quite possible). I'm sure you have good reason for not doing so *right* *now*, but keep it in mind.
If I do this and add the bit about the Reply-To, what would the code look like?
If you do it for all mail, you just delete the "if" line and shift everything left one dedent.
name, addr = parseaddr(msg.get('from'))
name = "%s (%s) via list" % (name if name else "Anonymous", addr)
fromaddr = mlist.GetListEmail()
del msg['from']
msg['from'] = formataddr((name, addr))
# reply-to handling goes hereI'm not comfortable trying to say what to do about reply-to, because it's quite complicated depending on how you want to handle each of a large number of variations: what to do with a preexisting Reply-To and whether to put the list and/or the from address there. See the Mailman/Handlers/CookHeaders.py file in the Mailman distribution.
I wonder why then I got a bunch of issues with btopenworld.com, which apparently is Yahoo based. I just checked btopenworld.com with the ‘host’ command and as you say, it has no ‘reject’:
$ host -t TXT _dmarc.btopenworld.com
_dmarc.btopenworld.com descriptive text "v=DMARC1\; p=none\; fo=1\; rua=mailto:dmarcagg@btinternet.com, mailto:dmarc_agg@auth.returnpath.net\;"
$ host -t TXT _dmarc.yahoo.com
_dmarc.yahoo.com descriptive text "v=DMARC1\; p=reject\; sp=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com, mailto:dmarc_y_rua@yahoo.com\;”
Here is the reject notice:
Final-Recipient: rfc822; subscriber@aol.com Original-Recipient: rfc822;subscriber@aol.com Action: failed Status: 5.2.1 Remote-MTA: dns; mailin-04.mx.aol.com Diagnostic-Code: smtp; 521 5.2.1 : AOL will not accept delivery of this message.
Date: May 13, 2015 at 07:52:17 PDT From: <sender@btopenworld.com> To: <list address> Subject: subject Reply-To: sender@btopenworld.com
And yes, as I just wrote, I have good reasons for keeping this as simple as I possibly can. Upgrading is not simple, I suspect, though I’d love to move to 3.0, as I have a lot of lists, with subscribers on many lists simulteneously.
Yours,
AllanOn May 24, 2015, at 11:14 , Stephen J. Turnbull <stephen@xemacs.org> wrote:
Allan Hansen writes:
Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form.
Oddly enough, it turns out that they only use DMARC p=reject at their principal domain (aol.com and yahoo.com). You can check for any given domain by prepending _dmarc. and checking the TXT record. For example, for aol.com it would be "host -t TXT _dmarc.aol.com" if you have the host utility for doing DNS lookups.
I would have to do this for all addresses, to be safe.
If you're worried about safety and care about conforming to standards, you really should upgrade to at least Mailman 2.1.18-1. That allows you to be nonconformant only for authors whose addresses are in troublesome domains, and handles the reply-to issue as well as possible (making everybody happy isn't quite possible). I'm sure you have good reason for not doing so *right* *now*, but keep it in mind.
If I do this and add the bit about the Reply-To, what would the code look like?
If you do it for all mail, you just delete the "if" line and shift everything left one dedent.
name, addr = parseaddr(msg.get('from')) name = "%s (%s) via list" % (name if name else "Anonymous", addr) fromaddr = mlist.GetListEmail() del msg['from'] msg['from'] = formataddr((name, addr)) # reply-to handling goes here
I'm not comfortable trying to say what to do about reply-to, because it's quite complicated depending on how you want to handle each of a large number of variations: what to do with a preexisting Reply-To and whether to put the list and/or the from address there. See the Mailman/Handlers/CookHeaders.py file in the Mailman distribution.
On 05/24/2015 03:19 PM, Allan Hansen wrote:
$ host -t TXT _dmarc.btopenworld.com _dmarc.btopenworld.com descriptive text "v=DMARC1\; p=none\; fo=1\; rua=mailto:dmarcagg@btinternet.com, mailto:dmarc_agg@auth.returnpath.net\;"
The domain publishes DMARC p=none. Thus, no ISP should treat a message From: someone@btopenworld.com any differently than the same message From: someone@elsewhere.com.
Here is the reject notice:
Final-Recipient: rfc822; subscriber@aol.com Original-Recipient: rfc822;subscriber@aol.com Action: failed Status: 5.2.1 Remote-MTA: dns; mailin-04.mx.aol.com Diagnostic-Code: smtp; 521 5.2.1 : AOL will not accept delivery of this message.
I see this exact rejection reliably from AOL. When an AOL user posts to a list, the list post sent back to that user is rejected in this way, even though AOL accepts the same post for delivery to other AOL users.
I have experimented with this using my own AOL address to send and reflecting various versions of the message back. I munged a lot of headers including I think Message-Id:, and I always got rejected. I gave up trying to figure out what AOL is looking at, but this reject occurs to list posts from aol.com, even though the From: is munged to the list address.
In any case, that's not the reject reason uses for a reject due to DMARC policy.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/24/2015 09:02 AM, Allan Hansen wrote:
Checking for aol.com and yahoo.com here alone will not work. I have a bunch of other subscribers that have accounts with providers that are owned by Yahoo (mostly) and AOL, but whose addresses are not of this form. I would have to do this for all addresses, to be safe.
You only have to mung those addresses for which the domain publishes a DMARC p=reject policy. This almost certainly does not include aol and yahoo provided 'other' domains.
The code in Mailman 2.1.18+ actually does a DNS lookup of the DMARC policy. In all the myriad @python.org list posts, the only domains I see with p=reject are aol.com, yahoo.com and paypal.com.
If I do this and add the bit about the Reply-To, what would the code look like?
I'm currently between flights in an airport. When I get a chance, I'll post some code.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/24/2015 12:56 PM, Mark Sapiro wrote:
The code in Mailman 2.1.18+ actually does a DNS lookup of the DMARC policy. In all the myriad @python.org list posts, the only domains I see with p=reject are aol.com, yahoo.com and paypal.com.
If I do this and add the bit about the Reply-To, what would the code look like?
I have attached two files. Cookheaders.py.txt is a patched version of the 2.1.14 CookHeaders which will mung the From: address for messages with a From: containing @yahoo.com or @aol.com in the same way that current Mailman's Mung From action does it.
This would replace your existing Mailman/Handlers/CookHeaders.py, and you also need to remove your changes to Mailman/Handlers/Cleanse.py.
Cookheaders.py.patch.txt is just the diff between the 2.1.14 base CookHeaders.py and the attached Cookheaders.py.txt.
If you want to base the munging on an actual DNS lookup of DMARC policy, you would need to:
Add the IsDMARCProhibited function at lines 1136-1223 at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/U...> and the import of dns.resolver at lines 74-79 of the same file to your Mailman/Utils.py,
Add
# Parameters for DMARC DNS lookups. If you are seeing 'DNSException: # Unable to query DMARC policy ...' entries in your error log, you may need # to adjust these. # The time to wait for a response from a name server before timeout. DMARC_RESOLVER_TIMEOUT = seconds(3) # The total time to spend trying to get an answer to the question. DMARC_RESOLVER_LIFETIME = seconds(5)
to Mailman/Defaults.py
Ensure the dnspython package <http://www.dnspython.org/> package is available in Python. This package can be downloaded from dnspython.org or from the CheeseShop <https://pypi.python.org/pypi/dnspython/> or installed with pip.
Then you could replace the lines
dre = re.compile('@(yahoo\.com|aol\.com)\W', re.IGNORECASE)
if dre.search(msg['from']) and not fasttrack:in the attached CookHeaders.py.txt with (all one line, watch out for wrap):
if IsDMARCProhibited(mlist, parseaddr(msg.get('from'))[1]) and notfasttrack:
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Stephen,
You’re right. AOL does not accept these messages with ‘invalid’ at the end.
You’re recommending this:
name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from’] msg['from'] = formataddr((name, addrs))
Can I copy this code directly into the file? Is <list-post address> valid syntax? (I have 40+ lists)
Yours,
AllanOn May 24, 2015, at 6:10 , Stephen J. Turnbull <stephen@xemacs.org> wrote:
Allan Hansen writes:
69,74d68 <
< # Added to deal with DMARC issuej < name, addrs = parseaddr(msg.get('from')) < addrs += '.invalid'This is known to be a bad idea, as it increases the spam score at many sites (because the author's mail domain doesn't resolve). Subscribers at such sites may have trouble receiving mail, and your list(s) may be tagged as suspicious.
I would recommend the From-munging approach:
name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from'] msg['from'] = formataddr((name, addr))
Mark (or you) probably have better code, and in some cases you may want to add the addr to the Reply-To field.
< del msg['from'] < msg['from'] = formataddr((name, addrs)) \ No newline at end of file
On 05/28/2015 08:25 AM, Allan Hansen wrote:
Hi Stephen,
You’re right. AOL does not accept these messages with ‘invalid’ at the end.
You’re recommending this:
name, addr = parseadder(msg.get('from')) if addr.endswith('aol.com') or addr.endswith('yahoo.com'): # I forget what happens if it's a bare address name = "%s (%s) via list" % (name if name else "Anonymous", addr) addr = <list-post address> del msg['from’] msg['from'] = formataddr((name, addrs))
Can I copy this code directly into the file? Is <list-post address> valid syntax? (I have 40+ lists)
No, it is meant to be replaced with the actual list posting address, but is there some reasone you don't want to do what I posted at <https://mail.python.org/pipermail/mailman-users/2015-May/079190.html>?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (5)
-
Allan Hansen -
Andrew Hodgson -
Cedric Knight -
Mark Sapiro -
Stephen J. Turnbull