Last week I upgraded to Mailman 2.1.18 in order to get the DMARC workarounds new in that version. I set dmarc_moderation_action for my list to "Munge From" and waited to see what would happen. Despite that I seem to still be having DMARC bounces, as this morning I got 22 "Bounce action notifications" for list subscribers from Mailman.
I'm running the just-released RPM for 2.1.18 on Fedora 20. I have the python-dns package installed, which I read was required for DMARC checks. When I grep my logs for 'DMARC', all I see are the bounce messages in my Postfix log---so if the DMARC checks that should be made when a post from a yahoo.com account comes to the list are failing, they're failing in a way which isn't showing up.
There is one unusual thing about my list---namely that it sits at one end of a bridge to a phpbb forum. That is, all of the posts from the forum are posted to the list with their Sender set to a special address which is subscribed to the list, and all post from the list are received by that special address and posted to the forum from there. This means that a lot of the addresses in From headers of messages going out over the list are not actually subscribers to the list. Could this be tripping up the dmarc_moderation_action?
-- J.
On 05/28/2014 05:41 AM, Joel Uckelman wrote:
I'm running the just-released RPM for 2.1.18 on Fedora 20. I have the python-dns package installed, which I read was required for DMARC checks.
The required package is dnspython. This is not the same as PyDNS. It looks like the Fedora python-dns package is the right one, but I'm not sure.
What happens when you invoke the python that Mailman is using and type
import dns.resolver from dns.exception import DNSException
If you get an ImportError, something is wrong. Otherwise things should be OK. You can see what python Mailman is using by looking at the command lines reported by
ps -fAw | grep qrunner
When I grep my logs for 'DMARC', all I see are the bounce messages in my Postfix log---so if the DMARC checks that should be made when a post from a yahoo.com account comes to the list are failing, they're failing in a way which isn't showing up.
There will normally be an entry in Mailman's vette log for every DMARC p=reject (and p=quarantine if enabled) found and possible entries in Mailman's error log for lookup errors and other unusual conditions.
If there are no 'DMARC' entries in Mailman's logs, it most likely means the imports I show above didn't succeed in the python that Mailman is using, in which case dmarc_moderaction_action will not be done at all.
There is one unusual thing about my list---namely that it sits at one end of a bridge to a phpbb forum. That is, all of the posts from the forum are posted to the list with their Sender set to a special address which is subscribed to the list, and all post from the list are received by that special address and posted to the forum from there. This means that a lot of the addresses in From headers of messages going out over the list are not actually subscribers to the list. Could this be tripping up the dmarc_moderation_action?
What do you mean by Sender? Do you mean the Sender: header or the From: header or what?
Mailman does DMARC checks on the From: domain of the message it sees, but then recipient MTAs do DMARC checks on the From: domain of the message they see.
Perhaps you can explain more precisely what you mean by the above in terms of the From: header seen by Mailman and the From: header in the list message that recipients see.
If all you are saying is that a lot of posts are From: non-members because they come via the phpbb forum, that shouldn't matter. Mailman should still check the From: domain for DMARC and apply the dmarc_moderation_action as required regardless of list membership.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
If there are no 'DMARC' entries in Mailman's logs, it most likely means the imports I show above didn't succeed in the python that Mailman is using, in which case dmarc_moderaction_action will not be done at all.
If dmarc_moderation_action is not none (precisely speaking, "accept", I think?), and the dnspython import fails, Mailman should log this.
Thus spake Mark Sapiro:
On 05/28/2014 05:41 AM, Joel Uckelman wrote:
I'm running the just-released RPM for 2.1.18 on Fedora 20. I have the python-dns package installed, which I read was required for DMARC checks.
The required package is dnspython. This is not the same as PyDNS. It looks like the Fedora python-dns package is the right one, but I'm not sure.
I'm certain I have the correct package: The URL 'rpm -qi' gives for the pacakge is http://www.dnspython.org/, which is the same as the one given by the 2.1.18 release announcement.
What happens when you invoke the python that Mailman is using and type
import dns.resolver from dns.exception import DNSException
[uckelman@one ~]$ python Python 2.7.5 (default, Feb 19 2014, 13:47:28) [GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import dns.resolver from dns.exception import DNSException
The dns module appears to be found.
If you get an ImportError, something is wrong. Otherwise things should be OK. You can see what python Mailman is using by looking at the command lines reported by
ps -fAw | grep qrunner
[uckelman@one ~]$ ps -fAw | grep -m 1 qrunner mailman 2733 2700 0 May22 ? 00:01:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
Looks like /usr/bin/python, which is the same one as on the path:
[uckelman@one ~]$ which python /usr/bin/python
There will normally be an entry in Mailman's vette log for every DMARC p=reject (and p=quarantine if enabled) found and possible entries in Mailman's error log for lookup errors and other unusual conditions.
I have five vette logs handy, going back as far as 5 May (which would be before I installed 2.1.18). Three are empty; the other two contain one message each about rejecting a post by a nonsubscriber. There's nothing about DMARC in any of them.
If there are no 'DMARC' entries in Mailman's logs, it most likely means the imports I show above didn't succeed in the python that Mailman is using, in which case dmarc_moderaction_action will not be done at all.
Do you still think that given what I found above?
There is one unusual thing about my list---namely that it sits at one end of a bridge to a phpbb forum. That is, all of the posts from the forum are posted to the list with their Sender set to a special address which is subscribed to the list, and all post from the list are received by that special address and posted to the forum from there. This means that a lot of the addresses in From headers of messages going out over the list are not actually subscribers to the list. Could this be tripping up the dmarc_moderation_action?
What do you mean by Sender? Do you mean the Sender: header or the From: header or what?
Yes, exactly. By "Sender" I'm referring to the Sender: header.
Perhaps you can explain more precisely what you mean by the above in terms of the From: header seen by Mailman and the From: header in the list message that recipients see.
If all you are saying is that a lot of posts are From: non-members because they come via the phpbb forum, that shouldn't matter. Mailman should still check the From: domain for DMARC and apply the dmarc_moderation_action as required regardless of list membership.
This is exactly what I'm saying. Many messages posted to the list via the bridge have From: headers with non-list-member addresses in them. All messages posted to the list via the bridge have the Sender: address set to a special address which *is* a list subscriber, which is why (I believe) Mailman does not reject such messages as originating from non-subscribers.
-- J.
On 05/28/2014 10:28 PM, Stephen J. Turnbull wrote:
Mark Sapiro writes:
If there are no 'DMARC' entries in Mailman's logs, it most likely means the imports I show above didn't succeed in the python that Mailman is using, in which case dmarc_moderaction_action will not be done at all.
If dmarc_moderation_action is not none (precisely speaking, "accept", I think?), and the dnspython import fails, Mailman should log this.
Agreed.
<https://bugs.launchpad.net/mailman/+bug/1324541>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/29/2014 03:03 AM, Joel Uckelman wrote:
Do you still think that given what I found above?
Did you restart Mailman after installing the python-dns package?
Yes, I still think that. If you did restart Mailman and the package is available, something should be logged unless the DNS lookup of DMARC policy for the From: domain succeeds and doesn't find a p=reject (or quarantine).
I have attached a patch to Mailman/Utils.py which is a fix for <https://bugs.launchpad.net/mailman/+bug/1324541> which will log the unavailability of DNS lookup.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thus spake Mark Sapiro:
On 05/28/2014 10:28 PM, Stephen J. Turnbull wrote:
Mark Sapiro writes:
If there are no 'DMARC' entries in Mailman's logs, it most likely means the imports I show above didn't succeed in the python that Mailman is using, in which case dmarc_moderaction_action will not be done at all.
If dmarc_moderation_action is not none (precisely speaking, "accept", I think?), and the dnspython import fails, Mailman should log this.
Agreed.
I added two lines to Util.py as a diagnostic: --- /usr/lib/mailman/Mailman/Utils.py~ 2014-05-29 15:51:38.739320749 +0200 +++ /usr/lib/mailman/Mailman/Utils.py 2014-05-29 15:59:18.713685775 +0200 @@ -1069,8 +1069,10 @@ # This takes an email address, and returns True if DMARC policy is p=reject # or possibly quarantine. def IsDMARCProhibited(mlist, email): + syslog('error', 'in IsDMARCProhibited()') if not dns_resolver: - return False + syslog('error', 'dns_resolver == False') + return False email = email.lower() at_sign = email.find('@') After sending a message to the list, what I see in /var/log/mailman/error is: May 29 16:01:36 2014 (30524) in IsDMARCProhibited() and nothing else. So: 1) IsDMARCProhibited() is running for me. 2) dns_resolver was *not* false. It looks like this rules out an ImportError in my case, since the only way dns_resolver can be set to false is in the event of an ImportError (see line 79). It's good to fix the bug you noted regardless, but something else is failing silently for me. -- J.
Thus spake Mark Sapiro:
On 05/29/2014 03:03 AM, Joel Uckelman wrote:
Do you still think that given what I found above?
Did you restart Mailman after installing the python-dns package?
I think so, but I have no way of checking now.
Yes, I still think that. If you did restart Mailman and the package is available, something should be logged unless the DNS lookup of DMARC policy for the From: domain succeeds and doesn't find a p=reject (or quarantine).
I have attached a patch to Mailman/Utils.py which is a fix for <https://bugs.launchpad.net/mailman/+bug/1324541> which will log the unavailability of DNS lookup.
We produced equivalent patches concurrently---the one I just posted shows that dns_resolver is not false for me.
-- J.
On 05/29/2014 07:15 AM, Joel Uckelman wrote:
So:
IsDMARCProhibited() is running for me.
dns_resolver was *not* false. It looks like this rules out an ImportError in my case, since the only way dns_resolver can be set to false is in the event of an ImportError (see line 79).
It's good to fix the bug you noted regardless, but something else is failing silently for me.
What do you get on the local machine from
dig txt _dmarc.yahoo.com
If that gives the expected result, what do you get from the following Python
import dns.resolver from dns.exception import DNSException resolver = dns.resolver.Resolver() dmarc_domain = '_dmarc.yahoo.com' resolver.timeout = 3.0 resolver.lifetime = 5.0 txt_recs = resolver.query(dmarc_domain, dns.rdatatype.TXT) for x in txt_recs: print x
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thus spake Mark Sapiro:
What do you get on the local machine from
dig txt _dmarc.yahoo.com
[uckelman@one Mailman]$ dig txt _dmarc.yahoo.com
; <<>> DiG 9.9.4-P2-RedHat-9.9.4-12.P2.fc20 <<>> txt _dmarc.yahoo.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2747 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 7
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;_dmarc.yahoo.com. IN TXT
;; ANSWER SECTION: _dmarc.yahoo.com. 1777 IN TXT "v=DMARC1\; p=reject\; sp=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com, mailto:dmarc_y_rua@yahoo.com\;"
;; AUTHORITY SECTION: yahoo.com. 89848 IN NS ns1.yahoo.com. yahoo.com. 89848 IN NS ns2.yahoo.com. yahoo.com. 89848 IN NS ns6.yahoo.com. yahoo.com. 89848 IN NS ns3.yahoo.com. yahoo.com. 89848 IN NS ns4.yahoo.com. yahoo.com. 89848 IN NS ns5.yahoo.com.
;; ADDITIONAL SECTION: ns1.yahoo.com. 521834 IN A 68.180.131.16 ns2.yahoo.com. 521834 IN A 68.142.255.16 ns3.yahoo.com. 521834 IN A 203.84.221.53 ns4.yahoo.com. 521848 IN A 98.138.11.157 ns5.yahoo.com. 521834 IN A 119.160.247.124 ns6.yahoo.com. 3460 IN A 121.101.144.139
;; Query time: 0 msec ;; SERVER: 62.210.16.6#53(62.210.16.6) ;; WHEN: Thu May 29 16:42:14 CEST 2014 ;; MSG SIZE rcvd: 371
If that gives the expected result,
It looks like I got a DMARC record back. Is that the expected result?
what do you get from the following Python
import dns.resolver from dns.exception import DNSException resolver = dns.resolver.Resolver() dmarc_domain = '_dmarc.yahoo.com' resolver.timeout = 3.0 resolver.lifetime = 5.0 txt_recs = resolver.query(dmarc_domain, dns.rdatatype.TXT) for x in txt_recs: print x
The script prints:
"v=DMARC1; p=reject; sp=none; pct=100; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com, mailto:dmarc_y_rua@yahoo.com;"
-- J.
On 05/29/2014 07:44 AM, Joel Uckelman wrote:
;; ANSWER SECTION: _dmarc.yahoo.com. 1777 IN TXT "v=DMARC1\; p=reject\; sp=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com, mailto:dmarc_y_rua@yahoo.com\;"
...
It looks like I got a DMARC record back. Is that the expected result?
Yes.
The script prints:
"v=DMARC1; p=reject; sp=none; pct=100; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com, mailto:dmarc_y_rua@yahoo.com;"
OK. So if you look up the DMARC record for yahoo.com, you find p=reject.
Try the attached patch or similar to see what's going on.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Thu, May 29, 2014 at 6:03 AM, Joel Uckelman <uckelman@nomic.net> wrote:
Thus spake Mark Sapiro:
If all you are saying is that a lot of posts are From: non-members because they come via the phpbb forum, that shouldn't matter. Mailman should still check the From: domain for DMARC and apply the dmarc_moderation_action as required regardless of list membership.
This is exactly what I'm saying. Many messages posted to the list via the bridge have From: headers with non-list-member addresses in them. All messages posted to the list via the bridge have the Sender: address set to a special address which *is* a list subscriber, which is why (I believe) Mailman does not reject such messages as originating from non-subscribers.
Can you pastebin a full set of headers, distinctly obfuscating people's names, etc.?
-Jim P.
Thus spake Mark Sapiro:
Did you restart Mailman after installing the python-dns package?
I think this was the problem---I forgot to restart Mailman after installing python-dns. It took a few days after asking about this before I got any further posts which would trigger From munging, but now I am seeing the From munging.
Thanks, everyone, for your help with this.
-- J.
participants (4)
-
Jim Popovitch -
Joel Uckelman -
Mark Sapiro -
Stephen J. Turnbull