A DMARC munging issue ... ?
I won't call this a bug but I think it's an issue :-)
This is with 2.1.18-1 which my provider updated to a few weeks ago.
I have (via the GUI sender filters) dmarc_moderation_action set to 'munge
from' and yet a post from aol.com went through unmunged (and caused 100+
subscribers to be disabled).
I think (but may be wrong) the problem is that I also have(had) a spam
filter set to catch all posts from ISPs with a bad spam/hacking/leaking
history. This holds everything that matches:
^from:.*(aol|yahoo|hotmail|btinternet|btopenworld).*So, it appears to me that being held by this spam filter and then released
by a moderator results in the message escaping the DMARC munging :-(
My guess is that this is a natural consequence of the ordering of
processing actions in the pipeline rather than a bug but it would be good
to have confirmation from those who know the code.
Thanks in advance, Malcolm.
-- Malcolm Austen malcolm.austen@weald.org.uk GENUKI trustee genuki@weald.org.uk Oxfordshire FHS webmaster webmaster@ofhs.org.uk PUG chairman chairman@pugweb.org.uk FFHS Communications Officer communications@ffhs.org.uk
On 06/25/2014 02:41 PM, Malcolm Austen wrote:
I won't call this a bug but I think it's an issue :-)
It is an issue. I too am not sure whether it is a bug, but I think we need to do something about it. More below ...
I have (via the GUI sender filters) dmarc_moderation_action set to 'munge from' and yet a post from aol.com went through unmunged (and caused 100+ subscribers to be disabled).
This implies that the list's bounce_score_threshold is <= 1.0 so that a member's delivery is disabled on the first bounce. This may be too agressive.
Also, you may be interested in http://www.msapiro.net/scripts/reset_bounce.py.
I think (but may be wrong) the problem is that I also have(had) a spam filter set to catch all posts from ISPs with a bad spam/hacking/leaking history. This holds everything that matches:
^from:.*(aol|yahoo|hotmail|btinternet|btopenworld).*So, it appears to me that being held by this spam filter and then released by a moderator results in the message escaping the DMARC munging :-(
That is exactly what happened. header_filter_rules is processed by SpamDetect which is the first handler in the pipeline. The Munge From actions are actually done by CookHeaders and WrapMessage which process even moderator approved messages, but the check for dmarc_moderation_action is in Moderate which also processes member moderation and non-member actions.
The Moderate process is bypassed if the message has already been approved. When we implemented dmarc_moderation_action, we deliberately didn't include Hold as an action, because we didn't think it made sense, so following that reasoning, we shouldn't skip at least the Munge From and Wrap Message actions on pre-approved messages because these are message transformation actions, not message disposition actions. What to do in the case of Reject or Discard is the question.
If dmarc_moderation_action is say Reject and applies to this message, we shouldn't just accept the message because it has been pre-approved (probably without considering the consequences thereof).
This leaves two choices. Either honor the dmarc_moderation_action which will possibly reject or discard a pre-approved post, or fall back to Wrap Message which may produce list messages in an undesired format.
I have reported this at https://bugs.launchpad.net/mailman/+bug/1334450. I am interest in hearing either here or as bug comments what others think.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
That is exactly what happened. header_filter_rules is processed by SpamDetect which is the first handler in the pipeline.
I see why Barry created Chains of Rules for MM3. This is messy.
I think for this reason DMARC checking should come before spam detection, or be done as part of SpamDetect rather than moderation. If dmarc_moderation_action is reject/discard, let's not waste any further CPU cycles or queue space, let alone moderator time, on applicable messages.
This leaves two choices. Either honor the dmarc_moderation_action which will possibly reject or discard a pre-approved post, or fall back to Wrap Message which may produce list messages in an undesired format.
If the list's policy is to reject/discard, that's the policy. Consider what you're saying:
- A message looks very spammy to SpamDetect.
- The moderator sees it and decides otherwise.
- The message is from a "p=reject" Author Domain and gets trashed.
- List mourns.
But!
- A message looks legitimate to SpamDetect (because it *is*).
- The moderator never sees it.
- The message is from a "p=reject" Author Domain and gets trashed.
- List rejoices.
I should hope not!
If you don't like legitimate message being trashed, don't set DMARC action to reject/discard, because that setting *will* result in legitimate messages being lost.
On 06/25/2014 08:11 PM, Stephen J. Turnbull wrote:
I think for this reason DMARC checking should come before spam detection, or be done as part of SpamDetect rather than moderation. If dmarc_moderation_action is reject/discard, let's not waste any further CPU cycles or queue space, let alone moderator time, on applicable messages.
Yes, that seems right to me. Thanks.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Wed, 25 Jun 2014 23:32:23 +0100, Mark Sapiro mark@msapiro.net wrote:
On 06/25/2014 02:41 PM, Malcolm Austen wrote:
I won't call this a bug but I think it's an issue :-)
It is an issue. I too am not sure whether it is a bug, but I think we need to do something about it. More below ...
Thanks Mark. For the time being I have removed AOL and Yahoo from that
spam filter and marked all such addresses as 'moderated'. I'm guessing
though that allowing them through would still escape the DMARC munging -
so I've told the list that it's no longer accepting posts from AOL or
Yahoo.com.
(I note that neither yahoo.co.uk nor aol.co.uk have published a DMARC
policy.)
I have (via the GUI sender filters) dmarc_moderation_action set to 'munge from' and yet a post from aol.com went through unmunged (and caused 100+ subscribers to be disabled).
This implies that the list's bounce_score_threshold is <= 1.0 so that a member's delivery is disabled on the first bounce. This may be too agressive.
I'm not that mean :-)
I should have made it clear that this message was the trigger rather than
the lone cause.
I did (before I had 2.1.18-1 available) toy with setting it high (c.9 or
10) and setting bounce_info_stale_after down to 1 so that I could let
aol/yahoo posts out in a burst on one day and then hold them for a couple
of days to clear the bounce scores.
Also, you may be interested in http://www.msapiro.net/scripts/reset_bounce.py.
I would if I weren't restricted to the GUI :-(
Thanks though, if think get tough though, I could probably get my provider
to run the script for me.
Thanks for all your (and others) work on mailman, Malcolm.
-- Malcolm Austen malcolm.austen@weald.org.uk GENUKI trustee genuki@weald.org.uk Oxfordshire FHS webmaster webmaster@ofhs.org.uk PUG chairman chairman@pugweb.org.uk FFHS Communications Officer communications@ffhs.org.uk
Malcolm Austen writes:
I did (before I had 2.1.18-1 available) toy with setting it high (c.9 or
10) and setting bounce_info_stale_after down to 1 so that I could let
aol/yahoo posts out in a burst on one day and then hold them for a couple
of days to clear the bounce scores.
Of course the people whose addresses bounce aren't receiving the messages. That doesn't seem great, even if they don't get disabled/ unsubscribed.
On Thu, 26 Jun 2014 09:50:51 +0100, Stephen J. Turnbull
stephen@xemacs.org wrote:
Malcolm Austen writes:
I did (before I had 2.1.18-1 available) toy with setting it high (c.9
or 10) and setting bounce_info_stale_after down to 1 so that I could let aol/yahoo posts out in a burst on one day and then hold them for a
couple of days to clear the bounce scores.Of course the people whose addresses bounce aren't receiving the messages. That doesn't seem great, even if they don't get disabled/ unsubscribed.
Indeed. I tried all sorts of 'sideways thinking' but only solved
side-effects, never the real problem.
I did run the list anonymous for a while but some people objected to my
requiring them (through emergency moderation) to include their name and
email address in their signatures!
= Malcolm.
-- Malcolm Austen malcolm.austen@weald.org.uk GENUKI trustee genuki@weald.org.uk Oxfordshire FHS webmaster webmaster@ofhs.org.uk PUG chairman chairman@pugweb.org.uk FFHS Communications Officer communications@ffhs.org.uk
On 06/26/2014 01:22 AM, Malcolm Austen wrote:
Thanks Mark. For the time being I have removed AOL and Yahoo from that spam filter and marked all such addresses as 'moderated'. I'm guessing though that allowing them through would still escape the DMARC munging - so I've told the list that it's no longer accepting posts from AOL or Yahoo.com.
From or Wrap Message, those will apply to the post after approval and if dmarc_moderation_action is Reject or Discard, that will be done before
With 2.1.18-1, if a post is held because it's from a moderated member and is subsequently approved, dmarc_moderation_action will apply. If the post is From: a p=reject domain and dmarc_moderation_action is Munge the message is held for 'moderated member'.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 26/06/2014 09:22, Malcolm Austen wrote:
(I note that neither yahoo.co.uk nor aol.co.uk have published a DMARC policy.)
As a relevant aside, AOL in the UK was sold to TalkTalk a couple of years ago so is independent of AOL in the USA.
I note that the aol.co.uk domain is still owned by AOL Inc. in the USA so I presume it is used by TalkTalk under licence (if they use still use the domain -- I don't know if they do or not).
-- Mark Rousell
PGP public key: http://www.signal100.com/markr/pgp Key ID: C9C5C162
Mark Rousell writes:
On 26/06/2014 09:22, Malcolm Austen wrote:
(I note that neither yahoo.co.uk nor aol.co.uk have published a DMARC policy.)
As a relevant aside, AOL in the UK was sold to TalkTalk a couple of years ago so is independent of AOL in the USA.
yahoo.com says the same thing ("independent") about yahoo.co.jp (which is definitely in use, but -- much to my amusement -- seriously decreased since mid-April). Evidently they do make their own decisions about things like "p=reject" but I find it hard to believe they're fully independent. AOL OTOH never managed to achieve a presence in Japan at all, although it appears they still own aol.co.jp domain.
Both AOL sites are in use and use the AOL look and feel.
Steve
On 27/06/2014 00:54, Stephen J. Turnbull wrote:
Mark Rousell writes:
On 26/06/2014 09:22, Malcolm Austen wrote:
(I note that neither yahoo.co.uk nor aol.co.uk have published a DMARC policy.)
As a relevant aside, AOL in the UK was sold to TalkTalk a couple of years ago so is independent of AOL in the USA.
yahoo.com says the same thing ("independent") about yahoo.co.jp (which is definitely in use, but -- much to my amusement -- seriously decreased since mid-April). Evidently they do make their own decisions about things like "p=reject" but I find it hard to believe they're fully independent. AOL OTOH never managed to achieve a presence in Japan at all, although it appears they still own aol.co.jp domain.
Both AOL sites are in use and use the AOL look and feel.
Yup, AOL continues to provide web properties in its own name in the UK. Nevertheless, the sale of AOL UK's ISP operations to TalkTalk was/is definitely a real sale. It seems unlikely to me that AOL in the USA would have any say now in TalkTalk's/AOL UK's ISP mail operations.
A news articles from 2006 when the deal was done: http://news.techworld.com/mobile-wireless/7076/troubled-talktalk-buys-aol-uk...
The TalkTalk Wikipedia article also mentions it: http://en.wikipedia.org/wiki/TalkTalk_Group
-- Mark Rousell
PGP public key: http://www.signal100.com/markr/pgp Key ID: C9C5C162
participants (4)
-
Malcolm Austen -
Mark Rousell -
Mark Sapiro -
Stephen J. Turnbull