Re: [Mailman-Users] Ignore DMARC bounces?
On 05/02/2014 10:51 PM, Mark Sapiro wrote:
On 05/02/2014 05:21 PM, Andrew Partan wrote:
Is there some way of ignoring the DMCAC bounces? That way a message From: some...@yahoo.com will not not increase the bounce count of all Yahoo, AOL, Hotmail, ATT, MSN, and Comcast users.
It's difficult. If The local MTA is refused and reports directly to Mailman at SMTP time, Mailman will only see the SMTP status, e.g. 554, 521, or 550 in your examples. It is not possible to distinguish DMARC from other failures just by this 5xx status.
More likely, the local MTA accepted the message from Mailman and is now delivering a DSN. If every MTA delivered an RFC 3464 compliant DSN with an RFC 1893 extended status code, one could just ignore 5.7.x bounces, but even your example services don't all use a 5.7.x code even though the RFC is clear that that is the code for security or policy rejection.
Then there is the fact that many real world MTAs report in their own way and don't necessarily provide enough information to tell what the reason is. Take a look at Mailman/Bouncers/* to get an idea of what you'd be up against.
Yahoo & ATT say this: 554 5.7.9 Message not accepted for policy reasons. See http://postmaster.yahoo.com/errors/postmaster-28.html
AOL says this: 521 5.2.1 : (DMARC) This message failed DMARC Evaluation and is being refused due to provided DMARC Policy
Comcast says this: 550 5.2.0 x4fx1n03n5DGQ1A034fysP Message rejected due to DMARC. Please see http://postmaster.comcast.net/smtp-error-codes.php#DM000001
MSN/Hotmail say this: 550 5.7.0 (BAY0-MCn-Fn) Unfortunately, messages from (N.N.N.N) on behalf of (yahoo.com) could not be delivered due to domain owner policy restrictions.)
Yahoo, ATT, MSN, Hotmail, and Google all seem to respond with 5.7.x status codes. If ignoring 5.7.x responses is a good approach, and a large fraction (significant majority, I suspect) of users use services that give 5.7.x responses, and mailman is already able to parse those responses, then it sounds like ignoring 5.7.x bounces (or counting them differently) is a viable step to take.
On 14 Jun 2014, at 3:23 am, "Sparr" sparr0@gmail.com wrote:
Then there is the fact that many real world MTAs report in their own way and don't necessarily provide enough information to tell what the reason is. Take a look at Mailman/Bouncers/* to get an idea of what you'd be up against.
Yahoo & ATT say this: 554 5.7.9 Message not accepted for policy reasons. See http://postmaster.yahoo.com/errors/postmaster-28.html
AOL says this: 521 5.2.1 : (DMARC) This message failed DMARC Evaluation and is being refused due to provided DMARC Policy
Comcast says this: 550 5.2.0 x4fx1n03n5DGQ1A034fysP Message rejected due to DMARC. Please see http://postmaster.comcast.net/smtp-error-codes.php#DM000001
MSN/Hotmail say this: 550 5.7.0 (BAY0-MCn-Fn) Unfortunately, messages from (N.N.N.N) on behalf of (yahoo.com) could not be delivered due to domain owner policy restrictions.)
Yahoo, ATT, MSN, Hotmail, and Google all seem to respond with 5.7.x status codes. If ignoring 5.7.x responses is a good approach, and a large fraction (significant majority, I suspect) of users use services that give 5.7.x responses, and mailman is already able to parse those responses, then it sounds like ignoring 5.7.x bounces (or counting them differently) is a viable step to take.
It probably(?) can't hurt, but what's the point if each bounce represents an undelivered message? Isn't it better to modify the messages so they don't bounce?
Peter Shute
Peter Shute writes:
It probably(?) can't hurt, but what's the point if each bounce represents an undelivered message?
Each bounce is one bounce that takes a perfectly innocent user one bounce closer to getting disabled or unsubscribed. We need to do something about this. The easiest thing may be to simply ignore bounces on messages from p=reject domains.
Isn't it better to modify the messages so they don't bounce?
Lots of people have different opinions on that. Remember, the Author Domain wants those messages bounced, that's what p=reject means. A lot of people think that policy should be respected.
On Fri, Jun 13, 2014 at 3:24 PM, Peter Shute pshute@nuw.org.au wrote:
It probably(?) can't hurt, but what's the point if each bounce represents an undelivered message? Isn't it better to modify the messages so they don't bounce?
I'd be more inclined to remove the posting privileges of someone whose posts produce more delivery failures than a normal user's.
Modifying the messages bothers me (and a lot of other people, as indicated by the last dozen times similar conversations have been had, about changing Reply-To and From and Subject and ...) and should be the last resort.
Sparr writes:
Modifying the messages bothers me (and a lot of other people, as indicated by the last dozen times similar conversations have been had, about changing Reply-To and From and Subject and ...) and should be the last resort.
Well, actually the point is that lists need to do fewer modifications than they already do. DMARC has two tests, one for the domain in From being equivalent to the IP of the SMTP client, which will fail unless the author is at the mailing list's domain, and a DKIM signature. The signature will survive and be valid at the recipient in the case that the message is completely unmodified.
However, mailing lists typically make one or more of the following modifications: add a list tag to the Subject field, add a header or footer to the body, remove prohibited MIME bodies (.exes, text/html, etc), or transform text/html to text/plain. Any of those will cause the usual DKIM signature to be invalidated. DMARC-using domains typically sign From (required by the DKIM protocol), To, Cc, Subject, and the whole body (effectively including the end of the message, preventing appended material such as a footer).
My personal opinion is that these traditional changes are expected and desired by mailing list subscribers, and that posting from "p=reject" domains is thereby a violation of the policy of the "p=reject" domain, and places other subscribers at risk. I think mailing lists should reject such posts (if the signature is valid), or silently discard them (if it is not).
However, subscribers from those domains are unlikely to agree ....
Sent from my iPad
On 14 Jun 2014, at 10:42 pm, "Stephen J. Turnbull" stephen@xemacs.org wrote:
My personal opinion is that these traditional changes are expected and desired by mailing list subscribers, and that posting from "p=reject" domains is thereby a violation of the policy of the "p=reject" domain, and places other subscribers at risk. I think mailing lists should reject such posts (if the signature is valid), or silently discard them (if it is not).
In other words these members can receive only? That would have the desirable effect of encouraging them to get a new email address.
But from the member's perspective they're being asked to change something they've possibly had for many years, for a reason they don't fully understand, and which they may not even believe. After all, they can send ok to other recipients, so why not lists?
The end result might be to simply drive them away, so I'm in favour of getting their postings to the list somehow.
Peter Shute
Peter Shute writes:
But from the member's perspective they're being asked to change something they've possibly had for many years, for a reason they don't fully understand, and which they may not even believe.
That kind of thing happens to me all the time (I now live in Japan). Nothing new about that.
After all, they can send ok to other recipients, so why not lists?
If they don't want to learn why not, they'll just have to take someone's word for it. They chose Yahoo!, Yahoo! chose to deny them service. Not the list -- if the list changes *nothing* (that it has "possibly had for many years"), most subscribers will not receive any posts from Yahoo! users. What changed? "p=reject", that's all.
And remember, it's not just lists that Yahoo! is screwing with. It's also "on behalf of" services, including some that are more or less entertainment ("to send this image to your friend, enter her email address and yours"), and some that are the lifeblood of real businesses (eg, invoicing services such as QuickBooks Online from Intuit).[1]
The end result might be to simply drive them away, so I'm in favour of getting their postings to the list somehow.
Fine. I'm just saying what *I* am in favor of. Yahoo!'s tech staff has admitted that they're doing this because it serves their purpose and because nobody can stop them, not because they believe it generally improves the quality of life on the Internet. They under- stand that they're degrading service to their own users and many innocent third parties in order to patch up a security breach that they have not been able to explain, maybe because they don't know, and maybe because it would hurt their business to explain.
IMO, friends don't let friends use Yahoo!
YMMV of course.
Footnotes: [1] @Jim P. This is another reason why the word "transactional" has become unpopular on the DMARC list -- Q.B.O. is an easy example of an automated transactional service that "p=reject" interferes with.
Could we not send the message out as usual, then on a p=reject bounce, forward the original message (so it comes from the mailing list) along with an explanation of what is transpiring to the bounced user, plus to the message author? Maybe include a note suggesting the author change mail providers. This way if the message author's domain causes 20 bounces, they get 20 messages letting them know they need to change mail providers.
-----Original Message----- From: Mailman-Users [mailto:mailman-users-bounces+bryan.wright=rigaku.com@python.org] On Behalf Of Stephen J. Turnbull Sent: Saturday, June 14, 2014 7:42 AM To: Sparr Cc: mailman-users@python.org Subject: Re: [Mailman-Users] Ignore DMARC bounces?
Sparr writes:
Modifying the messages bothers me (and a lot of other people, as > indicated by the last dozen times similar conversations have been had, > about changing Reply-To and From and Subject and ...) and should be > the last resort.
Well, actually the point is that lists need to do fewer modifications than they already do. DMARC has two tests, one for the domain in From being equivalent to the IP of the SMTP client, which will fail unless the author is at the mailing list's domain, and a DKIM signature. The signature will survive and be valid at the recipient in the case that the message is completely unmodified.
However, mailing lists typically make one or more of the following modifications: add a list tag to the Subject field, add a header or footer to the body, remove prohibited MIME bodies (.exes, text/html, etc), or transform text/html to text/plain. Any of those will cause the usual DKIM signature to be invalidated. DMARC-using domains typically sign From (required by the DKIM protocol), To, Cc, Subject, and the whole body (effectively including the end of the message, preventing appended material such as a footer).
My personal opinion is that these traditional changes are expected and desired by mailing list subscribers, and that posting from "p=reject" domains is thereby a violation of the policy of the "p=reject" domain, and places other subscribers at risk. I think mailing lists should reject such posts (if the signature is valid), or silently discard them (if it is not).
However, subscribers from those domains are unlikely to agree ....
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/bryan.wright%40rigaku....
On 06/16/2014 10:10 AM, Bryan Wright wrote:
Could we not send the message out as usual, then on a p=reject bounce, forward the original message (so it comes from the mailing list) along with an explanation of what is transpiring to the bounced user, plus to the message author? Maybe include a note suggesting the author change mail providers. This way if the message author's domain causes 20 bounces, they get 20 messages letting them know they need to change mail providers.
Aside from the fact that this would be a real kludge to implement, inundating the message author who is in a very real sense an innocent victim of his ESPs policy with this kind of backscatter is unacceptable.
Note that on April 4, one post to one of my lists From: a yahoo.com address which was sent to only 236 individual message subscribers was bounced by 90 of them. Of those 90 bounces, I'm not sure how many I could identify reliably as DMARC bounces without just assuming they were because the From: domain published a DMARC p=reject policy. But, my point is close to 40% of the list member's ESPs honored the p=reject policy. YMMV, but that's a lot of extra handling.
While the idea of identifying a DMARC bounce and "forwarding" the original post to the bounced recipient wrapped in some boilerplate about the need to do so has some appeal, I think the implementation would be too messy to contemplate. If you wish to try, the code is at https://code.launchpad.net/~mailman-coders/mailman/2.1 aka 'bzr branch lp:mailman/2.1'.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
While the idea of identifying a DMARC bounce and "forwarding" the original post to the bounced recipient wrapped in some boilerplate about the need to do so has some appeal, I think the implementation would be too messy to contemplate. If you wish to try, the code is at https://code.launchpad.net/~mailman-coders/mailman/2.1 aka 'bzr branch lp:mailman/2.1'.
For wrapping the message, the logic in dmarc_moderation_action is much more efficient. It checks for DMARC p=reject at the Author Domain, and then does something. Wrapping the message so that the "outside" message is "From:" the list and the inside message is the original (modified per list policy) is one option. I'm not sure if there's any way to specify an explanatory message, but it would be easy to add such an option.
Note that for some lists (the ones I manage at XEmacs, for example) almost none of the members honor p=reject, and for others 95% or more of the subscribers are at GMail, Yahoo!, Hotmail, or AOL, which all (more or less, you may get lucky with list traffic to GMail) honor p=reject. If you have few enough Yahoo! and AOL subscribers, a direct appeal to them to use different posting addresses (followed up a couple weeks later by a ban on posting from those domains) might be very effective. (For example, at least some parts of the Japanese government did this in mid-April, at which point the 6 of 15 of the students on my list who had @yahoo addresses simply stopped using them for mail, although a couple of them still need them for Internet shopping.)
On 06/16/2014 06:36 PM, Stephen J. Turnbull wrote:
Mark Sapiro writes:
While the idea of identifying a DMARC bounce and "forwarding" the original post to the bounced recipient wrapped in some boilerplate about the need to do so has some appeal, I think the implementation would be too messy to contemplate. If you wish to try, the code is at https://code.launchpad.net/~mailman-coders/mailman/2.1 aka 'bzr branch lp:mailman/2.1'.
For wrapping the message, the logic in dmarc_moderation_action is much more efficient. It checks for DMARC p=reject at the Author Domain, and then does something. Wrapping the message so that the "outside" message is "From:" the list and the inside message is the original (modified per list policy) is one option. I'm not sure if there's any way to specify an explanatory message, but it would be easy to add such an option.
Yes, what Stephen says is correct. It would be a simple matter to add say a dmarc_wrapped_message_text field which text if provided would be added as a separate MIME text/plain part along with the wrapped message when the message was wrapped due to dmarc_moderation_action.
What I was saying was "too messy to contemplate" was the notion of sending the post with the original From: to all recipients and then resending a wrapped message only to those recipients that bounce the original.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro
For wrapping the message, the logic in dmarc_moderation_action is much more efficient. It checks for DMARC p=reject at the Author Domain, and then does something. Wrapping the message so that the "outside" message is "From:" the list and the inside message is the original (modified per list policy) is one option. I'm not sure if there's any way to specify an explanatory message, but it would be easy to add such an option.
Yes, what Stephen says is correct. It would be a simple matter to add say a dmarc_wrapped_message_text field which text if provided would be added as a separate MIME text/plain part along with the wrapped message when the message was wrapped due to dmarc_moderation_action.
I like the idea of the explanatory message, and it would be great if there was an option for that.
As well as explaining why a particular message is being treated differently, I'd like to be able to warn users not to simply Reply to the message expecting the reply to go to the original message's author.
Peter Shute
On 17 Jun 2014, at 3:25 am, "Bryan Wright" bryan.wright@rigaku.com wrote:
Could we not send the message out as usual, then on a p=reject bounce, forward the original message (so it comes from the mailing list) along with an explanation of what is transpiring to the bounced user, plus to the message author? Maybe include a note suggesting the author change mail providers. This way if the message author's domain causes 20 bounces, they get 20 messages letting them know they need to change mail providers.
I don't understand what you mean by "plus to the message author". Isn't the author the same person as the "bounced user"?
Do you mean the user whose server bounced it? If so, given that we can predict which messages are likely to bounce this way (but not which recipient servers), why not just forward (from the list) these messages to the whole list (including the author), along with the explanation and avoid the bouncing?
That way the author has the incentive of getting rid of these accompanying explanations to change addresses. I'd forward the messages quoted, not as attachments.
Peter Shute
participants (5)
-
Bryan Wright -
Mark Sapiro -
Peter Shute -
Sparr -
Stephen J. Turnbull