DMARC and Reply-To lines with from_is_list munging.
Greetings...
So I run a bunch of mailing lists, with a bunch of people who are not technically adept whatsoever. ("I am not getting list posts! "That's because you set yourself to no mail" "What's no mail?" "It means you set yourself to be a member of the list, but not to get any email from it." "Oh that's good." "So we're good then?" "But why am I not getting any emails from the list?" *headdesk*--yes this was an actual conversation with a user.)
People are, of course, bitching about the from_is_list setting removing the email addresses of people who are sending email to the lists. (And people aren't quite understanding that it's helpful to sign one's emails, etc.
So I updated to 2.1.18-1 today. Now we have a Reply-To that has the poster's email and the list's email address.
A few of the lists I run block emails with more than one recipient, so now this is going to be an adventure. (Ok, more like a nightmare, as right now it appears my choices are "make reply-to only the list" ("anonymous_list") or "make reply-to the poster and the list.")
I wonder if this solution might be more helpful here--something like what Google Groups is doing. Changing the From line to this:
'First Last <firstlast@domain.com>' via List Title <list-address@googlegroups.com>
This still shows the poster's email address (as the Real Name), which makes it easier for people to reply privately if they choose, and still addresses the DMARC issue.
Thoughts? Ideas?
Best, --Glenn
On 05/06/2014 12:47 PM, Glenn Sieb wrote:
So I updated to 2.1.18-1 today. Now we have a Reply-To that has the poster's email and the list's email address.
A few of the lists I run block emails with more than one recipient,
Do you mean Privacy options... -> Recipient filters -> max_num_recipients = 2
If so, ouch, but what do you do now when people reply-all to posts. Don't those replies get held?
I wonder if this solution might be more helpful here--something like what Google Groups is doing. Changing the From line to this:
'First Last <firstlast@domain.com>' via List Title <list-address@googlegroups.com>
This is specifically advised against by the DMARC community. See the NOTE: in the Requirements: section at <http://www.dmarc.org/supplemental/mailman-project-mlm-dmarc-reqs.html>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 5/6/14, 4:29 PM, Mark Sapiro wrote:
Do you mean Privacy options... -> Recipient filters -> max_num_recipients = 2
If so, ouch, but what do you do now when people reply-all to posts. Don't those replies get held?
Indeed. They get rejected. Policy on a couple particular lists. No cc's, no using the address on web-forms (i.e. "greeting card sites") etc.
This is specifically advised against by the DMARC community. See the NOTE: in the Requirements: section at <http://www.dmarc.org/supplemental/mailman-project-mlm-dmarc-reqs.html>.
Fair enough. So, basically I'm fsck'd. Set the lists to be "anonymous_list" or set an explicit reply-to to be the lists and hope that strips out the extraneous reply-to entry.
Or, as you said above, "ouch" and having to deal with a metric crapton of ID-10t users not cleaning up the To: line when they reply and dealing with clearing the moderation queue since we can't edit posts held for moderation easily.
Best, --G.
On 05/06/2014 02:17 PM, Glenn Sieb wrote:
Fair enough. So, basically I'm fsck'd. Set the lists to be "anonymous_list" or set an explicit reply-to to be the lists and hope that strips out the extraneous reply-to entry.
I went back and forth with this. Initially, if first_strip_reply_to was Yes and reply_goes_to_list was This list or Explicit address, I didn't put the poster's address in Reply-To:
I finally decided it was of overriding importance to expose the posters address to enable off list (or non-list member) replies, and this warranted breaking the previous Reply-To: header munging options semantics.
I am willing to consider changing this, either to treat Reply-To: differently for Wrap Message since the original headers are in the wrapped message in that case, or to just go back to not adding the poster's address to Reply-To: as in my initial paragraph above.
However, I need more feedback from the community before making changes. I could always add yet another setting, but I hate that idea for multiple reasons.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 5/6/14, 5:31 PM, Mark Sapiro wrote:
I went back and forth with this. Initially, if first_strip_reply_to was Yes and reply_goes_to_list was This list or Explicit address, I didn't put the poster's address in Reply-To:
I finally decided it was of overriding importance to expose the posters address to enable off list (or non-list member) replies, and this warranted breaking the previous Reply-To: header munging options semantics.
I am willing to consider changing this, either to treat Reply-To: differently for Wrap Message since the original headers are in the wrapped message in that case, or to just go back to not adding the poster's address to Reply-To: as in my initial paragraph above.
However, I need more feedback from the community before making changes. I could always add yet another setting, but I hate that idea for multiple reasons.
Can there be an option somewhere in between "anonymous_list" and "reply_goes_to_list?" One where it can strip the poster's email from the reply-to, but leave the other headers alone?
Best, --Glenn
On 05/06/2014 02:36 PM, Glenn Sieb wrote:
On 5/6/14, 5:31 PM, Mark Sapiro wrote:
I could always add yet another setting, but I hate that idea for multiple reasons.
Can there be an option somewhere in between "anonymous_list" and "reply_goes_to_list?" One where it can strip the poster's email from the reply-to, but leave the other headers alone?
That's covered in my sentence above.
Anyway, that's a decision for the next release, which hopefully isn't 'imminent'.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Is the existing change (making sure the poster's address is in the reply-to) available in a patch? I checked launchpad but if it's there I couldn't find it. I'd like to see if I can apply it to 2.1.17 while waiting for cPanel to upgrade to 2.1.18.
FWIW, I'd vote against a rollback to the earlier behavior. I got several complaints about the poster's email address going missing. So I ended up setting first_strip_reply_to to "No," which of course is also a problem because I have max_num_recipients set pretty low (4).
rac
On Tue, May 6, 2014 at 2:48 PM, Mark Sapiro <mark@msapiro.net> wrote:
On 05/06/2014 02:36 PM, Glenn Sieb wrote:
On 5/6/14, 5:31 PM, Mark Sapiro wrote:
I could always add yet another setting, but I hate that idea for multiple reasons.
Can there be an option somewhere in between "anonymous_list" and "reply_goes_to_list?" One where it can strip the poster's email from the reply-to, but leave the other headers alone?
That's covered in my sentence above.
Anyway, that's a decision for the next release, which hopefully isn't 'imminent'.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/rclemings%40gmail.com
On 05/06/2014 02:52 PM, Russell Clemings wrote:
Is the existing change (making sure the poster's address is in the reply-to) available in a patch? I checked launchpad but if it's there I couldn't find it. I'd like to see if I can apply it to 2.1.17 while waiting for cPanel to upgrade to 2.1.18.
The actual change is the CookHeaders.py diff at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1476>, but there are other changes in CookHeaders.py and other modules since 2.1.17 that impact this as well so you can't just apply that patch. In fact, the stuff that's being changed isn't even there in 2.1.17.
It's very convoluted and fragile and touches things like new list settings as well, and I don't know how it plays with cPanel's mods. It would almost turn into a full upgrade to 2.1.18.
I'm advising you to not try it.
FWIW, I'd vote against a rollback to the earlier behavior. I got several complaints about the poster's email address going missing. So I ended up setting first_strip_reply_to to "No," which of course is also a problem because I have max_num_recipients set pretty low (4).
Thanks for voting.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, 2014-05-06 at 14:31 -0700, Mark Sapiro wrote:
I am willing to consider changing this, either to treat Reply-To: differently for Wrap Message since the original headers are in the wrapped message in that case, or to just go back to not adding the poster's address to Reply-To: as in my initial paragraph above.
However, I need more feedback from the community before making changes. I could always add yet another setting, but I hate that idea for multiple reasons.
It's ugly, but having yet another switch seems to me to be the only way to handle this. Having the poster's address in Reply-To: is the only way to address the information loss implied by the necessary change to the From: header, especially for MUAs that expose only the address comment and not the actual address, and especially for subscribers who are not technically inclined and wish to simply hit "reply" and get a reply to the original author.
This _should_ be a matter of choice for list admins, even if it seems that they're already overloaded with choices pursuant to addressing the DMARC issue. Until something better comes along, we're just going to have to deal with it.
-- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | --- The Roadie http://www.fmp.com |
On May 06, 2014, at 05:17 PM, Glenn Sieb wrote:
Fair enough. So, basically I'm fsck'd. Set the lists to be "anonymous_list" or set an explicit reply-to to be the lists and hope that strips out the extraneous reply-to entry.
Yes, and sadly it's forced on us by external policies.
I must admit that I'm sympathetic to John Levine's solution over in
mailman-developers. His implementation adds .invalid to the domain in the
From header. Yes it breaks the standards and you'd still have to explicitly
modify the headers in the reply (the ease of which depends on your MUA), but
it avoids tricky interactions with the already fragile and overloaded Reply-To
header munging, and points the finger in the direction of the original problem.
I need to read that whole thread and think about it some more. It's painfully clear that DMARC as defined and implemented today is poison to mailing lists, and it's a shame that you, our dear users, are the canaries. I hope we can have some constructive discussions with the DMARC advocates about how to restore usability to mailing lists in a DMARC pervasive world.
Cheers, -Barry
Glenn Sieb writes:
So I updated to 2.1.18-1 today. Now we have a Reply-To that has the poster's email and the list's email address.
A few of the lists I run block emails with more than one recipient, so now this is going to be an adventure. (Ok, more like a nightmare, as right now it appears my choices are "make reply-to only the list" ("anonymous_list") or "make reply-to the poster and the list.")
What is the intent of the restriction? Are you trying to get the users to use "reply to author" by punishing them with a black hole if they don't, and then set Reply-To to list-post so that nobody ever gets a personal reply? Or is this intended to prevent people from including 3rd parties in the OP (of course, you can't -- they can always BCC and you'll never know)?
I suppose your users would get upset if you used dmarc_moderation_action = 'Wrap Message' instead of whichever_option = 'Mung From'?
Given Mark's reply, probably you'll need use a custom Handler, whatever the requirements. Is that acceptable (ie, you have the necessary accesses)? N.B. It's possible to restrict use of Handlers to particular lists by giving them list-specific pipelines.
Steve
On 5/7/14, 12:08 AM, Stephen J. Turnbull wrote:
What is the intent of the restriction? Are you trying to get the users to use "reply to author" by punishing them with a black hole if they don't, and then set Reply-To to list-post so that nobody ever gets a personal reply? Or is this intended to prevent people from including 3rd parties in the OP (of course, you can't -- they can always BCC and you'll never know)?
What my list owners want out of my lists, and what rules they decide on for their lists, is not my business. By extension, it is not yours. I provide them email lists, they ask for things that seem reasonable to me. When those things suddenly are yanked away, they complain, and I'm left holding the bag of trying to answer "why."
Your attempt to "explain away" the request by making it sound like some kind of absurd policy is disingenuous at best.
I suppose your users would get upset if you used dmarc_moderation_action = 'Wrap Message' instead of whichever_option = 'Mung From'?
Some use mobile devices. So there's the answer to that question.
Given Mark's reply, probably you'll need use a custom Handler, whatever the requirements. Is that acceptable (ie, you have the necessary accesses)? N.B. It's possible to restrict use of Handlers to particular lists by giving them list-specific pipelines.
I'm just trying to see if there are better options out there. This DMARC stuff is ridiculous. The providers aren't being blamed for this, we (the mailing-list providers) are. Doesn't help that the users on services responsible for the DMARC p=reject stuff aren't getting the bounces, it's other people whose ISPs are respecting it who are, and they're the ones who get bounced off of lists because it's *their* bounce score that gets increased.
It's ridiculous. And I want to know why, exactly, Yahoo Groups isn't being affected by this. They're not doing the "via YahooGroup" bit, or wrapping their mails. :-\ I'm betting they're not even honoring the DMARC from other providers.
*sigh* I hate this frustration.
Best, --Glenn
On 05/07/2014 12:45 PM, Glenn Sieb wrote:
It's ridiculous. And I want to know why, exactly, Yahoo Groups isn't being affected by this. They're not doing the "via YahooGroup" bit, or wrapping their mails. :-\ I'm betting they're not even honoring the DMARC from other providers.
Yahoo groups doesn't have problems with mail From: yahoo.com because they send the mail with envelope from ...@returns.groups.yahoo.com which passes SPF and aligns with the domain in From:, but the interesting question is what do they do with a post From: aol.com. I haven't had time to test that yet.
Note that google groups does the same From: munging that Mailman does, and only for From: domains that publish DMARC p=reject.
*sigh* I hate this frustration.
So do we all. The Mailman development community resents as much as anyone being forced into this "here's what *we're* doing, now *you* have to figure out how to deal with it" bind, but that's where we are. We are trying to talk with DMARC proponents, and we're trying to figure out how to mitigate the effects with the least possible disruption to users and to long term, established standards and practices.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Wed, May 7, 2014 at 6:47 PM, Mark Sapiro <mark@msapiro.net> wrote:
We are trying to talk with DMARC proponents,
You won't be successful until those people themselves figure out what they are doing (and then they agree to quit using the Internet as a testbed) :-) Honestly, they (one of the principal DMARC spec authors works for Yahoo) ignored their own advice, imagine how well that would go over in some other industries.
-Jim P.
Jim Popovitch writes:
On Wed, May 7, 2014 at 6:47 PM, Mark Sapiro <mark@msapiro.net> wrote:
We are trying to talk with DMARC proponents,
You won't be successful until those people themselves figure out what they are doing
That's true, but those folks (or, more accurately, their bosses) have their shorts in a knot over the recent attacks. I don't have a lot of sympathy for the corporations which have a long history of half-baked implementations, but our best bet is to help them figure it out.
(and then they agree to quit using the Internet as a testbed) :-)
But there is no other. I can't really blame them for eventually going live, I just wish they tried harder to work and play well with others.
Honestly, they (one of the principal DMARC spec authors works for Yahoo) ignored their own advice, imagine how well that would go over in some other industries.
Happens all the time. Ford Pinto gas tanks, space shuttle O-rings, the list goes on. Let's have some perspective: nobody died this time. And I doubt the principal authors ignored their own advice; some PHB did it.
"Stephen J. Turnbull" <stephen@xemacs.org> wrote:
Honestly, they (one of the principal DMARC spec authors works for Yahoo) ignored their own advice, imagine how well that would go over in some other industries.
Let's not overlook Agari, which has a financial stake in offering a solution to the problem they helped create. Notice how many dmarc domains direct the reports to agari, from which, for a fee, they will get nice reports and metrics for their CIO to show around, reports that will show how many times their domain was "faked". Agari has an interest in making those numbers big, and mailing lists help them do it. The Agari web page boasts how many users they "protect", and it features the kind of slick writing that impresses people who don't know nuts and bolts.
One of the great failings on Yahoo's part was introducing a Change without notice to those affected, not even their own customers (to my knowledge). Even sloppy business owners should know not to do that.
Agari introduced "Agari PRO" April 1. Dmarc was pulled from standards track April 2. Yahoo implemented dmarc April 4. What was the rush?
Let's have some perspective: nobody died this time.
So true. In 100 years who will know the difference.
Joseph Brennan Manager, Email and Systems Applications Columbia University Information Technology
Joseph Brennan writes:
"Stephen J. Turnbull" <stephen@xemacs.org> wrote:
Honestly, they (one of the principal DMARC spec authors works for Yahoo) ignored their own advice, imagine how well that would go over in some other industries.
I didn't write that, and I dissent from the implied sentiment.
Cheers
On Thu, May 8, 2014 at 11:44 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Joseph Brennan writes:
"Stephen J. Turnbull" <stephen@xemacs.org> wrote:
Honestly, they (one of the principal DMARC spec authors works for Yahoo) ignored their own advice, imagine how well that would go over in some other industries.
I didn't write that, and I dissent from the implied sentiment.
I wrote it, and I have no idea why you need to sound like a lawyer just to tell someone you didn't write something. :-) Or at least you could have just said "Jim wrote that, not me" :-)
-Jim P.
Glenn Sieb writes:
What my list owners want out of my lists, and what rules they decide on for their lists, is not my business. By extension, it is not yours.
If you just want to vent, please say so. I thought you were asking for help.
If you want help, then the questions I asked are essential to doing a good job for your list owners. There are two reasons for that.
(1) Users often request a feature that they believe accomplishes a certain goal, but it does not. All too often implementing that feature does not satisfy their need, with attendant frustration all around. Letting the developer design the feature to achieve the goal often (although not always) does a better job of satisfying needs.
(2) Often either the current implementation of the program or the nature of the world means that not all needs can be satisfied, and a compromise must be suggested. Knowing the goals (reasons why) can help the designer suggest a better (accomplishes more goals more fully) or more palatable (emphasizes more important goals) compromise.
See my dialog with Peter Shute for an example of how such design can succeed. It's rare that we get 95%[1] success that way because of (2), but my lack of understanding of his lists' requirements displayed at the start is the usual case.
I'm just trying to see if there are better options out there.
And I'm just trying to understand what "better" means to you and your list owners and subscribers.
Footnotes: [1] Not yet 100% success because of the increased resource requirement, which can be a blocker.
It is not necessary to cc: me. I get list emails. Emails can go to the list, unless you wish to take something private. Thank you.
On 5/7/14, 10:36 PM, Stephen J. Turnbull wrote:
If you just want to vent, please say so. I thought you were asking for help.
Then please work on your phrasing. You sounded very judgmental. "Are you...*snip*...punishing them with a black hole" "They can always BCC and you'll never know!"
They apparently set the max_num_recipients to 2 to help prevent spam from making it onto the lists, as SA is fine and all, but is generally crap for catching short URI spam.
And, again, what rules my list owners choose to have on their lists is not my business, but frankly, I see nothing *wrong* with this, and it makes a metric f*ckton of sense to me given the number of AOL and Yahoo subscribers on some of the lists. Which makes this whole DMARC stuff such an effing joke.
If you want help, then the questions I asked are essential to doing a good job for your list owners. There are two reasons for that.
If I felt what my users were asking for was unreasonable, I wouldn't have bothered to bring it here. They'd *like* to see who's posting so if they *choose* to reply privately they can. In the past, this was easy enough. The From: line was there with the OP's email address. Now, as far as I can tell, depending on the MUA the *poster* uses, there *might* be two Reply-Tos--one with the OP email, one with the list address. But that's not reliable, as it doesn't happen for ALL posters.
Hell, even a munged From: like:
"ges+lists at wingfoot dot org via Mailman-Users <mailman-users@python.org>"
would be a vast improvement over:
"ges+lists--- via Mailman-Users <mailman-users@python.org>"
Best, --Glenn
On 05/08/2014 12:42 PM, Glenn Sieb wrote:
In the past, this was easy enough. The From: line was there with the OP's email address. Now, as far as I can tell, depending on the MUA the *poster* uses, there *might* be two Reply-Tos--one with the OP email, one with the list address. But that's not reliable, as it doesn't happen for ALL posters.
There will only be one Reply-To: header in outgoing mail from Mailman per RFC 822/2822/5322.
With the DMARC mitigations, at least as of 2.1.18, this Reply-To: will always contain the posters original From: address. Depending on the first_strip_reply_to and reply_goes_to_list settings of the list and whether or not there was a Reply-To: in the incoming mail, it may contain other addresses too. It will never contain duplicate addresses.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I appreciate that the MailMan team is awesome & doing the best that can be expected given the new DMARC restrictions being forced on us all. Thanks guys!
FEATURE REQUEST: A setting to configure the new "From" line. Instead of FROM: Author_Name via ListName <ListName@lists.example.com>
I want to request that MailMan let the list admin config the text after the author's name in the FROM line. I would use this setting to say: FROM: Author_Name via Club Name <ListName@example.com>
- substituted test string "Club Name" for "List Name"
- substituted explicit reply_to_address instead of list address.
More details:
DreamHost just installed 2.1.17 so that's what I have to work with. I don't know why they didn't go all the way to the newest version this week.
My email lists are served from a sub-domain that I don't use for anything except lists. lists.example.com That is a given, not my option.
I have things figured out as well as can be for my situation. The settings we are going to use are: reply_goes_to_list = Explicit address reply_to_address = everybody@example.com (as opposed to everybody@lists.example.com ) from_is_list = Mung From first_strip_reply_to = Yes
The only difference is from_is_list = Mung From. The other settings are the same as before DEMARC.
This makes it almost as good as before Yahoo ruined all mailing lists worldwide.
Differences before & After DEMARC BEFORE: FROM: Author_Name <Author@example.com> TO: ListName <everybody@example.com> REPLY-TO: ListName <everybody@example.com>
We liked it like that. Everything was fine.
AFTER DEMARC, using the best settings for us that we can:
FROM: Author_Name via ListName <everybody@lists.example.com> TO: ListName <everybody@example.com> REPLY-TO: ListName <everybody@example.com>
This is *pretty good* except that
- I don't like the "via list name" in the from header, even though I understand that is the new reality.
- I don't like the lack of an author email address.
I also tried: first_strip_reply_to = No But that means a normal reply will go to both the list & the author. So the author will get 2 copies, and I don't feel that is desirable, even though it seems to be the only way to include the author's email address.
FEATURE REQUEST:
A setting to control the new "From" line. Instead of FROM: Author_Name via ListName <everybody@lists.example.com>
I want to request that it offer the option to let the list admin config what goes there after the author's name. I would use this setting to say:
FROM: Author_Name via Club Name <everybody@example.com>
substituted "Club Name" for "List Name", and used
explicit reply_to_address instead of list address.
Best, Dave Nathanson Mac Medix
On 05/08/2014 02:34 PM, Dave Nathanson wrote:
AFTER DEMARC, using the best settings for us that we can:
FROM: Author_Name via ListName <everybody@lists.example.com> TO: ListName <everybody@example.com> REPLY-TO: ListName <everybody@example.com>
This is *pretty good* except that
- I don't like the "via list name" in the from header, even though I understand that is the new reality.
- I don't like the lack of an author email address.
I also tried: first_strip_reply_to = No But that means a normal reply will go to both the list & the author. So the author will get 2 copies, and I don't feel that is desirable, even though it seems to be the only way to include the author's email address.
Well then you won't like 2.1.18 because it will put the original From: in Reply-To: even if first_strip_reply_to = Yes, but the author won't get 2 copies if the author's "Avoid duplicate copies of messages?" option (nodups in the admin Membership List) is Yes which is the normal default. The issue however is the author gets the direct reply, not the list copy which is the only way it can work, but may not be the copy she'd prefer to get.
FEATURE REQUEST: A setting to control the new "From" line. Instead of FROM: Author_Name via ListName <everybody@lists.example.com> I want to request that it offer the option to let the list admin config what goes there after the author's name. I would use this setting to say: FROM: Author_Name via Club Name <everybody@example.com>
substituted "Club Name" for "List Name", and used explicit reply_to_address instead of list address.
I'll consider this. It would help to keep my attention if you would submit this to the tracker at <https://bugs.launchpad.net/mailman/+filebug>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Thu, 2014-05-08 at 15:42 -0400, Glenn Sieb wrote:
If I felt what my users were asking for was unreasonable, I wouldn't have bothered to bring it here. They'd *like* to see who's posting so if they *choose* to reply privately they can. In the past, this was easy enough. The From: line was there with the OP's email address. Now, as far as I can tell, depending on the MUA the *poster* uses, there *might* be two Reply-Tos--one with the OP email, one with the list address. But that's not reliable, as it doesn't happen for ALL posters.
Hell, even a munged From: like:
"ges+lists at wingfoot dot org via Mailman-Users <mailman-users@python.org>"
would be a vast improvement over:
"ges+lists--- via Mailman-Users <mailman-users@python.org>"
I'm not as knowledgeable as Stephen or Mark, but I've been working with Internet email since the early 90s or so and have read the founding RFCs. One of the principles underlying the design of the Internet email system is that information should never be intentionally abandoned. Nothing gets dumped into the cosmic bit bucket, neither header information nor content, and NDRs and DSNs keep the sender appraised of problems with delivery. This has been a strong argument against munging of Reply-To headers going back quite a few years. Information may be _added_ by a component in the delivery chain (and generally is) but not deleted.
Arguably, the correct response to DMARC filtering _should_ be the MIME encapsulation of list mail, with appropriate RFC 2369 headers added to the enclosing MIME structure leaving the content un-munged, with all information from the original poster intact. Arguably, MUAs should be transparent to this. Arguably, this would have been the best design for the operation of mailing lists in email-space from the git-go.
We're stuck in the Real World, however, where Apple and probably other MUA authors and designers have cut corners in design and we're forced into a corner where information loss of some sort is imposed on us. From: header munging is decidedly ugly! It's perhaps the least ugly solution that still works reliably to deliver content to _everyone_ even though the information loss limits choice on the receiving end.
Your suggested partial solution ("ges+lists at wingfoot dot org via Mailman-Users ...") is also ugly, but given the situation we're in at this point, IMHO it has merit and should be worth some consideration in the design of Mailman. What goes into an address comment is, or should be, purely informational on a human level, and ignored on a computational level. Whether or not it would would confuse people is another matter. It ain't the kinder, gentler Internet I jumped into back in 1994!
-- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com |
Lindsay Haisley writes:
What goes into an address comment is, or should be, purely informational on a human level, and ignored on a computational level.
Unfortunately, we can't depend on that:
There are a few possible mechanisms that attempt mitigation of [display name] attacks, such as:
o If the display name is found to include an email address (as specified in [MAIL]), execute the DMARC mechanism on the domain name found there rather than the domain name discovered originally.
DMARC draft, sec. 15.2. This is discussion of matters outside the scope of DMARC itself, not a normative specification, and the document itself says there are legitimate uses of email addresses in display names (or comments). But that hasn't stopped the spam-fighters in the past; it may not stop them this time. AFAICS, putting an address from a DMARC domain anywhere in the mail leaves you subject to a possible DMARC reject unless you satisfy "from alignment" for that domain exactly as specified in DMARC.
That's not implemented by anyone now, and may never be. And obfuscating the address as in the OP may help, but for my previous work address that would be
stephen dot turnbull dot 1 at econ dot ohio-state dot eduwhich is 57 characters. You pays your money and you takes your choice, I guess.
On Sat, 2014-05-10 at 04:01 +0900, Stephen J. Turnbull wrote:
Lindsay Haisley writes:
What goes into an address comment is, or should be, purely informational on a human level, and ignored on a computational level.
Unfortunately, we can't depend on that:
The operational term is "or should be" :/
DMARC draft, sec. 15.2. This is discussion of matters outside the scope of DMARC itself, not a normative specification, and the document itself says there are legitimate uses of email addresses in display names (or comments). But that hasn't stopped the spam-fighters in the past; it may not stop them this time. AFAICS, putting an address from a DMARC domain anywhere in the mail leaves you subject to a possible DMARC reject unless you satisfy "from alignment" for that domain exactly as specified in DMARC.
That's not implemented by anyone now, and may never be. And obfuscating the address as in the OP may help, but for my previous work address that would be
stephen dot turnbull dot 1 at econ dot ohio-state dot eduwhich is 57 characters. You pays your money and you takes your choice, I guess.
DMARC is ugly, as AOL and Yahoo are using it. From: header munging is ugly. Ugly begets ugly when agreements start to break down. All we can do is ride with it and hope that smart people with cool heads and a sense of the real value of a smoothly working Internet to the larger community will ultimately prevail. I'm not overly optimistic at this point. AFAICS, this just another aspect of the general abandonment of net neutrality, one which has come in under the radar of the nightly news.
A nice fix, albeit probably total pie-in-the-sky, would be the establishment of a MIME Content-Type: multipart/list-post, a variation on (or extension of) mulpart/mixed. MUAs SHOULD (in the RFC 2119 sense) effectively hide the outermost enclosing MIME envelope with this Content-Type and present the contents according to rules that would apply were the enclosing MIME envelope not there. As far as the mail system is concerned, the headers on the envelope are the effective ones. As far as the MUA is concerned, for presentation purposes, the envelope content is what counts.
-- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com |
Lindsay Haisley writes:
A nice fix, albeit probably total pie-in-the-sky, would be the establishment of a MIME Content-Type: multipart/list-post, a variation on (or extension of) mulpart/mixed. MUAs SHOULD (in the RFC 2119 sense) effectively hide the outermost enclosing MIME envelope with this Content-Type and present the contents according to rules that would apply were the enclosing MIME envelope not there. As far as the mail system is concerned, the headers on the envelope are the effective ones. As far as the MUA is concerned, for presentation purposes, the envelope content is what counts.
The problem is that the DMARC people don't give a damn about the mail system (and the PHBs behind the actions at Yahoo and AOL could care less in both senses, apparently). They're entirely concerned with presentation.
And the technicians who designed DMARC are *right* to be concerned about presentation, because it is presentation that the crooks use to hook their prey. In other words, if we come up with a way to present mail that doesn't bear their signature[1] "as if" it came straight from one of their domains, that can be abused by the crooks.
When (not if!) that abuse happens, the forces behind DMARC will come back and say "Ooooohhhh no! You can't do THAT!" And they (the PHBs, I mean) will break the system again ... and again ... and again.
So, unfortunately, I think there is *no* fix based on presentation. The only real fix is users who are sophisticated enough to avoid spammers, which can't be perfect (some people just aren't, and everybody slips occasionally), but can certainly be enhanced by better filters.
Well, there's that other fix, the one that involves lists as we love them joining the dinosaurs. :-(
All-hail-Dave-Hayes-and-the-AI-newsreader!-ly y'rs,
Footnotes: [1] Any list that isn't a pure address exploder will be unable to maintain the signature.
Arguably, the correct response to DMARC filtering _should_ be the MIME encapsulation of list mail, with appropriate RFC 2369 headers added to the enclosing MIME structure leaving the content un-munged, with all information from the original poster intact. Arguably, MUAs should be transparent to this. Arguably, this would have been the best design for the operation of mailing lists in email-space from the git-go.
Unfortunately, this argument falls over when you note that spammers and phishers can encapsulate their paypal.com phishes and add list headers, too.
The correct response is either for senders to stop publishing DMARC policies that don't match the way their users use mail (fat chance), or for recipient systems to skip the DMARC checks on mail from sources that are known to send mail that recipients want but that doesn't match DMARC's narrow authentication model, e.g., mailing lists and the Wall Street Journal's mail-an-article button.
Failing that, all we have left is hacks, none of which are satisfactory.
R's, John
On 5/9/14, 10:13 PM, John Levine wrote:
Arguably, the correct response to DMARC filtering _should_ be the MIME encapsulation of list mail, with appropriate RFC 2369 headers added to the enclosing MIME structure leaving the content un-munged, with all information from the original poster intact. Arguably, MUAs should be transparent to this. Arguably, this would have been the best design for the operation of mailing lists in email-space from the git-go. Unfortunately, this argument falls over when you note that spammers and phishers can encapsulate their paypal.com phishes and add list headers, too.
The correct response is either for senders to stop publishing DMARC policies that don't match the way their users use mail (fat chance), or for recipient systems to skip the DMARC checks on mail from sources that are known to send mail that recipients want but that doesn't match DMARC's narrow authentication model, e.g., mailing lists and the Wall Street Journal's mail-an-article button.
Failing that, all we have left is hacks, none of which are satisfactory.
R's, John
But the wrapped message could pass the DMARC DKIM signature check, if it will exactly matchs the message that came from Yahoo/AOL. (which the phish won't). This says that the List Headers, modified subject, list headers and footers should be added to the wrapping message, not the wrapped message, which also says that the MUA shouldn't throw this away, but combine these with the original message (but in a way that makes it clear which is which).
-- Richard Damon
Richard Damon writes:
On 5/9/14, 10:13 PM, John Levine wrote:
The correct response is either for senders to stop publishing DMARC policies that don't match the way their users use mail (fat chance), or for recipient systems to skip the DMARC checks on mail from sources that are known to send mail that recipients want but that doesn't match DMARC's narrow authentication model, e.g., mailing lists and the Wall Street Journal's mail-an-article button.
GMail is already doing this, although we don't know the algorithm precisely. If GMail continues and others join, ostracism of providers who continue to use inflexible bouncing policies instead of smart filters becomes more plausible.
I know that's not satisfactory for people whose lists are populated by AOL and Yahoo users, but I don't know what to say to them. Their users are DoS'ing their mailing lists with their addresses, even if they don't know it.
But the wrapped message could pass the DMARC DKIM signature check, if it will exactly matchs the message that came from Yahoo/AOL. (which the phish won't). This says that the List Headers, modified subject, list headers and footers should be added to the wrapping message, not the wrapped message, which also says that the MUA shouldn't throw this away, but combine these with the original message (but in a way that makes it clear which is which).
Sure (and that is what I intended when I suggested wrapping in the first place), but (a) MUAs don't support DMARC yet, and all the signs say that the yahoos will deliberately delay implementing MUAs that do, and (b) many MUAs don't support wrapped messages well at all.
As John put it,
Failing that, all we have left is hacks, none of which are satisfactory.
We'll see how the on-going talks at the IETF go. Some results should be forthcoming "shortly" (that's hearsay, and I can't say any more because that's exactly what I was told by a source close to the center of the process).
On 10 May 2014, at 1:17 pm, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
and (b) many MUAs don't support wrapped messages well at all.
Could somebody please explain this wrapping that's been mentioned a lot? Is this something like "forward as attachment"?
Peter Shute
On 05/10/2014 01:48 AM, Peter Shute wrote:
Could somebody please explain this wrapping that's been mentioned a lot? Is this something like "forward as attachment"?
It's exactly like forward as attachment.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11 May 2014, at 12:39 am, "Mark Sapiro" <mark@msapiro.net> wrote:
On 05/10/2014 01:48 AM, Peter Shute wrote:
Could somebody please explain this wrapping that's been mentioned a lot? Is this something like "forward as attachment"?
It's exactly like forward as attachment.
Thanks for clarifying that. Someone pointed out that their iPhone handles it ok, but that it's an extra step to open the wrapped message. Same on my iPad, but the wrapped message then opens in an iPhone sized window. Better than nothing, but very inconvenient for long messages.
Outlook handles it ok once you open the attached message, but I had an unfortunate experience with one of these. A moderator forwarded a moderation alert to a separate moderators' list for discussion among moderators. I open the attached list message, then forgot there'd been that extra step and did a Reply All with a moderation comment querying the sender's motives, which then went to the main list instead of the moderators. I spent the rest of the evening explaining myself.
I think there's too much room for similar kinds of confusion with wrapping. I'd rather use forwarding as a quoted message, with an explanation at the top. This is what we're doing manually for our Yahoo and AOL users now, and people seem ok with it.
Peter Shute
On 05/10/2014 02:07 PM, Peter Shute wrote:
Thanks for clarifying that. Someone pointed out that their iPhone handles it ok, but that it's an extra step to open the wrapped message. Same on my iPad, but the wrapped message then opens in an iPhone sized window. Better than nothing, but very inconvenient for long messages.
When on April 6 I enabled Wrap Message on my lists I started to get these kinds of complaints. Some who had this experience were OK with adapting to it, but others didn't want it so after a day or so with Wrap Message, I went to Munge From. Now I see occasional misdirected replies from users whose MUAs apparently don't always handle the Reply-To: correctly
In fact, my own Apple Mail 4.2 on OS X 10.6 is randomly broken. I don't use this MUA regularly, but I have it on one machine, and when I look at messages from my lists, it will sometimes display the Reply-To: header and 'reply' will be addressed to the addresses therein. Other times, it will not display the Reply-To: header and 'reply' will be addressed to the address in the From: header, yet if I examine the raw headers, the Reply-To: is there and I have so far not been able to determine what the difference is that makes it fail to 'see' the header sometimes.
Bottom line - all these mitigations have issues for reading, replying or both.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/09/2014 07:27 PM, Richard Damon wrote:
But the wrapped message could pass the DMARC DKIM signature check, if it will exactly matchs the message that came from Yahoo/AOL. (which the phish won't). This says that the List Headers, modified subject, list headers and footers should be added to the wrapping message, not the wrapped message, which also says that the MUA shouldn't throw this away, but combine these with the original message (but in a way that makes it clear which is which).
Just for the record, this is how the Wrap Message action is implemented in Mailman. I.e. all the stuff Richard mentions is done to the outer message, not to the message/rfc822 part that is the original message. The one exception that will break DKIM is content filtering which by necessity is applied to the original message before it's wrapped. This is a big one, because I suspect almost all messages from Yahoo users are multipart/alternative to begin with (and has anyone else noticed what a horrible job Yahoo does in making the text/plain alternative, but I digress ...), and many lists collapse alternatives so the DKIM sig will be broken.
That notwithstanding, as Stephen and others have mentioned, the MUAs to deal with this are not here and are unlikely to be here anytime soon.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, 2014-05-10 at 02:13 +0000, John Levine wrote:
Arguably, the correct response to DMARC filtering _should_ be the MIME encapsulation of list mail, with appropriate RFC 2369 headers added to the enclosing MIME structure leaving the content un-munged, with all information from the original poster intact. Arguably, MUAs should be transparent to this. Arguably, this would have been the best design for the operation of mailing lists in email-space from the git-go.
Unfortunately, this argument falls over when you note that spammers and phishers can encapsulate their paypal.com phishes and add list headers, too.
There are no headers, except the local mail server's Received header, that can't be spoofed. The point here is that currently the response of MUAs to Content-Type: multipart/mixed or Content-Type: multipart/rfc822 varies widely, so that the Mailman option of "Wrap Message" becomes problematic for some recipients. My iPhone deals with it, but opening the actual content containing the list post requires an extra step. Some MUAs, such as Evolution, open the contained email by default, presenting a double set of headers - those of the wrapper and those of the contents. What I'm suggesting is functionally, from the point of view of DMARC, identical to Mailman's current "Wrap Message" method, but would standardize the way contents are presented, eliminating a bit of the geekyness which some MUAs require.
Stephen's point, and yours too as I take it, is that making it too easy and transparent to read wrapped email essentially legitimizes the social misbehavior of ESPs which have implemented DMARC, and Stephen has underlined the pie-in-the-sky-ness of the idea by pointing out that it will be a sub-zero day in Hell when the authors of MUAs would go along with such an idea.
As an email system administrator I've watched a lot of misbegotten attempts on the part of a lot of ESPs to fight spam and phishing, many of them breaking email (and email RFCs) in the process. It seems to me though, that adding a largely transparent extra layer of MIME encapsulation to email, with a predictable result on the MUA end, would be a good point from which to start. It's analogous to double-boxing delicate items when shipping them.
The correct response is either for senders to stop publishing DMARC policies that don't match the way their users use mail (fat chance), or for recipient systems to skip the DMARC checks on mail from sources that are known to send mail that recipients want but that doesn't match DMARC's narrow authentication model, e.g., mailing lists and the Wall Street Journal's mail-an-article button.
Agreed, but in the Real World spam and phishing have been persistent problems since the early 90s. The fundamental problem is the shifting of the cost of email onto the receiver, and no method yet devised has successfully dealt with it. The original email protocols, starting with RFC 822, didn't anticipate it. It is quite arguable that DMARC contains some sensible and effective elements to address the issue of email authentication, which is ultimately where the whole process needs to go. This is quite apart from the ham-handed attack on network neutrality demonstrated by the present implementation of it by Yahoo and AOL. As Stephen (I think) pointed out, the use of DMARC by PayPal and a number of banks is the correct way that it should be used. As you rightly point out, recipient systems should skip the DMARC checks on mail from sources that are known to send mail that recipients want but that doesn't match DMARC's narrow authentication model. Nonetheless a narrow authentication model of some sort may be what's needed here.
-- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com |
Lindsay Haisley writes:
Stephen's point, and yours too as I take it, is that making it too easy and transparent to read wrapped email essentially legitimizes the social misbehavior of ESPs which have implemented DMARC, and Stephen has underlined the pie-in-the-sky-ness of the idea by pointing out that it will be a sub-zero day in Hell when the authors of MUAs would go along with such an idea.
Legitimizing the ESPs does bother me, but that wasn't my point, and I think it probably wasn't John's either. My point was that anything that makes it appear that a message whose DKIM signature doesn't verify is actually from the user assigned that address can be emulated by spammers. That will cause these same ESPs to try to prevent that from happening, and the cycle starts again.
The MUA authors (at least the MUAs I use ;-) will catch up by the way. Some already do, actually.
On Sun, 2014-05-11 at 18:28 +0900, Stephen J. Turnbull wrote:
Legitimizing the ESPs does bother me, but that wasn't my point, and I think it probably wasn't John's either. My point was that anything that makes it appear that a message whose DKIM signature doesn't verify is actually from the user assigned that address can be emulated by spammers. That will cause these same ESPs to try to prevent that from happening, and the cycle starts again.
That's true, but what I suggested is, or should be simpler than that, and is fundamentally no different from what Mailman does now with wrapping. The points are:
implementation of a MIME multipart standard _specifically_ for the purpose of wrapping emails would be a good idea. The result would be not so much the visual elimination of information from a message but a standardization of MUA presentation so as to avoid putting off non-tech people. No effort need be made to make a message appear that it's something that it's not, or to disguise the presence or absence of DKIM verification.
the observation that such encapsulation, and a uniform way of handling and presenting it, may be a way forward if ANY methods of sender authentication become widespread, which IMHO is likely. If the industry requires standards and the IETF standards process doesn't keep up, the result is, and always has been, fragmented standards (a.k.a. no standards). Case in point: the non-standardization of HTML enhancement for email.
The MUA authors (at least the MUAs I use ;-) will catch up by the way. Some already do, actually.
I would guess that Xemacs VM isn't for the technologically faint of heart ;)
-- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com |
On 05/11/2014 11:24 AM, Lindsay Haisley wrote:
That's true, but what I suggested is, or should be simpler than that, and is fundamentally no different from what Mailman does now with wrapping. The points are:
- implementation of a MIME multipart standard _specifically_ for the purpose of wrapping emails would be a good idea.
Why multipart?
See <https://mail.python.org/pipermail/mailman-users/2014-May/076736.html>.
Granted, a decorated message with msg_header and/or msg_footer added is multipart, if these decorations are omitted, you are left with a multipart message with a single sub-part. While this is still RFC compliant (recursion is wonderful), it is (to me at least) esthetically unpleasing.
Perhaps a new Content-Type such as message/wrapped, although that raises the question of what to do when a message/wrapped part is not at the top level.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
it is (to me at least) esthetically unpleasing.
"There's no arguing with taste", but as a data point I have no problem with multipart/mixed with only one part.
How about multipart/alternative:
message header
multipart/alternative
part header
message/rfc822 # original message in all its glory
part header
<traditional cooked list message>> Perhaps a new Content-Type such as message/wrapped
AFAICS this is completely unnecessary?
message header
Content-Type: message/rfc822
original message header
original message body # or cooked if you preferThen amend the existing MIME RFCs to say that MUAs SHOULD (MAY?) simply display the original message in some appropriate way. No?
On 05/12/2014 01:25 AM, Stephen J. Turnbull wrote:
How about multipart/alternative:
message header multipart/alternative part header message/rfc822 # original message in all its glory part header <traditional cooked list message>
Interesting idea, but I think the part order is reversed. The simplest, most universally readable part is supposed to be first with parts of increasing complexity coming later.
Perhaps a new Content-Type such as message/wrapped
AFAICS this is completely unnecessary?
message header Content-Type: message/rfc822 original message header original message body # or cooked if you prefer
Which is essentially what the Wrap Message action does now.
Then amend the existing MIME RFCs to say that MUAs SHOULD (MAY?) simply display the original message in some appropriate way. No?
I really wonder if that would help. Section 5.2 of RFC 2046 doesn't say exactly that, but it does contain this note:
NOTE: It has been suggested that subtypes of "message" might be defined for forwarded or rejected messages. However, forwarded and rejected messages can be handled as multipart messages in which the first part contains any control or descriptive information, and a second part, of type "message/rfc822", is the forwarded or rejected message. Composing rejection and forwarding messages in this manner will preserve the type information on the original message and allow it to be correctly presented to the recipient, and hence is strongly encouraged.
A couple of things are significant in that. It basically agrees with Stephen that message/wrapped is unnecessary, but it also says the message/rfc822 type "will preserve the type information on the original message and allow it to be correctly presented to the recipient".
While this doesn't explicitly say MUAs SHOULD or MAY simply display the original message in some appropriate way, it certainly conveys that sentiment to me, yet here we are over 17 years later with apparently some mainstream MUAs that don't do that.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
On 05/12/2014 01:25 AM, Stephen J. Turnbull wrote:
How about multipart/alternative:
message header multipart/alternative part header message/rfc822 # original message in all its glory part header <traditional cooked list message>Interesting idea, but I think the part order is reversed. The simplest, most universally readable part is supposed to be first with parts of increasing complexity coming later.
That's precisely the point. Most MUAs choose to display the *last* form that they understand, but there's no guarantee that they'll understand earlier ones, so they should (but see below) keep trying.
As Bugs Bunny says, "Eh-he-he-eh, ain' I a stinka?!" ;-)
Then amend the existing MIME RFCs to say that MUAs SHOULD (MAY?) simply display the original message in some appropriate way. No?
I really wonder if that would help. Section 5.2 of RFC 2046 [...]. While this doesn't explicitly say MUAs SHOULD or MAY simply display the original message in some appropriate way, it certainly conveys that sentiment to me, yet here we are over 17 years later with apparently some mainstream MUAs that don't do that.
I know, but what can we do? There are very few of us who could get away with telling our subscribers, "well, then, get a *real* MUA!!", and even fewer who can do that, and want to.
participants (13)
-
Barry Warsaw -
Dave Nathanson -
Glenn Sieb -
Jim Popovitch -
John Levine -
Joseph Brennan -
Lindsay Haisley -
Mark Sapiro -
Peter Shute -
Richard Damon -
Russell Clemings -
Stephen J. Turnbull -
Stephen J. Turnbull