Re: [Mailman-Developers] Author_is_list option in upcoming mailman 2.1.16
On Sep 17, 2013, at 6:21 PM, Mark Sapiro mark@msapiro.net wrote:
On 09/17/2013 05:28 PM, Barry Warsaw wrote:
On Sep 15, 2013, at 08:24 AM, Mark Sapiro wrote:
Because the issue remains controversial, I will soon release 2.1.16 final with the feature disabled by default, and will consider the message encapsulation approach or other possibilities based on experience with 2.1.16 for a 2.1.17 release perhaps early next year.
This makes sense to me, although I would label the feature "provisional" or "experimental". There just isn't any good experience here we can draw on, so it seems reasonable to provide the knobs that will allow motivated folks to gather such experience, but generally keep it out of the way for the majority of users.
I intend to so indicate in the NEWS about the feature.
I have given some thought to the encapsulation approach and have some questions about it.
My thoughts on how to do it include the following if the feature is enabled.
CookHeaders saves the original Subject: before prefixing in the metadata.
A new handler before ToOutgoing but after ToDigest, ToArchive and ToUsenet creates a new message From: the list with Content-Type: message/rfc822, a unique Message-ID: and Subject:, References: and In-Reply-To: copied from the current message. It then replaces the Subject: in the current message with that saved in the metadata and sets the payload of the new message to the current message.
This will result in a single-part message with a payload of the content filtered original message. If content filtering hasn't removed anything, the original message's DKIM signatures if any should still be valid.
The message then goes to ToOutgoing and eventually OutgoingRunner and SMTPDirect which will call Decorate and if any msg_header or msg_footer is added, these will be added as is currently done which will result in the message becoming multipart/mixed with the addition of one or two text/plain, Content-Disposition: inline parts containing the header and/or footer.
The idea is the message/rfc822 part preserves as much of the original as possible so its DKIM sigs if any may still validate and to put enough into the headers of the new message so MUAs can still thread it properly and render it nicely. Also, the message is unchanged over current behavior for digests, archives and usenet.
The sticky questions are what to do with the original From: and Reply-To: in the face of possible Reply-To: munging options and so on. Presumably, we want to still be able to reply to the original author, and preserve the behavior of a simple MUA 'reply' going to the original author and not the list in the absence of Reply-To: munging, and that could be accomplished by putting the original author's Reply-To: (or the original From: if no original Reply-To:) in the new message's Reply-To:, but it's not yet clear to me how to handle the munging options (maybe just ignore them ;).
I could actually implement this approach for the 2.1.16 release, but that would negate the i18n work that's already been done as the descriptive string on General Options would change, so I'm more inclined to label this feature as experimental and subject to change significantly in a future release.
- If you keep the From: header as it is then, we will still have the same problems
- the purpose of the encapsulation message/rfc822 is not meant to preserve the DKIM signature, DKIM is not meant to be verified by MUAs
- encapsulation is not there to provide a transitive trust, there is another method explored for that which is Original-Authentication-Results (OAR). There is an expired Internet Draft on it.
I think, it is risky to code this encapsulation method directly and now, it requires a branch some testing and then merging back into the main branch.
The author_is_list has had deployment and testing for over a year in a DMARC environment. Limited testing I agree but nevertheless proved not to break things ( besides people ideal view of email ;) ) and be useable with many MTAs and especially MUAs
Here is a recent test, deployment and analysis: http://sys4.de/en/blog/2013/08/11/mailman-dmarc-konform-betreiben/
I'd like to see somebody operating a mailing list with this encapsulation method first, before merging.
On 09/17/2013 10:04 PM, Franck Martin wrote:
- If you keep the From: header as it is then, we will still have the same problems
Perhaps I wasn't clear. The From: of the outer message would contain the list address. The From: of its message/rfc822 payload would be unchanged from that of the incoming message.
My questions only had to do with the Reply-To: of the outer message in cases where the list was set to mung Reply-To: in some way.
- the purpose of the encapsulation message/rfc822 is not meant to preserve the DKIM signature, DKIM is not meant to be verified by MUAs
No, but that is a side effect, at least where content filtering has not altered the message.
Franck Martin writes:
I think, it is risky to code this encapsulation method directly and now, it requires a branch some testing and then merging back into the main branch.
First, the risk is zero, except to volunteers, as long as it's not default.
Second, it's been tested for decades. A MIME digest is nothing but one or more encapsulated message/rfc822 parts. In multipart/digest form, it has an obvious defect of requiring an extra click for readers using the MUAs I refuse to use (the MUAs I use can be taught to explode digests automatically, or read them as mini-folders). It would be nice to find alternative ways to accomplish the same goals, but this is already proof of concept.
Third, it has the advantage of preserving as much or as little of the original message as the list would like without interfering with DKIM validation of the encapsulation.
The author_is_list has had deployment and testing for over a year in a DMARC environment. Limited testing I agree but nevertheless proved
Limited testing is not "proof" that something works. Limited testing can only *prove* that something is *broken*. More extensive testing is still not *proof*, but it can give you confidence that it's not *too* broken.
Here is a recent test, deployment and analysis: http://sys4.de/en/blog/2013/08/11/mailman-dmarc-konform-betreiben/
I don't read German, but I don't see anything that looks like data, nor is there room for "analysis." Nor does the blog by Patrick Koettner referenced therein. (The Google translations confirm that.) Please show us something that looks like data and analysis. Specifically of interest:
Number of lists, number of users on each list (min, mean, max
would do), duration of operation in this mode, type of users (mail
admins vs. general technical vs non-technical), the MUAs in use,
any discussion from the users themselves.
I'd like to see somebody operating a mailing list with this encapsulation method first, before merging.
Any list with MIME digests enabled and in use is a test of the basic usability. Do so on a site with DKIM-signing and you're done. All my proposal does is tune the encapsulation a bit. It might or might not work.
On Sep 18, 2013, at 05:04 PM, Stephen J. Turnbull wrote:
I don't read German, but I don't see anything that looks like data, nor is there room for "analysis." Nor does the blog by Patrick Koettner referenced therein. (The Google translations confirm that.) Please show us something that looks like data and analysis. Specifically of interest:
Number of lists, number of users on each list (min, mean, max would do), duration of operation in this mode, type of users (mail admins vs. general technical vs non-technical), the MUAs in use, any discussion from the users themselves.
Indeed. I'm skeptical about how well encapsulation will go over with end-users who have no understanding about these issues (nor should they).
End users just care about how the email looks in their mail readers. I'm concerned that this will be a nice, RFC-compliant feature that makes things easy and workable for all the automated systems involved, but will look horrible to end-users and just make them upset. If that's the case then IMHO, it a failure.
OTOH, maybe we won't know for sure until it gets *a lot* more testing. But I think it's a mistake to say "well, we just have to force MUA developers to catch up". As we've seen with something presumably as simple as reply-to-list, it (almost) never happens.
-Barry
On 09/22/2013 02:12 PM, Barry Warsaw wrote:
OTOH, maybe we won't know for sure until it gets *a lot* more testing. But I think it's a mistake to say "well, we just have to force MUA developers to catch up". As we've seen with something presumably as simple as reply-to-list, it (almost) never happens.
I think what we already know for sure is that people will continue to use their favorite, brain dead MUA regardless of how well or how often we point out the better choices. I think we have some limited influence, but we are definitely not the 600 lb. gorilla in this jungle.
Because of that and because of the need for better real world data, I am convinced that I need to release 2.1.16 final with the options of going with simple From: munging (as in RC2) or encapsulation and a default of neither.
To that end, I will work on getting it out as soon as I can and will definitely describe the feature as experimental and subject to change in future releases, but encourage people to try it and report so their experience can influence future direction.
OT - I just registered for PyCon. I haven't been to Montreal since 1975. I'm committed. I'm stoked!
On Sun, Sep 22, 2013 at 2:12 PM, Barry Warsaw barry@list.org wrote:
End users just care about how the email looks in their mail readers. I'm concerned that this will be a nice, RFC-compliant feature that makes things easy and workable for all the automated systems involved, but will look horrible to end-users and just make them upset. If that's the case then IMHO, it a failure.
OTOH, maybe we won't know for sure until it gets *a lot* more testing. But I think it's a mistake to say "well, we just have to force MUA developers to catch up". As we've seen with something presumably as simple as reply-to-list, it (almost) never happens.
We as developers and standards people often avoid engaging UI people (MUAs, in our case) and issues specifically because it's a space that doesn't follow rules, which is what we're used to. That partition allows us to be able to declare victory on our side of the line most of the time, but it leaves us with the frustrations you've described here.
I wonder how long we can hold out before we start trying to drag them into our conversations, which might be the only way to solve these pain points long term. It seems to me that Gmail, Yahoo Mail, Thunderbird, etc., must have either a team or an individual that spends time thinking about and testing user-visible solutions to these problems, so perhaps it's time to start asking them for help.
-MSK
participants (5)
-
Barry Warsaw -
Franck Martin -
Mark Sapiro -
Murray S. Kucherawy -
Stephen J. Turnbull