Hi Steve,
your observations put me on the right track. Thank you so much!
Long post below:
On Thu 24/Sep/2020 12:31:29 +0200 Stephen J. Turnbull wrote:
Alessandro Vesely writes:
First, what Mailman are you talking about? Only Mailman 3 is likely to get these improvements, as Mailman 2 is end-of-life. However, Mailman 2 installations are likely to be around in large numbers for several years, and if Mailman 2 is any evidence, likely few Mailman 3 installations would use these features unless forced to by a disaster like the Yahoo/AOL sudden switch to DMARC p=reject.
Reversing transformation should work with any Mailman version, and with other mailing list managers as well. Hence, one cannot rely on some precise indications, but rather on common MLM behavior.
Yet, it is possible to undo the transformation that Mailman put in place, thereby validating the original DKIM signature. > It would always be possible to undo all transformations by supplying the original email as part of a multipart/alternative, or perhaps a new multipart subtype, maybe with some kind of device to make reading the message/rfc822 original difficult in standard MUAs.
Comparing to the original can make it more difficult to check that the transformed version does not deviate from the original in unacceptable ways. While limiting the cases where original signatures can be recovered, a set of accepted transformations also limits the attack surface.
(In the case of Microsoft MUAs, if Mailman is configured to strip HTML, the result might be less than 10% bigger than the original! ;-)
Even if plain text is safer, stripping HTML is irreversible.
It is sometimes possible to reverse transformations with only the information in the post after Mailman processing. However, some very desirable changes are destructive (eg, anonymized lists, conversion of HTML to plain text, removal of prohibitive attachments). Some non-destructive changes (headers and footers) are highly customizable. So the question is what are the transformations that users want to reverse, and whether that's really possible.
It has to be the responsibility of the list owner to configure Mailman so that the transformations can be reversed. Some options, like anonymized lists, are clearly at odds with the need to recover the original author domain's signature.
This kind of transformation reversal probably requires no changes to Mailman, just an addition of a Handler which could be written independently and "dropped in" (with a configuration change to the default pipeline).
On the other hand, a new header field can be abused. List-*, for example, are often found in a number of messages which don't come from mailing lists, but just aim at not being classified as UCE.
The necessary information about transformations that are configured would be available from Mailman in the usual way (existing Handlers need that information).
There could be more info than just the confguration, see the heuristic below.
Mailman carries out some irreversible changes, such as rewriting To: or Cc: changing the order of the mailboxes,
Does this happen outside of DMARC mitigation? Can you show examples?
I checked a few messages and couldn't find a switched To:. Switched Cc: seems to happen when one of the recipients is the list itself, which is then moved to the last place. (I try to reproduce that behavior with this message.)
or rewriting Content-Transfer-Encoding: irrespective of quotation marks and case (for example "7bit" even if the original, signed field was spelled as "7Bit"). > I'm not aware of such behavior *unless other modifications were done*. In that case, Mailman is specifying the C-T-E it uses, it is not rewriting the original C-T-E.
I don't know if it's Mailman or a DKIM signing tool running afterwards, but many plain text messages from mailing lists come rendered as base64. Since the footer is part of the only MIME entity, the reversal has to decode base 64, remove the footer, and re-encode as base64 if that was the original C-T-E. In the latter case, one needs to know the column width of the original encoding...
I guess this behavior is coded deeply in Python libraries,
I don't think so. As far as I know, the email module in Python 3 provides some support for parsing header fields but I don't know why this would change order or spelling of field contents.
Aha, you're right. It is probably Mailman writing its own C-T-E, which happens to be the same as the original, albeit spelt differently.
I would guess that to the extent that it happens it has to do with Mailman-level processing (for example, collecting addresses from the same domain so they can be presented as multiple RCPT TO with a single DATA).
No, the MTA doesn't care about header addresses. Domain optimization has to be done after MX lookup.
I can say for sure that some care was taken to ensure that the order of header fields, including multiple instances of the same field is carefully preserved.
Good news. In any case, DKIM header canonicalization has to be "relaxed", because fields are re-wrapped.
but would like to know developers' opinions. Is that something that could be fixed? > First, the issues with headers could be improved, though not entirely fixed, in DKIM itself by further canonicalizing structured headers before signing or verifying. > I'm not saying that this is the right way forward, but it should be considered.
There have been various proposals about a MIME-compatible DKIM. It's not going to happen any time soon. There's not enough traction.
The second question is about producing a hint to the verifier telling which transformation(s) have been applied to the message. That would come as an additional header field, for example: >> DKIM-Transform: footer
This could be done easily, but it would be at best a hint. Among other things, it might be desirable to identify the agent that performs the transformation, as well as the algorithm and perhaps the host and/or the list. Mailman adds footers in different ways, specifically appending text and adding a MIME part. Third party patches are available that dig into HTML structure (at least for Mailman 2).
DKIM-Transform would ease the reversing filter's job greatly. I wrote my prototype relaying on it, then I wrote a bash script to add the hint. The script was easy only because it /knew/ that a footer was added one way or the other.
There are lists that feed into lists, and apply their own transformations.
IMHO, transformation reversing must not be stackable. That is, no attempt to recover a middle mediator's signature. If multiple footers or multiple subject tags are added, reversibility is lost.
or as an extra tag in a DKIM signature, for example:
DKIM-Signature: v=1; (...) tf=footer; (...)Not possible without a lot of effort and specific cooperation from MTAs. Mailman doesn't DKIM sign messages, really doesn't want to (there are Python modules for this, but use and configuration would be our responsibility so we'd like to have specialists do it), and probably shouldn't (we're not specialists) -- that should be left to the border MTA of the administrative domain.
Thank you for confirming that.
That hint could spare the verifier one pass over the message. Is it something that could be implemented? If not, I'd try guessing, according to this scheme: > You're going to have to guess a lot for a long time anyway, because very few installations will implement this header. It's not obvious to me that guessing won't be nearly as accurate as the header might be.
That's a key observation!
outermost Content-Type: | first entity Content-Type: | transformation | ------------------------+-----------------------------+-----------------+ text/plain | any | footer | ------------------------+-----------------------------+-----------------+ multipart/mixed | multipart/mixed | add-part | ------------------------+-----------------------------+-----------------+ multipart/mixed + any other | mime-wrap | ------------------------+-----------------------------+-----------------+ any other | any | non-reversible | ------------------------+-----------------------------+-----------------+
Does that look correct?
Not 100%. I'm not sure what you mean by "mime-wrap", but if it's Mailman's "Wrap Message" DMARC mitigation, as far as I know nobody uses it.
No, those are in the initial set of transformations in the draft i cited.
*mime-wrap* is when the original message was, say, multipart/alternative, as in an HTML message with plain text equivalent. In that case, Mailman creates a new overall part with two entities. The first is the original body of the message, the second the added footer.
*add-part* is when the original message was already multipart/mixed, as in the case of an attachment. Mailman keeps the existing structure and adds a part at the bottom, with the footer.
The mime-wrap case is so easy to reverse that it took me a good deal of time to realize that I cannot take advantage of it.
So, the kind of transformation says a bit more than the list configuration. How a footer is added depends on the message at hand.
I suspect that pretty much any multipart/mixed may have an added part containing a footer, but it might not.
Here's the *heuristic* I came up with (not yet implemented):
First of all, find out a purported author domain, in any of From:, Original-From:, or Reply-To:. Check if that domain differs from the one in the Sender:. Then check if there is a failed DKIM signature by the purported author domain. In that case, try and reverse the transformation.
For the header, if the From: was rewritten, put it back. If the Subject: starts with a bracket, remove it. If any Original-* (or whatever) are present, replace them.
For the body, look for a line consisting of underscores or dashes. Check it is in a text/plain MIME entity, either its own entity or the whole body. Check that line is not followed by more than, say, 10 lines of text at the end of the message. If found, remove it as per the table above.
I think that would work for most of the mailing lists I'm subscribed to, for "mild" DKIM signers. "Hard" DKIM signers, e.g. those who sign Sender:, will have to adjust Original-* fields by trial and error.
At this point, I'm not so sure that specifying a DKIM-Transform: header field to ease transformation reversal is a good idea. Obviously, it would apply to future transformation only. In the future we'll have more powerful machines, so if the only advantage is to gain some efficiency, it becomes questionable.
Currently, there are mailing lists which don't do any change, not even subject tags, in order to avoid breaking DKIM signatures. A somewhat Procrustean solution. > It's the ONLY guaranteed solution, though, because avoiding rewriting is only possible if you *know* that you're distributing modified posts only to sites participating in your reversible modifications protocol (or ignore DMARC p=reject).
Exactly.
I don't think From: rewriting is going to be disabled any time soon. > You're right. You need universal deployment of reverse transformation to make disabling rewriting palatable. > Reply-To: usually comes after From:, thereby requiring to go back to change already parsed fields. > That's not a problem, since DKIM requires reordering fields anyway. The expensive part is not fiddling with the header, it's multiple passes of the signature algorithm.
Good point!
Knowing the kind of transformation beforehand can save one pass through the message.
As an alternative, I'd provide for yet another field to be put near the top of the header. > It's not an alternative. The changes to Reply-To or Cc are *necessary* (in the opinion of the list admin, not Mailman) to preserve the ability of the recipient MTA to respond to author.
The goal is different. Transformation reversal recognizes the original signature, thereby affecting the aggregate reports that senders receive. According to DMARK marketing, the latter might influence their decision to switch to a strict DMARC policy.
As a side effect, when the MDA sees that an Original-From: was authenticated, it can restore it in place of From:, after any external forwarding but before storing the message. That way, users recover the ability to reply privately to the author.
Original-From:, say. This may seem redundant, however it serves a different goal. In addition, if the Original-From: is put in place by the original signer, it ratifies its knowledge that From: will be rewritten and its willingness to recover it afterwards. > Could work, but addition of Original-From should be done by DMARC originators, not by Mailman.
Yes.
The name should probably be DMARC-Original-From, as well.
Yeah, or DKIM-Original-From, or whatever. It's important that it be new. I sometimes see X-Original-From:, which the new field shouldn't be conflated with. (Or should it?)
Note that, as mentioned above, if the author domain encodes a plain text message in base64, it should also add something like:
Original-Content-Transfer-Encodig: base64; column-width=76Is this endeavor completely useless, given that the current settings work well enough? Or could it help keeping a consistent DMARC semantics among participants yearning to do so? I'd be glad to hear your opinions... > I don't think it's useless, but I don't see any reason for Mailman to participate until there's a (1) specification of transformations that people want to be reversible, or (2) specific defects that if fixed, or (3) features that if added, would enable reversibility. > For (1), we would just guarantee a particular recognizable format for transformations that should be reversible, and (2) and (3) would be addressed as usual. > As mentioned, the hinting function can be done well-enough by a user-supplied Handler that looks up the list's configuration, determines the transformations that are applied, and inserting the hints in the appropriate place.
The draft I cited provided for a IANA registry containing the set of reversible transformation accepted. I'll try and propose an alternative specification, based on the above heuristic (when I'll have implemented it). The question of a DKIM-Transform: header field and a registry of its possible content should probably be discussed at that time.
One case I can think of is a list with an HTML footer. It may use an <hr> instead of many underscores. How would that heuristically be discovered? Adding an entry in a registry would be a way to add new possibilities. On the other hand, HTML presents a wider attack surface. Section 8.2 of DKIM still deprecates using l=, as it can be used for "exploiting lax HTML parsing in the MUA" in order for "the appended content to completely replace the original content in the end recipient's eyes".
Finally, contrary to what we all wish were true, this is not really a choice for mailing lists. It's a choice for recipient ad-dom border MTAs. If they don't buy in in large numbers, I'm not particularly interested in doing the work. I don't see why they would.
IME, MTAs are much more interested in DKIM signing than verifying. Anyway, sooner or later I'll release my filter with reverting capabilities. Albeit it is not widely used, a site can at least recognize their own signatures when messages come back from a list.
Most lists I participate in do things like strip large attachments and strip prohibited executables. I think those are very common in general. Most lists I participate in also strip text/html alternatives and many convert text/html to text/plain. If that's the common case, why would a postmaster bother (unless they're a DMARC purist, of course, which may be a good thing but I don't think there are very many of them)?
That is the list's choice.
If the postmasters aren't bothering, why would list admins? If the list admins aren't bothering, why should Mailman?
Perhaps it's me, but I feel a bit of reluctance in From: rewriting. It /has/ to be done, so savvy list admins do it. However, nobody seems to like it. If that feeling is correct, a little bit of adoption should show up...
On the other hand, I do think that Mailman can and should enable better analytics on posts by ensuring that we only change parts of a message that we intentionally change. I guess the might include situations like the case where Mailman changes a MIME part, reassembles the whole message, and assigns a C-T-E that happens to have the same semantics as the original C-T-E.
Those changes are benign.
I encourage you (or anyone in the reversible transformation effort) to report inadvertant changes as bugs, and to suggest candidates for "standard formats" we could adopt to make ad hoc reversals more reliable (eg, the list name in a Subject tag should be enclosed in square brackets and match the last component of the List-ID -- not clear how that works with internationalized lists though).
We need to experiment. For example, author domains may want to try adding an Original-Subject:.
I can't speak for other developers at this point, so I can't promise any proposals would be implemented, but I'm certainly interested and in some cases would definitely be an "advocate on the inside".
Thank you for your commitment.