Long lines in mail composed by HyperKitty

Hi,
[I prefer discussion be directed to mailman-developers, so Reply-To is set. But if you're not subscribed to -developers, following up here is OK, and I'll eventually summarize to -developers. Just don't followup to both.]
I just noticed that among the agents that send non-conformant long lines[1] is HyperKitty. Nowadays violating the 80 octet limit is common and MUAs mostly handle it, but the 1000 octet limit is enforced by many MTAs[2]
I think that HyperKitty should format mail per RFC 3676 format=flowed https://tools.ietf.org/html/rfc3676#section-4. However, I don't use "modern" (aka crappy HTML-oriented) clients, so I don't know whether they handle format=flowed properly.
Discussion greatly desired.
Steve
Footnotes: [1] RFC 5321: MTAs MAY impose a limit >= 1000 octets including CRLF. RFC 5322: MUST be <= 1000 octets including CRLF, SHOULD be <= 80 octets including CRLF.
[2] In practice, most (?) MTAs just break the line at 998 octets and insert CRLF rather than reject the message. I assume they would break happily in the middle of a multi-byte character, i.e., any non-ASCII in a UTF-8 text.

Is this something new that you’ve noticed?
I think previously, Hyperkitty used to wrap lines at 90 characters when displaying them in UI and then I think the replies would’ve been wrapped at 90 + “ >” (2 chars) per line maximum. I may need to confirm this because we don’t do anything else when sending the message or any recent changes to that part of the code.
But recently, when adding support for rich text rendering, I think I removed that 90 char wrap in the UI, so the original line length is preserved and you might be seeing long lines from lists.mailman3.org or mail.python.org servers which run the git master branches.
Support for format=flowed is good in the web client (Fastmail) and Mac client that I’ve been using.
Just to confirm this, but the way to implement that would be to add format=flowed to the generated email’s content-type header and then add a CRLF after LINE_LENGTH octects, right?
We use EmailMessage1 from Django2, which is itself a wrapper over Message class form standard library. I don’t know if the BytesGenerator supports some sort of policy when serializing the body, but if not, I guess we can handle breaking lines with CRLF before we pass it to Django.
-- thanks, Abhilash Raj (maxking)

Abhilash Raj writes:
Is this something new that you’ve noticed?
I've been noticing long-lines in format=fixed (ie, the default setting) for a while, but hadn't paid attention to the sending clients. For some reason I was rooting around in the headers of this particular message, and saw HyperKitty and that kinda tickled my RFC wonk reflex. :-)
I think previously, Hyperkitty used to wrap lines at 90 characters when displaying them in UI
UI shouldn't have anything to do with the message as formatted for the wire, though. I'm not seeing this in the UI, I'm seeing a message *sent from the HyperKitty UI*, viewed in my usual Emacs-based MUA.
This shouldn't be related to my issue.
Good!
No, basically the idea is to take a message formatted for fixed-length lines, and allow the receiving client to reflow paragraphs as it wants to.
The method is to add "format=flowed" to the generated MIME body's content-type header field, and then append an ASCII space to the end of each line that is part of a flowable paragraph. Each paragraph ends with a fixed line (no trailing space).
Preformatted tables and ASCII art are supported by simply not adding the space. Obviously that is not an easy thing to do for a plain text message body source, but should be easy for Markdown source.
The rules for handling quoted material are non-trivial and not very smart. Eg, the quoting style where the writer's initials appear before the ">" as in "sjt>" is not supported, and will be very ugly if flowed. See the RFC for that. https://tools.ietf.org/html/rfc3676#section-4
I would expect that email.message supports generating format=flowed message bodies, at least the simple case of a series of flowed paragraphs, but I haven't checked.
That's helpful, thanks! Yes, I suspect that we should do our own rendering.
Steve

Is this something new that you’ve noticed?
I think previously, Hyperkitty used to wrap lines at 90 characters when displaying them in UI and then I think the replies would’ve been wrapped at 90 + “ >” (2 chars) per line maximum. I may need to confirm this because we don’t do anything else when sending the message or any recent changes to that part of the code.
But recently, when adding support for rich text rendering, I think I removed that 90 char wrap in the UI, so the original line length is preserved and you might be seeing long lines from lists.mailman3.org or mail.python.org servers which run the git master branches.
Support for format=flowed is good in the web client (Fastmail) and Mac client that I’ve been using.
Just to confirm this, but the way to implement that would be to add format=flowed to the generated email’s content-type header and then add a CRLF after LINE_LENGTH octects, right?
We use EmailMessage1 from Django2, which is itself a wrapper over Message class form standard library. I don’t know if the BytesGenerator supports some sort of policy when serializing the body, but if not, I guess we can handle breaking lines with CRLF before we pass it to Django.
-- thanks, Abhilash Raj (maxking)

Abhilash Raj writes:
Is this something new that you’ve noticed?
I've been noticing long-lines in format=fixed (ie, the default setting) for a while, but hadn't paid attention to the sending clients. For some reason I was rooting around in the headers of this particular message, and saw HyperKitty and that kinda tickled my RFC wonk reflex. :-)
I think previously, Hyperkitty used to wrap lines at 90 characters when displaying them in UI
UI shouldn't have anything to do with the message as formatted for the wire, though. I'm not seeing this in the UI, I'm seeing a message *sent from the HyperKitty UI*, viewed in my usual Emacs-based MUA.
This shouldn't be related to my issue.
Good!
No, basically the idea is to take a message formatted for fixed-length lines, and allow the receiving client to reflow paragraphs as it wants to.
The method is to add "format=flowed" to the generated MIME body's content-type header field, and then append an ASCII space to the end of each line that is part of a flowable paragraph. Each paragraph ends with a fixed line (no trailing space).
Preformatted tables and ASCII art are supported by simply not adding the space. Obviously that is not an easy thing to do for a plain text message body source, but should be easy for Markdown source.
The rules for handling quoted material are non-trivial and not very smart. Eg, the quoting style where the writer's initials appear before the ">" as in "sjt>" is not supported, and will be very ugly if flowed. See the RFC for that. https://tools.ietf.org/html/rfc3676#section-4
I would expect that email.message supports generating format=flowed message bodies, at least the simple case of a series of flowed paragraphs, but I haven't checked.
That's helpful, thanks! Yes, I suspect that we should do our own rendering.
Steve
participants (2)
-
Abhilash Raj
-
Stephen J. Turnbull