[Mailman-Developers] EOL handling in Mailman

[Sylvain Beucler]

On Sat, Mar 19, 2005 at 09:59:46AM -0800, Mark Sapiro wrote:
> Sylvain Beucler wrote:
> >
> >I'm a Savane (gna.org/p/savane) developer, and we noticed that when we
> >send mail with newlines in the \r\n format, Mailman converts them to
> >\n\n.
> >
> >I would like to get your point on what we should do to fix the
> >problem. I had a look at RFC 822; the EOL (end of line) description is
> >not very clear, but it seems that the standard EOL convention is \r\n
> >(CRLF).
> 
> RFC 822 is obsolete. The current standards are RFC 2821 SMTP and RFC
> 2822 Internet Message Format. Both these standards are very clear. For
> example RFC 2821 says:
> 
> <quote>
> 2.3.7 Lines
> 
>     SMTP commands and, unless altered by a service extension, message
> data, are transmitted in "lines". Lines consist of zero or more data
> characters terminated by the sequence ASCII character "CR" (hex value
> 0D) followed immediately by ASCII character "LF" (hex value 0A). This
> termination sequence is denoted as <CRLF> in this document. Conforming
> implementations MUST NOT recognize or generate any other character or
> character sequence as a line terminator. Limits MAY be imposed on line
> lengths by servers (see section 4.5.3).
> 
>     In addition, the appearance of "bare" "CR" or "LF" characters in
> text (i.e., either without the other) has a long history of causing
> problems in mail implementations and applications that use the mail
> system as a tool. SMTP client implementations MUST NOT transmit these
> characters except when they are intended as line terminators and then
> MUST, as indicated above, transmit them only as a <CRLF> sequence. 
> </quote>
> 
> RFC 2822 says
> 
> <quote>
> 2.3 Body
> 
>     The body of a message is simply lines of US-ASCII characters. The
> only two limitations on the body are as follows:
> 
>    - CR and LF MUST only occur together as CRLF; they MUST NOT appear
>      independently in the body.
> 
>    - Lines of characters in the body MUST be limited to 998 characters,
>      and SHOULD be limited to 78 characters, excluding the CRLF.
> 
>     Note: As was stated earlier, there are other standards documents,
> specifically the MIME documents [RFC2045, RFC2046, RFC2048, RFC2049]
> that extend this standard to allow for different sorts of message
> bodies. Again, these mechanisms are beyond the scope of this document.
> </quote>

Thanks for the pointer. So we should use CRLF anywhere.


> >The issue can be fixed if I make Savane convert all \r\n to \n before
> >to send the notifications, but I am not sure this is very
> >compliant. Anyway I do think this is a bug in Mailman. What do you
> >think we (Mailman and Savane) should do?
> 
> I don't see any issues with mail delivered from Mailman having any
> "doubling" of line terminators resulting in extra blank lines.
> 
> How does Savane inject its messages into the internet mail transport
> system? If it is sending via SMTP, then it MUST send line terminators
> as <CRLF> (or \r\n). If this results in doubling, then the receiving
> SMTP server is non compliant. If it is sending messages via some non
> SMTP intermediary, then it needs to format the messages as expected by
> the intermediary.
> 
> Does Savane perhaps pipe outgoing messages directly to the Mailman
> wrapper in the same way that an incoming MTA might do? If so, this is
> _not_ an SMTP transfer and end of line sequences should be those of
> the platform, i.e just \n for Unix.

Savane uses the PHP "mail()" function, that executes the local
sendmail-compatible command.

At both GNU Savannah (savannah.gnu.org) and Gna! (gna.org), we use the
Exim version packaged by Debian. In both cases, the mail we receive
via mailing lists (and in the Mailman archives) have the newlines
doubled. The mails we receive at our personnal mail addresses is
correct (no doubling). As far as I'm concerned, line doubling _does_
look ugly and unprofessional ;)

As a result, I was under the impression that Mailman is the one that
doubles the newlines. Now maybe Exim as an SMTP server (not as a
sendmail-compatible command) is doubling lines. But are you completely
sure Mailman (or a Python library used by Mailman) is not converting
\r\n to \n\n?

-- 
Sylvain