Brad Knowles wrote:

At 3:20 PM -0800 2005-11-28, Nathan Herring wrote:

...

Specifically, when it (re)folds the Subject: and/or Thread-Topic: headers, it replaces a space character in the original subject with a tab character.

Also keep in mind that Mailman uses a lot of built-in Python routines for handling stuff, and some of those routines might include the handling and formatting of e-mail messages. You want to make sure that you're using the latest version of Python that is compatible with the version of Mailman you've got. Unfortunately, while the people working on Mailman tend to be pretty well aware of the various mail-related RFCs, the people writing the code and libraries within Python itself may not be.

Brad is correct here. Mailman represents messages as instances of the Python email.Message.Message class and is at the mercy of the methods in that class as far header folding and unfolding are concerned. And, for the record, even the Python 2.4.2 email module folds with a <tab>.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Nathan Herring wrote:

I've increased the subject length of this mail, and removed the "[Mailman-developers]" attribution, so we can see firsthand whether or not this list, which purports to be running 2.1.6, and I presume is running a suitable version of Python, has the same issue.

and Mark Sapiro previously wrote:

Brad is correct here. Mailman represents messages as instances of the Python email.Message.Message class and is at the mercy of the methods in that class as far header folding and unfolding are concerned. And, for the record, even the Python 2.4.2 email module folds with a <tab>.

Clearly what I said above is not the whole story as Nathan's message that I received from the list had the Subject: folded with a <space>, yet other Mailman related headers in the message, namely List-Unsubscribe: and List-Subscribe: are folded with <tab>.

Upon closer examination, I see that the email.Header.Header class supports a continuation_ws argument which as Brad notes is used in CookHeaders. The prefix_subject function in CookHeaders attempts to determine the continuation-ws character from the existing header by looking at the first character of the first continuation line of the original subject header. If the header isn't continued or if the first character of the first continuation is not a <space> or <tab>, it defaults to a <tab>.

Thus it will try to preserve the continuation-ws of an already folded incoming subject, but will default to <tab>. Thus if the incoming subject is not folded, but addition of the prefix lengthens it sufficiently so it folds, it will be continued with a <tab>.

BTW, this code has been in CookHeaders since 2.1.1, so I don't think anything will change in this respect between 2.1.2 and 2.1.6.

It does seem however, that given RFC 2822, the default continuation-ws character in CookHeaders should be <space> and not <tab>.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Nathan Herring wrote:

On 12/2/05 6:13:20 PM, "Mark Sapiro" <msapiro@value.net> wrote:

<snip>

The other interesting one from an Outlook or Entourage perspective is the Thread-Topic, which ends up having a tab inserted into it.

Yes, and this is significant in that Thread-Topic is an MUA (in your case Microsoft-Entourage) header that is completely untouched by Mailman.

Therefore, it would appear simply instantiating an email.Message.Message object and later writing it out is sufficient to cause the header continuation white space to change from <space> to <tab>.

...

Thus it will try to preserve the continuation-ws of an already folded incoming subject, but will default to <tab>. Thus if the incoming subject is not folded, but addition of the prefix lengthens it sufficiently so it folds, it will be continued with a <tab>.

It seems that for other headers, like Thread-Topic, even regularly folded items get its folding space turned into a folding tab.

Yes. The logic of trying to determine and preserve the continuation white space of an already continued, incoming header is specific to Mailman's manipulation of Subject: headers in CookHeaders. It is not done elsewhere in Mailman.

From my reading, it seems that email.Header doesn't preserve the FWS in the original header as it should. It would seem that the only time the continuation-ws parameter should be used is if there are sets of characters that need to be turned into multiple adjacent RFC 2047 encoded-words, as that FWS is not considered to be logically part of the header value, but merely an artifact of encoding. If email.Header did the correct preservation, then it would not matter whether you passed in <space> or <tab>.

Actually, the email.Header.Header class is a constructor for Header objects. It does not take a 'header' as an input argument. It makes a class instance from a non-continued string or a list of (data, charset) pairs to be RFC 2047 encoded. See the Python library reference, sec. 12.2.5. The resulting Header instance is continued as necessary using the continuation-ws argument as the leading white space on continued lines. Within the class, continuation-ws defaults to <space>, not <tab>.

There are really two issues here. The first is that CookHeaders manipulates Subject: headers as Header class objects and unless the incoming Subject: header is already continued with a <space>, it specifies a <tab> for the continuation-ws character. This is strictly a Mailman issue.

The other issue is that the Python email.Parser class API parses message headers into strings, not Header class instances, and the methods for flattening messages continue long header strings with <tab>. This is a Python email library issue. To see it in action, just do the following in an interactive Python session.

...

...

import email from cStringIO import StringIO from email.Generator import Generator x = email.message_from_string('Header: a long string of words that will be ultimately continued because it\n is too long for one line') x['Subject'] = 'Some other long line which we build here and stick into a header to see what happens' fp = StringIO() g = Generator(fp, maxheaderlen=60) g.flatten(x) text = fp.getvalue() text 'Header: a long string of words that will be ultimately\n\tcontinued because it is too long for one line\nSubject: Some other long line which we build here and stick\n\tinto a header to see what happens\n\n'

(Lines 4, 5 and 11 above are each one long line in the original, although they may get wrapped in emailing.)

Note that the original Header: was continued with a <space>. The flatten method properly unfolds it but then refolds it with a <tab>. Likewise it folds the Subject: with a <tab>.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan