Subject's being mangled

Folks:
I've noticed some odd behavior in the latest CVS that I updated to ... the subject line is being treated as if it's blank.
Here's an example of a message that was processed by one of my lists ...
Notice that the subject is on a seperate line.
This list does NOT have a prefix configured.
Any ideas?
Thanks!
david

David Gibbs wrote:
Notice that the subject is on a seperate line.
This list does NOT have a prefix configured.
Make sure your prefix is blank. Invisible space may cause this effect, I suspect.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

David Gibbs wrote:
Sorry but how about putting a space in your blank prefix. (effect is reverse, now I suppose)
It happens on lists with subject prefixes also.
Hmm, could you show me an example?
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

It happens on lists with subject prefixes also. Hmm, could you show me an example?
Here's one ...
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

This makes no problem, I believe.
Try putting a white space or deleting for no-subject_prefix case.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

David, Try this patch. Tokio --- CookHeaders.py.orig Sun Feb 9 15:13:27 2003 +++ CookHeaders.py Sat Feb 8 15:06:46 2003 @@ -219,6 +219,8 @@ # tracked (e.g. internally crafted, delivered to a single user such as the # list admin). prefix = mlist.subject_prefix + if not prefix: + return subject = msg.get('subject', '') # Try to figure out what the continuation_ws is for the header if isinstance(subject, Header):

Tokio: Didn't help ... it seems to have something to do with the length of the subject line. A long subject gets wrapped around to the next line of the message. david At 12:45 AM 2/9/2003, Tokio Kikuchi wrote:
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

At 11:32 AM 2/9/2003, Robby Griffin wrote:
I have no problem with the subject getting wrapped ... however I do have a problem with the WAY the subject is getting wrapped.
In the examples I posted, the messages entire subject was getting wrapped to another line ... it was not getting wrapped when the text exceeded a specific length. This has the affect of making most mail client software (that I'm using, Eudora and Squirrelmail) showing a blank subject or just the subject prefix.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

MUA developers should take care the RFC. In RFC-2822,
subject = "Subject:" unstructured CRLF unstructured = *([FWS] utext) [FWS] ..., any CRLF that appears in FWS is semantically "invisible."
So, any MUA which treat the folding CR/LF as a meaningful line separation is not RFC compliant.
I know it's weired but will take a lot of hacking because this is caused by email package on which mailman relies.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

At 08:53 PM 2/9/2003, Tokio Kikuchi wrote:
Well, this has never been a problem with mailman before ... so when it starts being a problem, I tend to blame the software that changed last ... and my mail client hasn't changed for a long time, and Mailman has changed recently.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

Hi,
I've uploaded a patch for pythonlib/email/Header.py https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id=25568 patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Tokio Kikuchi wrote:
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E4F4781.9080703@is.kochi-u.ac.jp...
I've uploaded a patch for pythonlib/email/Header.py
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id =25568
patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Ah, that didn't seem to help ... in fact, it seems to have made it somewhat worse.
Here's what I would get BEFORE the patch ...
From: David Gibbs <david@midrange.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean Subject: [testlist] RE: FTP from QNTC prepends "?" to file name X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
... and here's what I get AFTER the patch ...
From: David Gibbs <david@midrange.com> Subject: [testlist] RE: FTP from QNTC prepends "?" to file name Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
As you can see, the list prefix is now put on the next line as well.
david

Ack, this also happened ...
==> error <== Feb 16 07:55:49 2003 (12529) Uncaught runner exception: string index out of range Feb 16 07:55:49 2003 (12529) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 71, in process g.flatten(msg, unixfrom=1) File "/usr/local/mailman/pythonlib/email/Generator.py", line 100, in flatten self._write(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 135, in _write self._write_headers(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 165, in _write_headers text = '%s: %s' % (h, v) File "/usr/local/mailman/pythonlib/email/Header.py", line 183, in __str__ return self.encode() File "/usr/local/mailman/pythonlib/email/Header.py", line 480, in encode return self._encode_chunks(newchunks) File "/usr/local/mailman/pythonlib/email/Header.py", line 447, in _encode_chunks if len(chunks) >0 and chunks[-1][-1] in ' \t': IndexError: string index out of range "David Gibbs" <david@midrange.com> wrote in message news:b2o3mm$mur$1@main.gmane.org...
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id

At 01:20 AM 2/18/2003, Tokio Kikuchi wrote:
Will you please send me the offending subject line. As to the other case, the problem you stated did not reproduce.
Tokio:
Sorry it took so long to get back to you on this ... work got kind of hectic ...
And example of a subject line that caused the problem is: "RE: FTP from QNTC prepends "£" to file name"
I have a test list that exhibits this behavior every time setup on my system ... if you would like, I can subscribe you to that list so you can see the behavior yourself.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

Hi,
David Gibbs wrote:
Use of Latin-1 character in english list is a different matter. If the subscribers use these character in your list, then see http://mail.python.org/pipermail/mailman-users/2003-February/026558.html
Tokio

"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E597D05.4000407@is.kochi-u.ac.jp...
I don't think that the character set relates to how the subject is being wrapped ... look at this thread ... there are a bunch of TAB characters in the text that (IMHO) shouldn't be there.
david

How about applying this patch also https://sourceforge.net/tracker/?func=detail&atid=300103&aid=601117&group_id=103
David Gibbs wrote:

"David Gibbs" <david@midrange.com> wrote in message news:b2165s$209$1@main.gmane.org...
I've noticed some odd behavior in the latest CVS that I updated to ... the subject line is being treated as if it's blank.
I have been able to determine a bit more information about the messages that are having the subjects mangled.
- It only happens on messages that have very long subjects
- When it happens, the entire subject is being put on a line by itself AFTER the subject header
- The subject is being prefixed, on the next line, with a tab character.
Here's an example:
X-MailScanner: Found to be clean Subject: [Interlug] [tab]Get COMMON's web banner... and "March 9-13 in Indy" materials X-BeenThere: interlug@midrange.com
(I replaced the actual tab character with [tab], in case it isn't obvious).
Note: The *SAME* thing happened on a thread with the subject "Odd message body "/root/8e8Ta4: Permission denied" (2.1b2)" started by Nick Arnett in this mailing list.
Does this help at all?
david

On Tuesday, February 11, 2003, at 04:37 PM, David Gibbs wrote:
Ah, this sounds like wrong behavior on Mailman's part. The next line of a wrapped subject should be prefixed with nothing, i.e., it should begin simply with whatever whitespace character followed the wrap point in the original subject line. Usually a single space. To use a tab means that an RFC-compliant MUA will display the unfolded subject with a tab (instead of the original amount of whitespace) at the wrap point. Apple Mail and Lotus Notes do this.
That still doesn't excuse your MUA's decision to just drop the wrapped text on the floor. Please consider reporting any such bugs to their authors as well.
--Robby

On Tuesday, February 11, 2003, at 05:42 PM, Robby Griffin wrote:
But wait, there's more. Mailman's wrapping the line _after_ a whitespace character instead of before. If you look closely, you'll see there's a space dangling at the end of the first line of the subject. Adding the tab on the next line makes the resulting header valid, but does actually add a tab to the content of the subject.
--Robby

"Robby Griffin" <rmg@terc.edu> wrote in message news:94EEC4AA-3E14-11D7-B928-00039383CAAE@terc.edu...
I think the real issue is this: Why is Mailman reformatting the subject line in the first place? If the subject is too long to fit in a certian amount of space, then it's fine ... but there is *NO* space problem in the examples I have referenced.
david

David Gibbs wrote:
Notice that the subject is on a seperate line.
This list does NOT have a prefix configured.
Make sure your prefix is blank. Invisible space may cause this effect, I suspect.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

David Gibbs wrote:
Sorry but how about putting a space in your blank prefix. (effect is reverse, now I suppose)
It happens on lists with subject prefixes also.
Hmm, could you show me an example?
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

It happens on lists with subject prefixes also. Hmm, could you show me an example?
Here's one ...
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

This makes no problem, I believe.
Try putting a white space or deleting for no-subject_prefix case.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

David, Try this patch. Tokio --- CookHeaders.py.orig Sun Feb 9 15:13:27 2003 +++ CookHeaders.py Sat Feb 8 15:06:46 2003 @@ -219,6 +219,8 @@ # tracked (e.g. internally crafted, delivered to a single user such as the # list admin). prefix = mlist.subject_prefix + if not prefix: + return subject = msg.get('subject', '') # Try to figure out what the continuation_ws is for the header if isinstance(subject, Header):

Tokio: Didn't help ... it seems to have something to do with the length of the subject line. A long subject gets wrapped around to the next line of the message. david At 12:45 AM 2/9/2003, Tokio Kikuchi wrote:
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

At 11:32 AM 2/9/2003, Robby Griffin wrote:
I have no problem with the subject getting wrapped ... however I do have a problem with the WAY the subject is getting wrapped.
In the examples I posted, the messages entire subject was getting wrapped to another line ... it was not getting wrapped when the text exceeded a specific length. This has the affect of making most mail client software (that I'm using, Eudora and Squirrelmail) showing a blank subject or just the subject prefix.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

MUA developers should take care the RFC. In RFC-2822,
subject = "Subject:" unstructured CRLF unstructured = *([FWS] utext) [FWS] ..., any CRLF that appears in FWS is semantically "invisible."
So, any MUA which treat the folding CR/LF as a meaningful line separation is not RFC compliant.
I know it's weired but will take a lot of hacking because this is caused by email package on which mailman relies.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

At 08:53 PM 2/9/2003, Tokio Kikuchi wrote:
Well, this has never been a problem with mailman before ... so when it starts being a problem, I tend to blame the software that changed last ... and my mail client hasn't changed for a long time, and Mailman has changed recently.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

Hi,
I've uploaded a patch for pythonlib/email/Header.py https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id=25568 patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Tokio Kikuchi wrote:
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E4F4781.9080703@is.kochi-u.ac.jp...
I've uploaded a patch for pythonlib/email/Header.py
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id =25568
patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Ah, that didn't seem to help ... in fact, it seems to have made it somewhat worse.
Here's what I would get BEFORE the patch ...
From: David Gibbs <david@midrange.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean Subject: [testlist] RE: FTP from QNTC prepends "?" to file name X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
... and here's what I get AFTER the patch ...
From: David Gibbs <david@midrange.com> Subject: [testlist] RE: FTP from QNTC prepends "?" to file name Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
As you can see, the list prefix is now put on the next line as well.
david

Ack, this also happened ...
==> error <== Feb 16 07:55:49 2003 (12529) Uncaught runner exception: string index out of range Feb 16 07:55:49 2003 (12529) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 71, in process g.flatten(msg, unixfrom=1) File "/usr/local/mailman/pythonlib/email/Generator.py", line 100, in flatten self._write(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 135, in _write self._write_headers(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 165, in _write_headers text = '%s: %s' % (h, v) File "/usr/local/mailman/pythonlib/email/Header.py", line 183, in __str__ return self.encode() File "/usr/local/mailman/pythonlib/email/Header.py", line 480, in encode return self._encode_chunks(newchunks) File "/usr/local/mailman/pythonlib/email/Header.py", line 447, in _encode_chunks if len(chunks) >0 and chunks[-1][-1] in ' \t': IndexError: string index out of range "David Gibbs" <david@midrange.com> wrote in message news:b2o3mm$mur$1@main.gmane.org...
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id

At 01:20 AM 2/18/2003, Tokio Kikuchi wrote:
Will you please send me the offending subject line. As to the other case, the problem you stated did not reproduce.
Tokio:
Sorry it took so long to get back to you on this ... work got kind of hectic ...
And example of a subject line that caused the problem is: "RE: FTP from QNTC prepends "£" to file name"
I have a test list that exhibits this behavior every time setup on my system ... if you would like, I can subscribe you to that list so you can see the behavior yourself.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)

Hi,
David Gibbs wrote:
Use of Latin-1 character in english list is a different matter. If the subscribers use these character in your list, then see http://mail.python.org/pipermail/mailman-users/2003-February/026558.html
Tokio

"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E597D05.4000407@is.kochi-u.ac.jp...
I don't think that the character set relates to how the subject is being wrapped ... look at this thread ... there are a bunch of TAB characters in the text that (IMHO) shouldn't be there.
david

How about applying this patch also https://sourceforge.net/tracker/?func=detail&atid=300103&aid=601117&group_id=103
David Gibbs wrote:

"David Gibbs" <david@midrange.com> wrote in message news:b2165s$209$1@main.gmane.org...
I've noticed some odd behavior in the latest CVS that I updated to ... the subject line is being treated as if it's blank.
I have been able to determine a bit more information about the messages that are having the subjects mangled.
- It only happens on messages that have very long subjects
- When it happens, the entire subject is being put on a line by itself AFTER the subject header
- The subject is being prefixed, on the next line, with a tab character.
Here's an example:
X-MailScanner: Found to be clean Subject: [Interlug] [tab]Get COMMON's web banner... and "March 9-13 in Indy" materials X-BeenThere: interlug@midrange.com
(I replaced the actual tab character with [tab], in case it isn't obvious).
Note: The *SAME* thing happened on a thread with the subject "Odd message body "/root/8e8Ta4: Permission denied" (2.1b2)" started by Nick Arnett in this mailing list.
Does this help at all?
david

On Tuesday, February 11, 2003, at 04:37 PM, David Gibbs wrote:
Ah, this sounds like wrong behavior on Mailman's part. The next line of a wrapped subject should be prefixed with nothing, i.e., it should begin simply with whatever whitespace character followed the wrap point in the original subject line. Usually a single space. To use a tab means that an RFC-compliant MUA will display the unfolded subject with a tab (instead of the original amount of whitespace) at the wrap point. Apple Mail and Lotus Notes do this.
That still doesn't excuse your MUA's decision to just drop the wrapped text on the floor. Please consider reporting any such bugs to their authors as well.
--Robby

On Tuesday, February 11, 2003, at 05:42 PM, Robby Griffin wrote:
But wait, there's more. Mailman's wrapping the line _after_ a whitespace character instead of before. If you look closely, you'll see there's a space dangling at the end of the first line of the subject. Adding the tab on the next line makes the resulting header valid, but does actually add a tab to the content of the subject.
--Robby

"Robby Griffin" <rmg@terc.edu> wrote in message news:94EEC4AA-3E14-11D7-B928-00039383CAAE@terc.edu...
I think the real issue is this: Why is Mailman reformatting the subject line in the first place? If the subject is too long to fit in a certian amount of space, then it's fine ... but there is *NO* space problem in the examples I have referenced.
david
participants (3)
-
David Gibbs
-
Robby Griffin
-
Tokio Kikuchi