Subject's being mangled
Folks:
I've noticed some odd behavior in the latest CVS that I updated to ... the subject line is being treated as if it's blank.
Here's an example of a message that was processed by one of my lists ...
Content-Type: text/plain; charset="US-ASCII" X-Content-Filtered-By: Mailman/MimeDel 2.1+ Subject: Re: V4R4 to V5R1 upgrade [was Re: Off-topic, and not sorry one iota] X-BeenThere: midrange-l@midrange.com X-Mailman-Version: 2.1+ Precedence: list
Notice that the subject is on a seperate line.
This list does NOT have a prefix configured.
Any ideas?
Thanks!
david
David Gibbs wrote:
X-Content-Filtered-By: Mailman/MimeDel 2.1+ Subject: Re: V4R4 to V5R1 upgrade [was Re: Off-topic, and not sorry one iota]
Notice that the subject is on a seperate line.
This list does NOT have a prefix configured.
Make sure your prefix is blank. Invisible space may cause this effect, I suspect.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
Tokio Kikuchi said:
Notice that the subject is on a seperate line. This list does NOT have a prefix configured. Make sure your prefix is blank. Invisible space may cause this effect, I suspect.
It happens on lists with subject prefixes also.
david
-- "Who said I couldn't have it all?" -B Gates
David Gibbs wrote:
Tokio Kikuchi said:
Notice that the subject is on a seperate line. This list does NOT have a prefix configured.
Make sure your prefix is blank. Invisible space may cause this effect, I suspect.
Sorry but how about putting a space in your blank prefix. (effect is reverse, now I suppose)
It happens on lists with subject prefixes also.
Hmm, could you show me an example?
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
It happens on lists with subject prefixes also. Hmm, could you show me an example?
Here's one ...
Sensitivity: To: interlug@midrange.com X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: <OF9E329149.2604A847-ON86256CC6.00586FEE@xxxxxxxxx> From: "Jennifer Alten" <xxxxxxxxxxxx@xxxxxxxxxxxx> Date: Fri, 7 Feb 2003 10:06:50 -0600 X-MIMETrack: Serialize by Router on Notes1/xxxxxxx(Release 5.0.11 |July 24, 2002) at 02/07/2003 10:07:27 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii X-MailScanner: Found to be clean Subject: [Interlug] Get COMMON's web banner... and "March 9-13 in Indy" materials X-BeenThere: interlug@midrange.com X-Mailman-Version: 2.1+ Precedence: list Reply-To: interlug@midrange.com List-Id: <interlug.midrange.com> List-Unsubscribe: <http://lists.midrange.com/mailman/listinfo/interlug>, <mailto:interlug-request@midrange.com?subject=unsubscribe> List-Archive: <http://lists.midrange.com/mailman/private/interlug> List-Post: <mailto:interlug@midrange.com> List-Help: <mailto:interlug-request@midrange.com?subject=help> List-Subscribe: <http://lists.midrange.com/mailman/listinfo/interlug>, <mailto:interlug-request@midrange.com?subject=subscribe> Sender: interlug-bounces@midrange.com Errors-To: interlug-bounces@midrange.com
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)
It happens on lists with subject prefixes also.
Hmm, could you show me an example?
Here's one ...
Subject: [Interlug] Get COMMON's web banner... and "March 9-13 in Indy" materials
This makes no problem, I believe.
Try putting a white space or deleting for no-subject_prefix case.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
David, Try this patch. Tokio --- CookHeaders.py.orig Sun Feb 9 15:13:27 2003 +++ CookHeaders.py Sat Feb 8 15:06:46 2003 @@ -219,6 +219,8 @@ # tracked (e.g. internally crafted, delivered to a single user such as the # list admin). prefix = mlist.subject_prefix + if not prefix: + return subject = msg.get('subject', '') # Try to figure out what the continuation_ws is for the header if isinstance(subject, Header):
Tokio: Didn't help ... it seems to have something to do with the length of the subject line. A long subject gets wrapped around to the next line of the message. david At 12:45 AM 2/9/2003, Tokio Kikuchi wrote:
David,
Try this patch.
Tokio
--- CookHeaders.py.orig Sun Feb 9 15:13:27 2003 +++ CookHeaders.py Sat Feb 8 15:06:46 2003 @@ -219,6 +219,8 @@ # tracked (e.g. internally crafted, delivered to a single user such as the # list admin). prefix = mlist.subject_prefix + if not prefix: + return subject = msg.get('subject', '') # Try to figure out what the continuation_ws is for the header if isinstance(subject, Header):
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)
On Sunday, Feb 9, 2003, at 10:43 US/Eastern, David Gibbs wrote:
Didn't help ... it seems to have something to do with the length of the subject line. A long subject gets wrapped around to the next line of the message.
Hmm. Yes, it does. Perhaps you should have a look at RFC2822, section 2.2.3, and then try to identify a specific problem with subject wrapping.
--RobbyAt 11:32 AM 2/9/2003, Robby Griffin wrote:
Didn't help ... it seems to have something to do with the length of the subject line. A long subject gets wrapped around to the next line of the message. Hmm. Yes, it does. Perhaps you should have a look at RFC2822, section 2.2.3, and then try to identify a specific problem with subject wrapping.
I have no problem with the subject getting wrapped ... however I do have a problem with the WAY the subject is getting wrapped.
In the examples I posted, the messages entire subject was getting wrapped to another line ... it was not getting wrapped when the text exceeded a specific length. This has the affect of making most mail client software (that I'm using, Eudora and Squirrelmail) showing a blank subject or just the subject prefix.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)
I have no problem with the subject getting wrapped ... however I do have a problem with the WAY the subject is getting wrapped.
In the examples I posted, the messages entire subject was getting wrapped to another line ... it was not getting wrapped when the text exceeded a specific length. This has the affect of making most mail client software (that I'm using, Eudora and Squirrelmail) showing a blank subject or just the subject prefix.
MUA developers should take care the RFC. In RFC-2822,
subject = "Subject:" unstructured CRLF unstructured = *([FWS] utext) [FWS] ..., any CRLF that appears in FWS is semantically "invisible."
So, any MUA which treat the folding CR/LF as a meaningful line separation is not RFC compliant.
I know it's weired but will take a lot of hacking because this is caused by email package on which mailman relies.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
At 08:53 PM 2/9/2003, Tokio Kikuchi wrote:
MUA developers should take care the RFC. In RFC-2822, So, any MUA which treat the folding CR/LF as a meaningful line separation is not RFC compliant. I know it's weired but will take a lot of hacking because this is caused by email package on which mailman relies.
Well, this has never been a problem with mailman before ... so when it starts being a problem, I tend to blame the software that changed last ... and my mail client hasn't changed for a long time, and Mailman has changed recently.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)
Hi,
I've uploaded a patch for pythonlib/email/Header.py https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id=25568 patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Tokio Kikuchi wrote:
I have no problem with the subject getting wrapped ... however I do have a problem with the WAY the subject is getting wrapped.
In the examples I posted, the messages entire subject was getting wrapped to another line ... it was not getting wrapped when the text exceeded a specific length. This has the affect of making most mail client software (that I'm using, Eudora and Squirrelmail) showing a blank subject or just the subject prefix.
MUA developers should take care the RFC. In RFC-2822,
subject = "Subject:" unstructured CRLF unstructured = *([FWS] utext) [FWS] ..., any CRLF that appears in FWS is semantically "invisible."
So, any MUA which treat the folding CR/LF as a meaningful line separation is not RFC compliant.
I know it's weired but will take a lot of hacking because this is caused by email package on which mailman relies.
-- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E4F4781.9080703@is.kochi-u.ac.jp...
I've uploaded a patch for pythonlib/email/Header.py
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id =25568
patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Ah, that didn't seem to help ... in fact, it seems to have made it somewhat worse.
Here's what I would get BEFORE the patch ...
From: David Gibbs <david@midrange.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean Subject: [testlist] RE: FTP from QNTC prepends "?" to file name X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
... and here's what I get AFTER the patch ...
From: David Gibbs <david@midrange.com> Subject: [testlist] RE: FTP from QNTC prepends "?" to file name Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
As you can see, the list prefix is now put on the next line as well.
david
Ack, this also happened ...
==> error <== Feb 16 07:55:49 2003 (12529) Uncaught runner exception: string index out of range Feb 16 07:55:49 2003 (12529) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 71, in process g.flatten(msg, unixfrom=1) File "/usr/local/mailman/pythonlib/email/Generator.py", line 100, in flatten self._write(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 135, in _write self._write_headers(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 165, in _write_headers text = '%s: %s' % (h, v) File "/usr/local/mailman/pythonlib/email/Header.py", line 183, in __str__ return self.encode() File "/usr/local/mailman/pythonlib/email/Header.py", line 480, in encode return self._encode_chunks(newchunks) File "/usr/local/mailman/pythonlib/email/Header.py", line 447, in _encode_chunks if len(chunks) >0 and chunks[-1][-1] in ' \t': IndexError: string index out of range "David Gibbs" <david@midrange.com> wrote in message news:b2o3mm$mur$1@main.gmane.org...
"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E4F4781.9080703@is.kochi-u.ac.jp...
I've uploaded a patch for pythonlib/email/Header.py
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id
=25568
patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Ah, that didn't seem to help ... in fact, it seems to have made it somewhat worse.
Here's what I would get BEFORE the patch ...
From: David Gibbs <david@midrange.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean Subject: [testlist] RE: FTP from QNTC prepends "?" to file name X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
... and here's what I get AFTER the patch ...
From: David Gibbs <david@midrange.com> Subject: [testlist] RE: FTP from QNTC prepends "?" to file name Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
As you can see, the list prefix is now put on the next line as well.
david
David,
Will you please send me the offending subject line. As to the other case, the problem you stated did not reproduce.
Subject: [testlist] RE: FTP from QNTC prepends "?" to file name
Tokio
David Gibbs wrote:
Ack, this also happened ...
==> error <== Feb 16 07:55:49 2003 (12529) Uncaught runner exception: string index out of range Feb 16 07:55:49 2003 (12529) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 71, in process g.flatten(msg, unixfrom=1) File "/usr/local/mailman/pythonlib/email/Generator.py", line 100, in flatten self._write(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 135, in _write self._write_headers(msg) File "/usr/local/mailman/pythonlib/email/Generator.py", line 165, in _write_headers text = '%s: %s' % (h, v) File "/usr/local/mailman/pythonlib/email/Header.py", line 183, in __str__ return self.encode() File "/usr/local/mailman/pythonlib/email/Header.py", line 480, in encode return self._encode_chunks(newchunks) File "/usr/local/mailman/pythonlib/email/Header.py", line 447, in _encode_chunks if len(chunks) >0 and chunks[-1][-1] in ' \t': IndexError: string index out of range "David Gibbs" <david@midrange.com> wrote in message news:b2o3mm$mur$1@main.gmane.org...
At 01:20 AM 2/18/2003, Tokio Kikuchi wrote:
Will you please send me the offending subject line. As to the other case, the problem you stated did not reproduce.
Tokio:
Sorry it took so long to get back to you on this ... work got kind of hectic ...
And example of a subject line that caused the problem is: "RE: FTP from QNTC prepends "£" to file name"
I have a test list that exhibits this behavior every time setup on my system ... if you would like, I can subscribe you to that list so you can see the behavior yourself.
david
-- | Internet: david@midrange.com | WWW: http://david.fallingrock.net | AIM: MidrangeMan | | We're not in the middle of nowhere... | we're on the outskirts of everywhere! | - DMRoth (adapted)
Hi,
David Gibbs wrote:
At 01:20 AM 2/18/2003, Tokio Kikuchi wrote:
Will you please send me the offending subject line. As to the other case, the problem you stated did not reproduce.
Tokio:
Sorry it took so long to get back to you on this ... work got kind of hectic ...
And example of a subject line that caused the problem is: "RE: FTP from QNTC prepends "£" to file name"
Use of Latin-1 character in english list is a different matter. If the subscribers use these character in your list, then see http://mail.python.org/pipermail/mailman-users/2003-February/026558.html
Tokio
"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E597D05.4000407@is.kochi-u.ac.jp...
Use of Latin-1 character in english list is a different matter. If the subscribers use these character in your list, then see http://mail.python.org/pipermail/mailman-users/2003-February/026558.html
I don't think that the character set relates to how the subject is being wrapped ... look at this thread ... there are a bunch of TAB characters in the text that (IMHO) shouldn't be there.
david
Have you tried my latest patch ? You should make sure that patched version is properly installed and restart mailmanctl.
Tokio
David Gibbs wrote:
"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E597D05.4000407@is.kochi-u.ac.jp...
Use of Latin-1 character in english list is a different matter. If the subscribers use these character in your list, then see http://mail.python.org/pipermail/mailman-users/2003-February/026558.html
I don't think that the character set relates to how the subject is being wrapped ... look at this thread ... there are a bunch of TAB characters in the text that (IMHO) shouldn't be there.
david
How about applying this patch also https://sourceforge.net/tracker/?func=detail&atid=300103&aid=601117&group_id=103
David Gibbs wrote:
"Tokio Kikuchi" <tkikuchi@is.kochi-u.ac.jp> wrote in message news:3E4F4781.9080703@is.kochi-u.ac.jp...
I've uploaded a patch for pythonlib/email/Header.py
https://sourceforge.net/tracker/?func=detail&atid=384680&aid=687338&group_id =25568
patching in the install dir would be OK because the installer doesn't seem to overwrite it in the next make install of mailman.
Ah, that didn't seem to help ... in fact, it seems to have made it somewhat worse.
Here's what I would get BEFORE the patch ...
From: David Gibbs <david@midrange.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean Subject: [testlist] RE: FTP from QNTC prepends "?" to file name X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
... and here's what I get AFTER the patch ...
From: David Gibbs <david@midrange.com> Subject: [testlist] RE: FTP from QNTC prepends "?" to file name Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-MailScanner: Found to be clean X-BeenThere: testlist@midrange.com X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Test List <testlist@midrange.com> List-Id: Test List <testlist.midrange.com>
As you can see, the list prefix is now put on the next line as well.
david
"David Gibbs" <david@midrange.com> wrote in message news:b2165s$209$1@main.gmane.org...
I've noticed some odd behavior in the latest CVS that I updated to ... the subject line is being treated as if it's blank.
I have been able to determine a bit more information about the messages that are having the subjects mangled.
- It only happens on messages that have very long subjects
- When it happens, the entire subject is being put on a line by itself AFTER the subject header
- The subject is being prefixed, on the next line, with a tab character.
Here's an example:
X-MailScanner: Found to be clean Subject: [Interlug] [tab]Get COMMON's web banner... and "March 9-13 in Indy" materials X-BeenThere: interlug@midrange.com
(I replaced the actual tab character with [tab], in case it isn't obvious).
Note: The *SAME* thing happened on a thread with the subject "Odd message body "/root/8e8Ta4: Permission denied" (2.1b2)" started by Nick Arnett in this mailing list.
Does this help at all?
david
On Tuesday, February 11, 2003, at 04:37 PM, David Gibbs wrote:
- The subject is being prefixed, on the next line, with a tab character.
Ah, this sounds like wrong behavior on Mailman's part. The next line of a wrapped subject should be prefixed with nothing, i.e., it should begin simply with whatever whitespace character followed the wrap point in the original subject line. Usually a single space. To use a tab means that an RFC-compliant MUA will display the unfolded subject with a tab (instead of the original amount of whitespace) at the wrap point. Apple Mail and Lotus Notes do this.
That still doesn't excuse your MUA's decision to just drop the wrapped text on the floor. Please consider reporting any such bugs to their authors as well.
--RobbyOn Tuesday, February 11, 2003, at 05:42 PM, Robby Griffin wrote:
Ah, this sounds like wrong behavior on Mailman's part. The next line of a wrapped subject should be prefixed with nothing, i.e., it should begin simply with whatever whitespace character followed the wrap point in the original subject line. Usually a single space.
But wait, there's more. Mailman's wrapping the line _after_ a whitespace character instead of before. If you look closely, you'll see there's a space dangling at the end of the first line of the subject. Adding the tab on the next line makes the resulting header valid, but does actually add a tab to the content of the subject.
--Robby"Robby Griffin" <rmg@terc.edu> wrote in message news:94EEC4AA-3E14-11D7-B928-00039383CAAE@terc.edu...
But wait, there's more. Mailman's wrapping the line _after_ a whitespace character instead of before. If you look closely, you'll see there's a space dangling at the end of the first line of the subject. Adding the tab on the next line makes the resulting header valid, but does actually add a tab to the content of the subject.
I think the real issue is this: Why is Mailman reformatting the subject line in the first place? If the subject is too long to fit in a certian amount of space, then it's fine ... but there is *NO* space problem in the examples I have referenced.
david
participants (3)
-
David Gibbs -
Robby Griffin -
Tokio Kikuchi