Problem with inappropriate breaks in messages
Hi all,
Newbie here.
I have recently moved a list over to Mailman running under Plesk on a VPS. The version of Mailman that I currently have access to is 2.1.9.
, although that now means that ">From " appears in the archive, but I
I had a mbox file to import from my old mail-list system. Originally, I had some problems - the mbox file was large (7 years of archives) and some messages had a literal "\nFrom ", which cause the archiver to break the messages inappropriately. For the moment, I escaped these incidents with a thought it was better than having a corrupted archive, and I could rebuild at a later date when I fully understood how Mailman copes with this.
However, I have very quickly discovered that if a post to the list contains
- in the actual message text - a newline (a single newline, not a double) followed by the word From, Mailman interprets that as a new message and breaks the message at that point, creating a fragment message with no subject line.
Note that this is not just the archive - this actually affects messages being sent to the subscribers - i.e. a message containing a newline followed by "From " will be split in two before going out.
This really surprised me, as it is not at all unlikely that sometime or other someone will post "From " at the start of a line!
Is this a known bug (I did search, and couldn't spot anything), and is it fixed in other versions? Or do I have a rogue version of Mailman installed on my VPS? The behaviour is consistent and repeatable.
Regards
Chris
Further to my earlier post.
Note that this is not just the archive - this actually affects messages being sent to the subscribers - i.e. a message containing a newline followed by "From " will be split in two before going out.
Further testing shows that this is incorrect. The problem affects only the archive, not the mail-list messages going to the subscribers.
However, the archive problem is repeatable. Any instance in the message text of "From " following a single newline is interpreted as a new message.
There's a program in bin called "cleanarch" that can be run on your archive to fix this problem. It cleans up the offending From lines from older mbox files so that you can run arch again and generate correct html versions of the archives.
Terri
Chris Malme wrote:
Further to my earlier post.
Note that this is not just the archive - this actually affects messages being sent to the subscribers - i.e. a message containing a newline followed by "From " will be split in two before going out.
Further testing shows that this is incorrect. The problem affects only the archive, not the mail-list messages going to the subscribers.
However, the archive problem is repeatable. Any instance in the message text of "From " following a single newline is interpreted as a new message.
Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/terri%40zone12.com
Terri Oda wrote:
There's a program in bin called "cleanarch" that can be run on your archive to fix this problem. It cleans up the offending From lines from older mbox files so that you can run arch again and generate correct html versions of the archives.
Terri is correct. cleanarch will escape the unescaped From_ lines in the .mbox, but it seems you have done that yourself in some way, and the issue is with new messages.
So the question is why is this happening with new messages? Again, what Mailman version is this?
Also note that escaping From_ by preceding it with '>' is the accepted way to deal with this. Many MUAs will do it before sending the message and MDAs will do it too before delivering a message. It is unusual to be able to pass a From_ through email from end to end without it being escaped to >From_ somewhere between source and destination.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Chris Malme wrote:
Further testing shows that this is incorrect. The problem affects only the archive, not the mail-list messages going to the subscribers.
However, the archive problem is repeatable. Any instance in the message text of "From " following a single newline is interpreted as a new message.
What Mailman version is this?
This was a bug at one time, but it was fixed years ago.
There was a change in this area between 2.1.12 and 2.1.13, but that change only affected outgoing messages, not archives or digests.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 20/04/2010 18:14, Mark Sapiro wrote:
That is the normal way of dealing with messages containing From_ in the message body. It's not just Mailman or pipermail, and it's problematic to unescape them for display, because while the escaping is normal, there is no standard for escaping/unescaping so when you see >From_ in a message, you don't know if it is an escaped From_, a quoted From_ or a literal >From_.
OK - that is different behaviour from the software I had been using, but if that is how it works, then I am happy - I was more concerned that it was an indication my installation might be broken than anything else. I take your point about not knowing which it is, but in my experience, a quoted From_ is most usually >_From_ not >From_, and I had been happy to take a potential hit on the exceptions to this, as they would be far less common than the escaped From_. But this is fairly unimportant, compared to the other problem of archive corruption.
The issue with unescaped From_ in the body causing archive corruption was fixed long before Mailman 2.1.9
To go to a more recent version is not impossible, but not trivial for me (a Linux VPS newbie), so I wanted to see if it was the solution before >> rolling my sleeves up.
You shouldn't need to. Mailman 2.1.9 should not have this problem.
OK - that means the weirdness in my specific system, not in Mailman. Not ideal, but at least I have narrowed things down a bit.
I believe you, so I don't think I need to see the test list. The question is why isn't Mailman escaping the From_ when it archives and sends the message.
It actually relies on the Python email library to do this, but Mailman 2.1.9 should install its own version of the email package in Mailman's pythonlib/ directory, and this should always escape From_ lines when converting an email.Message.Message object to text. Why it doesn't is the question.
Also curious is that I think you said the problem occurs with "text\nFrom " in the body, but not with "text\n\nFrom ". If I understood that correctly, that is really strange.
Ah, no, it is happening with both "text\nFrom " and "text\n\nFrom ". It's just that I encountered "text\nFrom " very early on. (the mail-list is primarily about songwriting, which means we get lyrics posted to the list. Hence there is a higher than normal chance that you will get new lines starting with capitals.)
Anyway, many thanks for your help. While it hasn't resolved my immediate problem, it has told me what it isn't, which is a great help. I'll go away and have a fiddle. I am actually tempted to reinstall 2.1.9 myself from scratch, which should have less issues regarding support than going to a new version. Originally, my VPS was supposed to have Mailman (not sure what version), but I had no access to it via Plesk (this is before I knew how to configure it direct). The VPS support then said they had upgraded it, which seemed to fix the problem. I can't help but wonder if they did a botched job of it. First thing to check is if Mailman's pythonlib/ directory, which you mention exists.
If I get anywhere, I will let you know!
Chris
participants (3)
-
Chris Malme -
Mark Sapiro -
Terri Oda