Terri Oda wrote:

There's a program in bin called "cleanarch" that can be run on your archive to fix this problem. It cleans up the offending From lines from older mbox files so that you can run arch again and generate correct html versions of the archives.

Terri is correct. cleanarch will escape the unescaped From_ lines in the .mbox, but it seems you have done that yourself in some way, and the issue is with new messages.

So the question is why is this happening with new messages? Again, what Mailman version is this?

Also note that escaping From_ by preceding it with '>' is the accepted way to deal with this. Many MUAs will do it before sending the message and MDAs will do it too before delivering a message. It is unusual to be able to pass a From_ through email from end to end without it being escaped to >From_ somewhere between source and destination.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 20/04/2010 18:14, Mark Sapiro wrote:

That is the normal way of dealing with messages containing From_ in the message body. It's not just Mailman or pipermail, and it's problematic to unescape them for display, because while the escaping is normal, there is no standard for escaping/unescaping so when you see >From_ in a message, you don't know if it is an escaped From_, a quoted From_ or a literal >From_.

OK - that is different behaviour from the software I had been using, but if that is how it works, then I am happy - I was more concerned that it was an indication my installation might be broken than anything else. I take your point about not knowing which it is, but in my experience, a quoted From_ is most usually >_From_ not >From_, and I had been happy to take a potential hit on the exceptions to this, as they would be far less common than the escaped From_. But this is fairly unimportant, compared to the other problem of archive corruption.

The issue with unescaped From_ in the body causing archive corruption was fixed long before Mailman 2.1.9

...

To go to a more recent version is not impossible, but not trivial for me (a Linux VPS newbie), so I wanted to see if it was the solution before >> rolling my sleeves up.

You shouldn't need to. Mailman 2.1.9 should not have this problem.

OK - that means the weirdness in my specific system, not in Mailman. Not ideal, but at least I have narrowed things down a bit.

I believe you, so I don't think I need to see the test list. The question is why isn't Mailman escaping the From_ when it archives and sends the message.

It actually relies on the Python email library to do this, but Mailman 2.1.9 should install its own version of the email package in Mailman's pythonlib/ directory, and this should always escape From_ lines when converting an email.Message.Message object to text. Why it doesn't is the question.

Also curious is that I think you said the problem occurs with "text\nFrom " in the body, but not with "text\n\nFrom ". If I understood that correctly, that is really strange.

Ah, no, it is happening with both "text\nFrom " and "text\n\nFrom ". It's just that I encountered "text\nFrom " very early on. (the mail-list is primarily about songwriting, which means we get lyrics posted to the list. Hence there is a higher than normal chance that you will get new lines starting with capitals.)

Anyway, many thanks for your help. While it hasn't resolved my immediate problem, it has told me what it isn't, which is a great help. I'll go away and have a fiddle. I am actually tempted to reinstall 2.1.9 myself from scratch, which should have less issues regarding support than going to a new version. Originally, my VPS was supposed to have Mailman (not sure what version), but I had no access to it via Plesk (this is before I knew how to configure it direct). The VPS support then said they had upgraded it, which seemed to fix the problem. I can't help but wonder if they did a botched job of it. First thing to check is if Mailman's pythonlib/ directory, which you mention exists.

If I get anywhere, I will let you know!

Chris

Terri Oda wrote:

There's a program in bin called "cleanarch" that can be run on your archive to fix this problem. It cleans up the offending From lines from older mbox files so that you can run arch again and generate correct html versions of the archives.

Terri is correct. cleanarch will escape the unescaped From_ lines in the .mbox, but it seems you have done that yourself in some way, and the issue is with new messages.

So the question is why is this happening with new messages? Again, what Mailman version is this?

Also note that escaping From_ by preceding it with '>' is the accepted way to deal with this. Many MUAs will do it before sending the message and MDAs will do it too before delivering a message. It is unusual to be able to pass a From_ through email from end to end without it being escaped to >From_ somewhere between source and destination.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 20/04/2010 18:14, Mark Sapiro wrote:

That is the normal way of dealing with messages containing From_ in the message body. It's not just Mailman or pipermail, and it's problematic to unescape them for display, because while the escaping is normal, there is no standard for escaping/unescaping so when you see >From_ in a message, you don't know if it is an escaped From_, a quoted From_ or a literal >From_.

OK - that is different behaviour from the software I had been using, but if that is how it works, then I am happy - I was more concerned that it was an indication my installation might be broken than anything else. I take your point about not knowing which it is, but in my experience, a quoted From_ is most usually >_From_ not >From_, and I had been happy to take a potential hit on the exceptions to this, as they would be far less common than the escaped From_. But this is fairly unimportant, compared to the other problem of archive corruption.

The issue with unescaped From_ in the body causing archive corruption was fixed long before Mailman 2.1.9

...

To go to a more recent version is not impossible, but not trivial for me (a Linux VPS newbie), so I wanted to see if it was the solution before >> rolling my sleeves up.

You shouldn't need to. Mailman 2.1.9 should not have this problem.

OK - that means the weirdness in my specific system, not in Mailman. Not ideal, but at least I have narrowed things down a bit.

I believe you, so I don't think I need to see the test list. The question is why isn't Mailman escaping the From_ when it archives and sends the message.

It actually relies on the Python email library to do this, but Mailman 2.1.9 should install its own version of the email package in Mailman's pythonlib/ directory, and this should always escape From_ lines when converting an email.Message.Message object to text. Why it doesn't is the question.

Also curious is that I think you said the problem occurs with "text\nFrom " in the body, but not with "text\n\nFrom ". If I understood that correctly, that is really strange.

Ah, no, it is happening with both "text\nFrom " and "text\n\nFrom ". It's just that I encountered "text\nFrom " very early on. (the mail-list is primarily about songwriting, which means we get lyrics posted to the list. Hence there is a higher than normal chance that you will get new lines starting with capitals.)

Anyway, many thanks for your help. While it hasn't resolved my immediate problem, it has told me what it isn't, which is a great help. I'll go away and have a fiddle. I am actually tempted to reinstall 2.1.9 myself from scratch, which should have less issues regarding support than going to a new version. Originally, my VPS was supposed to have Mailman (not sure what version), but I had no access to it via Plesk (this is before I knew how to configure it direct). The VPS support then said they had upgraded it, which seemed to fix the problem. I can't help but wonder if they did a botched job of it. First thing to check is if Mailman's pythonlib/ directory, which you mention exists.

If I get anywhere, I will let you know!

Chris