[Mailman-Developers] Re: [Mailman-Users] 2.1.1 mbox archive doesn't handle lines starting with "From " correctly?

[Barry Warsaw]

On Tue, 2003-04-15 at 20:51, Eric D. Christensen wrote:
> I'm seeing problems with the mbox archives after upgrading mailman from
> 2.0 to 2.1.1. I started to report these as bugs, but then thought that
> I'd better check in here first just in case I'm missing something silly.
> Apologies for cross posting to both users and developers lists.... 
> 
> First, a little background...
> 
> I use both pipermail and mbox format archives for all of our lists. I
> use the mbox format mainly as a backup so we can regenerate the
> pipermail archives (via 'arch --wipe list'). Since some of our lists
> have over 5 years of mailman archives now, having the mbox archives
> around  has save my butt several times. Too bad servers don't last as
> long as the mailing lists running on them! :-)
> 
> I'll admit up front that I'm NOT a python programmer.... perl, C, java,
> PHP, but not python. So I'm only slightly familiar with the syntax and
> totally clueless beyond that. I'm hoping to NOT have to use this problem
> as a reason to learn python (though I'd like to someday when I'm not
> quite so busy). 
> 
> Anyway, here are the two problems I'm seeing with mbox archives:
> 
> After upgrading mailman from 2.0 to 2.1.1 (and python itself to 2.2.2) I
> regenerated the pipermail archives from the mbox archives and found that
> suddenly I had a bunch of messages with "[no subject]", all together
> starting just about the time I switched the lists over the 2.1.1. Upon
> further investigation I found two issues that make the mbox file invalid
> (or at least suspect):
> 
> 1. No newline before "From " lines in the mbox with 2.1.1.
>         Sine the 2.1.1 update it appears the the mbox archiver is no
>         longer instering a newline before starting a new message. This
>         results in the "From_" line being directly below the last line
>         of the previous message. This confuses the mbox parser something
>         awful.... it also confuses elm, mutt, and mh if I try to read
>         the mbox files with them. 
>         
>         I found the code in Mailbox.py (in AppendMessage @ line 46) that
>         gets called from Archiver/Archiver.py to handle inserting the
>         newline if the last thing in the mbox file isn't already a
>         newline before appending the message, but it doesn't seem to be
>         working correctly. 
>         
>         Am I missing something or is AppendMessage broken in this
>         respect?
>         
> 2. Lines beginning with "From " inside of a message body are not
> handled.
>         If a line inside a message body starts with the string "From ",
>         it is being mis-interpreted as the beginning of a new message
>         (i.e. it's being treated as an envelope "From " line. 
>         
>         I'm not quite sure who to fault on this one... I believe that
>         it's common practice to somehow quote this case, in which case
>         it s Generator that's not doing the right thing. I think this is
>         supported by the fact the other mail agents (elm, mutt, etc...)
>         are confused by this unquoted "From " in the message body. It's
>         probably a bit much to ask utilities like arch to try to discern
>         body from envelope on the fly while reading in the mbox file. 
> 
> Any insight, pointers or ideas on these before I report them as bugs?

These are genuine bugs.  Please submit a bug report on them.

Thanks,
-Barry