Ivan Van Laningham wrote:

But I have one list for which I used archives from two previous incarnations of the list, plus the current archive mbox, as input to arch. I made sure that the previous archives were in mbox format and that they contained only one "From " line per message.

Are you sure? Did you run bin/cleanarch against the .mbox file to check it?

Once I was convinced they were all ready, I combined the old archive mbox with the current archive mbox using cat, and ran arch.

It worked perfectly, creating archive pages going all the way back to 1999, except that in the archive page for the month in which I ran arch (May) for the day on which I ran it (May 7), I have in the vicinity of 5000 entries for messages with "No subject" and no body. The index page for May looks like this:

# [Guppies] Malice 2008 Suzanne Williams # No subject # No subject # No subject ... 5000 entries # No subject # No subject # [Guppies] harsh words for cheating peg908 at aol.com # [Guppies] harsh words for cheating Vwright

This usually results from a message containing an embedded "From " somewhere in the message body. The message is archived properly under its correct date and subject, but that entry is truncated at the line that begins with "From ". Then the rest of the message is archived as a separate message. Since it has no From:, Subject: or Date: headers, it is archived with the current date and no subject. Also , text following the "From " up to the first totally empty (not just blank) line is considered part of the header and is not archived with this 'second' message.

I tried to find these mysterious entries in the current archive mbox, but they don't appear.

If there is any message body text in the 'No subject' archived entry, you should be able to find that in the .mbox.

The _only_ thing I can see, in the current mbox, is that the end of the last message from the old archives ends on one line and the "From " line for the next message begins on the very next line, with no blank lines between,

That shouldn't cause this.

and everywhere else there are either one or more blank lines or one of those message separator lines from AOL:

...

"----------MB_8C9379FAFA8ECEC_DAC_6C2A_WEBMAIL-MC05.sysops.aol.com--"<

These bogus entries aren't really hurting anything, I suppose, but they are annoying and it is irritating to have to scroll down 5000 lines to get to the next real message.

They are actually, because they represent missing pieces of other messages.

What is causing this? And is there anything I can do to get rid of the problem? I am willing to live with it if I have to, but I would prefer having a fix.

I think you have unescaped "From " lines in the bodies of messages. Run bin/cleanarch (with the -n/--dry-run option) to check.

Another possibility is you have real looking but extraneous (duplicate?) "From " lines not followed by a real message with Subject: and Date: headers prior to the next "From ".

-- Mark Sapiro msapiro@value.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Ivan Van Laningham wrote:

I ran cleanarch, yes, but all it did was to escape every single "From " line, which would make arch think there was only one message.

Then either the From line doesn't match the pattern mailbox.UnixMailbox._fromlinepattern or it is not followed immediately (with no intervening lines or maybe even '\r') by a line that looks like a message header.

If there is intervening whitespace between the "From " line and the message headers, that may cause the spurious archived empty messages.

Do lines beginning with whitespace before a From count? There are about a hundred of those in the input mbox.

They shouldn't be a problem.

-- Mark Sapiro msapiro@value.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Ivan Van Laningham wrote:

Ah. Now we're getting somewhere. Here are some sample "From " lines:

1. From the current list.mbox (leading '> ' not part of actual line):

...

From Lizzelvin@aol.com Sun Mar 18 18:17:56 2007

This is a normal Unix From_

2. From the old mbox which I want to incorporate (leading '> ' inserted):

...

From "robyn m. fritz" rfritz@nwlink.com or From Mochie@webtv.net (C Ryplansky)

These are non-standard separators

And here is the _fromlinepattern:

_fromlinepattern = r"From \s*[^\s]+\s+\w\w\w\s+\w\w\w\s+\d?\d\s+"
r"\d?\d:\d\d(:\d\d)?(\s+[^\s]+)?\s+\d\d\d\d\s*$"

Now, I don't understand much of this pattern, but it looks to me as if a) there's no provision for matching " or < or > characters; and b) some sort of date/time mark is required.

This pattern is used by cleanarch to try to separate a standard Unix From_ separator from other lines that just happen to begin with "From ". It matches "From " followed by whitespace-delimited fields containing any non-whitespace - email address 3 alphanumercs - day of week 3 alphanumerics - month 1 or 2 digits - day of month 1 or 2 digits, colon, 2 digits, optional colon and 2 digits - hh:mm(:ss) optional any non whitespace - time zone offset 4 digits - year

So yes, it looks for a single email address and a date in a specific format. The email address can be bracketed - user@example.com and doesn't really have to look like a valid email address, but it can't contain whitespace, thus it can't have a 'real name' unless it has no whitespace such as johnsmithjsmith@example.com.

This is only used by cleanarch. Pipermail doesn't care about the contents of the From_ separator. It assumes any line that begins with "From " is a separator and ignores the rest of the line.

All the "From " lines are terminated with a \n, and all are followed immediately by what look like valid message header lines, so I don't think those are problems. There do appear to be 1006 unescaped "From " lines in the old mbox:

$ grep '^From ' guppies-out.mbox | wc 46295 163728 1800087 $ grep '^From: ' guppies-out.mbox | wc 45289 159710 1803623

This seems to indicate a problem, but still doesn't account for 5000 spurious archive entries.

So, if I process the old mbox and convert the "From " lines without dates into "From " lines without " and <> and add a date/time stamp, and THEN run cleanarch, cleanarch should escape only the 1006 non-matching "From " lines, and I should end up with an mbox I can combine with March, April and May of 2007 from the current list. Is that a correct assessment?

That is correct, but if you can process the old mbox and identify which "From " lines without dates are actually message separators, then you should be able to identify which ones are not message separators and just escape those. I.e. create your own archive cleaner specific to this situation.

-- Mark Sapiro msapiro@value.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On May 25, 2007, at 9:30 AM, Ivan Van Laningham wrote:

Editing 122 MB of raw archive is going to take some time, since I have to throw away 110 MB of it. I'd like to prevent new messages from coming into the system while I'm editing it, and I seem to be overlooking instructions on how to lock the list. I find that the
help message for withlist tells me how to lock the list while I operate
on it using withlist, but is that what I want? Can I vi/emacs the mbox
while it is locked with withlist?

Am I simply obtuse, or is there no way to lock the list while I'm editing? Or do I throw caution to the winds and blithely edit without concern for incoming messages?

Hi Ivan,

Can you just turn off mailmanctl while you're editing the inbox?
Okay, this will shut down your Mailman system globally, which you
might not want to do, but it's as safe as it gets.

If you specifically want to block messages just to the list your
editing, then yes, lock it with bin/withlist and just edit the mbox.

Another option is to restart mailmanctl but temporarily disable the
ArchRunner. You should be safe to edit the mbox file then, and just
delay updating all your archives (including the mbox archives) until
you've finished your surgery, while the rest of the system continues
to churn away.

Cheers,

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin)

iQCVAwUBRlbpP3EjvBPtnXfVAQJS1wP/Qdgn08VKwRS0IHkZCy5RIYsylZGGEfEV FCNuZgi538HjwZy6sXlOGrFmDInSRvPVhXDlzhuUbbZulzjY3iqQcZh63FqDodjd DKZ1+W1V0S0c0m0dDu/ehVi5sexIrfE289ogWKahK9iEXDGAl4AXvyWT8TP927xR Xsfb5sLUKic= =iQFc -----END PGP SIGNATURE-----