[Mailman-Users] Old messages in bounces and in

[Mark Sapiro]

Sebastian Hagedorn wrote:
>
>we've been running Mailman for several years now (currently version 2.1.9), 
>but up to now we've neglected some housekeeping. Actually that's a 
>testament to how well it works for the most part! Still, I guess there are 
>things we should be doing. Recently we've started running the Daily Status 
>Report script. There are some issues that I don't understand. Here's one:
>
>Subdirectory: bounces
>==============================
>Entries: 37
>------------------------------
>total 304
>drwxrws---    2 mailman  mailman      8192 Mar  6 23:57 .
>drwxrwsr-x   12 root     mailman      4096 Dec 27  2006 ..
>-rw-rw----    1 mail     mailman      3115 Jan  9  2007 
>1168304393.8014059+653abf0ec76b61ef5f5245e7f08d944c441bc306.bak
>-rw-rw----    1 mail     mailman      3074 Jan 27  2007 
>1169866007.5216539+598123641f4820025a7c1b576d18b61e04948bd7.bak
<snip>
>
>Subdirectory: in
>==============================
>Entries: 159
>------------------------------
>total 1100
>drwxrws---    2 mailman  mailman    212992 Mar  6 23:58 .
>drwxrwsr-x   12 root     mailman      4096 Dec 27  2006 ..
>-rw-rw----    1 mail     mailman      4140 Jan  3  2007 
>1167840137.109689+79891fb75a203b544dd071b9c7b1f03932b60587.bak
>-rw-rw----    1 mail     mailman      5837 Jan  4  2007 
>1167894274.0668249+207e2bf67b4335f2085b20b5afdffdbb9d7a36fe.bak
<snip>
>
>What are those? Can I just delete them? How do they end up there?


Good questions.

Beginning with Mailman 2.1.9, a mechanism was implemented to prevent
message loss in case of a system crash or queue runner hard death. The
mechanism is when dequeueing an entry, we rename the queue file with a
.bak extension. Then when we are finished processing that entry we
remove the .bak file.

If a runner dies and is restarted, it recovers any .bak files by
renaming them to .pck and they will be processed normally.

There are two problems in 2.1.9 as released that are fixed in 2.1.10.

It is possible for a message itself to cause a runner to die which in
turn causes the runner to be restarted and reprocess the message and
die again until the restart limit (10) is reached. This is fixed by
limiting the number of times a specific .bak file can be recovered
before it is logged and moved aside (to a .psv file in the shunt
queue).

Also, There was an oversight having to do with unparseable messages. I
think this is the issue in your case. If you look in mailman's error
log, I think you'll see for each one of these files, an entry like
"Ignoring unparseable message:
1168304393.8014059+653abf0ec76b61ef5f5245e7f08d944c441bc306". If so,
what is happening is malformed MIME messages which are almost
certainly spam are being sent to your list and list-bounces addresses.
This causes the parse error to be logged and the message dropped, but
we neglected to deal with the .bak file. In 2.1.10 we move it aside to
a .psv file in the shunt queue.

The bottom line, if I am correct, is these are spam and can be safely
just deleted.

If you are not ready to upgrade to 2.1.10, you can patch 2.1.9. Find
lines 107-109 in Mailman/Queue/Runner.py. They should be

                self._log(e)
                syslog('error', 'Ignoring unparseable message: %s',
filebase)
                continue

(the syslog line is wrapped here, but not in the module). Add a line to
make it

                self._log(e)
                syslog('error', 'Ignoring unparseable message: %s',
filebase)
                self._switchboard.finish(filebase)
                continue

That won't preserve the entry in the shunt queue, but it will prevent
.bak files from accumulating.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan