[Mailman-Developers] Problems if shunting fails

[Mark Sapiro]

There is a bug report at
<https://sourceforge.net/tracker/index.php?func=detail&aid=1656289&group_id=103&atid=100103>.

The basic issue here is there is an extremely large message in the 'in'
queue. In followup emails with the submitter, he said "The file was
nearly half a terabyte in size".

Not surprisingly, IncomingRunner threw a MemoryError when SpamDetect
tried to flatten() the message, but then Runner tried to shunt the
message and Switchboard.enqueue() threw another MemoryError in the
attempt to pickle the new shunt queue entry.

The second MemoryError is uncaught in Runner._oneloop, so it causes the
main loop in Runner.run to exit.

Pre-Mailman 2.1.9, the master just restarts IncomingRunner - the
message is lost, but everything else is OK.

Because of the changes in 2.1.9 to prevent message loss in case of
disaster, there is now a .bak file left in the 'in' queue. When
IncomingRunner restarts, it recovers the .bak file and the whole
scenario repeats until the master reaches MAX_RESTARTS on
IncomingRunner and we are left with no IncomingRunner and the .bak
file still in the 'in' queue.

In order to fix this, I suggest we protect the shunt enqueue in a try.
I have worked up a patch for this which is attached. This patch also
adds a 'preserve' argument to Switchboard.finish such that if it is
called with preserve=True, instead of removing the .bak file, it just
renames it with a .psv extension. These changes ensure that
IncomingRunner doesn't exit, and no .bak file is left to cause a
subsequent problem while still preserving the original queue entry for
further analysis if possible.

I would like some feedback on whether or not this is the right approach.

-- 
Mark Sapiro <msapiro at value.net>       The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch.txt
Url: http://mail.python.org/pipermail/mailman-developers/attachments/20070210/a5da4088/attachment.txt