[Mailman-Users] Mailman Borks on Large Attachments

[Mark Sapiro]

Lachlan Michael wrote:
>
>I have a problem when using mailman. The problem manifests itself when a
>user sends a largish attachment (say 300kB) such as a digital camera
>picture to a list. The lists I run are small private groups and if
>possible  I would like to allow users to send these kind of attachments.
>
>The server is question is vanilla FreeBSD 6.3 machine, using mailman
>2.1.9_5 from ports with sendmail. The machine has 1GB of physical memory.
>Python 2.5 is used (default settings). The maximum message size for this
>list was set at 150kB, in an effort to stop this problem (which it didn't
>seem to do), but I would like to set this limit higher (maybe to 1MB).


Setting max_message_size doesn't help because the error occurs before
the message size is checked.


<snip>
>Mailman fails with a "memory error" (see logs below).
>
>This error then caused ALL mail to ALL other lists on the same machine to
>be stop being processed, although they were queued and could be sent
>later. Eventually some days later I realized that mail was not getting
>through and investigated.
>
>A simple restart of mailman did not help, and I found a suspicious file in
>qfiles. The suspicious file had a .bak extension I moved it away. Then a
>restart of mailman allowed all queued messages to be sent correctly (with
>the exception of the suspicious mail file).


This part of the problem is fixed in Mailman 2.1.10b3 which is now
available (see <http://www.list.org/download.html>).

Prior to Mailman 2.1.9, the error would have occurred once and the
message would have been lost. Beginning in 2.1.9, a queue file
backup/recovery mechanism was implemented which allowed messages that
were in process when a crash occurred to be recovered, but the problem
here is that if the message itself causes the crash, a loop occurs
such as you experienced. This is addressed in 2.1.10b3 by counting the
number of times a .bak file has been recovered and moving it aside
after 3 tries.

It would be wonderful if you could install the latest
Mailman/Queue/Switchboard.py from
<http://codebrowse.launchpad.net/~mailman-coders/mailman/2.1/files>
and see if this mitigates the problem.



>Using less I saw that the mail
>included a large picture file.  There are 11 members in the list that this
>mail was sent to. The offending mail file size is shown below (ls -lh).
>
>-rw-rw----   1 mailnull  mailman   599K Jan 30 23:36
>1201703766.518079+a5beef9fbd8e4eea60ab77049b3bed4e877a98fc.bak
>
>My questions is, is this a bug, a mis-configuration on my part, or
>expected behaviour of mailman under these conditions for this machine
>(i.e. am I hitting python or FreeBSD limits)?


MemoryError is beyond the control of both Mailman and Python. Python is
requesting additional memory form the OS for the process and is being
denied. This is some kind of hardware or OS limitation. It could be
that the OS itself is running out of real or virtual memory or the
process is running into some process limit. This is a FreeBSD question.


>My expectation would be that
>1) the user gets an error that the file sent was too big


We can't do that in this case, because we haven't successfully parsed
the message so we don't even know who the sender is. If there were no
MemoryError, and the message just exceeded max_message_size, this is
exactly what would happen.


>2) all other lists and other mail are not affected


The 2.1.10b3 fix addresses this. However, the way things are currently
set, Incoming runner will crash 3 times before moving this message
aside. If this happens again with a second message, IncomingRunner
will crash 3 more times. On the fourth such message, IncomingRunner
will reach its limit of 10 restarts, so if these errors continue,
Mailman's mail processing will eventually stop.


>I guess many people would be jumping up and down if this occurred
>regularly, so I imagine that this a local configuration problem, but I
>would appreciate if anybody had any advice for debugging or changing my
>environment to remove this error.


500KB is not a big message. This should not be happening. I am only
guessing, but I think there must be some per-process memory limitation
in your OS that is set way too low.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan