Approving large messages results in MemoryError
Hi, I'm Running mailman 2.1.7 on gentoo. When i want to approve a message which needs for some reason my approval (e.g. from a wrong adress or sent BCC) i "hit a bug". It seems that it does not give any problems when the message to be approved is very small.... But when an attachment is included mailman hits a bug. Very funny is that i can still discard the message only that is not what i want. What can i do to get rid of this annoying error? Peter -------------- THe resulting error in my logs is: Mar 19 16:00:11 2006 admin(22759): @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ admin(22759): [----- Mailman Version: 2.1.7 -----] admin(22759): [----- Traceback ------] admin(22759): Traceback (most recent call last): admin(22759): File "/usr/local/mailman/scripts/driver", line 101, in run_main admin(22759): main() admin(22759): File "/usr/local/mailman/Mailman/Cgi/admindb.py", line 164, in main admin(22759): process_form(mlist, doc, cgidata) admin(22759): File "/usr/local/mailman/Mailman/Cgi/admindb.py", line 714, in process_form admin(22759): forward, forwardaddr) admin(22759): File "/usr/local/mailman/Mailman/ListAdmin.py", line 167, in HandleRequest admin(22759): forward, addr) admin(22759): File "/usr/local/mailman/Mailman/ListAdmin.py", line 272, in __handlepost admin(22759): msg = readMessage(path) admin(22759): File "/usr/local/mailman/Mailman/ListAdmin.py", line 581, in readMessage admin(22759): msg = cPickle.load(fp) admin(22759): MemoryError admin(22759): [----- Python Information -----] admin(22759): sys.version = 2.4.2 (#1, Oct 15 2005, 14:18:45) [GCC 3.3.5-20050130 (Gentoo 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie-8.7.7. admin(22759): sys.executable = /usr/bin/python admin(22759): sys.prefix = /usr admin(22759): sys.exec_prefix = /usr admin(22759): sys.path = /usr admin(22759): sys.platform = linux2 admin(22759): [----- Environment Variables -----] admin(22759): HTTP_COOKIE: spam+admin=28020000006962621d44732800000062613662313033616262666335363731353 330303339306234393064616232636432663431356230 admin(22759): SERVER_SOFTWARE: Apache admin(22759): SCRIPT_NAME: /mailman/admindb admin(22759): SERVER_SIGNATURE: <address>Apache Server at xxx.xxxxxx.xxx Port 80</address> admin(22759): admin(22759): REQUEST_METHOD: POST admin(22759): HTTP_KEEP_ALIVE: 300 admin(22759): SERVER_PROTOCOL: HTTP/1.1 admin(22759): QUERY_STRING: admin(22759): CONTENT_LENGTH: 136 admin(22759): HTTP_ACCEPT_CHARSET: ISO-8859-1,utf-8;q=0.7,*;q=0.7 admin(22759): HTTP_USER_AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1 admin(22759): HTTP_CONNECTION: keep-alive admin(22759): HTTP_REFERER: http:// <http://dory.kabelfoon.net/mailman/admindb/spam> xxx.xxxxxx.xxx/mailman/admindb/spam admin(22759): SERVER_NAME: xxx.xxxxxx.xxx admin(22759): REMOTE_ADDR: xxx.xxxxxx.xxx admin(22759): PATH_TRANSLATED: /data/hosting/localhost/htdocs/spam admin(22759): SERVER_PORT: 80 admin(22759): SERVER_ADDR: xxx.xxxxxx.xxx admin(22759): DOCUMENT_ROOT: /data/hosting/localhost/htdocs/ admin(22759): PYTHONPATH: /usr/local/mailman admin(22759): SCRIPT_FILENAME: /usr/local/mailman/cgi-bin/admindb admin(22759): SERVER_ADMIN: pveerman@ <mailto:pveerman@nemo.kabelfoon.net> xxx.xxxxxx.xxx admin(22759): HTTP_HOST: xxx.xxxxxx.xxx admin(22759): REQUEST_URI: /mailman/admindb/spam admin(22759): HTTP_ACCEPT: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q= 0.8,image/png,*/*;q=0.5 admin(22759): GATEWAY_INTERFACE: CGI/1.1 admin(22759): REMOTE_PORT: 2641 admin(22759): HTTP_ACCEPT_LANGUAGE: en-us,en;q=0.5 admin(22759): CONTENT_TYPE: application/x-www-form-urlencoded admin(22759): HTTP_ACCEPT_ENCODING: gzip,deflate admin(22759): PATH_INFO: /spam
Peter Veerman wrote:
I'm Running mailman 2.1.7 on gentoo.
When i want to approve a message which needs for some reason my approval (e.g. from a wrong adress or sent BCC) i "hit a bug". It seems that it does not give any problems when the message to be approved is very small.... But when an attachment is included mailman hits a bug.
How big are these attachments?
<snip>
admin(22759): File "/usr/local/mailman/Mailman/ListAdmin.py", line 581, in readMessage admin(22759): msg = cPickle.load(fp) admin(22759): MemoryError
The Python process run via cgi from the web server has run out of memory in attempting to retrieve the message from the held message pickle where it's stored. This is a file in Mailman's 'data' directory named heldmsg-listname-nnn.pck (where listname is the name of the list and nnn is a sequence number). How big is this file? can you successfully dump it with bin/dumpdb? Does your OS or web server somehow limit the size of cgi programs?
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Peter Veerman wrote:
I'm Running mailman 2.1.7 on gentoo.
When i want to approve a message which needs for some reason my approval (e.g. from a wrong adress or sent BCC) i "hit a bug". It seems that it does not give any problems when the message to be approved is very small.... But when an attachment is included mailman hits a bug. How big are these attachments?
Ranging from 1 to 5 MB
<snip>
admin(22759): File "/usr/local/mailman/Mailman/ListAdmin.py", line 581, in readMessage admin(22759): msg = cPickle.load(fp) admin(22759): MemoryError
The Python process run via cgi from the web server has run out of memory in attempting to retrieve the message from the held message pickle where it's stored. This is a file in Mailman's 'data' directory named heldmsg-listname-nnn.pck (where listname is the name of the list and nnn is a sequence number). How big is this file?
This file is 1.4 MB
can you successfully dump it with bin/dumpdb?
Yes...Just 1.4 MB of data flashed ran across my terminal :)
Does your OS or web server somehow limit the size of cgi programs?
Well I run apache2 and to be onest, i have not got a clue about this. I searched my configuration files for any indications for this and also googled around without any result :(
Peter Veerman wrote:
How big are these attachments?
Ranging from 1 to 5 MB
<snip>
How big is this file?
This file is 1.4 MB
can you successfully dump it with bin/dumpdb?
Yes...Just 1.4 MB of data flashed ran across my terminal :)
Does your OS or web server somehow limit the size of cgi programs?
Well I run apache2 and to be onest, i have not got a clue about this. I searched my configuration files for any indications for this and also googled around without any result :(
This seems to be an issue beyond either Mailman or Python. Clearly Mailman's IncomingRunner process can receive the incoming post, parse it into a message instance and ultimately pickle it to the heldmsg file.
The stand alone bin/dumpdb can unpickle it and display it, but when the admindb interface running as a cgi attempts to unpickle it, it gets a MemoryError exception. I think it has to be due to some limit on the cgi from apache2, but I don't know what that would be.
As a workaround, you could try approving the message by email. I didn't ask whether you have admin_immed_notify on so you receive individual notices of these posts, but if you do, the notice (in addition to being additional evidence that Mailman's IncomingRunner has no problem with this size message) has an attached message/rfc822 part at the end following the attached original post.
This final attached message is From: the listname-request address and has a subject like 'confirm 72bd7cc3e10add5cd7577039f77be82112f237d1' (with different hex digits). It has instructions in the body, but simply, if you send a message back to the listname-request address with the same subject and the first line of the body being
Approved: list_password
where list_password is the actual list admin or moderator password, the original message will be approved. Without the Approved: line, the original message will be discarded.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
How big are these attachments?
Ranging from 1 to 5 MB
<snip>
I solved it thanks to your (Mark Sapiro) clear explanation about the inner working of mailman.
As you mentioned that it wasnt mailman causing the problem i went into my configfiles of apache again and this time i found something i looked over before:
In the virtual host config running the mailman cgi script there were the following lines:
RlimitCPU 50 RlimitMem 15000000
After commenting them out....IT WORKED!
As i have seen quite a lot of messages with alike errors without any solution it might be interesting to add this to your FAQ?
Thanks!
Peter
Peter Veerman wrote:
As i have seen quite a lot of messages with alike errors without any solution it might be interesting to add this to your FAQ?
I just added <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.018.htp>.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark Sapiro -
Peter Veerman