[Mailman-Developers] [Mailman-Users] Python process size grows 30x in 8 hours (memory

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 2, 2008, at 11:15 AM, Mark Sapiro wrote:

> At some point in the next day or so, I'm going to make a modified
> scripts/post script which will queue incoming messages in qfiles/bad
> and then move them to qfiles/in only if they are under a certain size.
> I'm really curious to see if that will help.

This should be moved to mailman-developers, but in general it's an  
interesting idea. In MM3 I've split the incoming queue into two  
separate queues.  The incoming queue now solely determines the  
disposition of the message, i.e. held, rejected, discarded or  
accepted.  If accepted, the message is moved to a pipeline queue where  
it's munged for delivery (i.e. headers and footers added, etc.).

MM3 also has an LMTP queue runner, which I'd like to make the default  
delivery mechanism for 3.0 and possibly 2.2 (yes, I still have a todo  
to back port MM3's new process architecture to 2.2).  Although it's  
not there right now, it would be trivial to add a check on the raw  
size of the message before it's parsed.  If it's too large then it can  
be rejected before the email package attempts to parse it, and that  
would give the upstream LTMP client (i.e. your MTA) a better diagnostic.

It still makes sense to put a size limit in your MTA so it never hits  
the LMTP server because the string will still be in the Python  
process's memory.  But at least you won't pay the penalty for parsing  
such a huge message just to reject it later.

>> Trying to find a way to look at the contents of the heap or at  
>> least limit
>> its growth.
>> Or is there not a way expire & restart mailman processes analogous  
>> to the
>> apache httpd process expiration (designed to mitigate this kind of  
>> resource
>> growth over time)?
>
>
> bin/mailmanctl could be modified to do this automatically, but
> currently only does it on command (restart) or signal (SIGINT), but I
> gather you're already running a cron that does a periodic restart.

This is a good idea.  It might be better to do this in  
Runner._doperiodic().

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkhrpt4ACgkQ2YZpQepbvXHAtgCgj0E1IJjf4kkv4TNKkzcB+RFF
VxAAn1k01dLfPeKPcOgMxDneSyEB/5Ro
=qiO5
-----END PGP SIGNATURE-----