[Mailman-Users] Memory usage climbs for 3rd and 4th outgoingqrunner process

[Ivan Fetch]

Hi Mark,

On Aug 22, 2011, at 7:42 PM, Mark Sapiro wrote:

> Ivan Fetch wrote:
>> 
>> I am curious what would cause an outgoing queue runner process to increase in memory usage, when Mailman is idle?
> 
> 
> [...]
>> 
>> Even when Mailman is idle and not processing messages, the memory usage for the fourth outgoing runner process remains at 200Mb and continues to rise.
>> 
>> Last week, after running this way for 3 days, this 4th process was using 800 Mb of memory - at the time, Mailman runners were restarted (although Mailman was idle, and would handle messages quickly). WE may be at that point again tomorrow - I'd like to gather what info I can about this process before it runs away.
> 
> 
> I can almost guarantee it won't run away. Eventually it will reach a
> point where every list object is cached and a maximum sized message
> has been handled, and it will stop growing. Eventually, all 4
> OutgoingRunners will reach this size (unless this size is due to
> processing an anomalously large message, but that doesn't appear to be
> the case here as large messages also affect IncomingRunner).
> 
> There are threads on this in the archives. Some relevant ones are at
> <http://mail.python.org/pipermail/mailman-users/2007-December/059332.html>,
> <http://mail.python.org/pipermail/mailman-users/2008-July/062359.html>
> and
> <http://mail.python.org/pipermail/mailman-users/2010-March/069075.html>.
> 
> In particular, see the post at
> <http://mail.python.org/pipermail/mailman-users/2010-March/069078.html>
> which includes a patch to disable list caching in the qrunners if you
> want to try that, but I suggest you first just wait, and I predict the
> memory growth will slow and eventually stop and it won't impact
> performance because at any given time most of the memory is inactive
> and will be swapped out if needed for other processes.
> 

I've let things run, and although there is still some growth, the box has not hit 97% (of 2Gb) memory usage like it did once before. 

# ps -Umailman -o pmem -o size -o args |sort -n
%MEM    SZ COMMAND
 0.2  7388 /usr/bin/python /mail/mailman/mailman/bin/mailmanctl start
 0.4  7408 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
 0.8 12740 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
 1.3 23032 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
 1.3 24872 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
 2.4 48280 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
 2.6 50060 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:0:4 -s
 3.4 67028 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:1:4 -s
 3.6 70140 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
 3.8 84208 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
 6.6 133052 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:2:4 -s
10.1 205020 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:3:4 -s


You mentioned large messages causing similar behavior, but also impacting the incomingrunner. How would I tell whether that is the case?


> Also, you can add up all the sizes of the lists/*/config.pck files to
> get a very rough idea of how much things may grow.

This came out to 32Mb, for 1360 lists.


Thanks,

Ivan.























.