Memory usage climbs for 3rd and 4th outgoing qrunner process
Hello,
I am curious what would cause an outgoing queue runner process to increase in memory usage, when Mailman is idle?
I am running 4 outgoing qrunner slices, using this in mm_cfg.py:
try: QRUNNERS.remove(('OutgoingRunner', 1)) QRUNNERS.append(('OutgoingRunner', 4)) except ValueError: pass
I've noticed that the first two processes use 20 MB of memory, the third has climbed to 29MB, and the fourth has climbed to 200Mb - over the past two days.
Even when Mailman is idle and not processing messages, the memory usage for the fourth outgoing runner process remains at 200Mb and continues to rise.
Last week, after running this way for 3 days, this 4th process was using 800 Mb of memory - at the time, Mailman runners were restarted (although Mailman was idle, and would handle messages quickly). WE may be at that point again tomorrow - I'd like to gather what info I can about this process before it runs away.
Here is what these look like right now:
# ps -ax -o pmem -o size -o args |sort -n |tail Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ 0.9 184936 /usr/sbin/named -u named 1.2 20712 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=BounceRunner:0:1 -s 1.3 23032 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=CommandRunner:0:1 -s 1.3 25164 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=RetryRunner:0:1 -s 1.4 25400 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:1:4 -s 1.7 31936 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=VirginRunner:0:1 -s 2.1 41292 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:2:4 -s 2.2 42612 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s 2.9 55456 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:0:4 -s 10.1 205020 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:3:4 -
Thanks,
Ivan.
.
Ivan Fetch wrote:
I am curious what would cause an outgoing queue runner process to increase in memory usage, when Mailman is idle?
[...]
Even when Mailman is idle and not processing messages, the memory usage for the fourth outgoing runner process remains at 200Mb and continues to rise.
Last week, after running this way for 3 days, this 4th process was using 800 Mb of memory - at the time, Mailman runners were restarted (although Mailman was idle, and would handle messages quickly). WE may be at that point again tomorrow - I'd like to gather what info I can about this process before it runs away.
I can almost guarantee it won't run away. Eventually it will reach a point where every list object is cached and a maximum sized message has been handled, and it will stop growing. Eventually, all 4 OutgoingRunners will reach this size (unless this size is due to processing an anomalously large message, but that doesn't appear to be the case here as large messages also affect IncomingRunner).
There are threads on this in the archives. Some relevant ones are at <http://mail.python.org/pipermail/mailman-users/2007-December/059332.html>, <http://mail.python.org/pipermail/mailman-users/2008-July/062359.html> and <http://mail.python.org/pipermail/mailman-users/2010-March/069075.html>.
In particular, see the post at <http://mail.python.org/pipermail/mailman-users/2010-March/069078.html> which includes a patch to disable list caching in the qrunners if you want to try that, but I suggest you first just wait, and I predict the memory growth will slow and eventually stop and it won't impact performance because at any given time most of the memory is inactive and will be swapped out if needed for other processes.
Also, you can add up all the sizes of the lists/*/config.pck files to get a very rough idea of how much things may grow.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
On Aug 22, 2011, at 7:42 PM, Mark Sapiro wrote:
Ivan Fetch wrote:
I am curious what would cause an outgoing queue runner process to increase in memory usage, when Mailman is idle?
[...]
Even when Mailman is idle and not processing messages, the memory usage for the fourth outgoing runner process remains at 200Mb and continues to rise.
Last week, after running this way for 3 days, this 4th process was using 800 Mb of memory - at the time, Mailman runners were restarted (although Mailman was idle, and would handle messages quickly). WE may be at that point again tomorrow - I'd like to gather what info I can about this process before it runs away.
I can almost guarantee it won't run away. Eventually it will reach a point where every list object is cached and a maximum sized message has been handled, and it will stop growing. Eventually, all 4 OutgoingRunners will reach this size (unless this size is due to processing an anomalously large message, but that doesn't appear to be the case here as large messages also affect IncomingRunner).
There are threads on this in the archives. Some relevant ones are at <http://mail.python.org/pipermail/mailman-users/2007-December/059332.html>, <http://mail.python.org/pipermail/mailman-users/2008-July/062359.html> and <http://mail.python.org/pipermail/mailman-users/2010-March/069075.html>.
In particular, see the post at <http://mail.python.org/pipermail/mailman-users/2010-March/069078.html> which includes a patch to disable list caching in the qrunners if you want to try that, but I suggest you first just wait, and I predict the memory growth will slow and eventually stop and it won't impact performance because at any given time most of the memory is inactive and will be swapped out if needed for other processes.
I've let things run, and although there is still some growth, the box has not hit 97% (of 2Gb) memory usage like it did once before.
# ps -Umailman -o pmem -o size -o args |sort -n %MEM SZ COMMAND 0.2 7388 /usr/bin/python /mail/mailman/mailman/bin/mailmanctl start 0.4 7408 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=NewsRunner:0:1 -s 0.8 12740 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=ArchRunner:0:1 -s 1.3 23032 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=CommandRunner:0:1 -s 1.3 24872 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=RetryRunner:0:1 -s 2.4 48280 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s 2.6 50060 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:0:4 -s 3.4 67028 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:1:4 -s 3.6 70140 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=BounceRunner:0:1 -s 3.8 84208 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=VirginRunner:0:1 -s 6.6 133052 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:2:4 -s 10.1 205020 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:3:4 -s
You mentioned large messages causing similar behavior, but also impacting the incomingrunner. How would I tell whether that is the case?
Also, you can add up all the sizes of the lists/*/config.pck files to get a very rough idea of how much things may grow.
This came out to 32Mb, for 1360 lists.
Thanks,
Ivan.
.
On 8/24/2011 7:59 PM, Ivan Fetch wrote:
I've let things run, and although there is still some growth, the box has not hit 97% (of 2Gb) memory usage like it did once before.
But the question is when the box is at 97% memory utilization, how much of that memory is actually needed vs. inactive pages that are memory resident simply because no other process actually needs the memory.
# ps -Umailman -o pmem -o size -o args |sort -n %MEM SZ COMMAND 0.2 7388 /usr/bin/python /mail/mailman/mailman/bin/mailmanctl start 0.4 7408 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=NewsRunner:0:1 -s 0.8 12740 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=ArchRunner:0:1 -s 1.3 23032 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=CommandRunner:0:1 -s 1.3 24872 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=RetryRunner:0:1 -s 2.4 48280 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s 2.6 50060 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:0:4 -s 3.4 67028 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:1:4 -s 3.6 70140 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=BounceRunner:0:1 -s 3.8 84208 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=VirginRunner:0:1 -s 6.6 133052 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:2:4 -s 10.1 205020 /usr/bin/python /mail/mailman/mailman/bin/qrunner --runner=OutgoingRunner:3:4 -s
You mentioned large messages causing similar behavior, but also impacting the incomingrunner. How would I tell whether that is the case?
A large message is queued for Mailman and the queue entry is at least as large as the raw message. When the message is processed, each runner that handles it winds up with the entire message in memory and thus must grow large enough to hold the message.
What I was saying is that in a situation like that above, it would not be the case that slices 2 or 3 of OutgoingRunner grew to 133 and 205 MB respectively because of large messages because those messages would have passed through IncomingRunner and it would have grown comparably large.
Also, you can add up all the sizes of the lists/*/config.pck files to get a very rough idea of how much things may grow.
This came out to 32Mb, for 1360 lists.
The implication of this is that if you leave things long enough, eventually all the runners will grow to > 32 MB. Four have already, one is significantly larger which says that I don't really know how to calculate/explain all the memory usage, but the aggregate size of all the list objects is definitely a lower bound on how big the qrunners will grow.
If you find that this is a real problem, you can try the patch at <http://mail.python.org/pipermail/mailman-users/attachments/20100312/24957034...> which removes the qrunner list cache. But, I don't think this is actually necessary because even though they are large, most of the memory will be swapped out for long periods if it is needed by other processes.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Ivan Fetch -
Mark Sapiro