[Mailman-Users] Python process size grows 30x in 8 hours (memory

[Mark Sapiro]

Fletcher Cocquyt wrote:

>Pmap shows its the heap
>
>god at irt-smtp-02:in 8:08pm 64 # pmap 24167
>24167:    /bin/python /opt/mailman-2.1.9/bin/qrunner
>--runner=IncomingRunner:5:8
>08038000      64K rwx--    [ stack ]
>08050000     940K r-x--  /usr/local/stow/Python-2.5.2/bin/python
>0814A000     172K rwx--  /usr/local/stow/Python-2.5.2/bin/python
>08175000  312388K rwx--    [ heap ]
>CF210000      64K rwx--    [ anon ]
><--many small libs -->
> total    318300K
>
>Whether its a leak or not - we need to understand why the heap is growing
>and put a limit on its growth to avoid exausting memory and swapping into
>oblivion...


At this point, I don't think it's a leak.

Your runners start out at about 9.5 MB. Most of your working runners
grow to about the 20-40 MB range which I don't think is unusual for a
site with some config.pck files approaching 1.4 MB.

Only your IncomingRunners seem to grow really big, and I think that is
because you are seeing occasional, very large messages, or perhaps it
has something to do with your custom spam filtering interface.

Does your MTA limit incoming message size?

In any case, I know you're reluctant to just let it run, but I think if
you did let it run for a couple of days that the IncomingRunners
wouldn't get any bigger than the 310 +- MB that you're already seeing
after 3 hours, and the rest of the runners would remain in the 10 - 50
MB range.

I don't think you'll see a lot of paging activity of that 300+MB
because I suspect that most of the time nothing is going on in most of
that memory


>None of the lists seem too big:
>god at irt-smtp-02:lists 8:24pm 73 # du -sk */*pck | sort -nr | head | awk
>'{print $1}'
>1392
>1240
>1152
>1096
>912
>720
>464
>168
>136
>112
>
>Researching python heap alloaction....


You may also be interested in the FAQ article at
<http://wiki.list.org/x/94A9>.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan