[Mailman-Users] Python process size grows 30x in 8 hours (memory

[Brad Knowles]

On 7/1/08, Fletcher Cocquyt wrote:

>  Pmap shows its the heap
>
>  god at irt-smtp-02:in 8:08pm 64 # pmap 24167
>  24167:    /bin/python /opt/mailman-2.1.9/bin/qrunner
>  --runner=IncomingRunner:5:8
>  08038000      64K rwx--    [ stack ]
>  08050000     940K r-x--  /usr/local/stow/Python-2.5.2/bin/python
>  0814A000     172K rwx--  /usr/local/stow/Python-2.5.2/bin/python
>  08175000  312388K rwx--    [ heap ]
>  CF210000      64K rwx--    [ anon ]
>  <--many small libs -->
>  total    318300K

And when I do the same thing on the mail server for python.org (which 
hosts over 100 lists, including some pretty active lists with large 
numbers of subscribers), on the largest queue runner we have 
(ArchRunner at 41m), I see:

# pmap 1040 | sort -nr -k 2 | head
  total    45800K
0815f000  23244K rwx--    [ anon ]
40f61000   4420K rw---    [ anon ]
40a0f000   2340K rw---    [ anon ]
408aa000   1300K rw---    [ anon ]
40745000   1300K rw---    [ anon ]
40343000   1160K r-x--  /usr/lib/i686/cmov/libcrypto.so.0.9.8
4009c000   1092K r-x--  /lib/libc-2.3.6.so
41844000   1040K rw---    [ anon ]
08048000    944K r-x--  /usr/local/bin/python

No heap showing up anywhere.  Doing the same for our IncomingRunner, I get:

# pmap 1043 | sort -nr -k 2 | head
  total    23144K
0815f000   7740K rwx--    [ anon ]
40b12000   1560K rw---    [ anon ]
40745000   1300K rw---    [ anon ]
40cb8000   1168K rw---    [ anon ]
40347000   1160K r-x--  /usr/lib/i686/cmov/libcrypto.so.0.9.8
4009c000   1092K r-x--  /lib/libc-2.3.6.so
4098d000   1040K rw---    [ anon ]
08048000    944K r-x--  /usr/local/bin/python
4063b000    936K rw---    [ anon ]

Again, no heap.

>  None of the lists seem too big:
>  god at irt-smtp-02:lists 8:24pm 73 # du -sk */*pck | sort -nr | head | awk
>  '{print $1}'
>  1392
>  1240
>  1152
>  1096
>  912
>  720
>  464
>  168
>  136
>  112

Where did you do this?  In the /usr/local/mailman directory?

When I did this in /usr/local/mailman, all of the .pck files that 
showed up were actually held messages in the data/ directory, not in 
lists/.  This would mean that they were individual messages that had 
been pickled and then held for moderation, not pickles for lists.

Doing the same in /usr/local/mailman/lists, I find that one of our 
smaller mailing lists (python-help, seventeen recipients) has the 
largest list pickle (1044 kilobytes).  We have a total of 150 lists, 
and here's the current subscription count of the five biggest lists:

    4075 Python-list
    3305 Tutor
    2600 Mailman-Users
    2329 Mailman-announce
    1528 Python-announce-list

Of these, python-list and tutor frequently gets between twenty to a 
hundred or more messages in a day.  However, here's their respective 
list.pck files, using the same "du -sk" script from above:

904     tutor/config.pck
652     python-list/config.pck
476     mailman-users/config.pck
324     mailman-announce/config.pck
208     python-announce-list/config.pck

-- 
Brad Knowles <brad at shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>