Why on CentOS, python consumes too much memory ?

snowingbear at gmail.com snowingbear at gmail.com
Thu Jul 18 06:45:01 CEST 2013


   Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M.
On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M).

    And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python:
import time

    I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/<pid>/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS 6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M.

     I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much.

Best Regards

More information about the Python-list mailing list