On Wed, Oct 14, 2015 at 3:11 PM, Chris Withers <chris@simplistix.co.uk> wrote:

I'm having trouble with some python processes that are using 3GB+ of memory but when I inspect them with either heapy or meliae, injected via pyrasite, those tools only report total memory usage to be 119Mb.

This feels like the old "python high water mark" problem, but I thought that was fixed in 2.6/3.0? Under what circumstances can a Python process still exhibit high memory usage that tools like heapy don't know about?

Which Python version are you experiencing this with? I know that in Python 2.7, having many floats (and I think also ints) active at once creates a high water situation. Python 2.7 is what I have experience with -- with heap sizes around 40 GB sometimes.

Hi, You may also try tracemalloc to get stats of the Python memory usage ;-) The Python memory allocator was optimized in Python 3.3: it now uses mmap() when available (on UNIX), it helps to reduce the fragmentation of the heap memory. Since Python 3.4, VirtualAlloc() is used for the same purpose on Windows. Please mention your OS, OS version and Python version. The Python memory allocator allocates chunks of memory of 256 KB (see ARENA_SIZE in Objects/obmalloc.c). A chunk cannot be released to the system before all objects stored in the chunk are released. The Python memory allocator is only used for allocations smaller than 256 bytes in Python <= 3.2, or allocations smaller than 512 bytes in Python 3.3. Otherwise, malloc() and free() are used. The GNU libc uses brk() or mmap() depending on a threshold: 128 KB by default. The threshold is dynamic nowadays. Use mallopt(M_MMAP_THRESHOLD, nbytes) to change this threshold. The fragmentation of the heap memory is an hard problem not fully solved in CPython. A moving garbage collector would reduce the fragmentation of the "arenas" objects, but I don't think that it's possible to change the Python garbage collector... My test for memory fragmentation: https://bitbucket.org/haypo/misc/src/56c61c3815a6f0a604cda0314d44081f21e8a786/memory/python_memleak.py?at=default&fileviewer=file-view-default Other example in pure C: https://bitbucket.org/haypo/misc/src/56c61c3815a6f0a604cda0314d44081f21e8a786/memory/tu_malloc.c?at=default&fileviewer=file-view-default Victor 2015-10-14 15:11 GMT+02:00 Chris Withers <chris@simplistix.co.uk>:

Hi All,

I'm having trouble with some python processes that are using 3GB+ of memory but when I inspect them with either heapy or meliae, injected via pyrasite, those tools only report total memory usage to be 119Mb.

This feels like the old "python high water mark" problem, but I thought that was fixed in 2.6/3.0? Under what circumstances can a Python process still exhibit high memory usage that tools like heapy don't know about?

cheers,

Chris

PS: Full details here of libraries being used and versions here:

https://groups.google.com/forum/#!topic/celery-users/SsTRZ7-mDMI

This post feels related and seems to suggest the high water mark problem is still there:

http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-pytho... _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...