[Python-Dev] under what circumstances can python still exhibit "high water mark" memory usage?

[Victor Stinner]

Hi,

You may also try tracemalloc to get stats of the Python memory usage ;-)

The Python memory allocator was optimized in Python 3.3: it now uses
mmap() when available (on UNIX), it helps to reduce the fragmentation
of the heap memory. Since Python 3.4, VirtualAlloc() is used for the
same purpose on Windows.

Please mention your OS, OS version and Python version.

The Python memory allocator allocates chunks of memory of 256 KB (see
ARENA_SIZE in Objects/obmalloc.c). A chunk cannot be released to the
system before all objects stored in the chunk are released.

The Python memory allocator is only used for allocations smaller than
256 bytes in Python <= 3.2, or allocations smaller than 512 bytes in
Python 3.3. Otherwise, malloc() and free() are used. The GNU libc uses
brk() or mmap() depending on a threshold: 128 KB by default. The
threshold is dynamic nowadays. Use mallopt(M_MMAP_THRESHOLD, nbytes)
to change this threshold.

The fragmentation of the heap memory is an hard problem not fully
solved in CPython. A moving garbage collector would reduce the
fragmentation of the "arenas" objects, but I don't think that it's
possible to change the Python garbage collector...

My test for memory fragmentation:
https://bitbucket.org/haypo/misc/src/56c61c3815a6f0a604cda0314d44081f21e8a786/memory/python_memleak.py?at=default&fileviewer=file-view-default

Other example in pure C:
https://bitbucket.org/haypo/misc/src/56c61c3815a6f0a604cda0314d44081f21e8a786/memory/tu_malloc.c?at=default&fileviewer=file-view-default

Victor

2015-10-14 15:11 GMT+02:00 Chris Withers <chris at simplistix.co.uk>:
> Hi All,
>
> I'm having trouble with some python processes that are using 3GB+ of memory
> but when I inspect them with either heapy or meliae, injected via pyrasite,
> those tools only report total memory usage to be 119Mb.
>
> This feels like the old "python high water mark" problem, but I thought that
> was fixed in 2.6/3.0?
> Under what circumstances can a Python process still exhibit high memory
> usage that tools like heapy don't know about?
>
> cheers,
>
> Chris
>
> PS: Full details here of libraries being used and versions here:
>
> https://groups.google.com/forum/#!topic/celery-users/SsTRZ7-mDMI
>
> This post feels related and seems to suggest the high water mark problem is
> still there:
>
> http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com