Reduce memory fragmentation with MALLOC_MMAP_THRESHOLD_ and MALLOC_MMAP_MAX_

Lorenzo Bolla lbolla at gmail.com
Fri Feb 26 11:43:28 EST 2016


Hi all,

I've been experimenting with MALLOC_MMAP_THRESHOLD_ and MALLOC_MMAP_MAX_ env variables to affect memory management in a long-running Python 2 process.
See http://man7.org/linux/man-pages/man3/mallopt.3.html

I got the idea from this bug report: http://bugs.python.org/issue11849

The results I have are encouraging: memory fragmentation is reduced and the typical high-water mark visible in memory used by long-running processes is lower.

My only concern is if there are other side effects that may bite back, when using such low level tweaks. Does anyone have any experience in using them?

Here is an example script that shows how those variables affect RSS memory in a script that generate a large dictionary:
https://gist.github.com/lbolla/8e2640133032b0a6bb9c
Just run "alloc.sh" and compare the output.

Thanks,
L.


More information about the Python-list mailing list