[Python-Dev] Add a new tracemalloc module to trace memory allocations

[Victor Stinner]

Hi,

Thanks to the PEP 445, it becomes possible to trace easily memory
allocations. I propose to add a new tracemalloc module computing the
memory usage per file and per line number. It has also a private
method to retrieve the location (filename and line number) of a memory
allocation of an object.

tracemalloc is different than Heapy or PySizer because it is focused
on the location of a memory allocation rather that the object type or
object content.

I have an implementation of the module for Python 2.5-3.4, but it
requires to patch and recompile Python:
https://pypi.python.org/pypi/pytracemalloc


My proposed implementation for Python 3.4 is different:

* reuse the PEP 445 to hook memory allocators

* use a simple C implementation of an hash table called "cfuhash"
(coming from the libcfu project, BSD license) instead of depending on
the glib library. I simplified and adapted cfuhash for my usage

* no enable() / disable() function: tracemalloc can only be enabled
before startup by setting PYTHONTRACEMALLOC=1 environment variable

* traces (size of the memory block, Python filename, Python line
number) are stored directly in the memory block, not in a separated
hash table

I chose PYTHONTRACEMALLOC env var instead of enable()/disable()
functions to be able to really trace *all* memory allocated by Python,
especially memory allocated at startup, during Python initialization.


The (high-level) API should be reviewed and discussed. The most
interesting part is to take "snapshots" and compare snapshots. The
module can load snapshots from disk and compare them later for deeper
analysis (ex: ignore some files).

For the documentation, see the following page:
https://pypi.python.org/pypi/pytracemalloc

I created the following issue to track the implementation:
http://bugs.python.org/issue18874

The implementation:
http://hg.python.org/features/tracemalloc

* * *

I also created a "pyfailmalloc" project based on the PEP 445 to inject
MemoryError exceptions. I used this module to check if Python handles
correctly memory allocation failures. The answer is no, I fixed many
bugs (see issue #18408).

Project homepage:
https://bitbucket.org/haypo/pyfailmalloc

Charles-François Natali and Serhiy Storchaka asked me to add this
module somewhere in Python 3.4: "how about adding pyfailmalloc to the
main repo (maybe under Tools), with a script making it easy to run the
tests suite with it enabled?"

What is the best place for such module? Add it to Modules/ directory
but as a private module: "_failmalloc"?

* * *

Example of tracemalloc output (it is more readable with colors, try in
a terminal). The first top is sorted by total size, whereas the second
top is sorted (automatically) with the size difference. You can see
for example that the linecache module likes caching data (1.5 MB after
10 seconds of tests).


$ PYTHONTRACEMALLOC=1 ./python -m test
...
== CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)]
==   Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow
little-endian
==   /home/haypo/prog/python/tracemalloc/build/test_python_11087
...
[  1/380] test_grammar
[  2/380] test_opcodes
[  3/380] test_dict
[  4/380] test_builtin
[  5/380] test_exceptions
[  6/380] test_types
[  7/380] test_unittest

2013-08-29 02:06:22: Top 25 allocations per file and line
#1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B
#2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706,
average=118 B
#3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B
#4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B
#5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB
#6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B
#7: ???:?: size=511 KiB, count=4445, average=117 B
#8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B
#9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B
#10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B
#11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B
#12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170,
average=113 B
#13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B
#14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B
#15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672,
average=156 B
#16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B
#17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B
#18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B
#19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B
#20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B
#21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B
#22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B
#23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115,
average=485 B
#24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B
#25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB
7333 more: size=4991 KiB, count=28051, average=182 B
Total Python memory: size=17 MiB, count=136358, average=136 B
Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB)

[  8/380] test_doctest
[  9/380] test_doctest2
[ 10/380] test_support
[ 11/380] test___all__

2013-08-29 02:08:44: Top 25 allocations per file and line (compared to
2013-08-29 02:08:39)
#1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB),
count=80879 (+24652), average=109 B
#2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB),
count=13669 (+4964), average=117 B
#3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB),
count=6415 (+1811), average=152 B
#4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB),
count=3136 (+966), average=108 B
#5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483
(+211), average=314 B
#6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969
(+969), average=75 B
#7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB),
count=1025 (+1025), average=59 B
#8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB),
count=507 (+507), average=113 B
#9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB),
count=199 (+86), average=460 B
#10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB),
count=505 (+242), average=136 B
#11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB),
count=332 (+138), average=188 B
#12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688
(+48), average=522 B
#13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB),
count=521 (+236), average=109 B
#14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190
(+190), average=132 B
#15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1)
#16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1)
#17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283
(+283), average=84 B
#18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB),
count=438 (+174), average=140 B
#19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB),
count=263 (+114), average=189 B
#20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B
#21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164
(+164), average=104 B
#22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145
(+3), average=560 B
#23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB),
count=23 (+7), average=2255 B
#24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165
(+165), average=84 B
#25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB),
count=170 (+170), average=81 B
10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B
Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805),
average=143 B
Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB)

Victor