[Python-Dev] Add a new tracemalloc module to trace memory allocations

[Brett Cannon]

On Wed, Aug 28, 2013 at 8:16 PM, Victor Stinner <victor.stinner at gmail.com>wrote:

> Hi,
>
> Thanks to the PEP 445, it becomes possible to trace easily memory
> allocations. I propose to add a new tracemalloc module computing the
> memory usage per file and per line number. It has also a private
> method to retrieve the location (filename and line number) of a memory
> allocation of an object.
>
> tracemalloc is different than Heapy or PySizer because it is focused
> on the location of a memory allocation rather that the object type or
> object content.
>
> I have an implementation of the module for Python 2.5-3.4, but it
> requires to patch and recompile Python:
> https://pypi.python.org/pypi/pytracemalloc
>
>
> My proposed implementation for Python 3.4 is different:
>
> * reuse the PEP 445 to hook memory allocators
>
> * use a simple C implementation of an hash table called "cfuhash"
> (coming from the libcfu project, BSD license) instead of depending on
> the glib library. I simplified and adapted cfuhash for my usage
>
> * no enable() / disable() function: tracemalloc can only be enabled
> before startup by setting PYTHONTRACEMALLOC=1 environment variable
>
> * traces (size of the memory block, Python filename, Python line
> number) are stored directly in the memory block, not in a separated
> hash table
>
> I chose PYTHONTRACEMALLOC env var instead of enable()/disable()
> functions to be able to really trace *all* memory allocated by Python,
> especially memory allocated at startup, during Python initialization.
>
>
> The (high-level) API should be reviewed and discussed. The most
> interesting part is to take "snapshots" and compare snapshots. The
> module can load snapshots from disk and compare them later for deeper
> analysis (ex: ignore some files).
>
> For the documentation, see the following page:
> https://pypi.python.org/pypi/pytracemalloc
>
> I created the following issue to track the implementation:
> http://bugs.python.org/issue18874
>
> The implementation:
> http://hg.python.org/features/tracemalloc


Without looking at the code or docs I can the concept sounds very cool!


>
>
> * * *
>
> I also created a "pyfailmalloc" project based on the PEP 445 to inject
> MemoryError exceptions. I used this module to check if Python handles
> correctly memory allocation failures. The answer is no, I fixed many
> bugs (see issue #18408).
>
> Project homepage:
> https://bitbucket.org/haypo/pyfailmalloc
>
> Charles-François Natali and Serhiy Storchaka asked me to add this
> module somewhere in Python 3.4: "how about adding pyfailmalloc to the
> main repo (maybe under Tools), with a script making it easy to run the
> tests suite with it enabled?"
>
> What is the best place for such module? Add it to Modules/ directory
> but as a private module: "_failmalloc"?
>

Would extension module authors find it useful? If so maybe we need a malloc
package with trace and fail submodules?

And if we add it we might want to add to running the tool as part of the
devguide something people can work on.

-Brett


>
> * * *
>
> Example of tracemalloc output (it is more readable with colors, try in
> a terminal). The first top is sorted by total size, whereas the second
> top is sorted (automatically) with the size difference. You can see
> for example that the linecache module likes caching data (1.5 MB after
> 10 seconds of tests).
>
>
> $ PYTHONTRACEMALLOC=1 ./python -m test
> ...
> == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02)
> [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)]
> ==   Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow
> little-endian
> ==   /home/haypo/prog/python/tracemalloc/build/test_python_11087
> ...
> [  1/380] test_grammar
> [  2/380] test_opcodes
> [  3/380] test_dict
> [  4/380] test_builtin
> [  5/380] test_exceptions
> [  6/380] test_types
> [  7/380] test_unittest
>
> 2013-08-29 02:06:22: Top 25 allocations per file and line
> #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227,
> average=105 B
> #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706,
> average=118 B
> #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B
> #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B
> #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB
> #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604,
> average=156 B
> #7: ???:?: size=511 KiB, count=4445, average=117 B
> #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B
> #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B
> #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B
> #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B
> #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170,
> average=113 B
> #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250,
> average=159 B
> #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B
> #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672,
> average=156 B
> #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193,
> average=78 B
> #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104,
> average=784 B
> #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B
> #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B
> #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B
> #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20,
> average=3321 B
> #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B
> #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115,
> average=485 B
> #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B
> #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB
> 7333 more: size=4991 KiB, count=28051, average=182 B
> Total Python memory: size=17 MiB, count=136358, average=136 B
> Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB)
>
> [  8/380] test_doctest
> [  9/380] test_doctest2
> [ 10/380] test_support
> [ 11/380] test___all__
>
> 2013-08-29 02:08:44: Top 25 allocations per file and line (compared to
> 2013-08-29 02:08:39)
> #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB),
> count=80879 (+24652), average=109 B
> #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB),
> count=13669 (+4964), average=117 B
> #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB),
> count=6415 (+1811), average=152 B
> #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB),
> count=3136 (+966), average=108 B
> #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483
> (+211), average=314 B
> #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969
> (+969), average=75 B
> #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB),
> count=1025 (+1025), average=59 B
> #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB),
> count=507 (+507), average=113 B
> #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB),
> count=199 (+86), average=460 B
> #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB),
> count=505 (+242), average=136 B
> #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB),
> count=332 (+138), average=188 B
> #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688
> (+48), average=522 B
> #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB),
> count=521 (+236), average=109 B
> #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190
> (+190), average=132 B
> #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1)
> #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1)
> #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283
> (+283), average=84 B
> #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB),
> count=438 (+174), average=140 B
> #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB),
> count=263 (+114), average=189 B
> #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B
> #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164
> (+164), average=104 B
> #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145
> (+3), average=560 B
> #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB),
> count=23 (+7), average=2255 B
> #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165
> (+165), average=84 B
> #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB),
> count=170 (+170), average=81 B
> 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B
> Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805),
> average=143 B
> Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130829/c3d4b9cb/attachment.html>