Hi, Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object. tracemalloc is different than Heapy or PySizer because it is focused on the location of a memory allocation rather that the object type or object content. I have an implementation of the module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc My proposed implementation for Python 3.4 is different: * reuse the PEP 445 to hook memory allocators * use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project, BSD license) instead of depending on the glib library. I simplified and adapted cfuhash for my usage * no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable * traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization. The (high-level) API should be reviewed and discussed. The most interesting part is to take "snapshots" and compare snapshots. The module can load snapshots from disk and compare them later for deeper analysis (ex: ignore some files). For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc I created the following issue to track the implementation: http://bugs.python.org/issue18874 The implementation: http://hg.python.org/features/tracemalloc * * * I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. I used this module to check if Python handles correctly memory allocation failures. The answer is no, I fixed many bugs (see issue #18408). Project homepage: https://bitbucket.org/haypo/pyfailmalloc Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?" What is the best place for such module? Add it to Modules/ directory but as a private module: "_failmalloc"? * * * Example of tracemalloc output (it is more readable with colors, try in a terminal). The first top is sorted by total size, whereas the second top is sorted (automatically) with the size difference. You can see for example that the linecache module likes caching data (1.5 MB after 10 seconds of tests). $ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest 2013-08-29 02:06:22: Top 25 allocations per file and line #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B #7: ???:?: size=511 KiB, count=4445, average=117 B #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, average=113 B #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB 7333 more: size=4991 KiB, count=28051, average=182 B Total Python memory: size=17 MiB, count=136358, average=136 B Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB) [ 8/380] test_doctest [ 9/380] test_doctest2 [ 10/380] test_support [ 11/380] test___all__ 2013-08-29 02:08:44: Top 25 allocations per file and line (compared to 2013-08-29 02:08:39) #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB), count=80879 (+24652), average=109 B #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB), count=13669 (+4964), average=117 B #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB), count=6415 (+1811), average=152 B #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB), count=3136 (+966), average=108 B #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483 (+211), average=314 B #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969 (+969), average=75 B #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB), count=1025 (+1025), average=59 B #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB), count=507 (+507), average=113 B #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB), count=199 (+86), average=460 B #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB), count=505 (+242), average=136 B #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB), count=332 (+138), average=188 B #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688 (+48), average=522 B #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB), count=521 (+236), average=109 B #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190 (+190), average=132 B #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1) #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1) #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283 (+283), average=84 B #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB), count=438 (+174), average=140 B #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB), count=263 (+114), average=189 B #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164 (+164), average=104 B #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145 (+3), average=560 B #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB), count=23 (+7), average=2255 B #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165 (+165), average=84 B #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB), count=170 (+170), average=81 B 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805), average=143 B Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB) Victor