Add a new tracemalloc module to trace memory allocations
Hi, Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object. tracemalloc is different than Heapy or PySizer because it is focused on the location of a memory allocation rather that the object type or object content. I have an implementation of the module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc My proposed implementation for Python 3.4 is different: * reuse the PEP 445 to hook memory allocators * use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project, BSD license) instead of depending on the glib library. I simplified and adapted cfuhash for my usage * no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable * traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization. The (high-level) API should be reviewed and discussed. The most interesting part is to take "snapshots" and compare snapshots. The module can load snapshots from disk and compare them later for deeper analysis (ex: ignore some files). For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc I created the following issue to track the implementation: http://bugs.python.org/issue18874 The implementation: http://hg.python.org/features/tracemalloc * * * I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. I used this module to check if Python handles correctly memory allocation failures. The answer is no, I fixed many bugs (see issue #18408). Project homepage: https://bitbucket.org/haypo/pyfailmalloc Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?" What is the best place for such module? Add it to Modules/ directory but as a private module: "_failmalloc"? * * * Example of tracemalloc output (it is more readable with colors, try in a terminal). The first top is sorted by total size, whereas the second top is sorted (automatically) with the size difference. You can see for example that the linecache module likes caching data (1.5 MB after 10 seconds of tests). $ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest 2013-08-29 02:06:22: Top 25 allocations per file and line #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B #7: ???:?: size=511 KiB, count=4445, average=117 B #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, average=113 B #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB 7333 more: size=4991 KiB, count=28051, average=182 B Total Python memory: size=17 MiB, count=136358, average=136 B Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB) [ 8/380] test_doctest [ 9/380] test_doctest2 [ 10/380] test_support [ 11/380] test___all__ 2013-08-29 02:08:44: Top 25 allocations per file and line (compared to 2013-08-29 02:08:39) #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB), count=80879 (+24652), average=109 B #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB), count=13669 (+4964), average=117 B #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB), count=6415 (+1811), average=152 B #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB), count=3136 (+966), average=108 B #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483 (+211), average=314 B #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969 (+969), average=75 B #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB), count=1025 (+1025), average=59 B #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB), count=507 (+507), average=113 B #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB), count=199 (+86), average=460 B #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB), count=505 (+242), average=136 B #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB), count=332 (+138), average=188 B #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688 (+48), average=522 B #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB), count=521 (+236), average=109 B #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190 (+190), average=132 B #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1) #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1) #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283 (+283), average=84 B #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB), count=438 (+174), average=140 B #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB), count=263 (+114), average=189 B #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164 (+164), average=104 B #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145 (+3), average=560 B #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB), count=23 (+7), average=2255 B #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165 (+165), average=84 B #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB), count=170 (+170), average=81 B 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805), average=143 B Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB) Victor
2013/8/29 Victor Stinner <victor.stinner@gmail.com>:
My proposed implementation for Python 3.4 is different:
* no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable
* traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table
I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization.
I'm not sure that having to set an environment variable is the most convinient option, especially on Windows. Storing traces directly into memory blocks should use less memory, but it requires to start tracemalloc before the first memory allocation. It is possible to add again enable() and disable() methods to dynamically install/uninstall the hook on memory allocators. I solved this issue in the current implementation by using a second hash table (pointer => trace). We can keep the environment variable as PYTHONFAULTHANDLER which enables faulthandler at startup. faulthandler has also a command line option: -X faulthandler. We may add -X tracemalloc. Victor
First, I really like this. +1 On Wed, Aug 28, 2013 at 6:07 PM, Victor Stinner <victor.stinner@gmail.com>wrote:
2013/8/29 Victor Stinner <victor.stinner@gmail.com>:
My proposed implementation for Python 3.4 is different:
* no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable
* traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table
I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization.
I'm not sure that having to set an environment variable is the most convinient option, especially on Windows.
Storing traces directly into memory blocks should use less memory, but it requires to start tracemalloc before the first memory allocation. It is possible to add again enable() and disable() methods to dynamically install/uninstall the hook on memory allocators. I solved this issue in the current implementation by using a second hash table (pointer => trace).
We can keep the environment variable as PYTHONFAULTHANDLER which enables faulthandler at startup. faulthandler has also a command line option: -X faulthandler. We may add -X tracemalloc.
We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() functions? Users are likely to want snapshots captured by enable()/disable() around particular pieces of code just as much as whole program information. Think of the possibilities, you could even setup a test runner to enable/disable before and after each test, test suite or test module to gather narrow statistics as to what code actually _caused_ the allocations rather than the ultimate individual file/line doing it. Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful. (Yes, this gets messy fast as hooks should not trigger calls back into themselves when they allocate or free, similar to the "fun" involved in writing coverage tools) let me know if you think i'm crazy. :) -gps
Le 31 août 2013 19:09, "Gregory P. Smith" <greg@krypto.org> a écrit :
First, I really like this. +1
Current votes: +3 (i also want tracemalloc!). No opposition against such addition.
We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() functions?
Users are likely to want snapshots captured by enable()/disable() around
Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to
(Yes, this gets messy fast as hooks should not trigger calls back into
The reason was technical but I have issues with enabling tracemalloc before Python is fully initialized. Enabling tracemalloc when Python is almost initialized and disabling it before Python exit is more reliable. In the last implementation, enable() and disable() are back. PYTHONTRACEMALLOC is still available. I will also add -X tracemalloc command line. particular pieces of code just as much as whole program information. enable()/disable() are useful for tests of the tracemalloc module! figure out what call patterns lead allocations is powerful. There is no portable function to retrieve the C traceback. For the Python traceback: it may be possible to get it but you have to remember that the hook is on PyMem_Malloc(), a low level memory allocator. The GIL is hold. It is possible to call Python code in some cases, but they are corner cases where it would lead to a loop, deadlock or worse. I prefer to only read available Python data without calling any Python code and try to write a reliable debug tool. I still have some issues like issues with reentrant calls to the hook, but I'm working on them. A nice improvement compared to the implementation on PyPI would be to hook PyMem_RawMalloc(), to trace also the memory used by gzip, bz2, lzma and other C modules. Locking the GIL in PyMem_RawMalloc() to get the filename and locking internals tracemalloc structures causes also new issues (like a funny deadlock related to subinterpreter). themselves when they allocate or free, similar to the "fun" involved in writing coverage tools) tracemalloc uses a "reentrant" variable to do nothing on a reentrant call to the hook.
let me know if you think i'm crazy. :)
You are not crazy. It is just hard to implement it. I'm not sure that it is possible. Victor
+1 from me for both tracemalloc and failmalloc in the same vein as faulthandler (and using similar API concepts to faulthandler). While I like the flat top level naming structure, we should clearly document these as implementation dependent runtime services. Other implementations may not provide them at all, and even if they do, many details will likely be different. The gc module may even fall into the same category. Cheers, Nick.
Le Sun, 1 Sep 2013 11:36:51 +1000, Nick Coghlan <ncoghlan@gmail.com> a écrit :
+1 from me for both tracemalloc and failmalloc in the same vein as faulthandler (and using similar API concepts to faulthandler).
While I like the flat top level naming structure, we should clearly document these as implementation dependent runtime services. Other implementations may not provide them at all, and even if they do, many details will likely be different.
So, it sounds like Victor may write a new PEP soon :-) +1 from me on a flat namespace. Non-designed naming hierarchies are just a pain to work with. Regards Antoine.
2013/9/1 Nick Coghlan <ncoghlan@gmail.com>:
+1 from me for both tracemalloc and failmalloc in the same vein as faulthandler (and using similar API concepts to faulthandler).
While I like the flat top level naming structure, we should clearly document these as implementation dependent runtime services. Other implementations may not provide them at all, and even if they do, many details will likely be different.
The gc module may even fall into the same category.
I updated the PEP 454 to mention that tracemalloc has been written for CPython. Victor
2013/8/31 Gregory P. Smith <greg@krypto.org>:
Think of the possibilities, you could even setup a test runner to enable/disable before and after each test, test suite or test module to gather narrow statistics as to what code actually _caused_ the allocations rather than the ultimate individual file/line doing it.
It may be used with "-m test -R" to detect memory leaks, you need to repeat a test.
Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful.
I modified the PEP 454 and the implementation. By default, tracemalloc only stores 1 "frame" instance (filename and line number). But you can now use tracemalloc.set_number_frame(10) to store 10 frames per memory allocation. Don't store too much frames, remember that tracemalloc traces all Python memory allocations! So when Python allocates 1 byte, tracemalloc may need to allocate 1000 bytes to store the trace... tracemalloc cannot get the C traceback. It was discussed in the design of the PEP 445 (Add new APIs to customize Python memory allocators) how to pass the C filename and line number. It was decided to not add them to have a simpler API. In general, a C function is binded to a Python function. Knowing the Python function should be enough. Victor
On Wed, Aug 28, 2013 at 8:16 PM, Victor Stinner <victor.stinner@gmail.com>wrote:
Hi,
Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object.
tracemalloc is different than Heapy or PySizer because it is focused on the location of a memory allocation rather that the object type or object content.
I have an implementation of the module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc
My proposed implementation for Python 3.4 is different:
* reuse the PEP 445 to hook memory allocators
* use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project, BSD license) instead of depending on the glib library. I simplified and adapted cfuhash for my usage
* no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable
* traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table
I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization.
The (high-level) API should be reviewed and discussed. The most interesting part is to take "snapshots" and compare snapshots. The module can load snapshots from disk and compare them later for deeper analysis (ex: ignore some files).
For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc
I created the following issue to track the implementation: http://bugs.python.org/issue18874
The implementation: http://hg.python.org/features/tracemalloc
Without looking at the code or docs I can the concept sounds very cool!
* * *
I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. I used this module to check if Python handles correctly memory allocation failures. The answer is no, I fixed many bugs (see issue #18408).
Project homepage: https://bitbucket.org/haypo/pyfailmalloc
Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?"
What is the best place for such module? Add it to Modules/ directory but as a private module: "_failmalloc"?
Would extension module authors find it useful? If so maybe we need a malloc package with trace and fail submodules? And if we add it we might want to add to running the tool as part of the devguide something people can work on. -Brett
* * *
Example of tracemalloc output (it is more readable with colors, try in a terminal). The first top is sorted by total size, whereas the second top is sorted (automatically) with the size difference. You can see for example that the linecache module likes caching data (1.5 MB after 10 seconds of tests).
$ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest
2013-08-29 02:06:22: Top 25 allocations per file and line #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B #7: ???:?: size=511 KiB, count=4445, average=117 B #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, average=113 B #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB 7333 more: size=4991 KiB, count=28051, average=182 B Total Python memory: size=17 MiB, count=136358, average=136 B Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB)
[ 8/380] test_doctest [ 9/380] test_doctest2 [ 10/380] test_support [ 11/380] test___all__
2013-08-29 02:08:44: Top 25 allocations per file and line (compared to 2013-08-29 02:08:39) #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB), count=80879 (+24652), average=109 B #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB), count=13669 (+4964), average=117 B #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB), count=6415 (+1811), average=152 B #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB), count=3136 (+966), average=108 B #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483 (+211), average=314 B #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969 (+969), average=75 B #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB), count=1025 (+1025), average=59 B #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB), count=507 (+507), average=113 B #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB), count=199 (+86), average=460 B #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB), count=505 (+242), average=136 B #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB), count=332 (+138), average=188 B #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688 (+48), average=522 B #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB), count=521 (+236), average=109 B #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190 (+190), average=132 B #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1) #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1) #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283 (+283), average=84 B #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB), count=438 (+174), average=140 B #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB), count=263 (+114), average=189 B #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164 (+164), average=104 B #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145 (+3), average=560 B #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB), count=23 (+7), average=2255 B #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165 (+165), average=84 B #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB), count=170 (+170), average=81 B 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805), average=143 B Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB)
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
2013/8/29 Brett Cannon <brett@python.org>:
I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. (...)
Would extension module authors find it useful?
I don't know, I created two months ago and I didn't made a public annoucement.
If so maybe we need a malloc package with trace and fail submodules?
I read somewhere "flat is better than nested". failmalloc and tracemalloc are not directly related. I guess that they can be used at the same time, but I didn't try.
And if we add it we might want to add to running the tool as part of the devguide something people can work on.
There are still tricky open issues related to failmalloc :-) - frame_fasttolocals.patch: fix for PyFrame_FastToLocals(), I didn't commit the patch because it is not atomic (it does not handle errors correctly). I should modify it to copy the locals before modifying the dict, so it can be restored in case of errors. - #18507: import_init() should not use Py_FatalError() but return an error - #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error - fix tests hang when an exception occurs in a thread Victor
2013/8/29 Victor Stinner <victor.stinner@gmail.com>:
Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?"
There are two reasons I think it would be a great addition: - since OOM conditions are - almost - never tested, the OOM handling code is - almost - always incorrect: indeed, Victor has found and fixed several dozens crashes thanks to this module - this module is actually really simple (~150 LOC) I have two comments on the API: 1) failmalloc.enable(range: int=1000): schedule a memory allocation failure in random.randint(1, range) allocations. That's one shot, i.e. only one failure will be triggered. So if this failure occurs in a place where the code is prepared to handle MemoryError (e.g. bigmem tests), no failure will occur in the remaining test. It would be better IMO to repeat this (i.e. reset the next failure counter), to increase the coverage. 2) It's a consequence of 1): since only one malloc() failure is triggered, it doesn't really reflect how a OOM condition would appear in real life: usually, it's either because you've exhausted your address space or the machine is under memory pressure, which means that once you've hit OOM, you're likely to encounter it again on subsequent allocations, for example if your OOM handling code allocates new memory (that's why it's so complicated to properly handle OOM, and one might want to use "memory parachutes"). It might be interesting to be able to pass an absolute maximum memory usage, or an option where once you've triggered an malloc() failure, you record the current memory usage, and use it as ceiling for subsequent allocations.
participants (6)
-
Antoine Pitrou -
Brett Cannon -
Charles-François Natali -
Gregory P. Smith -
Nick Coghlan -
Victor Stinner