[Python-Dev] Add a new tracemalloc module to trace memory allocations

[Victor Stinner]

2013/8/31 Gregory P. Smith <greg at krypto.org>:
> Think of the possibilities, you could even setup a test runner to
> enable/disable before and after each test, test suite or test module to
> gather narrow statistics as to what code actually _caused_ the allocations
> rather than the ultimate individual file/line doing it.

It may be used with "-m test -R" to detect memory leaks, you need to
repeat a test.

> Taking that further: file and line information is great, but what if you
> extend the concept: could you allow for C API or even Python hooks to gather
> additional information at the time of each allocation or free? for example:
> Gathering the actual C and Python stack traces for correlation to figure out
> what call patterns lead allocations is powerful.

I modified the PEP 454 and the implementation. By default, tracemalloc
only stores 1 "frame" instance (filename and line number). But you can
now use tracemalloc.set_number_frame(10) to store 10 frames per memory
allocation. Don't store too much frames, remember that tracemalloc
traces all Python memory allocations! So when Python allocates 1 byte,
tracemalloc may need to allocate 1000 bytes to store the trace...

tracemalloc cannot get the C traceback. It was discussed in the design
of the PEP 445 (Add new APIs to customize Python memory allocators)
how to pass the C filename and line number. It was decided to not add
them to have a simpler API.

In general, a C function is binded to a Python function. Knowing the
Python function should be enough.

Victor