[Python-Dev] RFC: PEP 454: Add a new tracemalloc module

[Victor Stinner]

2013/9/8 Janzert <janzert at janzert.com>:
> It seems like most of this could live on PyPi for a while so the API can get
> hashed out in use?

The pytracemalloc is available on PyPI since 6 months. The only
feedback I had was something trying to compile it on Windows (which is
complex because of the dependency to glib, I don't think that it
succeed to install it on Windows). I guess that I didn't get more
feedback because it requires to patch and recompile Python, which is
not trivial.

I expect more feedback on python-dev with a working implementation (on
hg.python.org) and a PEP.

The version available on PyPI works and should be enough for most use
cases to be able to identify a memory leak.

Gregory P. Smith asked me if it would be possible to get more frames
(filename and line number) of the Python traceback, instead of just
the one frame (the last frame). I implemented it, but now I have new
issues (memory usage of the tracemalloc module itself), so I'm working
on filters directly implemented in the C module (_tracemalloc). It was
already possible to filter traces from a snapshot read from the disk.

I still have some tasks in my TODO list to finish the API and the
implementation. When I will be done, I will post post a new version of
the PEP on python-dev.

> If that's not the case is it because the PEP 445 API
> isn't rich enough?

The PEP 445 API is only designed to allow to develop new tools like
failmalloc or tracemalloc, without adding overhead if such debug tool
is not used.

The tracemalloc module reads the current Python traceback (filename
and line number) which is "not directly" accessible from
PyMem_Malloc(). I hope that existing tools like Heapy and Melia will
benefit directly from tracemalloc instead of having to develop their
own memory allocator hooks to get the same information (Python
traceback).

Victor