[Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!

Victor Stinner victor.stinner at gmail.com
Wed Oct 30 14:17:14 CET 2013


2013/10/30 Kristján Valur Jónsson <kristjan at ccpgames.com>:
> The point of a PEP is getting something into standard python.  The command line flag is also part of this.
> Piggybacking a lightweight client/server data-gathering version of this on top of the PEP
> could be beneficial in that respect.

In my opinion, your use case (log malloc/free and sent it to the
network) is completly different to what tracemalloc does. Reusing
tracemalloc for you would be inefficient (slow, use too much memory).

You can use tracemalloc if you want to send a snapshot of traces every
N minutes. It should not be hard (less than 100 lines of Python) to
implement that using a thread, pickle and a socket.

But I prefer to not include it to the PEP, Charles François wants a
minimal module and prefers to develop tools on top of the module. I
*now* agree with him (first I wanted to pack everything into the
stdlib! even escape sequences to write text with colors!).

For example, the old code using "tasks" to take automatically a
snapshot every N minutes or display the top 10 allocations  every N
minutes into the terminal with colors has been moved to a new project:

   https://github.com/haypo/pytracemalloctext/blob/master/doc/index.rst

(the project is not usable yet, I will finish it after the PEP 454,
and after updating the pytracemalloc module on PyPI)

> Unless I am mistaken, the Pep 445 hooks must be setup before calling Py_Initialize() and so using
> them is not trivial.

It depends on how you use the API. If you want to replace the memory
allocators (use your own "malloc"), you have to call
PyMem_SetAllocator() *before the first memory allocation* ! In Python,
the first memory allocation occurs much earlier than Py_Initialize(),
PyMem_RawMalloc() is the *first* instruction (!) executed by Python in
its main() function... (see Modules/python.c).

If you want to install an hook calling the previous allocator, you can
call PyMem_SetAllocator() anytime: that's why it's possible to call
tracemalloc.enable() anytime.

Victor


More information about the Python-Dev mailing list