[Python-Dev] Re: Snapshot formats in tracemalloc vs profiler

Well, then. I think I'm going to have some fun with this. :) Thank you! On Thu, Jun 27, 2019 at 4:17 PM Victor Stinner wrote:

...

Although while I have you hear, I do have a further question about how

Le ven. 28 juin 2019 à 01:03, Yonatan Zunger a écrit : tracemalloc works: If I'm reading the code correctly, traces get removed by tracemalloc when objects are free, which means that at equilibrium (e.g. at the end of a function) the trace would show just the data which leaked. That's very useful in most cases, but I'm trying to hunt down a situation where memory usage is transiently spiking -- which might be due to something being actively used, or to something building up and overwhelming the GC, or to evil elves in the CPU for all I can tell so far. Would it be completely insane for tracemalloc to have a mode where it either records frees separately (e.g. as a malloc of negative space, at the trace where the free is happening), or where it simply ignores frees altogether?

My very first implementation of tracemalloc produced a log of malloc and free calls. Problem: transferring the log from a slow set top box to a desktop computer was slow, and parsing the log was very slow. Parsing complexity is in O(n) where n is the number of malloc or free calls, knowning that Python calls malloc(), realloc() or free() 270,000 times per second in average: https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator

tracemalloc is built on top of PEP 445 -- Add new APIs to customize Python memory allocators: https://www.python.org/dev/peps/pep-0445/

Using these PEP 445 hooks, you should be able to do whatever you want on Python memory allocations and free :-)

Example of toy project to inject memory allocation failures: https://github.com/vstinner/pyfailmalloc

Victor