[issue18874] Add a new tracemalloc module to trace memory allocations

[STINNER Victor]

STINNER Victor added the comment:

New patch (version 2):

- an hash table entry can now contain the data directly instead of a pointer to the data
- _tracemalloc._get_stats() copies the hash table to not have to lock the hash table too long, which will be especially useful for TRACE_RAW_MALLOC
- Add get_object_trace() function which returns a namedtuple
- get_process_memory() returns a namedtuple
- DisplayGarbage class has been removed: the PEP 442 has been implemented, it is no more interesting to trace uncollectable objects because they became very rare. If you still want to do that, get_object_trace() is available


TODO:

- TRACE_RAW_MALLOC define is still disabled because it introduces a deadlock when PyMem_RawMalloc() is called indirectly by Py_NewInterpreter()
- minor pending FIXME in the code
- review the high level API: add an helper to build a diff between two snapshots and drop colors (not portable)?


test_capi deadlock with TRACE_RAW_MALLOC:

    test_subinterps (test.test_capi.SubinterpreterTest) ... ^C
    Program received signal SIGINT, Interrupt.
    0xb7fdd424 in __kernel_vsyscall ()
    (gdb) where
    ...
    #2  0x081abc1c in PyCOND_TIMEDWAIT (cond=0x83470e0, mut=0x8347110, us=5000) at Python/condvar.h:103
    #3  0x081ac55c in take_gil (tstate=0x8349c78) at Python/ceval_gil.h:224
    #4  0x081ad0a9 in PyEval_RestoreThread (tstate=0x8349c78) at Python/ceval.c:443
    #5  0x081f08e3 in PyGILState_Ensure () at Python/pystate.c:786
    #6  0x0808df1b in trace_get_filename (trace=0x84c26b8, gil_held=0) at ./Modules/_tracemalloc.c:555
    #7  0x0808e314 in trace_log_alloc (ptr=0x84c9f18, size=52, gil_held=0) at ./Modules/_tracemalloc.c:698
    #8  0x0808e449 in trace_malloc (ctx=0x8342890, size=52, gil_held=0) at ./Modules/_tracemalloc.c:744
    #9  0x0808e5b8 in trace_raw_malloc (ctx=0x8342890, size=52) at ./Modules/_tracemalloc.c:811
    #10 0x0805d32a in PyMem_RawMalloc (size=52) at Objects/obmalloc.c:256
    #11 0x081eeb79 in PyInterpreterState_New () at Python/pystate.c:63
    #12 0x08060c8b in Py_NewInterpreter () at Python/pythonrun.c:700
    #13 0xb74d4b90 in run_in_subinterp ()
    #14 0x080e9e7a in PyCFunction_Call ()
    ...

----------
Added file: http://bugs.python.org/file31545/tracemalloc-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue18874>
_______________________________________