[Python-Dev] [numpy wishlist] PyMem_*Calloc

[Julian Taylor]

Hi,
In NumPy what we want is the tracing, not the exchangeable allocators.
I don't think it is a good idea for the core of a whole stack of
C-extension based modules to replace the default allocator or allowing
other modules to replace the allocator NumPy uses.

I think it would be more useful if Python provides functions to
register memory allocations and frees and the tracemalloc module
registers handlers for these register functions.
If no trace allocation tracer is registered the functions just return
immediately.
That way the tracemalloc can be used with arbitrary allocators as long
as they register their allocations with Python.

For example a hugepage allocator, which you would not want to use that
as the default allocator for all python objects, but you may still
want to trace its usage:

my_hugetlb_alloc(size)
    p = mmap('hugepagefs', ..., MAP_HUGETLB);
    PyMem_Register_Alloc(p, size, __func__, __line__);
    return p

my_hugetlb_free(p);
    PyMem_Register_Free(p, __func__, __line__);
    munmap(p, ...);

normally the registers are nops, but if tracemalloc did register
tracers the memory is tracked, e.g. tracemodule does this on start():
tracercontext.register_alloc = trace_alloc
tracercontext.register_free = trace_free
tracercontext.data = mycontext
PyMem_SetTracer(&tracercontext)

Regards,
Julian Taylor