[Python-Dev] [numpy wishlist] PyMem_*Calloc

[Victor Stinner]

Hi,

2014-04-16 7:51 GMT-04:00 Julian Taylor <jtaylor.debian at googlemail.com>:
> In NumPy what we want is the tracing, not the exchangeable allocators.

Did you read the PEP 445? Using the new malloc API, in fact you can
have both: install new allocators and set up hooks on allocators.
http://legacy.python.org/dev/peps/pep-0445/

The PEP 445 has been implemented in Python 3.4, we don't plan to
rewrite it. So it's probably better to try to understand how it was
designed and why we chose this design.

See the talk I just have at Pycon Montreal for more information on how
tracemalloc works. Slides:
https://raw.githubusercontent.com/haypo/conf/master/2014-Pycon-Montreal/tracemalloc.pdf

Video:
http://pyvideo.org/video/2698/track-memory-leaks-in-python

> my_hugetlb_alloc(size)
>     p = mmap('hugepagefs', ..., MAP_HUGETLB);
>     PyMem_Register_Alloc(p, size, __func__, __line__);
>     return p
>
> my_hugetlb_free(p);
>     PyMem_Register_Free(p, __func__, __line__);
>     munmap(p, ...);

This is exactly how tracemalloc works. The advantage of the PEP 445 is
that you have a null overhead when tracemalloc is disabled. There is
no need to check if a trace function is present or not.

You can chain multiple hooks.

See also the calloc issue which was written for NumPy:
http://bugs.python.org/issue21233

Victor