[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>> detect misusage of the API.
>
> Yes, but people don't necessarily do this, e.g. I have
> for a very long time ignored debug builds completely
> and when I started to try them, I found that some of the
> things I had been doing with e.g. free list implementations
> did not work in debug builds.

I just added support for debug hooks on Python memory allocators on
Python compiled in *release* mode. Set the environment variable
PYTHONMALLOC to debug to try with Python 3.6.

I added a check on PyObject_Malloc() debug hook to ensure that the
function is called with the GIL held. I opened an issue to add a
similar check on PyMem_Malloc():
https://bugs.python.org/issue26563


> Yes, but those are part of the stdlib. You'd need to check
> a few C extensions which are not tested as part of the stdlib,
> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
> types in C since these will often need the memory management
> APIs).
>
> It may also be a good idea to check wrapper generators such
> as cython, swig, cffi, etc.

I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).

I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
https://github.com/numpy/numpy/pull/7404

Except of this bug, all other tests pass with PyMem_Malloc() using
pymalloc and all debug checks.

Victor