[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance
victor.stinner at gmail.com
Mon Mar 14 19:19:11 EDT 2016
2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>> detect misusage of the API.
> Yes, but people don't necessarily do this, e.g. I have
> for a very long time ignored debug builds completely
> and when I started to try them, I found that some of the
> things I had been doing with e.g. free list implementations
> did not work in debug builds.
I just added support for debug hooks on Python memory allocators on
Python compiled in *release* mode. Set the environment variable
PYTHONMALLOC to debug to try with Python 3.6.
I added a check on PyObject_Malloc() debug hook to ensure that the
function is called with the GIL held. I opened an issue to add a
similar check on PyMem_Malloc():
> Yes, but those are part of the stdlib. You'd need to check
> a few C extensions which are not tested as part of the stdlib,
> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
> types in C since these will often need the memory management
> It may also be a good idea to check wrapper generators such
> as cython, swig, cffi, etc.
I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).
I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
Except of this bug, all other tests pass with PyMem_Malloc() using
pymalloc and all debug checks.
More information about the Python-Dev