[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance
victor.stinner at gmail.com
Fri Mar 25 08:11:31 EDT 2016
So what do you think? Is it worth to change PyMem_Malloc() allocator
to pymalloc for a small speedup?
Should we do something else before doing that?
Or do you expect that too many applications use PyMem_Malloc() without
holding the GIL and will not run try to run their application with
2016-03-15 0:19 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>>> detect misusage of the API.
>> Yes, but people don't necessarily do this, e.g. I have
>> for a very long time ignored debug builds completely
>> and when I started to try them, I found that some of the
>> things I had been doing with e.g. free list implementations
>> did not work in debug builds.
> I just added support for debug hooks on Python memory allocators on
> Python compiled in *release* mode. Set the environment variable
> PYTHONMALLOC to debug to try with Python 3.6.
> I added a check on PyObject_Malloc() debug hook to ensure that the
> function is called with the GIL held. I opened an issue to add a
> similar check on PyMem_Malloc():
>> Yes, but those are part of the stdlib. You'd need to check
>> a few C extensions which are not tested as part of the stdlib,
>> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
>> types in C since these will often need the memory management
>> It may also be a good idea to check wrapper generators such
>> as cython, swig, cffi, etc.
> I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).
> I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
> Except of this bug, all other tests pass with PyMem_Malloc() using
> pymalloc and all debug checks.
More information about the Python-Dev