[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

So what do you think? Is it worth to change PyMem_Malloc() allocator
to pymalloc for a small speedup?

Should we do something else before doing that?

Or do you expect that too many applications use PyMem_Malloc() without
holding the GIL and will not run try to run their application with
PYTHONMALLOC=debug?

Victor

2016-03-15 0:19 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>>> detect misusage of the API.
>>
>> Yes, but people don't necessarily do this, e.g. I have
>> for a very long time ignored debug builds completely
>> and when I started to try them, I found that some of the
>> things I had been doing with e.g. free list implementations
>> did not work in debug builds.
>
> I just added support for debug hooks on Python memory allocators on
> Python compiled in *release* mode. Set the environment variable
> PYTHONMALLOC to debug to try with Python 3.6.
>
> I added a check on PyObject_Malloc() debug hook to ensure that the
> function is called with the GIL held. I opened an issue to add a
> similar check on PyMem_Malloc():
> https://bugs.python.org/issue26563
>
>
>> Yes, but those are part of the stdlib. You'd need to check
>> a few C extensions which are not tested as part of the stdlib,
>> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
>> types in C since these will often need the memory management
>> APIs).
>>
>> It may also be a good idea to check wrapper generators such
>> as cython, swig, cffi, etc.
>
> I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).
>
> I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
> https://github.com/numpy/numpy/pull/7404
>
> Except of this bug, all other tests pass with PyMem_Malloc() using
> pymalloc and all debug checks.
>
> Victor