[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

Hi,

2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>> Do you see any drawback of using pymalloc for PyMem_Malloc()?
>
> Yes: You cannot free memory allocated using pymalloc with the
> standard C lib free().

That's not completly new.

If Python is compiled in debug mode, you get a fatal error with a huge
error message if you free the memory allocated by PyMem_Malloc() using
PyObject_Free() or PyMem_RawFree().

But yes, technically it's possible to use free() when Python is *not*
compiled in debug mode.


> It would be better to go through the list of PyMem_*() calls
> in Python and replace them with PyObject_*() calls, where
> possible.

There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
and PyMem_Free().

I would prefer to modify a single place having to replace 536 calls :-/


>> Does anyone recall the rationale to have two families to memory allocators?
>
> The PyMem_*() APIs were needed to have a cross-platform malloc()
> implementation which returns standard C lib free()able memory,
> but also behaves well when passing 0 as size.

Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour.
But, why not PyObject_Malloc() & PObject_Free() were not used in the
first place?

An explanation can be that PyMem_Malloc() can be called without the
GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc()
called (indirectly) PyObject_Malloc() when Python was compiled in
debug mode, and PyObject_Malloc() requires the GIL to be held.

When I wrote the PEP 445, there was a discussion about the GIL. It was
proposed to allow to call PyMem_xxx() without the GIL:
https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc

This option was rejected.

Victor