[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Victor Stinner victor.stinner at gmail.com
Thu Feb 4 07:29:42 EST 2016


2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>> Do you see any drawback of using pymalloc for PyMem_Malloc()?
> Yes: You cannot free memory allocated using pymalloc with the
> standard C lib free().

That's not completly new.

If Python is compiled in debug mode, you get a fatal error with a huge
error message if you free the memory allocated by PyMem_Malloc() using
PyObject_Free() or PyMem_RawFree().

But yes, technically it's possible to use free() when Python is *not*
compiled in debug mode.

> It would be better to go through the list of PyMem_*() calls
> in Python and replace them with PyObject_*() calls, where
> possible.

There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
and PyMem_Free().

I would prefer to modify a single place having to replace 536 calls :-/

>> Does anyone recall the rationale to have two families to memory allocators?
> The PyMem_*() APIs were needed to have a cross-platform malloc()
> implementation which returns standard C lib free()able memory,
> but also behaves well when passing 0 as size.

Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour.
But, why not PyObject_Malloc() & PObject_Free() were not used in the
first place?

An explanation can be that PyMem_Malloc() can be called without the
GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc()
called (indirectly) PyObject_Malloc() when Python was compiled in
debug mode, and PyObject_Malloc() requires the GIL to be held.

When I wrote the PEP 445, there was a discussion about the GIL. It was
proposed to allow to call PyMem_xxx() without the GIL:

This option was rejected.


More information about the Python-Dev mailing list