On 04.02.2016 13:29, Victor Stinner wrote:
2016-02-04 11:17 GMT+01:00 M.-A. Lemburg email@example.com:
Do you see any drawback of using pymalloc for PyMem_Malloc()?
Yes: You cannot free memory allocated using pymalloc with the standard C lib free().
That's not completly new.
If Python is compiled in debug mode, you get a fatal error with a huge error message if you free the memory allocated by PyMem_Malloc() using PyObject_Free() or PyMem_RawFree().
But yes, technically it's possible to use free() when Python is *not* compiled in debug mode.
Debug mode is a completely different beast ;-)
It would be better to go through the list of PyMem_*() calls in Python and replace them with PyObject_*() calls, where possible.
There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().
I would prefer to modify a single place having to replace 536 calls :-/
You have a point there, but I don't think it'll work out that easily, since we are using such calls to e.g. pass dynamically allocated buffers to code in extensions (which then have to free the buffers again).
Does anyone recall the rationale to have two families to memory allocators?
The PyMem_*() APIs were needed to have a cross-platform malloc() implementation which returns standard C lib free()able memory, but also behaves well when passing 0 as size.
Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour. But, why not PyObject_Malloc() & PObject_Free() were not used in the first place?
Good question. I guess developers simply thought of PyObject_Malloc() being for PyObjects, not arbitrary memory buffers, most likely because pymalloc was advertised as allocator for Python objects, not random chunks of memory.
Also: PyObject_*() APIs were first introduced with pymalloc, and no one really was interested in going through all the calls to PyMem_*() APIs and convert those to use the new pymalloc at the time.
All this happened between Python 1.5.2 and 2.0.
One of the reasons probably also was that pymalloc originally did not return memory back to the system malloc(). This was changed only some years ago.
An explanation can be that PyMem_Malloc() can be called without the GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc() called (indirectly) PyObject_Malloc() when Python was compiled in debug mode, and PyObject_Malloc() requires the GIL to be held.
When I wrote the PEP 445, there was a discussion about the GIL. It was proposed to allow to call PyMem_xxx() without the GIL: https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc
This option was rejected.
AFAIR, the GIL was not really part of the consideration at the time. We used pymalloc for PyObject allocation, that's all.