[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[M.-A. Lemburg]

On 04.02.2016 13:29, Victor Stinner wrote:
> Hi,
> 
> 2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>> Do you see any drawback of using pymalloc for PyMem_Malloc()?
>>
>> Yes: You cannot free memory allocated using pymalloc with the
>> standard C lib free().
> 
> That's not completly new.
> 
> If Python is compiled in debug mode, you get a fatal error with a huge
> error message if you free the memory allocated by PyMem_Malloc() using
> PyObject_Free() or PyMem_RawFree().
> 
> But yes, technically it's possible to use free() when Python is *not*
> compiled in debug mode.

Debug mode is a completely different beast ;-)

>> It would be better to go through the list of PyMem_*() calls
>> in Python and replace them with PyObject_*() calls, where
>> possible.
> 
> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
> and PyMem_Free().
> 
> I would prefer to modify a single place having to replace 536 calls :-/

You have a point there, but I don't think it'll work out
that easily, since we are using such calls to e.g. pass
dynamically allocated buffers to code in extensions (which then
have to free the buffers again).

>>> Does anyone recall the rationale to have two families to memory allocators?
>>
>> The PyMem_*() APIs were needed to have a cross-platform malloc()
>> implementation which returns standard C lib free()able memory,
>> but also behaves well when passing 0 as size.
> 
> Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour.
> But, why not PyObject_Malloc() & PObject_Free() were not used in the
> first place?

Good question. I guess developers simply thought of PyObject_Malloc()
being for PyObjects, not arbitrary memory buffers, most likely
because pymalloc was advertised as allocator for Python objects,
not random chunks of memory.

Also: PyObject_*() APIs were first introduced with pymalloc, and
no one really was interested in going through all the calls to
PyMem_*() APIs and convert those to use the new pymalloc at the
time.

All this happened between Python 1.5.2 and 2.0.

One of the reasons probably also was that pymalloc originally
did not return memory back to the system malloc(). This was
changed only some years ago.

> An explanation can be that PyMem_Malloc() can be called without the
> GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc()
> called (indirectly) PyObject_Malloc() when Python was compiled in
> debug mode, and PyObject_Malloc() requires the GIL to be held.
> 
> When I wrote the PEP 445, there was a discussion about the GIL. It was
> proposed to allow to call PyMem_xxx() without the GIL:
> https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc
> 
> This option was rejected.

AFAIR, the GIL was not really part of the consideration at the time.
We used pymalloc for PyObject allocation, that's all.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/