[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[M.-A. Lemburg]

On 04.02.2016 14:25, Victor Stinner wrote:
> Thanks for your feedback, you are asking good questions :-)
> 
> 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
>>> and PyMem_Free().
>>>
>>> I would prefer to modify a single place having to replace 536 calls :-/
>>
>> You have a point there, but I don't think it'll work out
>> that easily, since we are using such calls to e.g. pass
>> dynamically allocated buffers to code in extensions (which then
>> have to free the buffers again).
> 
> Ah, interesting. But I'm not sure that we delegate the responsability
> of freeing the memory to external libraries. Usually, it's more the
> opposite: a library gives us an allocated memory block, and we have to
> free it. No?

Sometimes, yes, but we also do allocations for e.g.
parsing values in Python argument tuples (e.g. using
"es" or "et"):

https://docs.python.org/3.6/c-api/arg.html

We do document to use PyMem_Free() on those; not sure whether
everyone does this though.

> I checked if we call directly malloc() to pass the buffer to a
> library, but I failed to find such case.
>
> Again, in debug mode, calling free() on a memory block allocated by
> PyMem_Malloc() will likely crash. Since we run the Python test suite
> with a Python compiled in debug mode, we would already have detected
> such bug, no?

The Python test suite doesn't test Python C extensions,
so it's not surprising that it passes :-)

> See also my old issue http://bugs.python.org/issue18203 which replaced
> almost all direct calls to malloc() with PyMem_Malloc() or
> PyMem_RawMalloc().
> 
>> Good question. I guess developers simply thought of PyObject_Malloc()
>> being for PyObjects,
> 
> Yeah, I also understood that, but in practice, it looks like
> PyMem_Malloc() is slower than so using it makes the code less
> efficient than it can be.
> 
> Instead of teaching developers that well, in fact, PyObject_Malloc()
> is unrelated to object programming, I think that it's simpler to
> modify PyMem_Malloc() to reuse pymalloc ;-)

Perhaps if you add some guards somewhere :-)

Seriously, this may work if C extensions use the APIs
consistently, but in order to tell, we'd need to check
few. I know that I switched over all mx Extensions to
use PyObject_*() instead of PyMem_*() or native malloc()
several years ago and have not run into any issues.

I guess the main question then is whether pymalloc is good enough
for general memory allocation needs; and the answer may well be
"yes".

BTW: Tuning pymalloc for commonly used object sizes is
another area where Python could gain better performance,
i.e. reserve more / pre-allocate space for often used block
sizes. pymalloc will also only work well for small blocks
(up to 512 bytes). Everything else is routed to the
system malloc().

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/