On 04.02.2016 14:25, Victor Stinner wrote:
Thanks for your feedback, you are asking good questions :-)
2016-02-04 13:54 GMT+01:00 M.-A. Lemburg firstname.lastname@example.org:
There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().
I would prefer to modify a single place having to replace 536 calls :-/
You have a point there, but I don't think it'll work out that easily, since we are using such calls to e.g. pass dynamically allocated buffers to code in extensions (which then have to free the buffers again).
Ah, interesting. But I'm not sure that we delegate the responsability of freeing the memory to external libraries. Usually, it's more the opposite: a library gives us an allocated memory block, and we have to free it. No?
Sometimes, yes, but we also do allocations for e.g. parsing values in Python argument tuples (e.g. using "es" or "et"):
We do document to use PyMem_Free() on those; not sure whether everyone does this though.
I checked if we call directly malloc() to pass the buffer to a library, but I failed to find such case.
Again, in debug mode, calling free() on a memory block allocated by PyMem_Malloc() will likely crash. Since we run the Python test suite with a Python compiled in debug mode, we would already have detected such bug, no?
The Python test suite doesn't test Python C extensions, so it's not surprising that it passes :-)
See also my old issue http://bugs.python.org/issue18203 which replaced almost all direct calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc().
Good question. I guess developers simply thought of PyObject_Malloc() being for PyObjects,
Yeah, I also understood that, but in practice, it looks like PyMem_Malloc() is slower than so using it makes the code less efficient than it can be.
Instead of teaching developers that well, in fact, PyObject_Malloc() is unrelated to object programming, I think that it's simpler to modify PyMem_Malloc() to reuse pymalloc ;-)
Perhaps if you add some guards somewhere :-)
Seriously, this may work if C extensions use the APIs consistently, but in order to tell, we'd need to check few. I know that I switched over all mx Extensions to use PyObject_*() instead of PyMem_*() or native malloc() several years ago and have not run into any issues.
I guess the main question then is whether pymalloc is good enough for general memory allocation needs; and the answer may well be "yes".
BTW: Tuning pymalloc for commonly used object sizes is another area where Python could gain better performance, i.e. reserve more / pre-allocate space for often used block sizes. pymalloc will also only work well for small blocks (up to 512 bytes). Everything else is routed to the system malloc().