[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Nick Coghlan]

On 15 June 2013 11:54, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
>>> http://hg.python.org/cpython/rev/6661a8154eb3
>>> ...
>>>   Issue #3329: Add new APIs to customize memory allocators
>>>
>>> * Add a new PyMemAllocators structure
>>> * New functions:
>>>
>>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>>>     allocator functions
>>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>>   - PyMem_SetupDebugHooks()
>>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>>
>> My two cents, but I would prefer if this whole changeset was reverted.
>> I think it adds too much complexity in the memory allocation APIs,
>> for a pretty specialized benefit. IMHO, we should be able to get by with
>> less allocation APIs (why the new _Raw APIs) and less hook-setting
>> functions.
>
> Ok, I reverted my commit.
>
> I posted my initial patch 3 months ago on the bug tracker. I got some
> reviews and discussed with Kristján Valur Jónsson who heavily modified
> Python for his game at CCP. I started two threads on python-dev this
> week (ok, only two days ago). I thaugh that the last known issues were
> fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> GIL-free allocator).
>
> I will work on a PEP to explain all these new functions and their use cases.

I think the new APIs are mostly valid and well-justified, but agree a
PEP is a good idea.

Yes, it's a complex solution, but it's solving a complex problem that
arises when embedding CPython inside executables that need to run on
non-traditional platforms where the simple C defined
malloc/realloc/free trio is inadequate.

This is a complementary effort to PEP 432 - that aims to simplify
embedding CPython in general, while Victor's efforts here specifically
focus on situations where it is necessary to better map CPython to an
underlying memory model that differs from the traditional C one. While
the "single heap" model of memory enshrined in the C standard is
certainly the most common model, it's far from being the only one, and
these days CPython also gets used in those other environments.

About the only simplification I can see is that PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
preprocessor macros instead of permitting runtime reconfiguration.
Allowing the memory allocations for the CPython runtime to be handled
separately from those for arbitrary C libraries loaded into the
process seems reasonable, though.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia