[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Victor Stinner]

2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
>> http://hg.python.org/cpython/rev/6661a8154eb3
>> ...
>>   Issue #3329: Add new APIs to customize memory allocators
>>
>> * Add a new PyMemAllocators structure
>> * New functions:
>>
>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>>     allocator functions
>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>   - PyMem_SetupDebugHooks()
>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>
> My two cents, but I would prefer if this whole changeset was reverted.
> I think it adds too much complexity in the memory allocation APIs,
> for a pretty specialized benefit. IMHO, we should be able to get by with
> less allocation APIs (why the new _Raw APIs) and less hook-setting
> functions.

Ok, I reverted my commit.

I posted my initial patch 3 months ago on the bug tracker. I got some
reviews and discussed with Kristján Valur Jónsson who heavily modified
Python for his game at CCP. I started two threads on python-dev this
week (ok, only two days ago). I thaugh that the last known issues were
fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
GIL-free allocator).

I will work on a PEP to explain all these new functions and their use cases.

**

The addition of PyMem_RawMalloc() is motivated by the issue #18203
(Replace calls to malloc() with PyMem_Malloc()). The goal is to be
able to setup a custom allocator for *all* allocation made by Python,
so malloc() should not be called directly. PyMem_RawMalloc() is
required in places where the GIL is not held (ex: in os.getcwd() on
Windows).

PyMem_Malloc() is misused (called without the GIL held) in different
places. Examples: the readline modules and functions called at Python
startup, including main(). Replacing PyMem_Malloc() with malloc()
would not allow to use the custom allocator everywhere, so
PyMem_RawMalloc() is also required here.

The last point is an extension of the issue #18203: some external
libraries like zlib or OpenSSL are also calling malloc() directly. But
Python can configure these libraries to use a custom memory allocator.
I plan to configure external libraries to use PyMem_GetRawAllocators()
if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not
simply malloc) and if setting a custom allocator only affect a
function and not the whole library.

Victor