[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators
victor.stinner at gmail.com
Sat Jun 15 03:54:50 CEST 2013
2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
>> Issue #3329: Add new APIs to customize memory allocators
>> * Add a new PyMemAllocators structure
>> * New functions:
>> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>> allocator functions
>> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>> - PyMem_GetAllocators(), PyMem_SetAllocators()
>> - PyMem_SetupDebugHooks()
>> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
> My two cents, but I would prefer if this whole changeset was reverted.
> I think it adds too much complexity in the memory allocation APIs,
> for a pretty specialized benefit. IMHO, we should be able to get by with
> less allocation APIs (why the new _Raw APIs) and less hook-setting
Ok, I reverted my commit.
I posted my initial patch 3 months ago on the bug tracker. I got some
reviews and discussed with Kristján Valur Jónsson who heavily modified
Python for his game at CCP. I started two threads on python-dev this
week (ok, only two days ago). I thaugh that the last known issues were
fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
I will work on a PEP to explain all these new functions and their use cases.
The addition of PyMem_RawMalloc() is motivated by the issue #18203
(Replace calls to malloc() with PyMem_Malloc()). The goal is to be
able to setup a custom allocator for *all* allocation made by Python,
so malloc() should not be called directly. PyMem_RawMalloc() is
required in places where the GIL is not held (ex: in os.getcwd() on
PyMem_Malloc() is misused (called without the GIL held) in different
places. Examples: the readline modules and functions called at Python
startup, including main(). Replacing PyMem_Malloc() with malloc()
would not allow to use the custom allocator everywhere, so
PyMem_RawMalloc() is also required here.
The last point is an extension of the issue #18203: some external
libraries like zlib or OpenSSL are also calling malloc() directly. But
Python can configure these libraries to use a custom memory allocator.
I plan to configure external libraries to use PyMem_GetRawAllocators()
if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not
simply malloc) and if setting a custom allocator only affect a
function and not the whole library.
More information about the Python-Dev