
2013/6/15 Antoine Pitrou <solipsis@pitrou.net>:
http://hg.python.org/cpython/rev/6661a8154eb3 ... Issue #3329: Add new APIs to customize memory allocators
* Add a new PyMemAllocators structure * New functions:
- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory allocator functions - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() - PyMem_GetAllocators(), PyMem_SetAllocators() - PyMem_SetupDebugHooks() - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
My two cents, but I would prefer if this whole changeset was reverted. I think it adds too much complexity in the memory allocation APIs, for a pretty specialized benefit. IMHO, we should be able to get by with less allocation APIs (why the new _Raw APIs) and less hook-setting functions.
Ok, I reverted my commit. I posted my initial patch 3 months ago on the bug tracker. I got some reviews and discussed with Kristján Valur Jónsson who heavily modified Python for his game at CCP. I started two threads on python-dev this week (ok, only two days ago). I thaugh that the last known issues were fixed with the addition of PyMem_SetupDebugHooks() (to avoid an environment variable, as asked by Nick) and PyMem_RawMalloc() (have a GIL-free allocator). I will work on a PEP to explain all these new functions and their use cases. ** The addition of PyMem_RawMalloc() is motivated by the issue #18203 (Replace calls to malloc() with PyMem_Malloc()). The goal is to be able to setup a custom allocator for *all* allocation made by Python, so malloc() should not be called directly. PyMem_RawMalloc() is required in places where the GIL is not held (ex: in os.getcwd() on Windows). PyMem_Malloc() is misused (called without the GIL held) in different places. Examples: the readline modules and functions called at Python startup, including main(). Replacing PyMem_Malloc() with malloc() would not allow to use the custom allocator everywhere, so PyMem_RawMalloc() is also required here. The last point is an extension of the issue #18203: some external libraries like zlib or OpenSSL are also calling malloc() directly. But Python can configure these libraries to use a custom memory allocator. I plan to configure external libraries to use PyMem_GetRawAllocators() if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not simply malloc) and if setting a custom allocator only affect a function and not the whole library. Victor