fwiw i'm also supportive of adding these apis. Lets PEP away to iron out any details or document disagreements but overall I'd also like to see something a lot like these go in. -gps On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 15 June 2013 11:54, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/6/15 Antoine Pitrou <solipsis@pitrou.net>:
http://hg.python.org/cpython/rev/6661a8154eb3 ... Issue #3329: Add new APIs to customize memory allocators
* Add a new PyMemAllocators structure * New functions:
- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory allocator functions - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() - PyMem_GetAllocators(), PyMem_SetAllocators() - PyMem_SetupDebugHooks() - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
My two cents, but I would prefer if this whole changeset was reverted. I think it adds too much complexity in the memory allocation APIs, for a pretty specialized benefit. IMHO, we should be able to get by with less allocation APIs (why the new _Raw APIs) and less hook-setting functions.
Ok, I reverted my commit.
I posted my initial patch 3 months ago on the bug tracker. I got some reviews and discussed with Kristján Valur Jónsson who heavily modified Python for his game at CCP. I started two threads on python-dev this week (ok, only two days ago). I thaugh that the last known issues were fixed with the addition of PyMem_SetupDebugHooks() (to avoid an environment variable, as asked by Nick) and PyMem_RawMalloc() (have a GIL-free allocator).
I will work on a PEP to explain all these new functions and their use cases.
I think the new APIs are mostly valid and well-justified, but agree a PEP is a good idea.
Yes, it's a complex solution, but it's solving a complex problem that arises when embedding CPython inside executables that need to run on non-traditional platforms where the simple C defined malloc/realloc/free trio is inadequate.
This is a complementary effort to PEP 432 - that aims to simplify embedding CPython in general, while Victor's efforts here specifically focus on situations where it is necessary to better map CPython to an underlying memory model that differs from the traditional C one. While the "single heap" model of memory enshrined in the C standard is certainly the most common model, it's far from being the only one, and these days CPython also gets used in those other environments.
About the only simplification I can see is that PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with preprocessor macros instead of permitting runtime reconfiguration. Allowing the memory allocations for the CPython runtime to be handled separately from those for arbitrary C libraries loaded into the process seems reasonable, though.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org