[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Christian Heimes]

Am 15.06.2013 14:22, schrieb Nick Coghlan:
> However, it's still desirable to be able to monitor those direct
> allocations in debug mode, thus it makes sense to have a GIL protected
> direct allocation API as well. You could try to hide the existence of
> the latter behaviour and treat it as a private API, but why? For
> custom allocators, it's useful to be able to *ensure* you can bypass
> CPython's small object allocator, rather than having to rely on it
> being bypassed for allocations above a certain size.


There is even more to it. We like to keep track of memory allocations in
libraries that are wrapped by Python's extension modules, e.g. expat,
openssl etc. Almost every library has a hook to set a custom memory
allocator, either globally (CRYPTO_set_mem_functions) or for each object
(XML_ParserCreate_MM's XML_Memory_Handling_Suite).

Python releases the GIL around IO or CPU critical sections of the
library. But these sections may call the memory management functions. If
these memory functions use the GIL, then some speed ups of releasing the
GIL in the first place are lost. It might even be possible to walk into
dead lock situations.

For that reason it makes sense to have a set of low level memory
management functions, that don't rely on the GIL for locking. These
functions can still impose their own locking if they need to modify a
global state (e.g. allocation statistics). In normal release mode and on
most platforms the raw memory allocators should be thin wrappers around
malloc(), realloc() and free() -- or perhaps just macros.

Eventually I would like to ban direct usage of malloc() from Python's
core and patch all memory management through our API.

Christian