[Python-Dev] Enhancement of Python memory allocators

Victor Stinner victor.stinner at gmail.com
Thu Jun 13 01:06:30 CEST 2013


I would like to improve memory allocators of Python. My two use cases
are replacing memory allocators with custom allocators in embedded
system and hooking allocators to track usage of memory.

I wrote a patch for this, I'm going to commit it if nobody complains:

Using this patch, detecting memory corruptions (buffer underflow and
overflow) can be done without recompilation. We may add an environment
variable to enable Python debug functions at runtime, example:
PYDEBUGMALLOC=1. There is just a restriction: the environment variable
would not be ignored with -E command line option, because command line
options are parsed after the first memory allocation. What do you


The patch adds the following functions:

void PyMem_GetAllocators(
    void **ctx_p,
    void* (**malloc_p) (void *ctx, size_t size),
    void* (**realloc_p) (void *ctx, void *ptr, size_t size),
    void (**free_p) (void *ctx, void *ptr));

void PyMem_SetAllocators(
    void *ctx,
    void* (*malloc) (void *ctx, size_t size),
    void* (*realloc) (void *ctx, void *ptr, size_t size),
    void (*free) (void *ctx, void *ptr));

It adds 4 similar functions (get/set) for PyObject_Malloc() and
allocators of pymalloc arenas.


For the "track usage of memory" use case, see the following project
which hooks memory allocators using PyMem_SetAllocators() and
PyObject_SetAllocators() to get allocated bytes per filename and line


Another issue proposes to use VirtualAlloc() and VirtualFree() for
pymalloc arenas, see:

I don't know if it would be interesting, but it would now possible to
choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc,
...) at runtime, with an environment variable for example.


More information about the Python-Dev mailing list