[Python-Dev] Enhancement of Python memory allocators

[Nick Coghlan]

On 13 Jun 2013 09:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> Hi,
>
> I would like to improve memory allocators of Python. My two use cases
> are replacing memory allocators with custom allocators in embedded
> system and hooking allocators to track usage of memory.
>
> I wrote a patch for this, I'm going to commit it if nobody complains:
> http://bugs.python.org/issue3329
>
> Using this patch, detecting memory corruptions (buffer underflow and
> overflow) can be done without recompilation. We may add an environment
> variable to enable Python debug functions at runtime, example:
> PYDEBUGMALLOC=1. There is just a restriction: the environment variable
> would not be ignored with -E command line option, because command line
> options are parsed after the first memory allocation. What do you
> think?

The rest of it sounds fine, but please don't add the runtime switching
support to our existing main function. Interpreter startup is a mess
already. If you were interested in helping directly with PEP 432, though,
that would be good - I haven't been able to spend much time on it lately.

Cheers,
Nick.

>
> *****
>
> The patch adds the following functions:
>
> void PyMem_GetAllocators(
>     void **ctx_p,
>     void* (**malloc_p) (void *ctx, size_t size),
>     void* (**realloc_p) (void *ctx, void *ptr, size_t size),
>     void (**free_p) (void *ctx, void *ptr));
>
> void PyMem_SetAllocators(
>     void *ctx,
>     void* (*malloc) (void *ctx, size_t size),
>     void* (*realloc) (void *ctx, void *ptr, size_t size),
>     void (*free) (void *ctx, void *ptr));
>
> It adds 4 similar functions (get/set) for PyObject_Malloc() and
> allocators of pymalloc arenas.
>
> *****
>
> For the "track usage of memory" use case, see the following project
> which hooks memory allocators using PyMem_SetAllocators() and
> PyObject_SetAllocators() to get allocated bytes per filename and line
> number.
> https://pypi.python.org/pypi/pytracemalloc
>
> *****
>
> Another issue proposes to use VirtualAlloc() and VirtualFree() for
> pymalloc arenas, see:
> http://bugs.python.org/issue13483
>
> I don't know if it would be interesting, but it would now possible to
> choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc,
> ...) at runtime, with an environment variable for example.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130613/76c7f867/attachment.html>