[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

[Nick Coghlan]

On 20 Jun 2013 02:03, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2013/6/19 Kristján Valur Jónsson <kristjan at ccpgames.com>:
> > Oh, it should be public, in my opinion.
>
> Ok. And do you think that the PyMemMappingAllocator structure is
> complete, or that we should add something to be future-proof? At
> least, PyMemMappingAllocator is enough for pymalloc usage :-)
>
> Is PyMemMappingAllocator complete enough for your usage at CCP Games?

Can we go back to calling this the "Arena" allocator? Or at least "Mapped"?
When I see "Mapping" in the context of Python I think of the container API,
not a memory allocation API.

>
> > We do exactly that when we embed python into UnrealEngine.  We keep
pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for
its arenas.  That way, we can still keep track of python's memory usage
from with the larger application, even if the granularity of memory is now
on an "arena" level, rather than individual allocs.
>
> I hope that the PEP 445 is flexible enough to allow you to decide
> which functions are hooked and replaced, and which functions will be
> leaved unchanged. That's why I'm not in favor of the "Make
> PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.

It's also why I'm in favour of the "domain" API rather than separate
functions.

1. In the initial iteration, just have the three basic domains (raw,
interpreter, objects). Replacing allocators for third party libraries is
the responsibility of embedding applications.

2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs
so that extension modules can register new domains for wrapped libraries.
Replacing allocators is still the responsibility of embedding applications,
but there's a consistent API to do it.

(Alternatively, we could do both now)

And agreed PyMem_Raw* are the only new APIs that should be added to the
stable ABI.

Cheers,
Nick.

>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/73c9c52d/attachment.html>