[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators
Gregory P. Smith
greg at krypto.org
Sat Jun 15 09:54:10 CEST 2013
fwiw i'm also supportive of adding these apis. Lets PEP away to iron out
any details or document disagreements but overall I'd also like to see
something a lot like these go in.
On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 June 2013 11:54, Victor Stinner <victor.stinner at gmail.com> wrote:
> > 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> >>> http://hg.python.org/cpython/rev/6661a8154eb3
> >>> ...
> >>> Issue #3329: Add new APIs to customize memory allocators
> >>> * Add a new PyMemAllocators structure
> >>> * New functions:
> >>> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free
> >>> allocator functions
> >>> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
> >>> - PyMem_GetAllocators(), PyMem_SetAllocators()
> >>> - PyMem_SetupDebugHooks()
> >>> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
> >> My two cents, but I would prefer if this whole changeset was reverted.
> >> I think it adds too much complexity in the memory allocation APIs,
> >> for a pretty specialized benefit. IMHO, we should be able to get by with
> >> less allocation APIs (why the new _Raw APIs) and less hook-setting
> >> functions.
> > Ok, I reverted my commit.
> > I posted my initial patch 3 months ago on the bug tracker. I got some
> > reviews and discussed with Kristján Valur Jónsson who heavily modified
> > Python for his game at CCP. I started two threads on python-dev this
> > week (ok, only two days ago). I thaugh that the last known issues were
> > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> > GIL-free allocator).
> > I will work on a PEP to explain all these new functions and their use
> I think the new APIs are mostly valid and well-justified, but agree a
> PEP is a good idea.
> Yes, it's a complex solution, but it's solving a complex problem that
> arises when embedding CPython inside executables that need to run on
> non-traditional platforms where the simple C defined
> malloc/realloc/free trio is inadequate.
> This is a complementary effort to PEP 432 - that aims to simplify
> embedding CPython in general, while Victor's efforts here specifically
> focus on situations where it is necessary to better map CPython to an
> underlying memory model that differs from the traditional C one. While
> the "single heap" model of memory enshrined in the C standard is
> certainly the most common model, it's far from being the only one, and
> these days CPython also gets used in those other environments.
> About the only simplification I can see is that PyMem_RawMalloc(),
> PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
> preprocessor macros instead of permitting runtime reconfiguration.
> Allowing the memory allocations for the CPython runtime to be handled
> separately from those for arbitrary C libraries loaded into the
> process seems reasonable, though.
> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
> Python-Dev mailing list
> Python-Dev at python.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev