[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Gregory P. Smith]

fwiw i'm also supportive of adding these apis.  Lets PEP away to iron out
any details or document disagreements but overall I'd also like to see
something a lot like these go in.

-gps


On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 June 2013 11:54, Victor Stinner <victor.stinner at gmail.com> wrote:
> > 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> >>> http://hg.python.org/cpython/rev/6661a8154eb3
> >>> ...
> >>>   Issue #3329: Add new APIs to customize memory allocators
> >>>
> >>> * Add a new PyMemAllocators structure
> >>> * New functions:
> >>>
> >>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free
> memory
> >>>     allocator functions
> >>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
> >>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
> >>>   - PyMem_SetupDebugHooks()
> >>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
> >>
> >> My two cents, but I would prefer if this whole changeset was reverted.
> >> I think it adds too much complexity in the memory allocation APIs,
> >> for a pretty specialized benefit. IMHO, we should be able to get by with
> >> less allocation APIs (why the new _Raw APIs) and less hook-setting
> >> functions.
> >
> > Ok, I reverted my commit.
> >
> > I posted my initial patch 3 months ago on the bug tracker. I got some
> > reviews and discussed with Kristján Valur Jónsson who heavily modified
> > Python for his game at CCP. I started two threads on python-dev this
> > week (ok, only two days ago). I thaugh that the last known issues were
> > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> > GIL-free allocator).
> >
> > I will work on a PEP to explain all these new functions and their use
> cases.
>
> I think the new APIs are mostly valid and well-justified, but agree a
> PEP is a good idea.
>
> Yes, it's a complex solution, but it's solving a complex problem that
> arises when embedding CPython inside executables that need to run on
> non-traditional platforms where the simple C defined
> malloc/realloc/free trio is inadequate.
>
> This is a complementary effort to PEP 432 - that aims to simplify
> embedding CPython in general, while Victor's efforts here specifically
> focus on situations where it is necessary to better map CPython to an
> underlying memory model that differs from the traditional C one. While
> the "single heap" model of memory enshrined in the C standard is
> certainly the most common model, it's far from being the only one, and
> these days CPython also gets used in those other environments.
>
> About the only simplification I can see is that PyMem_RawMalloc(),
> PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
> preprocessor macros instead of permitting runtime reconfiguration.
> Allowing the memory allocations for the CPython runtime to be handled
> separately from those for arbitrary C libraries loaded into the
> process seems reasonable, though.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/23063972/attachment-0001.html>