[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Antoine Pitrou]

On Sun, 16 Jun 2013 00:12:02 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 June 2013 22:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 15 Jun 2013 22:22:33 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> For
> >> custom allocators, it's useful to be able to *ensure* you can bypass
> >> CPython's small object allocator, rather than having to rely on it
> >> being bypassed for allocations above a certain size.
> >
> > Which custom allocators?
> 
> Those used by companies like Dropbox to speed up frequent allocations
> (look up their PyCon 2011 keynote). If we don't provide suitable APIs
> that we can still hook in debug mode, they'll bypass our
> infrastructure completely and we'll miss significant memory accesses.

I don't understand the concern. People can ignore the Python
allocators, and then use their own debugging infrastructure. This is
what happens everytime you want to use your own infrastructure instead
of a 3rd party-provided one.

> > And I'm not even sure what "faux simplicity" you are talking about.
> > What is the supposed complexity that this API is supposed to address?
> 
> The fact that there is more to the world than x86/x86_64 and the very
> simplistic C memory model.

Then I think it needs a PEP to properly address it and explain it to
everyone.

Moreover, I think you are conflating two issues: the ability to add
memory allocation hooks (for tracing/debugging purposes), and the
adaptation to "non-traditional" memory models (whatever that means).
Those concerns don't necessarily come together.

Regards

Antoine.