[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

[Scott Dial]

On 6/18/2013 4:40 PM, Victor Stinner wrote:
> No context argument
> -------------------
> 
> Simplify the signature of allocator functions, remove the context
> argument:
> 
> * ``void* malloc(size_t size)``
> * ``void* realloc(void *ptr, size_t new_size)``
> * ``void free(void *ptr)``
> 
> It is likely for an allocator hook to be reused for
> ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
> ``PyMem_SetRawAllocator()``, but the hook must call a different function
> depending on the allocator. The context is a convenient way to reuse the
> same custom allocator or hook for different Python allocators.

I think there is a lack of justification for the extra argument, and the
extra argument is not free. The typical use-case for doing this
continuation-passing style is when the set of contexts is either
unknown, arbitrarily large, or infinite. In other words, when it would
be either impossible or impractical to enumerate all of the contexts.
However, in this case, we have only 3.

Your proposal already puts forward having 3 pairs of Get/Set functions,
so there is no distinct advantage in having a single typedef instance
that you pass in to all 3 of them. And, having all 3 pairs use the same
typedef is a bit of an attractive nuisance, in that one could pass the
wrong allocators to the wrong setter. With that, I could argue that
there should be 3 typedefs to prevent coding errors.

Nevertheless, the ctx argument buys the implementer nothing if they have
to begin their alloc function with "if(ctx == X)". In other words, there
is nothing simpler about:

"""
void *_alloc(void *ctx, size_t size) {
  if(ctx == PYALLOC_PYMEM)
    return _alloc_pymem(size);
  else if(ctx == PYALLOC_PYMEM_RAW)
    return _alloc_pymem_raw(size);
  else if(ctx == PYALLOC_PYOBJECT)
    return _alloc_pyobject(size);
  else
    abort();
}

PyMemBlockAllocator pymem_allocator =
  {.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pymem_raw_allocator =
  {.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pyobject_allocator =
  {.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free};
"""

In comparison to:

"""
PyMemBlockAllocator pymem_allocator =
  {.alloc=&_alloc_pymem, .free=&_free_pymem};
PyMemBlockAllocator pymem_raw_allocator =
  {.alloc=&_alloc_pymem_raw, .free=&_free_pymem};
PyMemBlockAllocator pyobject_allocator =
  {.alloc=&_alloc_pyobject, .free=&_free_pyobject};
"""

And in the latter case, there is no extra indirect branching in the
hot-path of the allocators.

Also, none of the external libraries cited introduce this CPS/ctx stuff.

-- 
Scott Dial
scott at scottdial.com