[Python-Dev] Python initialization and embedded Python

[Victor Stinner]

2017-11-20 22:35 GMT+01:00 Eric Snow <ericsnowcurrently at gmail.com>:
> I'm okay with that if we can't find another way.  However, shouldn't
> we be able to statically initialize the raw allocator in _PyRuntime,
> much as we were doing before in obmalloc.c?  I have a rough PR up:
>
>     https://github.com/python/cpython/pull/4481
>
> Also, I opened https://bugs.python.org/issue32096 for the regression.
> Thanks for bringing it up.

To statically initialize PyMemAllocatorEx fields, you need to export a
lot of allocator functions. I would prefer to not do that.

static void* _PyMem_DebugRawMalloc(void *ctx, size_t size);
static void* _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize);
static void* _PyMem_DebugRawRealloc(void *ctx, void *ptr, size_t size);
static void _PyMem_DebugRawFree(void *ctx, void *ptr);

static void* _PyMem_DebugMalloc(void *ctx, size_t size);
static void* _PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize);
static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
static void _PyMem_DebugFree(void *ctx, void *p);

static void* _PyObject_Malloc(void *ctx, size_t size);
static void* _PyObject_Calloc(void *ctx, size_t nelem, size_t elsize);
static void _PyObject_Free(void *ctx, void *p);
static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);

The rules to choose the allocator to each domain are also complex
depending if pymalloc is enabled, debug hooks are enabled by default,
etc. The memory allocator is also linked to _PyMem_Debug which is not
currently in Include/internals/ but Objects/obmalloc.c.

I understand that moving global variables to _PyRuntime helps to
clarify how these variables are initialized and then finalized, but
memory allocators are a complex corner case.

main(), Py_Main() and _PyRuntime_Initialize() now have to change
temporary the allocators to make sure that their initialization and
finalization use the same allocator.

I prefer to revert the change on memory allocators, and retry later to
fix it, once other initializations issues are fixed ;-)

Victor