[Python-Dev] head crashing

[Kristján Valur Jónsson]

> but I got to a different conclusion. If it really goes through the
> pymalloc pool (obmalloc), then it must hold the GIL while doing so.
> obmalloc itself is not thread-safe, and relies on the GIL for
> thread-safety.
>
> In release mode, PyMEM_FREE goes directly to free, which is thread-
> safe.

Yes.  It is quite unfortunate how PyMem_* gets redirected to the PyObject_*
functions in debug builds.
Even worse is how PyObject_Malloc gets #defined to PyObject_DebugMalloc
for debug builds, changing linkage of modules.  But that is a different
matter.

One thing I'd like to point out however, is that it is quite unnecessary
for the PyObject_DebugMalloc() functions to lie on top of PyObject_Malloc()
They can just call malloc() etc. directly, since in debug builds the
performance benefit of the block allocator is moot.  I'd suggest to
keep the debug functions as a thin layer on top of malloc to do basic
testing.  I'd even suggest that we reverse things, and move the debug
library to pymem.c.  This would keep the debug functionalty threadsafe
on top of regular malloc, rather than wrapping it in there with the
non-threadsafe object allocator.

We would then have
void *PyMem_DebugMalloc() /* layers malloc /*
void *PyMem_Malloc() /* calls PyMem_MALLOC */

#ifndef _DEBUG
#define PyMem_MALLOC malloc
#else
#define PyMem_MALLOC PyMem_DebugMalloc
#endif

PyObject_Malloc() would then just call PyMem_DebugMalloc in DEBUG builds.

The reason I have opinions on this is that at CCP we have spent considerable
effort on squeezing our own veneer functions into the memory allocators, both
for the PyMem ones and PyObject.  And the structure of the macros and their
interconnectivity really doesn't make it easy.  We ended up creating a set
of macros like PyMem_MALLOC_INNER()  and ease our functions between the
MALLOC and the INNER.  I'll try to show you the patch one day which is a
reasonable attempt at a slight reform in the structure of these
memory APIs.  Perhaps something for Py3K.


Kristjan