[Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)

[Tim Peters]

[Neal Norwitz]
>> In rev 54982 (the first time this crash was seen), I see something
>> which might create a problem.  In python/trunk/Modules/posixmodule.c
>> (near line 6300):
>>
>> +       PyMem_FREE(mode);
>>        Py_END_ALLOW_THREADS

Shouldn't do that.

[Brett Cannon]
> The PyMem_MALLOC call that creates 'mode' is also called without explicitly
> holding the GIL.

Or that ;-)

>> Can you call PyMem_FREE() without the GIL held?  I couldn't find it
>> documented either way.

> I believe the GIL does not need to be held, but obviously Tim or someone
> with more memory experience should step in to say definitively.

The GIL should be held.  The relevant docs are in the Python/C API
manual, section "8.1 Thread State and the Global Interpreter Lock":

    Therefore, the rule exists that only the thread that has acquired the global
    interpreter lock may operate on Python objects or call Python/C
API functions.

PyMem_XYZ is certainly a "Python/C API function".  There are functions
you can call without holding the GIL, and section 8.1 intends to give
an exhaustive list of those.  These are functions that can't rely on
the GIL, like PyEval_InitThreads() (which /creates/ the GIL), and
various functions that create and destroy thread and interpreter
state.

> If you look at Include/pymem.h, PyMem_FREE gets defined as PyObject_FREE in
> a debug build.  PyObject_Free is defined at _PyObject_DebugFree.  That
> function checks that the memory has not been written with the debug bit
> pattern and then calls PyObject_Free.  That call just sticks the memory back
> into pymalloc's memory pool which is implemented without using any Python
> objects.

But pymalloc's pools have a complex internal structure of their own,
and cannot be mucked with safely by multiple threads simultaneously.

> In other words no Python objects are used in pymalloc (to my knowledge) and
> thus is safe to use without the GIL.

Nope.  For example, if two threads simultaneously try to free objects
in the same obmalloc size class, there are a number of potential
thread-race disasters in linking the objects into the same size-class
chain.

In a release build this doesn't matter, since PyMem_XYZ map directly
to the platform malloc/realloc/free, and so inherit the thread safety
(or lack thereof) of the platform C implementations.

If it's necessary to do malloc/free kinds of things without holding
the GIL, then the platform malloc/free must be called directly.
Perhaps that's what posixmodule.c wants to do in this case.