[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

Aahz <aahz@pythoncraft.com> writes:

> On Sun, Dec 29, 2002, David Abrahams wrote:
>>
>> Python extension module B calls shared library Q and uses Q's callback
>> interface.  Because some of the callbacks need to use the Python API,
>> and *might* be invoked by threads, they must all acquire the GIL.
>> Because they also might be invoked by direct calls into Q, B must
>> always release the GIL before calling anything in Q.
>
> So you're saying that the callback functions in B acquire the GIL?

Yes.

>> Problem: using B while A is loaded breaks A: because B has installed
>> callbacks in Q that acquire the GIL, A must also release the GIL
>> before calling into Q.
>
> Why?  The callbacks in B will simply hang until they acquire the GIL.

If A doesn't release the GIL, one of its direct calls into Q from
Python may invoke a callback in B, which tries to acquire the lock
when it is already held.  This is a no-no.

I realize that the docs for PyEval_AcquireLock() say: "If this thread
already has the lock, a deadlock ensues", but the behavior we're
seeing is consistent with a scenario where trying to acquire an
already-held is a no-op and releasing it is unconditional.  Eventually
the GIL release in B's callback takes effect and when A returns to
Python there is no thread state.

> I think this thread might be better handled on c.l.py, at least
> until it's understood well enough to be clear whether something does
> need to change in Python.

I'm pretty certain I understand what's going on.  If it would be
better for you to take this to c.l.py, I'm happy to do so, but AFAICT
there _is_ a Python core issue here: there's no way to find out
whether you've already got the GIL**, so if you _might_ have to acquire
it, you must always acquire it.

**If I'm wrong about that, please let me know.  It isn't obvious from
  the documentation.

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution