[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> There was no deadlock, as I've said.=20=20
>
> Yes, you only said this was a no-no, in
>
> http://mail.python.org/pipermail/python-dev/2002-December/031449.html
>
> I inferred from that (apparently incorrectly) that the no-no is a
> deadlock.

In that very message, I also wrote:

    I realize that the docs for PyEval_AcquireLock() say: "If this
    thread already has the lock, a deadlock ensues", but the behavior
    we're seeing is consistent with a scenario where trying to acquire
    an already-held is a no-op and releasing it is unconditional.
    Eventually the GIL release in B's callback takes effect and when A
    returns to Python there is no thread state.

>> The symptom is that Python complains at some point that there's no
>> thread state.  It goes away if A releases the GIL before calling
>> into Qt, and reacquires the GIL afterwards.
>
> Now I'm confused. In
>
> http://mail.python.org/pipermail/python-dev/2002-December/031424.html
>
> you said "A must also release the GIL"=20

Yes, that's the inevitable conclusion.

> and then "the author of A may have had no reason to believe anyone
> would install Python callbacks in Q". From that I inferred that A
> does *not* release the GIL (as the author had no reason to).

Yes, not releasing the GIL in A was fine until B came along and
installed callbacks in Q which acquire the GIL.
>
> Now you are saying that A releases the GIL. Which one it is?

No, I am not saying A releases the GIL.  I am saying that A that must
release the GIL if it is to work properly in the presence of B.  A is
currently broken in the presence of B.  The addition of B to the
system places a new constraint on A.

>> I speculate that the callback releases the GIL when it is finished,
>> so that when A returns to Python there is no current thread.
>
> That would be a bug in the callback.=20

Not if it has previously acquired the GIL.

> If there was a thread state when it was called, there should be a
> thread state when it returns.

Yes, the whole problem is that there's no way to know whether there's
a thread state.

>> Hmm, it seems as though a mutex-protected record of which thread is
>> currently holding the GIL should be enough to handle it.=20
>
> It depends on what "it" is, here. This one?
>
> Q: Is there a way to find out whether the current thread holds the GIL?
>
> If so, a mutex-protected record might work, but also might be
> expensive.

Yes.  I assume that acquiring the GIL already needs to do
synchronization, though.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution