[Python-Dev] Extension modules, Threading, and the GIL

[Martin v. Löwis]

David Abrahams <dave@boost-consulting.com> writes:

> >> The symptom is that Python complains at some point that there's no
> >> thread state.  It goes away if A releases the GIL before calling
> >> into Qt, and reacquires the GIL afterwards.
[...]
> No, I am not saying A releases the GIL.  

"...there is no thread state. It [the thread state] goes away if A
releases the GIL ..."

>From that I inferred that A releases the GIL, since you said that
there is no thread state. Rereading your message, I now see that you
meant "It [the problem] goes away".

So I now understand that you reported that there is no deadlock, and
that A does not release the GIL, and that Python reports that there is
no thread state "when A returns to Python". You also report that B
acquires the GIL.

I can't understand why this happens. How does B acquire the GIL?

Assuming that B uses PyEval_AcquireThread/PyEval_ReleaseThread, I
would expect that 
a) there is a deadlock if this happens in a context of a call to
   A, since the GIL is already held, and (if, for some reason,
   locks are recursive on this platform), 
b) the code

	if (PyThreadState_Swap(tstate) != NULL)
		Py_FatalError(
			"PyEval_AcquireThread: non-NULL old thread state");

   should trigger, as there is an old thread state.

So I infer that B does not use
PyEval_AcquireThread/PyEval_ReleaseThread. What else does it use?

> >> I speculate that the callback releases the GIL when it is finished,
> >> so that when A returns to Python there is no current thread.
> >
> > That would be a bug in the callback. 
> 
> Not if it has previously acquired the GIL.

Even then, but I refrain from speculating how B's code to release the
lock actually looks.

> > If there was a thread state when it was called, there should be a
> > thread state when it returns.
> 
> Yes, the whole problem is that there's no way to know whether there's
> a thread state.

Wrong. If B acquires the GIL, B must use some thread state to do
so. It must install that thread state through PyThreadState_Swap,
directly or indirectly. That will return the old thread state, or
NULL.

> > If so, a mutex-protected record might work, but also might be
> > expensive.
> 
> Yes.  I assume that acquiring the GIL already needs to do
> synchronization, though.

Sure. But with that proposed change, you have not only the GIL lock
call (which is a single sem_wait call on Posix, and an
InterlockedCompareExchange call on Win32). You also get a mutex call,
and a call to find out the current thread.

Regards,
Martin