[Python-Dev] GIL vs thread state

[Guido van Rossum]

> > If pts->frame is not NULL, I'd expect a warning from
> > PyThreadState_Clear(): "PyThreadState_Clear: warning: thread still has
> > a frame\n".
> You mean this code, from Python/pystate.h?
> 
> void
> PyThreadState_Clear(PyThreadState *tstate)
> {
> 	if (Py_VerboseFlag && tstate->frame != NULL)
> 		fprintf(stderr,
> 		  "PyThreadState_Clear: warning: thread still has a frame\n");
> 
> 	ZAP(tstate->frame);
> 
> 	ZAP(tstate->dict);
> ...
> }

Yes.

> Py_VerboseFlag is 0 set in my case, so no warning is printed.

OK.

> > > "Fatal Python error: PyThreadState_Get: no current thread" in the call
> > > to PyThreadState_Clear().
> > 
> > That's strange, because I cannot trace the code in there to such a
> > call.  (Unless it is in a destructor.
> 
> It is in a destructor: frame_dealloc, called from ZAP(tstate->frame).

Aha.  That wasn't obvious from your description.

> > Can you tell more about where the PyThreadState_Get() call was?)
> 
> This function allocates the threadstate for me:
> 
> static void EnterPython(char *msg)
> {
> 	PyThreadState *pts;
> 	PyEval_AcquireLock();
> 	pts = PyThreadState_New(g_interp);
> 	if (!pts)
> 		Py_FatalError("wincall: Could not allocate ThreadState");
> 	if (NULL != PyThreadState_Swap(pts))
> 		Py_FatalError("wincall (EnterPython): thread state not == NULL?");
> }

Maybe you should have a look at Mark Hammond's PEP 311.  It describes
the problem and proposes a better solution.  (I think it requires you
to always use the existing thread state for the thread, rather than
making up a temporary thread state as is currently the idiom.)

> To explain the picture a little better, here is the sequence of calls:
> 
> Python calls into a C extension.
> The C extension does
> 
> Py_BEGIN_ALLOW_THREADS
>   call_a_C_function()
> Py_END_ALLOW_THREADS
> 
> The call_a_C_function calls back into C code like this:
> 
> void MyCallback(void)
> {
>     EnterPython();  /* acquire the lock, and create a thread state */
>     execute_some_python_code();
>     LeavePython(); /* destroy the thread state, and release the lock */
> }
> 
> Now, the execute_some_python_code() section is enclosed in a win32
> structured exception handling block, and it may return still with a
> frame in the threadstate, as it seems.

Ouch!  I don't know what structured exception handling is, but this
looks like it would be as bad as using setjmp/longjmp to get back to
right after execute_some_python_code().  That code could leak
arbitrary Python references!!!

> Oops, I just tried the code in CVS python, and the problem goes away.
> Same for 2.3a2.

I vaguely recall that someone fixed some things in this area... :-(

> But my code has to run in 2.2.2 as well...

If the docs are lying, they have to be fixed.  This is no longer my
prime area of expertise... :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)