[Python-Dev] GIL vs thread state

[Thomas Heller]

Guido van Rossum <guido@python.org> writes:

> > The docs for PyThreadState_Clear() state that the interpreter lock must
> > be held.
> > 
> > I had this code in ctypes to delete the thread state and release the lock:
> > 
> > static void LeavePython(char *msg)
> > {
> > 	PyThreadState *pts = PyThreadState_Swap(NULL);
> > 	if (!pts)
> > 		Py_FatalError("wincall (LeavePython): ThreadState is NULL?");
> > 	PyThreadState_Clear(pts);
> > 	PyThreadState_Delete(pts);
> > 	PyEval_ReleaseLock();
> > }
> > 
> > and (under certain coditions, when ptr->frame was not NULL), got
> 
> What is ptr->frame?  A typo for pts->frame?

Right, sorry.

> 
> If pts->frame is not NULL, I'd expect a warning from
> PyThreadState_Clear(): "PyThreadState_Clear: warning: thread still has
> a frame\n".
You mean this code, from Python/pystate.h?

void
PyThreadState_Clear(PyThreadState *tstate)
{
	if (Py_VerboseFlag && tstate->frame != NULL)
		fprintf(stderr,
		  "PyThreadState_Clear: warning: thread still has a frame\n");

	ZAP(tstate->frame);

	ZAP(tstate->dict);
...
}

Py_VerboseFlag is 0 set in my case, so no warning is printed.

> 
> > "Fatal Python error: PyThreadState_Get: no current thread" in the call
> > to PyThreadState_Clear().
> 
> That's strange, because I cannot trace the code in there to such a
> call.  (Unless it is in a destructor.

It is in a destructor: frame_dealloc, called from ZAP(tstate->frame).

>   Can you tell more about where
> the PyThreadState_Get() call was?)

This function allocates the threadstate for me:

static void EnterPython(char *msg)
{
	PyThreadState *pts;
	PyEval_AcquireLock();
	pts = PyThreadState_New(g_interp);
	if (!pts)
		Py_FatalError("wincall: Could not allocate ThreadState");
	if (NULL != PyThreadState_Swap(pts))
		Py_FatalError("wincall (EnterPython): thread state not == NULL?");
}


To explain the picture a little better, here is the sequence of calls:

Python calls into a C extension.
The C extension does

Py_BEGIN_ALLOW_THREADS
  call_a_C_function()
Py_END_ALLOW_THREADS

The call_a_C_function calls back into C code like this:

void MyCallback(void)
{
    EnterPython();  /* acquire the lock, and create a thread state */
    execute_some_python_code();
    LeavePython(); /* destroy the thread state, and release the lock */
}

Now, the execute_some_python_code() section is enclosed in a win32
structured exception handling block, and it may return still with a
frame in the threadstate, as it seems.

Oops, I just tried the code in CVS python, and the problem goes away.
Same for 2.3a2.
But my code has to run in 2.2.2 as well...

Thomas

Here's the stack from python 2.2.2:

NTDLL! 77f6f570()
PyThreadState_Get() line 246 + 10 bytes
PyErr_Fetch(_object * * 0x0012f944, _object * * 0x0012f954, _object * * 0x0012f948) line 215 + 5 bytes
call_finalizer(_object * 0x0095ef20) line 382 + 17 bytes
subtype_dealloc(_object * 0x0095ef20) line 434 + 9 bytes
_Py_Dealloc(_object * 0x0095ef20) line 1837 + 7 bytes
frame_dealloc(_frame * 0x00890c20) line 82 + 79 bytes
_Py_Dealloc(_object * 0x00890c20) line 1837 + 7 bytes
PyThreadState_Clear(_ts * 0x0095d2a0) line 174 + 86 bytes
LeavePython(char * 0x1001125c) line 41 + 10 bytes