[Python-Dev] Multiple interpreters not compatible with current thread module

[Michael Hudson]

Jeremy Maxfield <anothermax at gmail.com> writes:

> The current threadmodule.c does not seem to correctly support multiple
> (sub) interpreters.

This would seem to be an accurate statement.

A short history:

The GILState functions were implemented.

The way they work is that when you call PyGILState_Ensure,
(essentially) a thread local variable is checked to see if a thread
state is known for this thread.  If one is found, fine, it is used.
If not, one is created (using the interpreter state what was passed to
_PyGILState_Init() by Py_Initialize()) and stored in the thread local
variable.

This had a (pretty obvious in hindsight) problem (bug #1010677): when
you create a thread via thread.start_new_thread a thread state is
created, but not stored in the thread local variable consulted by
PyGILState_Ensure.  So if you call PyGILState_Ensure another thread
state is created for the thread (generally a no-no) and if you already
hold the GIL PyGILState_Ensure will attempt to acquire it again -- a
deadlock.

This was fixed by essentially using PyGILState_Ensure to create the
threadstate.  This has a (pretty obvious in hindsight) problem (bug
#1163563): PyGILState_Ensure always uses the interpreter state created
by Py_Initialize, ignoring the interpreter state carefully supplied to
t_bootstrap.  Hilarity ensues.

So, what's the fix?  Well, the original problem was only the lack of
association between a C level thread and a thread state.  This can be
fixed by setting up this association in t_bootstrap (and I've posted a
patch that does this to the report of #1163563).  This suffices for
all known problems, but it's a bit hackish.  Another approach is to
set up this association PyThreadState_New(), which is possibly a bit
more elegant, but carries a risk: is PyThreadState_New always called
from the new thread?  ISTM that it is, but I'm not sure.

I'm not expecting anyone else to think hard about this on recent form,
so I'll think about it for a bit and then fix it in the way that seems
best after that.  Feel free to surprise me.

Cheers,
mwh

-- 
  I would hereby duly point you at the website for the current pedal
  powered submarine world underwater speed record, except I've lost
  the URL.                                         -- Callas, cam.misc