Multiple interpreters not compatible with current thread module

The current threadmodule.c does not seem to correctly support multiple (sub) interpreters. This became apparent when using jep - (a Java/Python bridge) and also seems to cause problems with mod_python. The incompatibility began in Python version 2.3.5 and has been traced to changes to the 2.4 threadmodule.c that were backported to the 2.3 delivery. A bug report was raised on 2005-03-15 (http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470) which covers the problem in more detail. I've just submitted a patch (I hope it's correctly formatted) for threadmodule.c (http://sourceforge.net/tracker/index.php?func=detail&aid=1203393&group_id=5470&atid=305470) adapted from the pre-2.3.5 threadmodule.c (replacing the PyGILState_XXX calls with those from the earlier thread module). The patch works correctly but it will probably re-introduce the problem that the change for threadmodule.c version 2.59 fixed.("Fix for [ 1010677 ] thread Module Breaks PyGILState_Ensure(),and a test case.When booting a new thread, use the PyGILState API to manage the GIL."). The documentation (http://docs.python.org/api/threads.html) states "Note that the PyGILState_*() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python still supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_*() API is unsupported. ", so it looks like that using the PyGilState_XXX functions in the core threadmodule.c means the Py_NewInterpreter() call (i.e. multiple interpreters) is no longer supported when threads are involved. This problem is preventing us upgrading to Python 2.4 so we'd obviously like to see a resolution the next Python release if that were possible... Cheers, Max

Jeremy Maxfield <anothermax@gmail.com> writes:
The current threadmodule.c does not seem to correctly support multiple (sub) interpreters.
This would seem to be an accurate statement. A short history: The GILState functions were implemented. The way they work is that when you call PyGILState_Ensure, (essentially) a thread local variable is checked to see if a thread state is known for this thread. If one is found, fine, it is used. If not, one is created (using the interpreter state what was passed to _PyGILState_Init() by Py_Initialize()) and stored in the thread local variable. This had a (pretty obvious in hindsight) problem (bug #1010677): when you create a thread via thread.start_new_thread a thread state is created, but not stored in the thread local variable consulted by PyGILState_Ensure. So if you call PyGILState_Ensure another thread state is created for the thread (generally a no-no) and if you already hold the GIL PyGILState_Ensure will attempt to acquire it again -- a deadlock. This was fixed by essentially using PyGILState_Ensure to create the threadstate. This has a (pretty obvious in hindsight) problem (bug #1163563): PyGILState_Ensure always uses the interpreter state created by Py_Initialize, ignoring the interpreter state carefully supplied to t_bootstrap. Hilarity ensues. So, what's the fix? Well, the original problem was only the lack of association between a C level thread and a thread state. This can be fixed by setting up this association in t_bootstrap (and I've posted a patch that does this to the report of #1163563). This suffices for all known problems, but it's a bit hackish. Another approach is to set up this association PyThreadState_New(), which is possibly a bit more elegant, but carries a risk: is PyThreadState_New always called from the new thread? ISTM that it is, but I'm not sure. I'm not expecting anyone else to think hard about this on recent form, so I'll think about it for a bit and then fix it in the way that seems best after that. Feel free to surprise me. Cheers, mwh -- I would hereby duly point you at the website for the current pedal powered submarine world underwater speed record, except I've lost the URL. -- Callas, cam.misc

Michael Hudson <mwh@python.net> writes:
And so that's what I did. I think I got this right, but threads are tricky buggers so x-platform testing of CVS HEAD would be appreciated (heck, that would be appreciated anyway, I don't get the impression much of it is happening at the moment...). Cheers, mwh -- <moshez> glyph: I don't know anything about reality. -- from Twisted.Quotes

Jeremy Maxfield <anothermax@gmail.com> writes:
The current threadmodule.c does not seem to correctly support multiple (sub) interpreters.
This would seem to be an accurate statement. A short history: The GILState functions were implemented. The way they work is that when you call PyGILState_Ensure, (essentially) a thread local variable is checked to see if a thread state is known for this thread. If one is found, fine, it is used. If not, one is created (using the interpreter state what was passed to _PyGILState_Init() by Py_Initialize()) and stored in the thread local variable. This had a (pretty obvious in hindsight) problem (bug #1010677): when you create a thread via thread.start_new_thread a thread state is created, but not stored in the thread local variable consulted by PyGILState_Ensure. So if you call PyGILState_Ensure another thread state is created for the thread (generally a no-no) and if you already hold the GIL PyGILState_Ensure will attempt to acquire it again -- a deadlock. This was fixed by essentially using PyGILState_Ensure to create the threadstate. This has a (pretty obvious in hindsight) problem (bug #1163563): PyGILState_Ensure always uses the interpreter state created by Py_Initialize, ignoring the interpreter state carefully supplied to t_bootstrap. Hilarity ensues. So, what's the fix? Well, the original problem was only the lack of association between a C level thread and a thread state. This can be fixed by setting up this association in t_bootstrap (and I've posted a patch that does this to the report of #1163563). This suffices for all known problems, but it's a bit hackish. Another approach is to set up this association PyThreadState_New(), which is possibly a bit more elegant, but carries a risk: is PyThreadState_New always called from the new thread? ISTM that it is, but I'm not sure. I'm not expecting anyone else to think hard about this on recent form, so I'll think about it for a bit and then fix it in the way that seems best after that. Feel free to surprise me. Cheers, mwh -- I would hereby duly point you at the website for the current pedal powered submarine world underwater speed record, except I've lost the URL. -- Callas, cam.misc

Michael Hudson <mwh@python.net> writes:
And so that's what I did. I think I got this right, but threads are tricky buggers so x-platform testing of CVS HEAD would be appreciated (heck, that would be appreciated anyway, I don't get the impression much of it is happening at the moment...). Cheers, mwh -- <moshez> glyph: I don't know anything about reality. -- from Twisted.Quotes
participants (2)
-
Jeremy Maxfield
-
Michael Hudson