[Python-Dev] PEP 311 Simplified Global Interpreter Lock Acquisition for Extensions

[Harri Pasanen]

Hi,

Just prior to my vacation I ran into a bit of trouble with my Python 
embedding, as much to my dismay my C++ program that had been working 
fine under Linux deadlocked under Windows.

Apparently the underlying C threading API's were behaving in subtly 
different ways.   After boiling down from my original embedded Python 
app which features omniORB, omniORBpy, and bunch of other goodies, I 
got down to the following testcase:

#include <Python.h>

class PythonLocker {
public:
  PythonLocker ()  {   _tstate = PyGILState_Ensure ();  }

  ~PythonLocker() {   PyGILState_Release(_tstate); }
  }
private:
  PyGILState_STATE _tstate;
};

int
main (int argc, const char **argv)
{
  Py_Initialize();
  PyEval_InitThreads();
  PyEval_ReleaseLock ();  // This is the problem
  {
    PythonLocker pl;
    PyObject* usermod = PyImport_ImportModule ("echo");  
   // The previous line deadlocks on win32
  }
  return 0;
}

The above works fine on Linux, but deadlocks on Win32.

Tracing the execution on Win32 showed that it hanged in 
thread_nt.h:
DWORD EnterNonRecursiveMutex(PNRMUTEX mutex, BOOL wait)
{
 ...
		ret = InterlockedIncrement(&mutex->owned) ?
			/* Some thread owns the mutex, let's wait... */
			WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ;

The gotcha being that when entering above, mutex->owned was == -2, so 
it incorrectly entered the wait.

I had a private discussion with Mark about this, and he quickly 
pointed out that I should not call PyEval_ReleaseLock().  Although 
this is not exactly clear from the PEP, which says 

"Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and 
Py_END_ALLOW_THREADS macros, it is assumed that no additional thread 
state API functions will be used by the extension."

I did not think PyEval_ReleaseLock() would qualify for the above, but 
somehow it does, under win32 at least.

Now if I modify thread_nt.h, simple mindedly by changing the above to:

		ret = InterlockedIncrement(&mutex->owned) > 0 ?
			/* Some thread owns the mutex, let's wait... */
			WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ;

everything works for me on Win32 as well, including python regression 
tests.  But I haven't wrestled with the code in question in detail to 
know what gremlins that change might unleash elsewhere.

Without modifying Python, I can make my testcase working with the 
following change to my test program:

{
  Py_Initialize();
  PyEval_InitThreads();
  { PythonLocker init_pl };   // release the lock
  // PyEval_ReleaseLock (); 
  { // this block could potentially be called from another thread
    PythonLocker pl;
    PyObject* usermod = PyImport_ImportModule ("echo");  
   // The previous line deadlocks on win32
  }

In the C API world this is a bit more convoluted looking, as I need to 
call PyGILState_Ensure(), and PyGILState_Restore, even if after 
PyEval_InitThreads() I already know I have the lock.

***

Sorry for rambling, but this is in the hope that someone else can 
avoid banging their heads against this as I have.   And it is a bit 
unclear to me also if this kind of a platform difference between 
Linux and Win32 is a bug, or a feature.  Ideally the Python C API 
should be bug-compatible.   So possibly wrong usage should cause all 
platforms to fail.

Btw., the new PyGILState_* functions do not appear to be part of the 
Python/C API documentation, there is only the PEP.

Anyway, thanks to Mark for this definite improvement on the state of 
things.

Harri