problem to write a THREAD enabled python extension

[Andreas Otto]

Hi,

  the following scenario:
   
 1. using GIL
 2. a pthread is created in a library and have to be "announced" to python
 3. the thread is created in a "PyObject_CallObject" function call from
      extension c code using a other extension c-code function called
      from python code

 python CRASH with invalid thread-state object
  http://www.linuxjournal.com/article/3641
 has information that GIL is NOT able to keep thread state information
 for cross thread operation.

workaround ->
......
PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
// Call Python/C API...
PyGILState_Release(gilState); // PyEval_SaveThread
......
// Create new thread...
......
PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
// Call Python/C API...
PyGILState_Release(gilState); // PyEval_SaveThread
......

the problem is that PyGILState_Release have to be done in the call of
"PyObject_CallObject" and not in the extension c-code

-> for now i call it impossible to provide thread support if thread is 
created in extension code

-> more info by valgrind
==29105==
==29105== Invalid read of size 4
==29105==    at 0x4F099AB: PyEval_EvalFrameEx (ceval.c:3728)
==29105==    by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
==29105==    by 0x4E9DE50: function_call (funcobject.c:630)
==29105==    by 0x4E7610C: PyObject_Call (abstract.c:2160)
==29105==    by 0x4E8BEFB: method_call (classobject.c:323)
==29105==    by 0x4E7610C: PyObject_Call (abstract.c:2160)
==29105==    by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
==29105==    by 0x72132DF: pymsgque_ProcCall (misc_python.c:48)
==29105==    by 0x742A23E: pTokenInvoke (token.c:327)
==29105==    by 0x742E7E2: sMqEventStart (msgque.c:908)
==29105==    by 0x7425365: pEventStart (event.c:360)
==29105==    by 0x742ECBF: MqProcessEvent (msgque.c:1081)
==29105==    by 0x7215F24: pymsgque_ProcessEvent (context_python.c:176)
==29105==    by 0x4F09BFA: PyEval_EvalFrameEx (ceval.c:3961)
==29105==    by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
==29105==    by 0x4F0B58A: PyEval_EvalCode (ceval.c:650)
==29105==    by 0x4F2EE29: PyRun_FileExFlags (pythonrun.c:1697)
==29105==    by 0x4F2F134: PyRun_SimpleFileExFlags (pythonrun.c:1182)
==29105==    by 0x4F43CFB: Py_Main (main.c:625)
==29105==    by 0x400D11: main (python.c:152)
==29105==  Address 0x24 is not stack'd, malloc'd or (recently) free'd


-> extension code ... (including some debugging output)

M0
  PyGILState_STATE gstate = PyGILState_Ensure();
  PyObject *result;
  PyObject * const self = (PyObject*) CONTEXT->self;
  PyObject * const callable = (PyObject*) dataP;
  enum MqErrorE ret = MQ_OK;

  // clean Python and libmsgque error
  PyErr_Clear();
  MqErrorReset(msgque->error);

  // call the function
M1
printP(PyThreadState_Get())
  if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
    //PyGILState_Release(gstate);
    result = PyObject_CallObject(callable, NULL);
    //gstate = PyGILState_Ensure();
  } else {
    //PyGILState_Release(gstate);
    result = PyObject_CallFunctionObjArgs(callable, self, NULL);
    //gstate = PyGILState_Ensure();
  }
M2
printP(PyThreadState_Get())
  Py_XDECREF(result);

  // no error return OK
  if (PyErr_Occurred() != NULL) {
    NS(ErrorSet) (self);
    ret = MqErrorGetCode(msgque->error);
  }
  PyGILState_Release(gstate);
  return ret;
}

with GIL wrapper 'PyGILState_Release' and 'PyGILState_Ensure' enabled for 
'PyObject_CallObject' the following SEG happen

  if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
    PyGILState_Release(gstate);
    result = PyObject_CallObject(callable, NULL);
    gstate = PyGILState_Ensure();
  } else {
    PyGILState_Release(gstate);
    result = PyObject_CallFunctionObjArgs(callable, self, NULL);
    gstate = PyGILState_Ensure();
  }
pymsgque_ProcCall(misc_python.c:32) -> 00000000000000000
pymsgque_ProcCall(misc_python.c:44) -> 11111111111111111
pymsgque_ProcCall(misc_python.c:45) -> PyThreadState_Get()<0x68c2ad8>
==29223==
==29223== Thread 2:
==29223== Invalid read of size 4
==29223==    at 0x4E760EA: PyObject_Call (abstract.c:2158)
==29223==    by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
==29223==    by 0x72132F7: pymsgque_ProcCall (misc_python.c:48)
==29223==    by 0x7430092: MqLinkCreate (msgque.c:644)
==29223==    by 0x7430AB0: MqDefaultLinkCreate (msgque.c:681)
==29223==    by 0x742F73B: MqLinkCreate (msgque.c:346)
==29223==    by 0x7432C3D: sSysServerThreadCreate (sys.c:502)
==29223==    by 0x521606F: start_thread (in /lib64/libpthread-2.9.so)
==29223==    by 0x5B5B10C: clone (in /lib64/libc-2.9.so)
==29223==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
==29223==
==29223== Process terminating with default action of signal 11 (SIGSEGV)


mfg

  aotto1968