PyEval_CallObject and threads

[Stefan Migowsky]

> [Phlip]
>
> Tim Peters wrote:
> 
> > [Phlip]
> >> My colleague came over and asked me to write this:
> >>
> >> "We got multiple C++ pthreads calling PyEval_CallObject. 
> How threadsafe
> >> is the Python API? A C++ module using Boost BPL creates multiple
threads,
> >> which then call back into Python. When two threads simultaneously call
> >> PyEval_CallObject we go boom."
> >>
> >> I will now go examine his serialization primitives. But if anyone has
any
> >> better ideas...
> > 
> > With very few exceptions, all calls to Python C API functions must be
made
> > while holding the global interpreter lock.  The exceptions have mostly
to
> > do with a handful of functions related to initialization and shutdown.
See
> > the section "Thread State and the Global Interpreter Lock" in the
Python/C
> > API Reference Manual for details.
> 
> Further inspection reveals our architect(s) think we need to call Python 
> script asynchronously from multiple threads. By 'lock' we assume both you 
> and the section you cite mean 'block'. Isn't this what Stackless Python is

> for?

We had a similar problem having both : threads started with Python commands
and threads started and controlled by the C-runtime. Since we couldn't be
sure if a threadstate is always set( a NULL Threadstate would end up in a 
PyFatalError if calling PyEvalSaveThread) we emulated the creation of a
Python
thread in the C-Runtime thread. For this you have to store the
Interpreterstate
and the use some code like this: 

// This code is running is a thread spawned by the C-program.
....
if(NULL != g_ppyinstInterp)  // Check global interpreterstate
{
    return;
}
if(NULL != g_ppyoPyThreadFunc ) // Check the function to be called
{
    PyThreadState *ppythstState  = NULL;
    PyObject      *ppyobRes      = NULL;
    PyObject      *ppyobArgs     = NULL;
    PyObject      *ppyobFirstArg = NULL;

    // 1. Set up the args
    ppyobFirstArg = GetFirstArg();
    ppyobArgs     = PyTuple_New(1);
    if(NULL == ppyobArgs )
    {
        Py_XDECREF(ppyobFirstArg);
        return;
    }
    PyTuple_SetItem(ppyobArgs, 0, ppyobFirstArg);
    // 2. Create a new threadstate for the current thread. This thread 
    // state is needed to insert the current thread into the Python 
    // interpreter.
    ppythstState  = PyThreadState_New(g_ppyinstInterp);
 
    // 3. Acquire Thread
    PyEval_AcquireThread(ppythstState);

    // 4. Call the function
    ppyobRes = PyEval_CallObjectWithKeywords(g_ppyoPyThreadFunc, ppyobArgs,
NULL);

    Py_DECREF(ppyobArgs);

    // 5. check result of callback function
    if (res == NULL) 
    {
        if (PyErr_ExceptionMatches(PyExc_SystemExit))
        {
	     PyErr_Clear();
        }
    	  else 
        {
	    	fprintf(stderr, "Unhandled exception in thread:\n");
	      PyErr_PrintEx(0);
	  }
    }
    else
    {
        Py_DECREF(res);
    }

    // 6. Cleanup
    PyThreadState_Clear(ppythstState);
    PyEval_ReleaseThread(ppythstState);
    PyThreadState_Delete(ppythstState);
}

   Stefan