Multithreaded C API Python questions

robert no-spam at no-spam-no-spam.invalid
Thu Nov 16 16:05:55 CET 2006

Svein Seldal wrote:

You seem to use the functions in very rude manner - wanting to force the GIL around at lowest level. 
Better forget the word GIL and think about acquiring and releasing Threads. 
For each thread wanting to execute Python stuff there has to be a thread state (ts). And then you have to enter/leave for executing Python by using the thread state.

PyEval_InitThreads  enters Python (incl. the GIL) first time.
To let Python free in main thread do 
ts=PyEval_SaveThread()    // or more easy: Py_BEGIN_ALLOW_THREADS 

to reenter Python do simply: 

PyEval_RestoreThread(ts)  // or: Py_END_ALLOW_THREADS 

Forget all the low level PyGIL... functions.

when you really want another c thread to enter Python first time with no thread state existing through Python itsel (better make the thread in Python with thread.start_new ?), then you have to do once

ts_cthread = PyThreadState_New(interp)

then enter Python in this thread:

PyEval_AcquireThread( ts_cthread ) 

to leave again:

PyEval_ReleaseThread( ts_cthread ) 

If you just loop the call_py_function_send() mainly in this thread  you  probably better create the thread at all in Python and make the loop there. You probably stick too much to C-level at any price :-)   Probably you just do a PyRun_xxxx in main thread and then everything else in Python, and expose C-parts for the thread-loop to Python as function (in other_py_inits) - where in the c-function you probably have the usual Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS  bracket during time consuming/system/reenter-endangered message stuff.


> I think I've found the bug, but I need to confirm this behavior.
> My findings is that if you use PyEval_InitThreads(), it is crucial to 
> release the GIL with PyEval_ReleaseThread() afterwards.
> The API docs states that you can release the GIL with 
> PyEval_ReleaseLock() or PyEval_ReleaseThread() AFAICS. 
> under docs of void 
> PyEval_InitThreads().
> However, if I do use PyEval_ReleaseLock() it will crash shortly after 
> with "Fatal Python error: ceval: tstate mix-up" in a multithreaded C 
> environment.
> If I use PyEval_ReleaseThread() to release the GIL, my app seems stable. 
> I release the GIL like this:
>     PyThreadState *pts = PyGILState_GetThisThreadState();
>     PyEval_ReleaseThread(pts);
> Now, is this a feature or expected behavior in python (i.e. unclear API 
> documentation), or is this a bug in python itself?
> Regards,
> Svein
> PS:
> For reference I did something like this in pseudo-code:
>     Py_Initialize();
>     PyEval_InitThreads();
>     other_py_inits();   // Load py modules, etc.
>     PyEval_ReleaseLock();    // <-- MAKES THE APP UNSTABLE
>     create_c_thread();
>     PyGILState_STATE gstate;
>     gstate = PyGILState_Ensure();
>     call_py_function_main();  // Py main() wont return
>     PyGILState_Release(gstate);
> And the "main" of the C thread function looks like this:
>    while(1)
>    {
>         PyGILState_STATE gstate;
>         gstate = PyGILState_Ensure();
>         call_py_function_send();
>         PyGILState_Release(gstate);
>    }

More information about the Python-list mailing list