[Python-Dev] orphan tstate problem

Berserker berserker_r at hotmail.com
Wed Dec 2 11:45:22 CET 2009

Hi, I'm developing a software which embeds Python as scripting language.
The software acts as webserver and for each request it peeks a thread 
from a pool which in turn loads a python script to generate the response 
(something like mod_python).
Since each http request is "independent" from others one, I need to 
create a new interpreter with its own dictionary every time.
I'm exposing lots of C++ wrapped code and, in order to provide a sort of 
"parallelism" between each request, I'm using this mechanism for each 
registered callback:

P = python core
A = my application

P: invokes the registered C callback
A: PyEval_SaveThread (allows others threads to run/resume their scripts)
A: invoke C/C++ application's functions (which don't touch CPython API)
A: PyEval_RestoreThread (takes back the lock)
P: resume the execution of the script

(the above schema is the same of 
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS documented in ceval.h at 
line 83)

The problem I have encountered is that when I process two requests 
simultaneously, Python reports the fatal error "ceval: orphan tstate".
I'm not an expert of the CPython internals API but I take a look at the 
file ceval.c (actually I'm using python 2x but in the 3x version I 
noticed it's the same) and the code involved is:

/* Give another thread a chance */

if (PyThreadState_Swap(NULL) != tstate)
    Py_FatalError("ceval: tstate mix-up");

/* Other threads may run now */

PyThread_acquire_lock(interpreter_lock, 1);
if (PyThreadState_Swap(tstate) != NULL)
    Py_FatalError("ceval: orphan tstate");

Can anyone explain me the meaning of those fatal errors (in particular 
the "orphan tstate" one)? Why the return value should be null?
As far as I understand after the "PyThread_release_lock" others threads 
are allowed to run and, if you take a look again at my above schema, 
PyThreadState_Swap is supposed to be called between 
PyThread_release_lock/PyThread_acquire_lock, exactly where the comment 
"Other threads may run now" is placed. If i changed the code in order to 
not fire a fatal error after the second PyThreadState_Swap everything
seems to work fine, but I'm afraid it's not the "proper" solution and I 
need to understand the meaning of that fatal error.

Sorry for the long post and for my bad english, I hope that someone 
could really help me.


More information about the Python-Dev mailing list