[Python-Dev] Extension modules, Threading, and the GIL

["Martin v. Löwis"]

Mark Hammond wrote:
> While I concede that this is likely to work in the general case, I am not
> sure it is "correct".  If no threading semantics will be broken by having
> one thread use multiple thread-states, then I must ask what purpose
> thread-states (as opposed to the GIL) have.

That is easy to answer (even though it is out of my area of expertise): 
it carries the Python stack, in particular for exceptions.

Now, if you have multiple thread states in a single thread, the question 
is how a Python exception should propagate through the C stack. With 
multiple thread states, the exception "drops off" in the callback, which 
usually has no meaningful way to deal with it except to print it (in my 
application, the callback was always CORBA-initiated, so it was 
straight-forward to propagate it across the wire to the remote caller).

The only meaningful alternative would be to assume that there is a 
single thread state. In that case, the exception would be stored in the 
thread state, and come out in the original caller. Now, it is very 
questionable that you could unwind the C stack between the the entrance 
to the library and the callback: If, as David says, you don't even know 
that the API may invoke a callback, there is surely no way to indicate 
that an exception came out of it.

As a result, when returning to the bottom of the C stack, the extension 
suddenly finds an extension in its thread state. The extension probably 
doesn't expect that exception, so it is simply lost (when the next 
exception is set). Potentially, strange things happen as somebody might 
invoke PyErr_Occurred().

I question whether this is better than printing the exception, in the 
case of multiple thread states.

Regards,
Martin