Mark Hammond wrote:
While I concede that this is likely to work in the general case, I am not sure it is "correct". If no threading semantics will be broken by having one thread use multiple thread-states, then I must ask what purpose thread-states (as opposed to the GIL) have.
That is easy to answer (even though it is out of my area of expertise): it carries the Python stack, in particular for exceptions. Now, if you have multiple thread states in a single thread, the question is how a Python exception should propagate through the C stack. With multiple thread states, the exception "drops off" in the callback, which usually has no meaningful way to deal with it except to print it (in my application, the callback was always CORBA-initiated, so it was straight-forward to propagate it across the wire to the remote caller). The only meaningful alternative would be to assume that there is a single thread state. In that case, the exception would be stored in the thread state, and come out in the original caller. Now, it is very questionable that you could unwind the C stack between the the entrance to the library and the callback: If, as David says, you don't even know that the API may invoke a callback, there is surely no way to indicate that an exception came out of it. As a result, when returning to the bottom of the C stack, the extension suddenly finds an extension in its thread state. The extension probably doesn't expect that exception, so it is simply lost (when the next exception is set). Potentially, strange things happen as somebody might invoke PyErr_Occurred(). I question whether this is better than printing the exception, in the case of multiple thread states. Regards, Martin