[Python-Dev] Extension modules, Threading, and the GIL

[Mark Hammond]

[Martin]
> Mark Hammond wrote:
> > While I concede that this is likely to work in the general
> > case, I am not sure it is "correct".  If no threading semantics
> > will be broken by having one thread use multiple thread-states,
> > then I must ask what purpose thread-states (as opposed to the GIL) have.
>
> That is easy to answer (even though it is out of my area of
> expertise):
> it carries the Python stack, in particular for exceptions.

It also carries the profiler and debugger hooks, a general purpose "thread
state" dictionary and other misc. details such as the tick count, recursion
depth protection etc.

> Now, if you have multiple thread states in a single thread,
> the question
> is how a Python exception should propagate through the C stack.

Actually, I think the question is still "why would a single thread have
multiple thread-states?". (Or maybe "should a thread-state be renamed to,
say, an "invocation frame"?)

> With multiple thread states, the exception "drops off" in the
> callback, which usually

"usually" is the key word here.  Python isn't designed only to handle what
programs "usually" do.  A strategy I have seen recently here, which is to
argue that any other requirements are self-evidently broken, is not helpful.

We could possibly argue that exceptions are OK to handle this way.  Similar
amounts of text could also possibly convince that the profiler, debugger and
thread-switch items also will not be too badly broken by having multiple
thread states per thread, or that such breakage is "desirable" (ie, can be
justified).  You will have more trouble convincing me that future items
stored in a Python thread state will not be broken, but I am past arguing
about it.

Please correct me if I am wrong, but it seems your solution to this is:

* Every function which *may* trigger such callbacks *must* switch out the
current thread state (thereby dropping the GIL etc)
* Every entry-point which needs to call Python must *always* allocate and
switch to a new thread-state.
* Anything broken by having multiple thread-states per thread be either (a)
fixed, or (b) justified in terms of a specific CORBA binding implementation.
* Anyone wanting anything more remains out on their own, just as now.

If so, I am afraid I was hoping for just a little more <wink>.

Mark.