Extension libraries, spawned threads, and the global interpreter lock

[Damien Elmes]

Thanks for your feedback, Martin.

More below.

martin at v.loewis.de (Martin v. Loewis) writes:

> Damien Elmes <resolve at repose.REMOVE.cx> writes:
> 
> > So my question is: am I correct in thinking it's a problem with the GIL not
> > being held in the other threads?
> 
> Probably not. Do the other threads ever call back into Python code, or
> invoke Python interpreter API? If so, they need to hold the lock while
> doing so. If they are completely separate, they are not concerned with
> the GIL.

They're entirely seperate, and thus, I thought, safe from the GIL. This problem
appears to crop up when multiple threads receive SIGINT at once.

> > If so, would explicitly grabbing and freeing the lock in the xlib
> > threads solve the problem?
> 
> At what time would you free it? As long as one of the threads holds
> the lock, your main thread will not advance.

Thought of that while lying in bed last night :-) Silly of me to mention it in
the first place.

> More likely, you have a problem with signal handlers. Python installs
> a SIGINT handler, to generate the KeyboardInterrupt. This can break
> badly if the signal handler is not delivered to a Python thread, or
> more specifically, to the main Python thread.
> 
> You have a number of choices: 
> 
> - don't install the SIGINT handler, but live with the default SIGINT
>   processing (i.e. process termination)

The problem is that the interactive interpreter installs a sigint handler,
which is sort of rude for a library to remove when it's instantianted. I think
the 'default' behavior for interactive use is installed once per statement
anyway, so if I were to try modify the behavior, it wouldn't change this
problem.

> - block SIGINT in the other threads. How to do this much depends on
>   your operating system.

Maybe this is worth looking into. So far I've been able to determine:

* when this class calls the backend stuff, two threads are forked, to a total
of three "processes" in a ps listing.

* sending an INT signal to any one of them yields the correct behavior.

* sending an INT signal to the main program thread, and any of the other two
(but only one of the other two) yields the correct behavior

* extmodulethread1 + extmodulethread2 yields the crash.

This problem is compounded by the behaviour of the interactive interpreter - a
C-c seemingly generates an interrupt signal to each of the threads instead of
just the main one.

This problem also doesn't seem to crop up when catching a keyboard error in my
own code, which leads me to believe the erroneous interaction is with exception
stuff - but as I understand it, that's all been thread-safe for a long time.

This behaviour is really perplexing me :-)

-- 
Damien Elmes
resolve at repose.cx