[Python-Dev] Threading and callbacks - missing documentation

[Martin v. Loewis]

Jack Jansen <Jack.Jansen@oratrix.com> writes:

> As I'm currently very interested in this for two separate reasons (one
> work-related, and another one is that I want to make the Mac toolbox
> modules release the GIL when appropriate) I could write something up,
> but only if (a) it isn't documented somewhere already and (b) someone
> can tell me how it actually works:-).

The issue in a callback is that you need to obtain a thread state
before getting the interpreter lock back. There are two ways to do
this:

1. the _tkinter approach: You can guarantee that the call-back occurs
   in the context of the same thread that originally released the GIL.
   In that case, save the thread state in a global variable, and
   restore it inside the callback (see ENTER_TCL/ENTER_PYTHON).

2. the "free threading" approach: You assume that callback may occur
   in the context of an arbitrary thread, even in a thread that was
   not originally created by Python. In that case, inside the
   call-back, use PyThreadState_New to create a fresh thread state,
   then obtain the GIL for this tread state. When the callback is
   done, delete the thread state. You need to preserve a pointer to
   the interpreter somehow in a global variable (or provide other
   means to locate the interpreter).

The advantage of 1 is that tracebacks go right back into the main
module, even from the callback; in approach 2, tracebacks go only back
to the entry in the callback (the callback appears to come out of
nowhere). The advantage of 2 is that it is more general.

HTH,
Martin