[Python-Dev] Threading and callbacks - missing documentation

[Tim Peters]

[Jack Jansen]
> unless I'm looking in completely the wrong place (I looked in
> "extending and Embedding" and in "Python/C API") there is
> absolutely no information in the docs on the interaction between
> callbacks and the GIL.

Just the Prime Directive:  you must not call back into Python unless you
hold the GIL.  Terrible things will happen otherwise.

Actually getting the GIL can be a nightmare.  The worst example in the core
is posixmodule.c's _PyPclose, where a thread has to get the GIL, but has no
thread state to use, and has no interpreter state to use either (in order to
get a thread state).  It has to bootstrap all that stuff into existence,
then tear it down again when it's done.

There are two saving graces in this specific routine that allow the obscure
dance to work:  (1) it knows for sure that Python has already been
initialized; and, (2) it knows for sure that the thread calling it does not
hold the GIL at the time of the call.  If you can't count on both of those,
it can get much harder.  Anyway, the comments there will be a real help.

> There is preciously little information on callbacks and threading, and
> it isn't very helpful at that (i.e. there doesn't seem to be a recipy on
> how to obtain the GIL in a callback routine that is about to call Python
> code,

It would take a small book to cover all the cases that can arise.  You can
find tens of thousands of words about how to do this in the Thread-SIG
archives, some attempts at helper frameworks, and a few forgotten promises
to make it easier someday.  I believe Mark Hammond has a general set of C++
classes to help with this stuff on Windows, but IIRC they rely on
Windows-specific TLS (thread local storage) gimmicks.

> or how to test whether the current thread holds the GIL already).

Unfortunately, this isn't possible:  if you need to do this, you need to
build your own conventions for keeping track of who has the GIL.

[Thomas Heller, to Martin]
> Can't you at least partly combine the advantages of 1 and 2 by using
> thread local storage for the global variable? I have no idea if there
> is a portable way to do this...

There isn't a portable way.  The closest Python gets is the little-known
PyThreadState_GetDict(), which returns a dict object unique to the thread
calling it (so each thread can set up its own key->value mappings for a
common set of keys).  However, the calling thread must hold the GIL (or, as
always, terrible things can happen <wink>).  So it's useful only if you
already already know whether you have the GIL, and how to get the GIL if you
don't have it.