Hi all,
I am a bit confused about future proof GIL release with respect to potential subinterpreter development. Or maybe just the best practices.
In short, is the PyGILState_Ensure() function safe? I had the
impression that was definitely not, but now makes me wondered how even
PyErr_Occurred() can be safe when
EXPERIMENTAL_ISOLATED_SUBINTERPRETERS. Of course PyErr_Occurred()
is called with the GIL, so that may make a big difference.
I scanned through this thread: https://bugs.python.org/issue15751 which
suggested the addition of API to make it (almost?) always safe? But
most of that thread is very old, much older than the newer
EXPERIMENTAL_ISOLATED_SUBINTERPRETERS work.
Reading https://bugs.python.org/issue4202 makes me think there is no
current solution/safety?
In my case (NumPy), I may be able to assume that the
PyGILState_Ensure() calls I am talking about will always happen in
the same thread as the initial GIL release, which gives me some hope
that it might actually just work? [1]
My impression until now was that the only clean solution would be to pass the threadstate. But, there is some public API, where we would require new API to pass the threadstate (which means slow adoption at the very least). So if I can avoid it, that would be much easier.
The consumer (the person who needs to grab the GIL) can be different
from the user (the person releasing the GIL). So I am considering
passing PyThreadState ** with NULL indicating "unknown" state to the
consumer.
Much code may not even release the GIL, and in either case a move to a
new API has to take at least 1-2 years, so I can't really wait for that
to be done.
Cheers,
Sebastian
[1] To be honest, I am not quite sure on this. It is true for NumPy itself, but I am not sure it is true for potential consumers of NumPy API. Or maybe, it would just be something to document very clearly somewhere.