On Tue, 2020-08-11 at 16:05 +0200, Victor Stinner wrote:
Hi,
I'm not sure if it's going to help you, but here are my notes about the "Python Thread State": https://pythondev.readthedocs.io/pystate.html
Thanks, can't say I can make much of it after looking at it for a while. It currently seems to me that using PyGILState_Ensure is likely fine on the same thread which released the GIL (or can be made to work in Python). But it also seems to that passing in the threadstate is probably just cleaner/nicer when possible.
Maybe I need to give some pseudo code. The 3rd party function is a ufunc loop, slightly simplified (anyone can register it):
cdef int ufunc_loop(char **data, ssize_t n, ...): for i in range(n): # Do things with data if something_bad: with gil: PyErr_SetString(...) return -1 return 0
Now, I need to decide the signature for that function (it has to be updated), there is a bunch of potentially useful metadata, I was considering the pointer to a struct, at least for most of it:
typedef struct metadata { PyArray_Descr *dtypes, PyThreadState *tstate, void *user_static_data, void *user_data, /* ... */ }
which can include the tstate
(whether or not its a struct hardly
matters here).
Unfortunately, there is at least one place where I cannot guarantee I know the threadstate. Thus we may need to pass NULL
to indicate
an unknown state.
The GIL may or may not be released, but I guess I could pass the
threadstate even when the GIL is held. I was toying with the thought
of passing PyThreadState **
(mainly because it would make updating it
easier).
Antoine mentioned passing PyInterpreterState *
, so I am unsure if
there would be a reason to prefer that?
If there is no clear right thing, but passing something seems
preferred, I can also just pass: void *reserved = NULL
for now to
simplify eventual addition.
Cheers,
Sebastian
Victor
Le ven. 7 août 2020 à 19:05, Sebastian Berg <sebastian@sipsolutions.net> a écrit :
Hi all,
I am a bit confused about future proof GIL release with respect to potential subinterpreter development. Or maybe just the best practices.
In short, is the
PyGILState_Ensure()
function safe? I had the impression that was definitely not, but now makes me wondered how evenPyErr_Occurred()
can be safe whenEXPERIMENTAL_ISOLATED_SUBINTERPRETERS
. Of coursePyErr_Occurred()
is called with the GIL, so that may make a big difference.I scanned through this thread: https://bugs.python.org/issue15751 which suggested the addition of API to make it (almost?) always safe? But most of that thread is very old, much older than the newer
EXPERIMENTAL_ISOLATED_SUBINTERPRETERS
work. Reading https://bugs.python.org/issue4202 makes me think there is no current solution/safety?In my case (NumPy), I may be able to assume that the
PyGILState_Ensure()
calls I am talking about will always happen in the same thread as the initial GIL release, which gives me some hope that it might actually just work? [1]My impression until now was that the only clean solution would be to pass the threadstate. But, there is some public API, where we would require new API to pass the threadstate (which means slow adoption at the very least). So if I can avoid it, that would be much easier.
The consumer (the person who needs to grab the GIL) can be different from the user (the person releasing the GIL). So I am considering passing
PyThreadState **
with NULL indicating "unknown" state to the consumer. Much code may not even release the GIL, and in either case a move to a new API has to take at least 1-2 years, so I can't really wait for that to be done.Cheers,
Sebastian
[1] To be honest, I am not quite sure on this. It is true for NumPy itself, but I am not sure it is true for potential consumers of NumPy API. Or maybe, it would just be something to document very clearly somewhere.
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: sebastian@sipsolutions.net