Hi all,
I am still confused about this. Maybe shorter, we have:
User who release GIL → NumPy API → third-party C function
We can probably assume that the user will use NumPy API from random
threads, but only from threads already running Python (i.e. created by
Python), although there may be arguments that exceptions would be nice,
I do not think those are a priority.
In that case, is PyGILState_Ensure() considered safe and appropriate
API to be used within "third-party C function" (and/or NumPy API)? I
used to generally think its not, but now it seems to me it should be,
if the thread we are running on was previously spawned by Python?
Or should it be a priority that e.g. the PyThreadState *tstate is in
some form passed all the way down to the "third-part C function" so
that warnings or errors could be set more easily?
We could always try to design an API where the error setting is deferred until a time when the GIL is guaranteed to be held, but that seems inconvenient to me (it requires to pass out all error-information in some different way, only to convert it to an error later).
Cheers,
Sebastian
On Fri, 2020-08-07 at 12:04 -0500, Sebastian Berg wrote:
Hi all,
I am a bit confused about future proof GIL release with respect to potential subinterpreter development. Or maybe just the best practices.
In short, is the
PyGILState_Ensure()function safe? I had the impression that was definitely not, but now makes me wondered how evenPyErr_Occurred()can be safe whenEXPERIMENTAL_ISOLATED_SUBINTERPRETERS. Of coursePyErr_Occurred()is called with the GIL, so that may make a big difference.I scanned through this thread: https://bugs.python.org/issue15751 which suggested the addition of API to make it (almost?) always safe? But most of that thread is very old, much older than the newer
EXPERIMENTAL_ISOLATED_SUBINTERPRETERSwork. Reading https://bugs.python.org/issue4202 makes me think there is no current solution/safety?In my case (NumPy), I may be able to assume that the
PyGILState_Ensure()calls I am talking about will always happen in the same thread as the initial GIL release, which gives me some hope that it might actually just work? [1]My impression until now was that the only clean solution would be to pass the threadstate. But, there is some public API, where we would require new API to pass the threadstate (which means slow adoption at the very least). So if I can avoid it, that would be much easier.
The consumer (the person who needs to grab the GIL) can be different from the user (the person releasing the GIL). So I am considering passing
PyThreadState **with NULL indicating "unknown" state to the consumer. Much code may not even release the GIL, and in either case a move to a new API has to take at least 1-2 years, so I can't really wait for that to be done.Cheers,
Sebastian
[1] To be honest, I am not quite sure on this. It is true for NumPy itself, but I am not sure it is true for potential consumers of NumPy API. Or maybe, it would just be something to document very clearly somewhere.
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: sebastian@sipsolutions.net