On Mon, 18 Nov 2019 12:39:00 -0500 Random832 <random832@fastmail.com> wrote:
On Mon, Nov 18, 2019, at 05:26, Antoine Pitrou wrote:
For the first goal, I don't think this is possible, or desirable. Obviously if we remove the GIL somehow then at a minimum we'll need to make the global threadstate a thread-local. But I think we'll always have to keep it around as a thread-local, at least, because there are situations where you simply cannot pass in the threadstate as an argument. One example comes up when doing FFI: there are C libraries that take callbacks, and will run them later in some arbitrary thread. When wrapping these in Python, we need a way to bundle up a Python function into a C function that can be called from any thread. So, ctypes and cffi and cython all have ways to do this bundling, and they all start with some delicate dance to figure out whether or not the current thread holds the GIL, acquiring the GIL if not, then checking whether or not this thread has a Python threadstate assigned, creating it if not, etc. This is completely dependent on having the threadstate available in ambient context. If threadstates were always passed as arguments, then it would become impossible to wrap these C libraries.
Most well-designed C libraries let you pass an additional "void*" parameter for user callbacks to be called with. A couple of them don't, unfortunately (OpenSSL perhaps? I don't remember).
I think you've missed the fact that the C library runs the callback on an arbitrary thread. The threadstate associated with the thread that made the original call is therefore *not the one you want*; you want a threadstate associated with the thread the callback is run on.
Ah, right, I had overlooked that mention. This does complicate things a bit. In that case you would want to pass the interpreter state and then use this particular interpreter's mapping of OS thread to threadstate. (assuming that per-interpreter mapping exists, which is another question; but it will have to exist at some point for PEP 554) Regards Antoine.