On Mon, Nov 18, 2019, at 05:26, Antoine Pitrou wrote:
For the first goal, I don't think this is possible, or desirable. Obviously if we remove the GIL somehow then at a minimum we'll need to make the global threadstate a thread-local. But I think we'll always have to keep it around as a thread-local, at least, because there are situations where you simply cannot pass in the threadstate as an argument. One example comes up when doing FFI: there are C libraries that take callbacks, and will run them later in some arbitrary thread. When wrapping these in Python, we need a way to bundle up a Python function into a C function that can be called from any thread. So, ctypes and cffi and cython all have ways to do this bundling, and they all start with some delicate dance to figure out whether or not the current thread holds the GIL, acquiring the GIL if not, then checking whether or not this thread has a Python threadstate assigned, creating it if not, etc. This is completely dependent on having the threadstate available in ambient context. If threadstates were always passed as arguments, then it would become impossible to wrap these C libraries.
Most well-designed C libraries let you pass an additional "void*" parameter for user callbacks to be called with. A couple of them don't, unfortunately (OpenSSL perhaps? I don't remember).
I think you've missed the fact that the C library runs the callback on an arbitrary thread. The threadstate associated with the thread that made the original call is therefore *not the one you want*; you want a threadstate associated with the thread the callback is run on. Alternately, if a thread state is not in any sense associated with a thread (would these situations then mean you simply always create a brand-new interpreter state?), maybe it shouldn't be called a thread state at all.