Le jeu. 9 janv. 2020 à 13:35, Mark Shannon a écrit :
Passing the thread state explicitly creates a new class of errors that was not there before. What happens if the tstate argument is NULL,
You will likely get a crash.
or points to a different thread?
No idea what will happen. If it's a different subinterpreter, bad things can happen I guess :-) Well, I don't see this class of bug as a blocker issuer. If you pass invalid data, you get a crash. I'm not surprised by that. In my previous email, I proposed to continue to pass implicitly tstate when you call a function using the public C API. Only Python internals would pass explicitly tstate and so only internals should be carefully reviewed.
There is a one-to-one correspondence between Python threads and O/S threads.
I'm not convinced that this :-) So far, it's unclear to me how the transition from one interpreter to another happens with Python thread states. Let's say that the main thead spawns a thread A. The main thread and the thread A are running in the main interpreter. Then thread A calls _testcapi.run_in_subinterp() or _xxsubinterpreter.run_string() (or something else to run code in a subinterpreter). A subinterpreter is created and a new and different Python thread state is "attached" to thread A. The thread A gets 2 Python thread states, one per interpreter. It gets one or the other depending which in which interpreter the thread is running... We have to be careful to pass the proper thread state to internal C functions while doing this dance between two interpreters in the same thread. PyThreadState_Swap(tstate) must be called to set the current Python thread state. Full implementation of _testcapi.run_in_subinterp(), notice the PyThreadState_Swap() dance: /* To run some code in a sub-interpreter. */ static PyObject * run_in_subinterp(PyObject *self, PyObject *args) { const char *code; int r; PyThreadState *substate, *mainstate; if (!PyArg_ParseTuple(args, "s:run_in_subinterp", &code)) return NULL; mainstate = PyThreadState_Get(); PyThreadState_Swap(NULL); substate = Py_NewInterpreter(); if (substate == NULL) { /* Since no new thread state was created, there is no exception to propagate; raise a fresh one after swapping in the old thread state. */ PyThreadState_Swap(mainstate); PyErr_SetString(PyExc_RuntimeError, "sub-interpreter creation failed"); return NULL; } r = PyRun_SimpleString(code); Py_EndInterpreter(substate); PyThreadState_Swap(mainstate); return PyLong_FromLong(r); }
So the threadstate can, and should, be stored in a thread local variable. Accessing thread local storage is fast. x86/64 uses the fs register to point to it, whereas ARM dedicates R15 (I think).
*If* we managed to design subinterpreters in a way that a native thread is always associated to the same Python interpreter, moving to a thread local variable can make Python more efficient :-) It is likely to be more efficient than the current atomic variable design.
I started to move more and more things from "globals" to "per interpreter". For example, the garbage collector is now "per interpreter" (lives in PyThreadState). Small integer singletons are now also "per singleton": int("1") are now different objects in each interpreter, whereas they were shared previously. Later, even "None" singleton (and all other singletons) should be made "per interpreter". Getting a "per interpreter" object requires to state from the Python thread state: call _PyThreadState_GET(). Avoiding _PyThreadState_GET() calls reduces any risk of making Python slower with incoming subinterpreter changes.
Thread locals are not "global". Each sub-interpreter will have its own pool of threads. Each threadstate object should contain a pointer to its sub-interpreter.
Getting the interpreter from a Python thread state is trivial: interp = tstate->interp. The problem is how to get the current Python thread state. *Currently*, it's an atomic variable. But tomorrow with multiple interpreters running in parallel, I expect that it's going to me more expensive like first having the current the interpreter running the current native thread, and then get the Python thread state of this interpreter. Or something like that. We may get more and more indirections... Victor -- Night gathers, and now my watch begins. It shall not end until my death.