Le jeu. 19 mars 2020 à 02:17, Kyle Stanley <aeros167@gmail.com> a écrit :
Agreed; a PEP (even if it's just informational) would go a long way in helping to clear up some misunderstandings.
I am still moving blindly in the darkness of CPython internals and so I don't feel comfortable to write a PEP which would mean that I know what I am doing :-) I fixed dozens of bugs in the Python finalization related to subinterpreters and daemon threads: https://bugs.python.org/issue33608#msg364670 I managed to make pending calls (Py_AddPendingCall) per-interpreter, but it was really painful. So far, I have no strong opinion about passing explicitly tstate vs "implicit" TLS (Thread Local Storage) variable. I mean, I don't feel that I have enough feedback to see which option is the best. All I can say is that passing explicitly tstate helps me to validate that the code is *correct*. It helps me to understand where tstate comes from and helps me to understand if tstate is valid or not. It wasn't an issue in Python 3.7 where most critical structures lives in _PyRuntime which is shared by all subinterpreters. It became a big deal in Python 3.9 where more and more critical structures are "per-interpreter" and becomes invalid during Python finalization. For example, in Python 3.8, PyThreadState structures of daemon threads are never freed. There are still too many functions which don't work properly with subinterpreters. There are still a few design issues that needs to be addressed. Most functions of the C API don't even specify in their documentation if they require the GIL to be held. One good example is Py_AddPendingCall(). The documentation says that it's safe to call it without holding the GIL. Except that right now, there is no reliable way to get the correct interpreter in this case (correct me if I'm wrong!). The function uses PyGILState_GetThisThreadState() which may return a Python thread state of the wrong interpreter :-( Again, the PyGILState API should be fixed to support subinterpreters. I'm not even sure that I discovered all borders of the problem. I only started my list of issues. I would prefer to continue to experiment passing tstate explicitly in internal C APIs until most blocker issues will be fixed. Once early work on running two subinterpreters in parallel will start working (one "GIL" per interpreter), I will be more open to reconsider using a TLS variable. We are not so far from being able to *experiment* one "GIL" per interpreter. I succeeded to move the first half of _PyRuntimeState.ceval into a new PyInterpreterState.ceval: pending_calls, eval_breaker and tracing_possible. Moving eval_breaker was tricky, since it's tidely coupled to the GIL which currently remains shared by all interpreters. By the way, I took the opportunity to optimized the bytecode evaluation loop (ceval), eval_breaker was not properly set in multithreaded applications: "Inefficient signal handling in multithreaded applications" https://bugs.python.org/issue40010 The main interpreter benefits of this optimization as well, not only subinterpreters ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.