On 09Jan2020 1659, Victor Stinner wrote:
Le jeu. 9 janv. 2020 à 19:33, Steve Dower <steve.dower@python.org> a écrit :
Requiring an _active_ Python thread (~GIL held) to only run on a single OS thread is fine, but tying it permanently to a single OS thread makes things very painful. (Of course, this isn't the only thing that makes it painful, but if we're going to make this a deliberate design of CPython then we are essentially excluding entire interesting classes of embedding.)
Do you have an use case where you reuse the same Python thread state in different native threads?
I do, but I can't share exactly what it is (yet) :) But essentially, the main application has its own threading setup where callbacks happen on its own threadpool. Some of those callbacks sometimes have to raise Python events, but all of those events should be serialized (i.e. *no* Python code should run in parallel, even though the app's callbacks can). Note that this is not arbitrary CPython code. It is a very restricted context (only first-party modules, no threading/ctypes/etc.). Ideally, we'd just lock the Python thread and run the Python code and get the result back directly. Instead, we've had to set up a parallel Python thread that is constantly running, and build a complicated cross-thread marshalling setup in order to simulate synchronous calls. (And also argue with the application architects who are very against mixing threading models in their app, unsurprisingly.)
Currently, the Python runtime expects to have a different Python thread state per native thread. For example, PyGILState_Ensure() creates one if there is no Python thread state associated to the current thread.
And ultimately, everything above is just because of this assumption. Turns out that simply transferring the thread state and setting it in TLS is *nearly* okay most of the time, but not quite. Breaking down the assumption that each CPython thread is the only code running on its particular OS thread would help a lot here - making the CPython thread state a parameter rather than ambient state is the right direction. Cheers, Steve