Hi Petr,On 09/06/2020 2:24 pm, Petr Viktorin wrote:On 2020-06-05 16:32, Mark Shannon wrote:
Hi,
There have been a lot of changes both to the C API and to internal implementations to allow multiple interpreters in a single O/S process.
These changes cause backwards compatibility changes, have a negative performance impact, and cause a lot of churn.
While I'm in favour of PEP 554, or some similar model for parallelism in Python, I am opposed to the changes we are currently making to support it.
What are sub-interpreters?
--------------------------
A sub-interpreter is a logically independent Python process which supports inter-interpreter communication built on shared memory and channels. Passing of Python objects is supported, but only by copying, not by reference. Data can be shared via buffers.
Here's my biased take on the subject:
Interpreters are contexts in which Python runs. They contain configuration (e.g. the import path) and runtime state (e.g. the set of imported modules). An interpreter is created at Python startup (Py_InitializeEx), and you can create/destroy additional ones with Py_NewInterpreter/Py_EndInterpreter.
This is long-standing API that is used, most notably by mod_wsgi.
Many extension modules and some stdlib modules don't play well with the existence of multiple interpreters in a process, mainly because they use process-global state (C static variables) rather than some more granular scope.
This tends to result in nasty bugs (C-level crashes) when multiple interpreters are started in parallel (Py_NewInterpreter) or in sequence (several Py_InitializeEx/Py_FinalizeEx cycles). The bugs are similar in both cases.
Whether Python interpreters run sequentially or in parallel, having them work will enable a use case I would like to see: allowing me to call Python code from wherever I want, without thinking about global state. Think calling Python from an utility library that doesn't care about the rest of the application it's used in. I personally call this "the Lua use case", because light-weight, worry-free embedding is an area where Python loses to Lua. (And JS as well—that's a relatively recent development, but much more worrying.)
This seems like a worthwhile goal. However I don't see why this requires having multiple Python interpreters in a single O/S process.
The mod_wsgi use case seems to require this (he writes without having looked at its source code). I have another possible use case: Independent plugins written in Python in native applications written in other languages. That doesn’t mean that is worthwhile to complicate the CPython code base for these. I have no opinion on that, both because I haven’t been active for a while and because I haven’t looked at the impact the current work has had.