On 26 May 2017 at 22:08, Stephan Houben firstname.lastname@example.org wrote:
Personally I feel that the current subinterpreter support falls short in the sense that it still requires a single GIL across interpreters.
Here is a point-wise overview of what I am imagining. I realize the following is very ambitious, but I would like to bring it to your consideration.
- Multiple interpreters can be instantiated, each of which is
completely independent. To this end, all global interpreter state needs to go into an interpreter strucutre, including the GIL (which becomes per-interpreter) Interpreters share no state whatsoever.
There'd still be true process global state (i.e. anything managed by the C runtime), so this would be a tiered setup with a read/write GIL and multiple SILs. For the time being though, a single GIL remains much easier to manage.
- PyObject's are tied to a particular interpreter and cannot be
shared between interpreters. (This is because each interpreter now has its own GIL.) I imagine a special debug build would actually store the interpreter pointer in the PyObject and would assert everywhere that the PyObject is only manipulated by its owning interpreter.
Yes, something like Rust's ownership model is the gist of what we had in mind (i.e. allowing zero-copy transfer of ownership between subinterpreters, but only the owning interpreter is allowed to do anything else with the object).
- Practically all existing APIs, including Py_INCREF and Py_DECREF,
need to get an additional explicit interpreter argument. I imagine that we would have a new prefix, say MPy_, because the existing APIs must be left for backward compatibility.
This isn't necessary, as the active interpreter is already tracked as part of the thread local state (otherwise mod_wsgi et al wouldn't work at all).
- At most one interpreter can be designated the "main" interpreter. This is for backward compatibility of existing extension modules ONLY. All the existing Py_* APIs operate implicitly on this main interpreter.
Yep, this is part of the concept. The PEP 432 draft has more details on that: https://www.python.org/dev/peps/pep-0432/#interpreter-initialization-phases
- Extension modules need to explicitly advertise multiple interpreter support. If they don't, they can only be imported in the main interpreter. However, in that case they can safely use the existing Py_ APIs.
This is the direction we started moving the with multi-phase initialisation PEP for extension modules: https://www.python.org/dev/peps/pep-0489/
As Petr noted, the main missing piece there now is the fact that object methods (as opposed to module level functions) implemented in C currently don't have ready access to the module level state for the modules where they're defined.
Since PyObject's cannot be shared across interpreters, there needs to be an explicit function which takes a PyObject in interpreter A and constructs a similar object in interpreter B.
Conceptually this would be equivalent to pickling in A and
unpickling in B, but presumably more efficient. It would use the copyreg registry in a similar way to pickle.
This would be an ownership transfer rather than a copy (which carries the implication that all the subinterpreters would still need to share a common memory allocator)
- Extension modules would also be able to register their function
for copying custom types across interpreters . That would allow extension modules to provide custom types where the underlying C object is in fact not copied but shared between interpreters. I would imagine we would have a"shared memory" memoryview object and also Mutex and other locking constructs which would work across interpreters.
We generally don't expect this to be needed given an ownership focused approach. Instead, the focus would be on enabling efficient channel based communication models that are cost-prohibitive when object serialisation is involved.
- Finally, the main application: functionality similar to the current
`multiprocessing' module, but with multiple interpreters on multiple threads in a single process. This would presumably be more efficient than `multiprocessing' and also allow extra functionality, since the underlying C objects can in fact be shared. (Imagine two interpreters operating in parallel on a single OpenCL context.)
We're not sure how feasible it will be to enable this in general, but even without it, zero-copy ownership transfers enable a *lot* of interest concurrency models that Python doesn't currently offer great primitives to support (they're mainly a matter of using threads in certain ways, which means they not only run afoul of the GIL, but you also don't get any assistance from the interpreter in strictly enforcing object ownership rules).