[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

Nick Coghlan ncoghlan at gmail.com
Wed Sep 13 18:46:45 EDT 2017


On 13 September 2017 at 14:10, Nathaniel Smith <njs at pobox.com> wrote:
> Subinterpreters are basically an attempt to reimplement the OS's
> process isolation in user-space, right?

Not really, they're more an attempt to make something resembling
Rust's memory model available to Python programs - having the default
behaviour be "memory is not shared", but having the choice to share
when you want to be entirely an application level decision, without
getting into the kind of complexity needed to deliberately break
operating system level process isolation.

The difference is that where Rust was able to do that on a per-thread
basis and rely on their borrow checker for enforcement of memory
ownership, for PEP 554, we're proposing to do it on a per-interpreter
basis, and rely on runtime object space partitioning (where Python
objects and the memory allocators are *not* shared between
interpreters) to keep things separated from each other.

That's why memoryview is such a key part of making the proposal
interesting: it's what lets us relatively easily poke holes in the
object level partitioning between interpreters and provide zero-copy
messaging passing without having to share any regular reference counts
between interpreters (which in turn is what makes it plausible that we
may eventually be able to switch to a true GIL-per-interpreter model,
with only a few cross-interpreter locks for operations like accessing
the list of interpreters itself).

Right now, the closest equivalent to this programming model that
Python offers is to combine threads with queue.Queue, and it requires
a lot of programming discipline to ensure that you don't access an
object again once you've submitted to a queue.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Python-ideas mailing list