PEP 554 v2 (new "interpreters" module)
I've updated the PEP in response to some excellent feedback. Thanks to all who helped. Most notably, I've added a basic object passing mechanism. I've included the PEP below. Please let me know what you think. Thanks!
-eric
****************************************************
PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow ericsnowcurrently@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History:
Abstract ========
This proposal introduces the stdlib ``interpreters`` module. It exposes the basic functionality of subinterpreters that already exists in the C-API. Each subinterpreter runs with its own state (see ``Interpreter Isolation`` below). The module will be "provisional", as described by PEP 411.
Rationale =========
Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended.
CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation.
This proposal is focused on enabling the fundamental capability of multiple isolated interpreters in the same Python process. This is a new area for Python so there is relative uncertainly about the best tools to provide as companions to subinterpreters. Thus we minimize the functionality we add in the proposal as much as possible.
Concerns --------
* "subinterpreters are not worth the trouble"
Some have argued that subinterpreters do not add sufficient benefit to justify making them an official part of Python. Adding features to the language (or stdlib) has a cost in increasing the size of the language. So it must pay for itself. In this case, subinterpreters provide a novel concurrency model focused on isolated threads of execution. Furthermore, they present an opportunity for changes in CPython that will allow simulateous use of multiple CPU cores (currently prevented by the GIL).
Alternatives to subinterpreters include threading, async, and multiprocessing. Threading is limited by the GIL and async isn't the right solution for every problem (nor for every person). Multiprocessing is likewise valuable in some but not all situations. Direct IPC (rather than via the multiprocessing module) provides similar benefits but with the same caveat.
Notably, subinterpreters are not intended as a replacement for any of the above. Certainly they overlap in some areas, but the benefits of subinterpreters include isolation and (potentially) performance. In particular, subinterpreters provide a direct route to an alternate concurrency model (e.g. CSP) which has found success elsewhere and will appeal to some Python users. That is the core value that the ``interpreters`` module will provide.
* "stdlib support for subinterpreters adds extra burden on C extension authors"
In the ``Interpreter Isolation`` section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext]_ Until that is resolved, C extension authors will face extra difficulty to support subinterpreters.
Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state.
Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often their users will be affected, what the additional maintenance burden will be for projects, and what the overall benefit of subinterpreters is to offset those costs. The position of this PEP is that the actual extra maintenance burden will be small and well below the threshold at which subinterpreters are worth it.
Proposal ========
The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level ``_interpreters`` module. The proposed API is inspired by the ``threading`` module.
The module provides the following functions:
``list()``::
Return a list of all existing interpreters.
``get_current()``::
Return the currently running interpreter.
``get_main()``::
Return the main interpreter.
``create()``::
Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain idle until something is run in it.
The module also provides the following classes:
``Interpreter(id)``::
id:
The interpreter's ID (read-only).
is_running():
Return whether or not the interpreter is currently executing code.
destroy():
Finalize and destroy the interpreter. If called on a running interpreter then raise RuntimeError in the interpreter that called ``destroy()``.
run(code):
Run the provided Python code in the interpreter, in the current OS thread. If the interpreter is already running then raise RuntimeError in the interpreter that called ``run()``.
The current interpreter (which called ``run()``) will block until the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.
Supported code: source text.
get_fifo(name):
Return the FIFO object with the given name that is associated with this interpreter. If no such FIFO exists then raise KeyError. The FIFO will be either a "FIFOReader" or a "FIFOWriter", depending on which "add_*_fifo()" was called.
list_fifos():
Return a list of all fifos associated with the interpreter.
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOReader gets tied to this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_recv_fifo()" was called) then raise KeyError.
add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOWriter gets tied to this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_send_fifo()" was called) then raise KeyError.
remove_fifo(name):
Drop the association between the named FIFO and this interpreter. If the named FIFO is not found then raise KeyError.
``FIFOReader(name)``::
The receiving end of a FIFO. An interpreter may use this to receive objects from another interpreter. At first only bytes and None will be supported.
name:
The FIFO's name.
__next__():
Return the next bytes object from the pipe. If none have been pushed on then block.
pop(*, block=True):
Return the next bytes object from the pipe. If none have been pushed on and "block" is True (the default) then block. Otherwise return None.
``FIFOWriter(name)``::
The sending end of a FIFO. An interpreter may use this to send objects to another interpreter. At first only bytes and None will be supported.
name:
The FIFO's name.
push(object, *, block=True):
Add the object to the FIFO. If "block" is true then block until the object is popped off. If the FIFO does not support the object's type then TypeError is raised.
About FIFOs -----------
Subinterpreters are inherently isolated (with caveats explained below), in contrast to threads. This enables a different concurrency model than currently exists in Python. CSP (Communicating Sequential Processes), upon which Go's concurrency is based, is one example of this model.
A key component of this approach to concurrency is message passing. So providing a message/object passing mechanism alongside ``Interpreter`` is a fundamental requirement. This proposal includes a basic mechanism upon which more complex machinery may be built. That basic mechanism draws inspiration from pipes, queues, and CSP's channels.
The key challenge here is that sharing objects between interpreters faces complexity due in part to CPython's current memory model. Furthermore, in this class of concurrency, the ideal is that objects only exist in one interpreter at a time. However, this is not practical for Python so we initially constrain supported objects to ``bytes`` and ``None``. There are a number of strategies we may pursue in the future to expand supported objects and object sharing strategies.
Note that the complexity of object sharing increases as subinterpreters become more isolated, e.g. after GIL removal. So the mechanism for message passing needs to be carefully considered. Keeping the API minimal and initially restricting the supported types helps us avoid further exposing any underlying complexity to Python users.
Deferred Functionality ======================
In the interest of keeping this proposal minimal, the following functionality has been left out for future consideration. Note that this is not a judgement against any of said capability, but rather a deferment. That said, each is arguably valid.
Interpreter.call() ------------------
It would be convenient to run existing functions in subinterpreters directly. ``Interpreter.run()`` could be adjusted to support this or a ``call()`` method could be added::
Interpreter.call(f, *args, **kwargs)
This suffers from the same problem as sharing objects between interpreters via queues. The minimal solution (running a source string) is sufficient for us to get the feature out where it can be explored.
timeout arg to pop() and push() -------------------------------
Typically functions that have a ``block`` argument also have a ``timeout`` argument. We can add it later if needed.
Interpreter Isolation =====================
CPython's interpreters are intended to be strictly isolated from each other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. [c-api]_
However, there are ways in which interpreters share some state. First of all, some process-global state remains shared, like file descriptors. There are no plans to change this.
Second, some isolation is faulty due to bugs or implementations that did not take subinterpreters into account. This includes things like at-exit handlers and extension modules that rely on C globals. In these cases bugs should be opened (some are already).
Finally, some potential isolation is missing due to the current design of CPython. This includes the GIL and memory management. Improvements are currently going on to address gaps in this area.
Provisional Status ==================
The new ``interpreters`` module will be added with "provisional" status (see PEP 411). This allows Python users to experiment with the feature and provide feedback while still allowing us to adjust to that feedback. The module will be provisional in Python 3.7 and we will make a decision before the 3.8 release whether to keep it provisional, graduate it, or remove it.
References ==========
.. [c-api] https://docs.python.org/3/c-api/init.html#bugs-and-caveats
.. [petr-c-ext] https://mail.python.org/pipermail/import-sig/2016-June/001062.html https://mail.python.org/pipermail/python-ideas/2016-April/039748.html
Copyright =========
This document has been placed in the public domain.
On Fri, Sep 08, 2017 at 04:04:27PM -0700, Eric Snow wrote:
- "stdlib support for subinterpreters adds extra burden on C extension authors"
In the ``Interpreter Isolation`` section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext]_ Until that is resolved, C extension authors will face extra difficulty to support subinterpreters.
It's a bit of a hassle, and the enormous slowdown in some of the existing solutions is really a no go [1].
In the case of _decimal, the tls-context is already subinterpreter safe and reasonable fast due to caching.
The most promising model to me is to put *all* globals in a tls structure and cache the whole structure. Extrapolating from my experiences with the context, this might have a slowdown of "only" 4%.
Still, the argument "who uses subinterpreters?" of course still remains.
Stefan Krah
[1] I'm referring to the slowdown of heaptypes + module state.
Le 09/09/2017 à 01:28, Stefan Krah a écrit :
On Fri, Sep 08, 2017 at 04:04:27PM -0700, Eric Snow wrote:
- "stdlib support for subinterpreters adds extra burden on C extension authors"
In the ``Interpreter Isolation`` section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext]_ Until that is resolved, C extension authors will face extra difficulty to support subinterpreters.
It's a bit of a hassle, and the enormous slowdown in some of the existing solutions is really a no go [1].
In the case of _decimal, the tls-context is already subinterpreter safe and reasonable fast due to caching.
The most promising model to me is to put *all* globals in a tls structure and cache the whole structure. Extrapolating from my experiences with the context, this might have a slowdown of "only" 4%.
Still, the argument "who uses subinterpreters?" of course still remains.
For now, nobody. But if we expose it and web frameworks manage to create workers as fast as multiprocessing and as cheap as threading, you will find a lot of people starting to want to use it.
We can't know until we got the toy to play with.
Stefan Krah
[1] I'm referring to the slowdown of heaptypes + module state.
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.c...
On Fri, Sep 8, 2017 at 8:54 PM, Michel Desmoulin desmoulinmichel@gmail.com wrote:
Le 09/09/2017 à 01:28, Stefan Krah a écrit :
Still, the argument "who uses subinterpreters?" of course still remains.
For now, nobody. But if we expose it and web frameworks manage to create workers as fast as multiprocessing and as cheap as threading, you will find a lot of people starting to want to use it.
To temper expectations a bit here, it sounds like the first version might be more like: as slow as threading (no multicore), as expensive as multiprocessing (no shared memory), and -- on Unix -- slower to start than either of them (no fork).
-n
On Fri, Sep 8, 2017 at 8:54 PM, Michel Desmoulin desmoulinmichel@gmail.com wrote:
Le 09/09/2017 à 01:28, Stefan Krah a écrit :
Still, the argument "who uses subinterpreters?" of course still remains.
For now, nobody. But if we expose it and web frameworks manage to create workers as fast as multiprocessing and as cheap as threading, you will find a lot of people starting to want to use it.
Note that subinterpreters share the GIL currently, and the objects you can pass between interpreters will be quite restricted initially. However, the ultimate goal is to stop sharing the GIL between interpreters and to broaden the types that can be passed between interpreters. Regardless, the isolation inherent to subinterpreters provides benefits immediately.
-eric
On Fri, Sep 8, 2017 at 4:28 PM, Stefan Krah stefan@bytereef.org wrote:
The most promising model to me is to put *all* globals in a tls structure and cache the whole structure. Extrapolating from my experiences with the context, this might have a slowdown of "only" 4%.
Yeah, this is actually something I've been exploring and was one of the motivations for consolidating the C globals.
-eric
On 9 September 2017 at 00:04, Eric Snow ericsnowcurrently@gmail.com wrote:
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOReader gets tied to this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_recv_fifo()" was called) then raise KeyError.add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOWriter gets tied to this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_send_fifo()" was called) then raise KeyError.
Personally, I *always* read these names backwards - from the POV of the caller. So when I see "add_send_fifo", I then expect to be able to send stuff on the returned FIFO (i.e., I get a writer back). But that's not how it works.
I may be alone in this - I've had similar problems in the past with how people name pipes, for example - but I thought I'd raise it while there's still a possibility that it's not just me and the names can be changed.
Paul
On Sat, 9 Sep 2017 09:04:59 +0100 Paul Moore p.f.moore@gmail.com wrote:
On 9 September 2017 at 00:04, Eric Snow ericsnowcurrently@gmail.com wrote:
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOReader gets tied to this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_recv_fifo()" was called) then raise KeyError.add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOWriter gets tied to this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_send_fifo()" was called) then raise KeyError.Personally, I *always* read these names backwards - from the POV of the caller. So when I see "add_send_fifo", I then expect to be able to send stuff on the returned FIFO (i.e., I get a writer back). But that's not how it works.
Hmm, it's even worse for me, as it's not obvious by the quoted excerpt how those APIs are supposed to be used exactly.
How does the other interpreter get the FIFO "tied" to it? Is it `get_current().get_fifo(name)`? Something else? How does it get to learn the *name*?
Why are FIFOs unidirectional? Does it really make them significantly cheaper? With bidirectional pipes, the whole recv/send confusion would be avoided.
As a point of comparison, for default for `multiprocessing.Pipe()` is to create a bidirectional pipe (*), yet I'm sure a multiprocessing Pipe has more overhead than an in-process FIFO.
(*) https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe
Regards
Antoine.
On Sat, Sep 9, 2017 at 4:45 AM, Antoine Pitrou solipsis@pitrou.net wrote:
How does the other interpreter get the FIFO "tied" to it? Is it `get_current().get_fifo(name)`? Something else?
Yep, that's it. I've added some examples to the PEP to illustrate. That said, I'm re-working the method names since they confuse me too. :)
How does it get to learn the *name*?
The main way is that the programmer hard-codes it. I'd expect close proximity between the code that adds the FIFO and the code that get's run in the subinterpreter.
Why are FIFOs unidirectional? Does it really make them significantly cheaper? With bidirectional pipes, the whole recv/send confusion would be avoided.
As a point of comparison, for default for `multiprocessing.Pipe()` is to create a bidirectional pipe (*), yet I'm sure a multiprocessing Pipe has more overhead than an in-process FIFO.
The choice wasn't about cost or performance. In part it's a result of my experience with Go's channels. Bidirectional channels are problematic in certain situations, including when it relates to which goroutine is responsible for managing the channel's lifetime. Also, bidirectional channels require both the reader and the writer (of the code) to keep track of the two roles that the channel plays. It's easy to forget about one or the other, and accidentally do the wrong thing. Unidirectional channels resolve all those problems. Granted, the FIFOs from the PEP aren't the same as Go's channels. They are more rudimentary and they lack the support provided by a static compiler (e.g. deadlock detection), but my intention is that they provide all the functionality we need as a building block.
-eric
On 9 September 2017 at 01:04, Paul Moore p.f.moore@gmail.com wrote:
On 9 September 2017 at 00:04, Eric Snow ericsnowcurrently@gmail.com wrote:
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOReader gets tied to this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_recv_fifo()" was called) then raise KeyError.add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOWriter gets tied to this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_send_fifo()" was called) then raise KeyError.Personally, I *always* read these names backwards - from the POV of the caller. So when I see "add_send_fifo", I then expect to be able to send stuff on the returned FIFO (i.e., I get a writer back). But that's not how it works.
I had the same problem with the current names: as a message sender, I expect to request a send queue, as a message receiver, I expect to request a receive queue.
Having to request the opposite of what I want to do next seems backwards:
send_fifo = other_interpreter.add_recv_fifo(__name__) # Wut? send_fifo.push(b"Hello!")
I think it would be much clearer if the API looked like this for an inline subinterpreter invocation.:
# Sending messages other_interpreter = interpreters.create() send_fifo = other_interpreter.get_send_fifo(__name__) send_fifo.push(b"Hello!") send_fifo.push(b"World!") send_fifo.push(None)
# Receiving messages receiver_code = textwrap.dedent(f""" import interpreters recv_fifo = interpreters.get_current().get_recv_fifo({__name__}) while True: msg = recv_fifo.pop() if msg is None: break print(msg) """) other_interpreter.run(receiver_code)
To enable concurrent communication between the sender & receiver, you'd currently need to move the "other_interpreter.run()" call out to a separate thread, but I think that's fine for now.
The rules for get_recv_fifo() and get_send_fifo() would be:
- named fifos are created as needed and always stored on the *receiving* interpreter - you can only call get_recv_fifo() on the currently running interpreter: other interpreters can't access your receive fifos - get_recv_fifo() never fails - the receiving interpreter can have as many references to a receive FIFO as it likes - get_send_fifo() can fail if the named fifo already exists on the receiving interpreter and the designated sender is an interpreter other than the one calling get_send_fifo() - interpreters are free to use the named FIFO mechanism to send messages to themselves
To immediately realise some level of efficiency benefits from the shared memory space between the main interpreter and subinterpreters, I also think these low level FIFOs should be defined as accepting any object that supports the PEP 3118 buffer protocol, and emitting memoryview() objects on the receiving end, rather than being bytes-in, bytes-out.
Such a memoryview based primitive can then potentially be expanded in the future to support additional more-efficient-than-multiprocessing serailisation based protocols, where the sending interpreter serialises into a region of memory, shares that region with the subinterpreter via a FIFO memoryview, and then the subinterpreter deserialises directly from the sending interpreter's memory region, without the serialised form ever needing to be streamed or copied anywhere (which is much harder to avoid when coordinating across multiple operating system processes).
Cheers, Nick.
On Sep 9, 2017 9:07 AM, "Nick Coghlan" ncoghlan@gmail.com wrote:
To immediately realise some level of efficiency benefits from the shared memory space between the main interpreter and subinterpreters, I also think these low level FIFOs should be defined as accepting any object that supports the PEP 3118 buffer protocol, and emitting memoryview() objects on the receiving end, rather than being bytes-in, bytes-out.
Is your idea that this memoryview would refer directly to the sending interpreter's memory (as opposed to a copy into some receiver-owned buffer)? If so, then how do the two subinterpreters coordinate the buffer release when the memoryview is closed?
-n
On 10 September 2017 at 03:54, Nathaniel Smith njs@pobox.com wrote:
On Sep 9, 2017 9:07 AM, "Nick Coghlan" ncoghlan@gmail.com wrote:
To immediately realise some level of efficiency benefits from the shared memory space between the main interpreter and subinterpreters, I also think these low level FIFOs should be defined as accepting any object that supports the PEP 3118 buffer protocol, and emitting memoryview() objects on the receiving end, rather than being bytes-in, bytes-out.
Is your idea that this memoryview would refer directly to the sending interpreter's memory (as opposed to a copy into some receiver-owned buffer)?
There are two possibilities that I think can be made to work. Option one would be a memoryview subclass that:
1. Also stores a reference to the source interpreter 2. Temporarily switches back to that interpreter when acquiring or releasing a buffer view
The receiving interpreter would then also be able to acquire a suitable referencing to the sending interpreter for the message in order to establish bidirectional communications.
Option two would be the same general idea, but with a regular memoryview placed in front of the subinterpreter aware variant (although it's less clear how we'd establish bidirectional comms channels in that case).
Cheers, Nick.
On Sat, Sep 9, 2017 at 1:04 AM, Paul Moore p.f.moore@gmail.com wrote:
On 9 September 2017 at 00:04, Eric Snow ericsnowcurrently@gmail.com wrote:
add_recv_fifo(name=None): add_send_fifo(name=None):
Personally, I *always* read these names backwards - from the POV of the caller. So when I see "add_send_fifo", I then expect to be able to send stuff on the returned FIFO (i.e., I get a writer back). But that's not how it works.
I may be alone in this - I've had similar problems in the past with how people name pipes, for example - but I thought I'd raise it while there's still a possibility that it's not just me and the names can be changed.
Yeah, those names are bugging me too. I'm stewing over possible alternatives. It's the one piece of the PEP that I want to address before I re-post to the list.
-eric
On Fri, 8 Sep 2017 16:04:27 -0700 Eric Snow ericsnowcurrently@gmail.com wrote:
The module provides the following functions:
``list()``::
Return a list of all existing interpreters.
It's called ``enumerate()`` in the threading module. Not sure there's a point in choosing a different name here.
The module also provides the following classes:
``Interpreter(id)``::
run(code):
Run the provided Python code in the interpreter, in the current OS thread. If the interpreter is already running then raise RuntimeError in the interpreter that called ``run()``. The current interpreter (which called ``run()``) will blockuntil the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.
Why does it block? How is concurrency supposed to be achieved in that model? It would be more flexible if run(code) returned an object that can later be waited on. Something like... a Future :-)
And why guarantee that it executes in the "current OS thread"? I would say you don't want to specify where it executes exactly, as it opens the door for more sophisticated implementations (such as automatic assignment of subinterpreters inside a pool of threads).
get_fifo(name):
Return the FIFO object with the given name that is associated with this interpreter. If no such FIFO exists then raise KeyError. The FIFO will be either a "FIFOReader" or a "FIFOWriter", depending on which "add_*_fifo()" was called.list_fifos():
Return a list of all fifos associated with the interpreter.
If fifos are uniquely named, why not return a name->fifo mapping?
``FIFOReader(name)``:: [...]
I don't think the method naming choice is very adequate here. The API model for the FIFO objects can either be a (threading or multiprocessing) Queue or a multiprocessing Pipe.
- if a Queue, then it should have a get() / put() pair of methods - if a Pipe, then it should have a recv() / send() pair of methods
Now, since Queues are multi-producer multi-consumer, while Pipes are single-producer single-consumer (they aren't "synchronized"), the better analogy seems to the multiprocessing Pipe here, so I would vote for recv() / send().
But, in any case, definitely not a pop() / push() pair.
Has any thought been given to how FIFOs could integrate with async code driven by an event loop (e.g. asyncio)? I think the model of executing several asyncio (or Tornado) applications each in their own subinterpreter may prove quite interesting to reconcile multi-core concurrency with ease of programming. That would require the FIFOs to be able to synchronize on something an event loop can wait on (probably a file descriptor?).
Regards
Antoine.
On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow ericsnowcurrently@gmail.com wrote:
``list()``::
It's called ``enumerate()`` in the threading module. Not sure there's a point in choosing a different name here.
Yeah, in the first version of the PEP it was called "enumerate()". I changed it to "list()" at Raymond's recommendation. The reasoning is that it's less confusing to most people that way. TBH, I'd rather leave it "list()", but could be swayed. Perhaps it would be enough for the PEP to not mention any relationship to "threading"?
The current interpreter (which called ``run()``) will blockuntil the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.
Why does it block? How is concurrency supposed to be achieved in that model? It would be more flexible if run(code) returned an object that can later be waited on. Something like... a Future :-)
I expect this is more a problem with my description than with the feature. :) I've already re-written this bit to be more clear. It's not that the thread blocks. It's more like a function call, where the current frame is paused while the call is executed. Then it returns to the calling frame. Likewise the interpreter in the current thread gets swapped out with the target interpreter, where the code gets run, and then the original interpreter gets swapped back in. This is how you do it in the C-API and it made sense (to me) to do it the same way in Python.
And why guarantee that it executes in the "current OS thread"? I would say you don't want to specify where it executes exactly, as it opens the door for more sophisticated implementations (such as automatic assignment of subinterpreters inside a pool of threads).
Again, I had explained this poorly in the PEP. The key thing here is that subinterpreters don't do anything special relative to threading. If you want to call "Interpreter.run()" in a thread then you stick it in a "threading.Thread". If you want to auto-assign to a pool of threads then you treat it like any other function you would auto-assign to a pool of threads.
get_fifo(name): list_fifos():
If fifos are uniquely named, why not return a name->fifo mapping?
I suppose we could. Then we could get rid of "get_fifo()" too. I'm still mulling over the right API for the FIFO parts of the PEP.
``FIFOReader(name)``:: [...]
I don't think the method naming choice is very adequate here. The API model for the FIFO objects can either be a (threading or multiprocessing) Queue or a multiprocessing Pipe.
- if a Queue, then it should have a get() / put() pair of methods
- if a Pipe, then it should have a recv() / send() pair of methods
Now, since Queues are multi-producer multi-consumer, while Pipes are single-producer single-consumer (they aren't "synchronized"), the better analogy seems to the multiprocessing Pipe here, so I would vote for recv() / send().
But, in any case, definitely not a pop() / push() pair.
Thanks for pointing that out. Prior art, FTW! I'll factor that in.
Has any thought been given to how FIFOs could integrate with async code driven by an event loop (e.g. asyncio)? I think the model of executing several asyncio (or Tornado) applications each in their own subinterpreter may prove quite interesting to reconcile multi-core concurrency with ease of programming. That would require the FIFOs to be able to synchronize on something an event loop can wait on (probably a file descriptor?).
Personally I've given pretty much no thought to the relationship with async. TBH, the FIFO parts of the PEP were added only recently and haven't fully baked yet. I'd be interested more feedback on async relative to PEP, not just with the FIFO bits; my experience with async is pretty limited thus far.
-eric
On 13 September 2017 at 07:43, Eric Snow ericsnowcurrently@gmail.com wrote:
On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow ericsnowcurrently@gmail.com wrote:
``list()``::
It's called ``enumerate()`` in the threading module. Not sure there's a point in choosing a different name here.
Yeah, in the first version of the PEP it was called "enumerate()". I changed it to "list()" at Raymond's recommendation. The reasoning is that it's less confusing to most people that way. TBH, I'd rather leave it "list()", but could be swayed. Perhaps it would be enough for the PEP to not mention any relationship to "threading"?
I think you should just say that you're not using interpreters.enumerate() because that conflicts with the enumerate builtin.
However, replacing that with a *different* naming conflict with an even more commonly used builtin wouldn't be a good idea either :)
I also think the "list interpreters" question is more subtle than you think, as there are two potential sets of interpreters to consider:
1. Getting a listing of all interpreters in the process (creating Interpreter objects in the current interpreter for each of them) 2. Getting a listing of all other interpreters created by calling interpreters.create_interpreter() in *this* interpreter
Since it's not clear yet whether or not we're actually going to track the metadata needed to report the latter, I'd suggest only offering "interpreters.list_all()" initially, and requiring folks to do their own tracking if they want a record of the interpreters their own code has created.
The current interpreter (which called ``run()``) will blockuntil the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.
Why does it block? How is concurrency supposed to be achieved in that model? It would be more flexible if run(code) returned an object that can later be waited on. Something like... a Future :-)
I expect this is more a problem with my description than with the feature. :) I've already re-written this bit to be more clear. It's not that the thread blocks. It's more like a function call, where the current frame is paused while the call is executed. Then it returns to the calling frame. Likewise the interpreter in the current thread gets swapped out with the target interpreter, where the code gets run, and then the original interpreter gets swapped back in. This is how you do it in the C-API and it made sense (to me) to do it the same way in Python.
I'm not sure if it would create more confusion than it would resolve, but I'm wondering if it may be worth including an example where you delegate the subinterpreter call to another thread via concurrent.futures.ThreadExecutor.
get_fifo(name): list_fifos():
If fifos are uniquely named, why not return a name->fifo mapping?
I suppose we could. Then we could get rid of "get_fifo()" too. I'm still mulling over the right API for the FIFO parts of the PEP.
The FIFO parts of the proposal are feeling a lot like the Mailbox objects in TI-RTOS to me, just with a more user-friendly API since we're dealing with Python rather than C: http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/sysbios/6...
``FIFOReader(name)``:: [...]
I don't think the method naming choice is very adequate here. The API model for the FIFO objects can either be a (threading or multiprocessing) Queue or a multiprocessing Pipe.
- if a Queue, then it should have a get() / put() pair of methods
- if a Pipe, then it should have a recv() / send() pair of methods
Now, since Queues are multi-producer multi-consumer, while Pipes are single-producer single-consumer (they aren't "synchronized"), the better analogy seems to the multiprocessing Pipe here, so I would vote for recv() / send().
But, in any case, definitely not a pop() / push() pair.
Thanks for pointing that out. Prior art, FTW! I'll factor that in.
+1 from me for treating this as a split Pipe model, where the SendPipe and RecvPipe are distinct objects (since they'll exist in different interpreters).
Has any thought been given to how FIFOs could integrate with async code driven by an event loop (e.g. asyncio)? I think the model of executing several asyncio (or Tornado) applications each in their own subinterpreter may prove quite interesting to reconcile multi-core concurrency with ease of programming. That would require the FIFOs to be able to synchronize on something an event loop can wait on (probably a file descriptor?).
Personally I've given pretty much no thought to the relationship with async. TBH, the FIFO parts of the PEP were added only recently and haven't fully baked yet. I'd be interested more feedback on async relative to PEP, not just with the FIFO bits; my experience with async is pretty limited thus far.
The way TI-RTOS handles this is to allow event objects to be passed in that are used to report a "Mailbox not empty" event (when a new message is queued) and a "Mailbox not full" event (when the Mailbox was previously full, and a message is read). Code can then either block on the mailbox itself (using Mailbox_pend), or else wait on the underlying events directly.
It seems to me that a similar model would work here (and wouldn't necessarily need to be included in the initial version of the PEP): a SendPipe could accept an optional Event object that it calls set() on when the pipe ceases to be full, and a RecvPipe could do the same for when it ceases to be empty.
It would then be up to the calling code to decide whether to pass in a regular synchronous threading.Event() object (in which case it could probably just make blocking calls to the send()/recv() methods instead), or else to pass in asyncio.Event() objects (which a coroutine can wait on via the Event.wait() coroutine)
Cheers, Nick.
On Tue, Sep 12, 2017 at 2:43 PM, Eric Snow ericsnowcurrently@gmail.com wrote:
Yeah, in the first version of the PEP it was called "enumerate()". I changed it to "list()" at Raymond's recommendation. The reasoning is that it's less confusing to most people that way. TBH, I'd rather leave it "list()", but could be swayed. Perhaps it would be enough for the PEP to not mention any relationship to "threading"?
Really, you're going to change it from a name conflict with a builtin function to a name conflict with a builtin type?
On Tue, 12 Sep 2017 14:43:43 -0700 Eric Snow ericsnowcurrently@gmail.com wrote:
On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow ericsnowcurrently@gmail.com wrote:
``list()``::
It's called ``enumerate()`` in the threading module. Not sure there's a point in choosing a different name here.
Yeah, in the first version of the PEP it was called "enumerate()". I changed it to "list()" at Raymond's recommendation. The reasoning is that it's less confusing to most people that way. TBH, I'd rather leave it "list()", but could be swayed. Perhaps it would be enough for the PEP to not mention any relationship to "threading"?
Yeah, given the other clarifications you did, I agree not mentioning threading at all would be better, since the PEP itself does not strive to provide new concurrency primitives.
As for the naming, let's make it both unconfusing and explicit? How about three functions: `all_interpreters()`, `running_interpreters()` and `idle_interpreters()`, for example?
And why guarantee that it executes in the "current OS thread"? I would say you don't want to specify where it executes exactly, as it opens the door for more sophisticated implementations (such as automatic assignment of subinterpreters inside a pool of threads).
Again, I had explained this poorly in the PEP. The key thing here is that subinterpreters don't do anything special relative to threading. If you want to call "Interpreter.run()" in a thread then you stick it in a "threading.Thread". If you want to auto-assign to a pool of threads then you treat it like any other function you would auto-assign to a pool of threads.
Makes sense, thanks. Perhaps you want to mention that example quickly in the PEP? (or perhaps not)
Has any thought been given to how FIFOs could integrate with async code driven by an event loop (e.g. asyncio)? I think the model of executing several asyncio (or Tornado) applications each in their own subinterpreter may prove quite interesting to reconcile multi-core concurrency with ease of programming. That would require the FIFOs to be able to synchronize on something an event loop can wait on (probably a file descriptor?).
Personally I've given pretty much no thought to the relationship with async. TBH, the FIFO parts of the PEP were added only recently and haven't fully baked yet. I'd be interested more feedback on async relative to PEP, not just with the FIFO bits; my experience with async is pretty limited thus far.
I think the FIFO is the primary point where interaction with event loop would prove useful, since that's how you would communicate and synchronize with other interpreters.
Actually, even waiting on an interpreter's state (e.g. running / idle) could use a "hidden" FIFO sending the right messages (for example the strings "running" and "idle", for a crude proof-of-concept).
Regards
Antoine.
On Sep 8, 2017 4:06 PM, "Eric Snow" ericsnowcurrently@gmail.com wrote:
run(code):
Run the provided Python code in the interpreter, in the current OS thread. If the interpreter is already running then raise RuntimeError in the interpreter that called ``run()``.
The current interpreter (which called ``run()``) will block until the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.
This phrase "bubble up" here is doing a lot of work :-). Can you elaborate on what you mean? The text now makes it seem like the exception will just pass from one interpreter into another, but that seems impossible – it'd mean sharing not just arbitrary user defined exception classes but full frame objects...
-n
On 10 September 2017 at 04:04, Nathaniel Smith njs@pobox.com wrote:
On Sep 8, 2017 4:06 PM, "Eric Snow" ericsnowcurrently@gmail.com wrote:
run(code):
Run the provided Python code in the interpreter, in the current OS thread. If the interpreter is already running then raise RuntimeError in the interpreter that called ``run()``. The current interpreter (which called ``run()``) will block until the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.This phrase "bubble up" here is doing a lot of work :-). Can you elaborate on what you mean? The text now makes it seem like the exception will just pass from one interpreter into another, but that seems impossible – it'd mean sharing not just arbitrary user defined exception classes but full frame objects...
Indeed, I think this will need to be something more akin to https://docs.python.org/3/library/subprocess.html#subprocess.CalledProcessEr..., where the child interpreter is able to pass back encoded text data (perhaps including a full rendered traceback), but the exception itself won't be able to be propagated.
Cheers, Nick.
On Mon, Sep 11, 2017 at 8:51 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 10 September 2017 at 04:04, Nathaniel Smith njs@pobox.com wrote:
On Sep 8, 2017 4:06 PM, "Eric Snow" ericsnowcurrently@gmail.com wrote:
run(code):
Run the provided Python code in the interpreter, in the current OS thread. If the interpreter is already running then raise RuntimeError in the interpreter that called ``run()``. The current interpreter (which called ``run()``) will block until the subinterpreter finishes running the requested code. Any uncaught exception in that code will bubble up to the current interpreter.This phrase "bubble up" here is doing a lot of work :-). Can you
elaborate
on what you mean? The text now makes it seem like the exception will just pass from one interpreter into another, but that seems impossible – it'd mean sharing not just arbitrary user defined exception classes but full frame objects...
Indeed, I think this will need to be something more akin to https://docs.python.org/3/library/subprocess.html# subprocess.CalledProcessError, where the child interpreter is able to pass back encoded text data (perhaps including a full rendered traceback), but the exception itself won't be able to be propagated.
It would be helpful if at least the exception type could somehow be preserved / restored / mimicked. Otherwise you need if-elif statements in your try-excepts and other annoying stuff.
-- Koos
On Sat, Sep 9, 2017 at 11:04 AM, Nathaniel Smith njs@pobox.com wrote:
This phrase "bubble up" here is doing a lot of work :-). Can you elaborate on what you mean? The text now makes it seem like the exception will just pass from one interpreter into another, but that seems impossible
I've updated the PEP to clarify.
it'd mean sharing not just arbitrary user defined exception classes but full frame objects...
My current implementation does the right thing currently. However, it would probably struggle under a more tightly controlled inter-interpreter boundary. I'll take a look and also update the PEP.
-eric
participants (9)
-
Antoine Pitrou -
Eric Snow -
Guido van Rossum -
Koos Zevenhoven -
Michel Desmoulin -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Stefan Krah