
On 9 September 2017 at 01:04, Paul Moore <p.f.moore@gmail.com> wrote:
On 9 September 2017 at 00:04, Eric Snow <ericsnowcurrently@gmail.com> wrote:
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOReader gets tied to this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_recv_fifo()" was called) then raise KeyError.
add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter in which "add_recv_fifo()" was called. A FIFOWriter gets tied to this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter (or with the one in which "add_send_fifo()" was called) then raise KeyError.
Personally, I *always* read these names backwards - from the POV of the caller. So when I see "add_send_fifo", I then expect to be able to send stuff on the returned FIFO (i.e., I get a writer back). But that's not how it works.
I had the same problem with the current names: as a message sender, I expect to request a send queue, as a message receiver, I expect to request a receive queue. Having to request the opposite of what I want to do next seems backwards: send_fifo = other_interpreter.add_recv_fifo(__name__) # Wut? send_fifo.push(b"Hello!") I think it would be much clearer if the API looked like this for an inline subinterpreter invocation.: # Sending messages other_interpreter = interpreters.create() send_fifo = other_interpreter.get_send_fifo(__name__) send_fifo.push(b"Hello!") send_fifo.push(b"World!") send_fifo.push(None) # Receiving messages receiver_code = textwrap.dedent(f""" import interpreters recv_fifo = interpreters.get_current().get_recv_fifo({__name__}) while True: msg = recv_fifo.pop() if msg is None: break print(msg) """) other_interpreter.run(receiver_code) To enable concurrent communication between the sender & receiver, you'd currently need to move the "other_interpreter.run()" call out to a separate thread, but I think that's fine for now. The rules for get_recv_fifo() and get_send_fifo() would be: - named fifos are created as needed and always stored on the *receiving* interpreter - you can only call get_recv_fifo() on the currently running interpreter: other interpreters can't access your receive fifos - get_recv_fifo() never fails - the receiving interpreter can have as many references to a receive FIFO as it likes - get_send_fifo() can fail if the named fifo already exists on the receiving interpreter and the designated sender is an interpreter other than the one calling get_send_fifo() - interpreters are free to use the named FIFO mechanism to send messages to themselves To immediately realise some level of efficiency benefits from the shared memory space between the main interpreter and subinterpreters, I also think these low level FIFOs should be defined as accepting any object that supports the PEP 3118 buffer protocol, and emitting memoryview() objects on the receiving end, rather than being bytes-in, bytes-out. Such a memoryview based primitive can then potentially be expanded in the future to support additional more-efficient-than-multiprocessing serailisation based protocols, where the sending interpreter serialises into a region of memory, shares that region with the subinterpreter via a FIFO memoryview, and then the subinterpreter deserialises directly from the sending interpreter's memory region, without the serialised form ever needing to be streamed or copied anywhere (which is much harder to avoid when coordinating across multiple operating system processes). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia