[Python-Dev] PEP 554 v2 (new "interpreters" module)

[Nick Coghlan]

On 9 September 2017 at 01:04, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 September 2017 at 00:04, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>    add_recv_fifo(name=None):
>>
>>       Create a new FIFO, associate the two ends with the involved
>>       interpreters, and return the side associated with the interpreter
>>       in which "add_recv_fifo()" was called.  A FIFOReader gets tied to
>>       this interpreter.  A FIFOWriter gets tied to the interpreter that
>>       called "add_recv_fifo()".
>>
>>       The FIFO's name is set to the provided value.  If no name is
>>       provided then a dynamically generated one is used.  If a FIFO
>>       with the given name is already associated with this interpreter
>>       (or with the one in which "add_recv_fifo()" was called) then raise
>>       KeyError.
>>
>>    add_send_fifo(name=None):
>>
>>       Create a new FIFO, associate the two ends with the involved
>>       interpreters, and return the side associated with the interpreter
>>       in which "add_recv_fifo()" was called.  A FIFOWriter gets tied to
>>       this interpreter.  A FIFOReader gets tied to the interpreter that
>>       called "add_recv_fifo()".
>>
>>       The FIFO's name is set to the provided value.  If no name is
>>       provided then a dynamically generated one is used.  If a FIFO
>>       with the given name is already associated with this interpreter
>>       (or with the one in which "add_send_fifo()" was called) then raise
>>       KeyError.
>
> Personally, I *always* read these names backwards - from the POV of
> the caller. So when I see "add_send_fifo", I then expect to be able to
> send stuff on the returned FIFO (i.e., I get a writer back). But
> that's not how it works.

I had the same problem with the current names: as a message sender, I
expect to request a send queue, as a message receiver, I expect to
request a receive queue.

Having to request the opposite of what I want to do next seems backwards:

    send_fifo = other_interpreter.add_recv_fifo(__name__) # Wut?
    send_fifo.push(b"Hello!")

I think it would be much clearer if the API looked like this for an
inline subinterpreter invocation.:

    # Sending messages
    other_interpreter = interpreters.create()
    send_fifo = other_interpreter.get_send_fifo(__name__)
    send_fifo.push(b"Hello!")
    send_fifo.push(b"World!")
    send_fifo.push(None)

    # Receiving messages
    receiver_code = textwrap.dedent(f"""
    import interpreters
    recv_fifo = interpreters.get_current().get_recv_fifo({__name__})
    while True:
        msg = recv_fifo.pop()
        if msg is None:
            break
        print(msg)
    """)
    other_interpreter.run(receiver_code)

To enable concurrent communication between the sender & receiver,
you'd currently need to move the "other_interpreter.run()" call out to
a separate thread, but I think that's fine for now.

The rules for get_recv_fifo() and get_send_fifo() would be:

- named fifos are created as needed and always stored on the
*receiving* interpreter
- you can only call get_recv_fifo() on the currently running
interpreter: other interpreters can't access your receive fifos
- get_recv_fifo() never fails - the receiving interpreter can have as
many references to a receive FIFO as it likes
- get_send_fifo() can fail if the named fifo already exists on the
receiving interpreter and the designated sender is an interpreter
other than the one calling get_send_fifo()
- interpreters are free to use the named FIFO mechanism to send
messages to themselves

To immediately realise some level of efficiency benefits from the
shared memory space between the main interpreter and subinterpreters,
I also think these low level FIFOs should be defined as accepting any
object that supports the PEP 3118 buffer protocol, and emitting
memoryview() objects on the receiving end, rather than being bytes-in,
bytes-out.

Such a memoryview based primitive can then potentially be expanded in
the future to support additional more-efficient-than-multiprocessing
serailisation based protocols, where the sending interpreter
serialises into a region of memory, shares that region with the
subinterpreter via a FIFO memoryview, and then the subinterpreter
deserialises directly from the sending interpreter's memory region,
without the serialised form ever needing to be streamed or copied
anywhere (which is much harder to avoid when coordinating across
multiple operating system processes).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia