[Python-Dev] PEP 554 v2 (new "interpreters" module)

[Nick Coghlan]

On 13 September 2017 at 07:43, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>> ``list()``::
>>
>> It's called ``enumerate()`` in the threading module.  Not sure there's
>> a point in choosing a different name here.
>
> Yeah, in the first version of the PEP it was called "enumerate()".  I
> changed it to "list()" at Raymond's recommendation.  The reasoning is
> that it's less confusing to most people that way.  TBH, I'd rather
> leave it "list()", but could be swayed.  Perhaps it would be enough
> for the PEP to not mention any relationship to "threading"?

I think you should just say that you're not using
interpreters.enumerate() because that conflicts with the enumerate
builtin.

However, replacing that with a *different* naming conflict with an
even more commonly used builtin wouldn't be a good idea either :)

I also think the "list interpreters" question is more subtle than you
think, as there are two potential sets of interpreters to consider:

1. Getting a listing of all interpreters in the process (creating
Interpreter objects in the current interpreter for each of them)
2. Getting a listing of all other interpreters created by calling
interpreters.create_interpreter() in *this* interpreter

Since it's not clear yet whether or not we're actually going to track
the metadata needed to report the latter, I'd suggest only offering
"interpreters.list_all()" initially, and requiring folks to do their
own tracking if they want a record of the interpreters their own code
has created.


>>>       The current interpreter (which called ``run()``) will block
>>> until the subinterpreter finishes running the requested code.  Any
>>>       uncaught exception in that code will bubble up to the current
>>>       interpreter.
>>
>> Why does it block?  How is concurrency supposed to be achieved in that
>> model?  It would be more flexible if run(code) returned an object that
>> can later be waited on.  Something like... a Future :-)
>
> I expect this is more a problem with my description than with the
> feature. :)  I've already re-written this bit to be more clear.  It's
> not that the thread blocks.  It's more like a function call, where the
> current frame is paused while the call is executed.  Then it returns
> to the calling frame.  Likewise the interpreter in the current thread
> gets swapped out with the target interpreter, where the code gets run,
> and then the original interpreter gets swapped back in.  This is how
> you do it in the C-API and it made sense (to me) to do it the same way
> in Python.

I'm not sure if it would create more confusion than it would resolve,
but I'm wondering if it may be worth including an example where you
delegate the subinterpreter call to another thread via
concurrent.futures.ThreadExecutor.

>>>    get_fifo(name):
>>>    list_fifos():
>>
>> If fifos are uniquely named, why not return a name->fifo mapping?
>
> I suppose we could.  Then we could get rid of "get_fifo()" too.  I'm
> still mulling over the right API for the FIFO parts of the PEP.

The FIFO parts of the proposal are feeling a lot like the Mailbox
objects in TI-RTOS to me, just with a more user-friendly API since
we're dealing with Python rather than C:
http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/sysbios/6_35_01_29/exports/bios_6_35_01_29/docs/cdoc/ti/sysbios/knl/Mailbox.html

>>> ``FIFOReader(name)``::
>>> [...]
>>
>> I don't think the method naming choice is very adequate here.  The API
>> model for the FIFO objects can either be a (threading or
>> multiprocessing) Queue or a multiprocessing Pipe.
>>
>> - if a Queue, then it should have a get() / put() pair of methods
>> - if a Pipe, then it should have a recv() / send() pair of methods
>>
>> Now, since Queues are multi-producer multi-consumer, while Pipes are
>> single-producer single-consumer (they aren't "synchronized"), the
>> better analogy seems to the multiprocessing Pipe here, so I would vote
>> for recv() / send().
>>
>> But, in any case, definitely not a pop() / push() pair.
>
> Thanks for pointing that out.  Prior art, FTW!  I'll factor that in.

+1 from me for treating this as a split Pipe model, where the SendPipe
and RecvPipe are distinct objects (since they'll exist in different
interpreters).

>> Has any thought been given to how FIFOs could integrate with async code
>> driven by an event loop (e.g. asyncio)?  I think the model of executing
>> several asyncio (or Tornado) applications each in their own
>> subinterpreter may prove quite interesting to reconcile multi-core
>> concurrency with ease of programming.  That would require the FIFOs to
>> be able to synchronize on something an event loop can wait on (probably
>> a file descriptor?).
>
> Personally I've given pretty much no thought to the relationship with
> async.  TBH, the FIFO parts of the PEP were added only recently and
> haven't fully baked yet.  I'd be interested more feedback on async
> relative to PEP, not just with the FIFO bits; my experience with async
> is pretty limited thus far.

The way TI-RTOS handles this is to allow event objects to be passed in
that are used to report a "Mailbox not empty" event (when a new
message is queued) and a "Mailbox not full" event (when the Mailbox
was previously full, and a message is read). Code can then either
block on the mailbox itself (using Mailbox_pend), or else wait on the
underlying events directly.

It seems to me that a similar model would work here (and wouldn't
necessarily need to be included in the initial version of the PEP): a
SendPipe could accept an optional Event object that it calls set() on
when the pipe ceases to be full, and a RecvPipe could do the same for
when it ceases to be empty.

It would then be up to the calling code to decide whether to pass in a
regular synchronous threading.Event() object (in which case it could
probably just make blocking calls to the send()/recv() methods
instead), or else to pass in asyncio.Event() objects (which a
coroutine can wait on via the Event.wait() coroutine)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia