[Python-Dev] PEP 554 v3 (new interpreters module)

[Steve Dower]

On 03Oct2017 0755, Antoine Pitrou wrote:
> On Tue, 3 Oct 2017 08:36:55 -0600
> Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> On Tue, Oct 3, 2017 at 5:00 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> On Mon, 2 Oct 2017 22:15:01 -0400
>>> Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>>>
>>>> I'm still not convinced that sharing synchronization primitives is
>>>> important enough to be worth including it in the PEP.  It can be added
>>>> later, or via an extension module in the meantime.  To that end, I'll
>>>> add a mechanism to the PEP for third-party types to indicate that they
>>>> can be passed through channels.  Something like
>>>> "obj.__channel_support__ = True".
>>>
>>> How would that work?  If it's simply a matter of flipping a bit, why
>>> don't we do it for all objects?
>>
>> The type would also have to be safe to share between interpreters. :)
> 
> But what does it mean to be safe to share, while the exact degree
> and nature of the isolation between interpreters (and also their
> concurrent execution) is unspecified?
> 
> I think we need a sharing protocol, not just a flag.

The easiest such protocol is essentially:

* an object can represent itself as bytes (e.g. generate a bytes object 
representing some global token, such as a kernel handle or memory address)
* those bytes are sent over the standard channel
* the object can instantiate itself from those bytes (e.g. wrap the 
existing handle, create a memoryview over the same block of memory, etc.)
* cross-interpreter refcounting is either ignored (because the kernel is 
refcounting the resource) or manual (by including more shared info in 
the token)

Since this is trivial to implement over the basic bytes channel, and 
doesn't even require a standard protocol except for convenience, Eric 
decided to avoid blocking the core functionality on this. I'm inclined 
to agree - get the basic functionality supported and let people build on 
it before we try to lock down something we don't fully understand yet.

About the only thing that seems to be worth doing up-front is some sort 
of pending-call callback mechanism between interpreters, but even that 
doesn't need to block the core functionality (you can do it trivially 
with threads and another channel right now, and there's always room to 
make something more efficient later).

There are plenty of smart people out there who can and will figure out 
the best way to design this. By giving them the tools and the ability to 
design something awesome, we're more likely to get something awesome 
than by committing to a complete design now. Right now, they're all 
blocked on the fact that subinterpreters are incredibly hard to start 
running, let alone experiment with. Eric's PEP will fix that part and 
enable others to take it from building blocks to powerful libraries.

Cheers,
Steve