<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 6 October 2017 at 11:48, Eric Snow <span dir="ltr"><<a href="mailto:ericsnowcurrently@gmail.com" target="_blank">ericsnowcurrently@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> And that's the real pay-off that comes from defining this in terms of the<br>
> memoryview protocol: Py_buffer structs *aren't* Python objects, so it's only<br>
> a regular C struct that gets passed across the interpreter boundary (the<br>
> reference to the original objects gets carried along passively as part of<br>
> the CIV - it never gets *used* in the receiving interpreter).<br>
<br>
</span>Yeah, the (PEP 3118) buffer protocol offers precedent in a number of<br>
ways that are applicable to channels here. I'm simply reticent to<br>
lock PEP 554 into such a specific solution as the buffer-specific CIV.<br>
I'm trying to accommodate anticipated future needs while keeping the<br>
PEP as simple and basic as possible. It's driving me nuts! :P Things<br>
were *much* simpler before I added Channels to the PEP. :)<br></blockquote><div><br></div><div>Starting with memory-sharing only doesn't lock us into anything, since you can still add a more flexible kind of channel based on a different protocol later if it turns out that memory sharing isn't enough.</div><div><br></div><div>By contrast, if you make the initial channel semantics incompatible with multiprocessing by design, you *will* prevent anyone from experimenting with replicating the shared memory based channel API for communicating between processes :)</div><div><br></div><div>That said, if you'd prefer to keep the "Channel" name available for the possible introduction of object channels at a later date, you could call the initial memoryview based channel a "MemChannel".</div> <span class=""></span><br><div><span class=""></span></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> I don't think we should be touching the behaviour of core builtins solely to<br>
> enable message passing to subinterpreters without a shared GIL.<br>
<br>
</span>Keep in mind that I included the above as a possible solution using<br>
tp_share() that would work *after* we stop sharing the GIL. My point<br>
is that with tp_share() we have a solution that works now *and* will<br>
work later. I don't care how we use tp_share to do so. :) I long to<br>
be able to say in the PEP that you can pass bytes through the channel<br>
and get bytes on the other side.<br></blockquote><div><br></div><div>Memory views are a builtin type as well, and they emphasise the practical benefit we're trying to get relative to typical multiprocessing arranagements: zero-copy data sharing.</div><div><br></div><div>So here's my proposed experimentation-enabling development strategy:</div><div><br></div><div>1. Start out with a MemChannel API, that accepts any buffer-exporting object as input, and outputs only a cross-interpreter memoryview subclass</div><div>2. Use that as the basis for the work to get to a per-interpreter locking arrangement that allows subinterpreters to fully exploit multiple CPUs</div><div>3. Only then try to design a Channel API that allows for sharing builtin immutable objects between interpreters (bytes, strings, numbers), at a time when you can be certain you won't be inadvertently making it harder to make the GIL a truly per-interpreter lock, rather than the current process global runtime lock.<br></div><div><br></div><div>The key benefit of this approach is that we *know* MemChannel can work: the buffer protocol already operates at the level of C structs and pointers, not Python objects, and there are already plenty of interesting buffer-protocol-supporting objects around, so as long as the CIV switches interpreters at the right time, there aren't any fundamentally new runtime level capabilities needed to implement it.<br></div><div><br></div><div>The lower level MemChannel API could then also be replicated for multiprocessing, while the higher level more speculative object-based Channel API would be specific to subinterpreters (and probably only ever designed and implemented if you first succeed in making subinterpreters sufficiently independent that they don't rely on a process-wide GIL any more).</div><div><br></div><div>So I'm not saying "Never design an object-sharing protocol specifically for use with subinterpreters". I'm saying "You don't have a demonstrated need for that yet, so don't try to define it until you do".<br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My mind is drawn to the comparison between that and the question of<br>
CIV vs. tp_share(). CIV would be more like the post-451 import world,<br>
where I expect the CIV would take care of the data sharing operations.<br>
That said, the situation in PEP 554 is sufficiently different that I'm<br>
not convinced a generic CIV protocol would be better. I'm not sure<br>
how much CIV could do for you over helpers+tp_share.<br>
<br>
Anyway, here are the leading approaches that I'm looking at now:<br>
<br>
* adding a tp_share slot<br>
+ you send() the object directly and recv() the object coming out of<br>
tp_share()<br>
(which will probably be the same type as the original)<br>
+ this would eventually require small changes in tp_free for<br>
participating types<br>
+ we would likely provide helpers (eventually), similar to the new<br>
buffer protocol,<br>
to make it easier to manage sharing data<br></blockquote><div><br></div><div>I'm skeptical about this approach because you'll be designing in a vacuum against future possible constraints that you can't test yet: the inherent complexity in the object sharing protocol will come from *not* having a process-wide GIL, but you'll be starting out with a process-wide GIL still in place. And that means third parties will inevitably rely on the process-wide GIL in their tp_share implementations (despite their best intentions), and you'll end up with the same issue that causes problems for the rest of the C API.</div><div><br></div><div>By contrast, if you delay this step until *after* the GIL has successfully been shifted to being per-interpreter, then by the time the new protocol is defined, people will also be able to test their tp_share implementations properly.</div><div><br></div><div>At that point, you'd also presumably have evidence of demand to justify the introduction of a new core language protocol, as:</div><div><br></div><div>* folks will only complain about the limitations of MemChannel if they're actually using subinterpreters</div><div>* the complaints about the limitations of MemChannel would help guide the object sharing protocol design<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* simulating tp_share via an external global registry (or a registry<br>
on the Channel type)<br>
+ it would still be hard to make work without hooking into tp_free()<br>
* CIVs hard-coded in Channel (or BufferViewChannel, etc.) for specific<br>
types (e.g. buffers)<br>
+ you send() the object like normal, but recv() the view<br>
* a CIV protocol on Channel by which you can add support for more types<br>
+ you send() the object like normal but recv() the view<br>
+ could work through subclassing or a registry<br>
+ a lot of conceptual similarity with tp_share+tp_free<br>
* a CIV-like proxy<br>
+ you wrap the object, send() the proxy, and recv() a proxy<br>
+ this is entirely compatible with tp_share()<br></blockquote><div><br></div><div>* Allow for multiple channel types, such that MemChannel is merely the *first* channel type, rather than the *only* channel type</div><div> + Allows PEP 554 to be restricted to things we already know can be made to work</div><div> + Doesn't block the introduction of an object-sharing based Channel in some future release</div><div> + Allows for at least some channel types to be adapted for use with shared memory and multiprocessing<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Here are what I consider the key metrics relative to the utility of a<br>
solution (not in any significant order):<br>
<br>
* how hard to understand as a Python programmer?<br></blockquote><div><br></div><div>Not especially important yet - this is more a criterion for the final API, not the initial experimental platform.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* how much extra work (if any) for folks calling Channel.send()?<br>
* how much extra work (if any) for folks calling Channel.recv()?<br></blockquote><div><br></div><div>I don't think either are particularly important yet, although we also don't want to raise any pointless barriers to experimentation.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* how complex is the CPython implementation?<br></blockquote><div><br></div><div>This is critical, since we want to minimise any potential for undesirable side effects on regular single interpreter code.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* how hard to understand as a type author (wanting to add support for<br>
their type)?<br>
* how hard to add support for a new type?<br>
* what variety of types could be supported?<br>
* what breadth of experimentation opens up?<br></blockquote><div><br></div><div>You missed the big one: what risk does the initial channel design pose to the underlying objective of making the GIL a genuinely per-interpreter lock?</div><div><br></div><div>If we don't eventually reach the latter goal, then subinterpreters won't really offer much in the way of compelling benefits over just using a thread pool and queue.Queue.<br></div><div><br></div><div>MemChannel poses zero additional risk to that, since we wouldn't be sharing actual Python objects between interpreters, only C pointers and structs.</div><div><br></div><div>By contrast, introducing an object channel early poses significant new risks to that goal, since it will force you to solve hard protocol design and refcount management problems *before* making the switch, rather than being able to defer the design of the object channel protocol until *after* you've already enabled the ability to run subinterpreters in completely independent threads.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The most important thing to me is keeping things simple for Python<br>
programmers. After that is ease-of-use for type authors. However, I<br>
also want to put us in a good position in 3.7 to experiment<br>
extensively with subinterpreters, so that's a big consideration.<br>
<br>
Consequently, for PEP 554 my goal is to find a solution for object<br>
sharing that keeps things simple in Python while laying a basic<br>
foundation we can build on at the C level, so we don't get locked in<br>
but still maximize our opportunities to experiment. :)<br></blockquote><div><br></div><div>I think our priorities are quite different then, as I believe PEP 554 should be focused on defining a relatively easy to implement API that nevertheless makes it possible to write interesting programs while working on the goal of making the GIL per-interpreter, without worrying too much about whether or not the initial cross-interpreter communication channels closely resemble the final ones that will be intended for more general use.</div><div><br></div><div>Cheers,</div><div>Nick.<br></div></div><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Nick Coghlan | <a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a> | Brisbane, Australia</div>
</div></div>