On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
>
>
> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
> >
> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
> > >
> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > > exposing them in Python, and adding a mechanism to safely share
> > > objects between them.
> >
> > This all sounds really cool if you can pull it off, and shared-nothing threads do seem like the least impossible model to pull off.
>
> Agreed.
>
> > But "least impossible" and "possible" are different :-). From your email I can't tell whether this plan is viable while preserving backcompat and memory safety.
>
> I agree that those issues must be clearly solved in the proposal before it can be approved.  I'm confident the approach I'm pursuing will afford us the necessary guarantees.  I'll address those specific points directly when I can sit down and organize my thoughts.

I'd love to see just a hand wavy, verbal proof-of-concept walking through how this might work in some simple but realistic case. To me a single compelling example could make this proposal feel much more concrete and achievable.

> > Suppose I have a queue between two subinterpreters, and on this queue I place a list of dicts of user-defined-in-python objects, each of which holds a reference to a user-defined-via-the-C-api object. What happens next?
>
> You've hit upon exactly the trickiness involved and why I'm thinking the best approach initially is to only allow *strictly* immutable objects to pass between interpreters.  Admittedly, my description of channels is very vague.:)  There are a number of possibilities with them that I'm still exploring (CSP has particular opinions...), but immutability is a characteristic that may provide the simplest *initial* approach.  Going that route shouldn't preclude adding some sort of support for mutable objects later.

There aren't really many options for mutable objects, right? If you want shared nothing semantics, then transmitting a mutable object either needs to make a copy, or else be a real transfer, where the sender no longer has it (cf. Rust).

I guess for the latter you'd need some new syntax for send-and-del, that requires the object to be self contained (all mutable objects reachable from it are only referenced by each other) and have only one reference in the sending process (which is the one being sent and then destroyed).

> Keep in mind that by "immutability" I'm talking about *really* immutable, perhaps going so far as treating the full memory space associated with an object as frozen.  For instance, we'd have to ensure that "immutable" Python objects like strings, ints, and tuples do not change (i.e. via the C API). 

This seems like a red herring to me. It's already the case that you can't legally use the c api to mutate tuples, ints, for any object that's ever been, say, passed to a function. So for these objects, the subinterpreter setup doesn't actually add any new constraints on user code.

C code is always going to be *able* to break memory safety so long as you're using shared-memory threading at the c level to implement this stuff. We just need to make it easy not to.

Refcnts and garbage collection are another matter, of course.

-n