On Sat, Jun 20, 2015 at 11:25 PM, Nathaniel Smith <njs@pobox.com> wrote:
I'd love to see just a hand wavy, verbal proof-of-concept walking through how this might work in some simple but realistic case. To me a single compelling example could make this proposal feel much more concrete and achievable.
Here's a vague example: ------------------ from subinterpreters import Subinterpreter, Channel def handle_job(val): if not isinstance(val, (int, float)): raise RuntimeError("{!r} not a valid arg".format(val)) # something potentially expensive... def runner(ch): while True: value = ch.pop() # blocks if value is None: break handle_job(value) ch = Channel() sub = Subinterpreter() task = sub.run(runner, ch) data = get_data() for immutable_item in data: ch.push(immutable_item) if task.is_alive(): ch.push(None) task.join() exc = task.exception() if exc is not None: raise RuntimeError from exc def verify(data): # make sure runner did its job ... task = sub.run(verify, data) # do other stuff while we wait task.join() sub.destroy() ------------------
There aren't really many options for mutable objects, right? If you want shared nothing semantics, then transmitting a mutable object either needs to make a copy, or else be a real transfer, where the sender no longer has it (cf. Rust).
I guess for the latter you'd need some new syntax for send-and-del, that requires the object to be self contained (all mutable objects reachable from it are only referenced by each other) and have only one reference in the sending process (which is the one being sent and then destroyed).
Right. The idea of a self-contained object graph is something we'd need if we went that route. That's why initially we should focus on sharing only immutable objects.
Keep in mind that by "immutability" I'm talking about *really* immutable, perhaps going so far as treating the full memory space associated with an object as frozen. For instance, we'd have to ensure that "immutable" Python objects like strings, ints, and tuples do not change (i.e. via the C API).
This seems like a red herring to me. It's already the case that you can't legally use the c api to mutate tuples, ints, for any object that's ever been, say, passed to a function. So for these objects, the subinterpreter setup doesn't actually add any new constraints on user code.
Fair enough.
C code is always going to be *able* to break memory safety so long as you're using shared-memory threading at the c level to implement this stuff. We just need to make it easy not to.
Exactly.
Refcnts and garbage collection are another matter, of course.
Agreed. :) -eric