[Python-ideas] solving multi-core Python

[Eric Snow]

On Sat, Jun 20, 2015 at 11:25 PM, Nathaniel Smith <njs at pobox.com> wrote:
> I'd love to see just a hand wavy, verbal proof-of-concept walking through
> how this might work in some simple but realistic case. To me a single
> compelling example could make this proposal feel much more concrete and
> achievable.

Here's a vague example:

------------------

from subinterpreters import Subinterpreter, Channel

def handle_job(val):
    if not isinstance(val, (int, float)):
        raise RuntimeError("{!r} not a valid arg".format(val))
    # something potentially expensive...

def runner(ch):
    while True:
        value = ch.pop()  # blocks
        if value is None:
            break
        handle_job(value)

ch = Channel()

sub = Subinterpreter()
task = sub.run(runner, ch)

data = get_data()
for immutable_item in data:
    ch.push(immutable_item)

if task.is_alive():
    ch.push(None)
    task.join()

exc = task.exception()
if exc is not None:
    raise RuntimeError from exc

def verify(data):
    # make sure runner did its job
    ...

task = sub.run(verify, data)
# do other stuff while we wait
task.join()

sub.destroy()

------------------

> There aren't really many options for mutable objects, right? If you want
> shared nothing semantics, then transmitting a mutable object either needs to
> make a copy, or else be a real transfer, where the sender no longer has it
> (cf. Rust).
>
> I guess for the latter you'd need some new syntax for send-and-del, that
> requires the object to be self contained (all mutable objects reachable from
> it are only referenced by each other) and have only one reference in the
> sending process (which is the one being sent and then destroyed).

Right.  The idea of a self-contained object graph is something we'd
need if we went that route.  That's why initially we should focus on
sharing only immutable objects.

>
>> Keep in mind that by "immutability" I'm talking about *really* immutable,
>> perhaps going so far as treating the full memory space associated with an
>> object as frozen.  For instance, we'd have to ensure that "immutable" Python
>> objects like strings, ints, and tuples do not change (i.e. via the C API).
>
> This seems like a red herring to me. It's already the case that you can't
> legally use the c api to mutate tuples, ints, for any object that's ever
> been, say, passed to a function. So for these objects, the subinterpreter
> setup doesn't actually add any new constraints on user code.

Fair enough.

>
> C code is always going to be *able* to break memory safety so long as you're
> using shared-memory threading at the c level to implement this stuff. We
> just need to make it easy not to.

Exactly.

>
> Refcnts and garbage collection are another matter, of course.

Agreed. :)

-eric