[Python-ideas] solving multi-core Python

Eric Snow ericsnowcurrently at gmail.com
Wed Jun 24 04:18:36 CEST 2015

On Sat, Jun 20, 2015 at 11:25 PM, Nathaniel Smith <njs at pobox.com> wrote:
> I'd love to see just a hand wavy, verbal proof-of-concept walking through
> how this might work in some simple but realistic case. To me a single
> compelling example could make this proposal feel much more concrete and
> achievable.

Here's a vague example:


from subinterpreters import Subinterpreter, Channel

def handle_job(val):
    if not isinstance(val, (int, float)):
        raise RuntimeError("{!r} not a valid arg".format(val))
    # something potentially expensive...

def runner(ch):
    while True:
        value = ch.pop()  # blocks
        if value is None:

ch = Channel()

sub = Subinterpreter()
task = sub.run(runner, ch)

data = get_data()
for immutable_item in data:

if task.is_alive():

exc = task.exception()
if exc is not None:
    raise RuntimeError from exc

def verify(data):
    # make sure runner did its job

task = sub.run(verify, data)
# do other stuff while we wait



> There aren't really many options for mutable objects, right? If you want
> shared nothing semantics, then transmitting a mutable object either needs to
> make a copy, or else be a real transfer, where the sender no longer has it
> (cf. Rust).
> I guess for the latter you'd need some new syntax for send-and-del, that
> requires the object to be self contained (all mutable objects reachable from
> it are only referenced by each other) and have only one reference in the
> sending process (which is the one being sent and then destroyed).

Right.  The idea of a self-contained object graph is something we'd
need if we went that route.  That's why initially we should focus on
sharing only immutable objects.

>> Keep in mind that by "immutability" I'm talking about *really* immutable,
>> perhaps going so far as treating the full memory space associated with an
>> object as frozen.  For instance, we'd have to ensure that "immutable" Python
>> objects like strings, ints, and tuples do not change (i.e. via the C API).
> This seems like a red herring to me. It's already the case that you can't
> legally use the c api to mutate tuples, ints, for any object that's ever
> been, say, passed to a function. So for these objects, the subinterpreter
> setup doesn't actually add any new constraints on user code.

Fair enough.

> C code is always going to be *able* to break memory safety so long as you're
> using shared-memory threading at the c level to implement this stuff. We
> just need to make it easy not to.


> Refcnts and garbage collection are another matter, of course.

Agreed. :)


More information about the Python-ideas mailing list