[Python-ideas] solving multi-core Python

[Nathaniel Smith]

On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>
>
> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >
> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com>
wrote:
> > >
> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > > exposing them in Python, and adding a mechanism to safely share
> > > objects between them.
> >
> > This all sounds really cool if you can pull it off, and shared-nothing
threads do seem like the least impossible model to pull off.
>
> Agreed.
>
> > But "least impossible" and "possible" are different :-). From your
email I can't tell whether this plan is viable while preserving backcompat
and memory safety.
>
> I agree that those issues must be clearly solved in the proposal before
it can be approved.  I'm confident the approach I'm pursuing will afford us
the necessary guarantees.  I'll address those specific points directly when
I can sit down and organize my thoughts.

I'd love to see just a hand wavy, verbal proof-of-concept walking through
how this might work in some simple but realistic case. To me a single
compelling example could make this proposal feel much more concrete and
achievable.

> > Suppose I have a queue between two subinterpreters, and on this queue I
place a list of dicts of user-defined-in-python objects, each of which
holds a reference to a user-defined-via-the-C-api object. What happens next?
>
> You've hit upon exactly the trickiness involved and why I'm thinking the
best approach initially is to only allow *strictly* immutable objects to
pass between interpreters.  Admittedly, my description of channels is very
vague.:)  There are a number of possibilities with them that I'm still
exploring (CSP has particular opinions...), but immutability is a
characteristic that may provide the simplest *initial* approach.  Going
that route shouldn't preclude adding some sort of support for mutable
objects later.

There aren't really many options for mutable objects, right? If you want
shared nothing semantics, then transmitting a mutable object either needs
to make a copy, or else be a real transfer, where the sender no longer has
it (cf. Rust).

I guess for the latter you'd need some new syntax for send-and-del, that
requires the object to be self contained (all mutable objects reachable
from it are only referenced by each other) and have only one reference in
the sending process (which is the one being sent and then destroyed).

> Keep in mind that by "immutability" I'm talking about *really* immutable,
perhaps going so far as treating the full memory space associated with an
object as frozen.  For instance, we'd have to ensure that "immutable"
Python objects like strings, ints, and tuples do not change (i.e. via the C
API).

This seems like a red herring to me. It's already the case that you can't
legally use the c api to mutate tuples, ints, for any object that's ever
been, say, passed to a function. So for these objects, the subinterpreter
setup doesn't actually add any new constraints on user code.

C code is always going to be *able* to break memory safety so long as
you're using shared-memory threading at the c level to implement this
stuff. We just need to make it easy not to.

Refcnts and garbage collection are another matter, of course.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/c2b41fe0/attachment.html>