[Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry]

> On 17 Jul 2018, at 20:35, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> 
>> On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Mon, 16 Jul 2018 18:00:37 +0100
>> MRAB <python at mrabarnett.plus.com> wrote:
>>> Could you explicitly share an object in a similar way to how you
>>> explicitly open a file?
>>> 
>>> The shared object's refcount would be incremented and the sharing
>>> function would return a proxy to the shared object.
>>> 
>>> Refcounting in the thread/process would be done on the proxy.
>>> 
>>> When the proxy is closed or garbage-collected, the shared object's
>>> refcount would be decremented.
>>> 
>>> The shared object could be garbage-collected when its refcount drops to
>>> zero.
>> 
>> Yes, I'm assuming that would be how shareable buffers could be
>> implemented: a per-interpreter proxy (with a regular Python refcount)
>> mediating access to a shared object (which could have an atomic /
>> thread-safe refcount).
> 
> Nice! That's exactly how I'm doing it. :)  The buffer protocol makes
> it easier, but the idea could apply to arbitrary objects generally.
> That's something I'll look into in a later phase of the project.
> 
> In both cases the tricky part is ensuring that the proxy does not
> directly mutate the object (especially the refcount).  In fact, the
> decref part above is the trickiest.  The trickiness is a consequence
> of our goals.  In my multi-core project we're aiming for not sharing
> the GIL between interpreters.  That means reaching and keeping proper
> separation between interpreters.  Notably, without a GIL shared by
> interpreters, refcount operations are not thread-safe.  Also, in the
> decref case GC would happen under the wrong interpreter (which is
> problematic for several reasons).
> 
> With this in mind, here's how I'm approaching the problem:
> 
> 1. interp A "shares" an object with interp B (e.g. through a channel)
>    * the object is incref'ed under A before it is sent to B
> 2. the object is wrapped in a proxy owned by B
>    * the proxy may not make C-API calls that would mutate the object
> or even cause an incref/decref
> 3. when the proxy is GC'd, the original object is decref'ed
>    * the decref must happen in a thread in which A is running
> 
> In order to make all this work the missing piece is a mechanism by
> which the decref (#3) happens under the original interpreter.  At the
> moment Emily Morehouse and I are pursuing an approach that extends the
> existing ceval "pending call" machinery currently used for handling
> signals (see Py_AddPendingCall).  The new [*private*] API would work
> the same way but on a per-interpreter basis rather than just the main
> interpreter.  This would allow one interpreter to queue up a decref to
> happen later under another interpreter.

The decrement itself is not the problem, that can be made thread safe.

Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?

Barry

> 
> FWIW, this ability to decref an object under a different interpreter
> is a blocker right now for a number of things, including supporting
> buffers in PEP 554 channels.
> 
> -eric
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>