[Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry Scott]

> On 18 Jul 2018, at 08:21, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> Op 18 jul. 2018 om 08:02 heeft Barry <barry at barrys-emacs.org <mailto:barry at barrys-emacs.org>> het volgende geschreven:
> 
>> 
>> 
>>>> On 17 Jul 2018, at 21:00, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>>>> 
>>>> On Tue, Jul 17, 2018 at 1:44 PM Barry <barry at barrys-emacs.org> wrote:
>>>> The decrement itself is not the problem, that can be made thread safe.
>>> 
>>> Yeah, by using the GIL. <wink>  Otherwise, please elaborate.  My
>>> understanding is that if the decrement itself were not the problem
>>> then we'd have gotten rid of the GIL already.
>> 
>> All processors have thread safe ways to inc and dec and test, integers without holding a lock.
>> 
>> That is the mechanism that locks themselves are built out of. You can use that to avoid holding the GIL until the ref count reaches 0.
>> 
>> In c++ they built it into the language with std::atomic_int, you would have to find the way to do this C, i don’t have an answer at my finger tips for C.
>> 
> Some past attempts at getting rid of the GIL used atomic inc/dec, and that resulted in bad performance because these instructions  aren’t cheap. 

Isn't this class of problem what leads to the per-processor caches and other optimisations in Linux kernel?
I wonder if kernel optimisations could be applied to this problem?

> 
> My gut feeling is that you’d have to get rid of refcounts to get high performance when getting rid of the GIL in a single interpreter, which would almost certainly result in breaking the C API.

Working on the ref count costs might be the enabling tech.

We already have the problem of unchanging objects being copied after a fork because of the ref counts being inside the object.
It was suggested that the ref count would have to move out of the object to help with this problem.

If there is a desirable solution to the parallel problem we can think about the C API migration problem.

Barry

>  
> 
> Ronald
>> Barry
>> 
>>> 
>>>> Do you mean that once the ref reaches 0 you have to make the delete happen on the original interpreter?
>>> 
>>> Yep.  For one thing, GC can trigger __del__, which can do anything,
>>> including modifying other objects from the original interpreter (incl.
>>> decref'ing them).  __del__ should be run under the original
>>> interpreter.  For another thing, during GC containers often decref
>>> their items.  Also, separating the GIL between interpreters may mean
>>> we'll need an allocator per interpreter.  In that case the
>>> deallocation must happen relative to the interpreter where the object
>>> was allocated.
>>> 
>>> -eric
>>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org <mailto:Python-ideas at python.org>
>> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
>> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180718/f7b350ef/attachment-0001.html>