[Python-ideas] solving multi-core Python

[Eric Snow]

On Sun, Jun 21, 2015 at 3:54 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 20 Jun 2015 23:01:20 -0600
> Eric Snow <ericsnowcurrently at gmail.com>
> wrote:
>> The only consequential shared piece is the
>> GIL and my proposal should render the GIL irrelevant for the most part.
>
> All singleton objects, built-in types are shared and probably a number
> of other things hidden in dark closets...

Yep.  I expect we'll be able to sort those out under the assumption
that 99% of the time they can be treated as immutable.  We'll then
have to find a way to keep the corner cases from breaking the
subinterpreter isolation.

>Not to mention the memory allocator.

This is a sticky part that I've been considering from almost day 1.
It's not the #1 problem to solve, but it will be an important one if
we want to have truly parallel subinterpreters.

>
> By the way, what you're aiming to do is conceptually quite similar to
> Trent's PyParallel (thought Trent doesn't use subinterpreters, his main
> work is around trying to making object sharing safe without any GIL to
> trivially protect the sharing), so you may want to pair with him. Of
> course, you may end up with a Windows-only Python interpreter :-)

Right.  I read through Trent's work on several occasions and have
gleaned a couple lessons related to object sharing.  I was planning on
getting in touch with Trent in the near future.

>
> I'm under the impression you're underestimating the task at hand here.
> Or perhaps you're not and you're just willing to present it in a
> positive way :-)

I'd like to think it's the latter. :)

The main reason why I'm hopeful we can make a meaningful change for
3.6 is that I don't foresee any major changes to CPython's internals.
Nearly all the necessary pieces are already there. <handwave/>  I'm
also intent on taking a minimal approach initially.  We can build on
it from there, easing restrictions that allowed us to roll out the
initial implementation more quickly.  All that said, I won't be
surprised if it takes the entire 3.6 dev cycle to get it right.

-eric