[Python-ideas] solving multi-core Python

[Stefan Behnel]

Eric Snow schrieb am 20.06.2015 um 23:42:
> tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> exposing them in Python, and adding a mechanism to safely share
> objects between them.
> [...]
> In some personal correspondence Nick Coghlan, he summarized my
> preferred approach as "the data storage separation of multiprocessing,
> with the low message passing overhead of threading".
> 
> For Python 3.6:
> 
> * expose subinterpreters to Python in a new stdlib module: "subinterpreters"
> * add a new SubinterpreterExecutor to concurrent.futures
> * add a queue.Queue-like type that will be used to explicitly share
> objects between subinterpreters
> [...]
> C Extension Modules
> =================
> 
> Subinterpreters already isolate extension modules (and built-in
> modules, including sys).  PEP 384 provides some help too.  However,
> global state in C can easily leak data between subinterpreters,
> breaking the desired data isolation.  This is something that will need
> to be addressed as part of the effort.

I also had some discussions about these things with Nick before. Not sure
if you really meant PEP 384 (you might have) or rather PEP 489:

https://www.python.org/dev/peps/pep-0489/

I consider that one more important here, as it will eventually allow Cython
modules to support subinterpreters. Unless, as you mentioned, they use
global C state, but only in external C code, e.g. wrapped libraries. Cython
should be able to handle most of the module internal global state on a
per-interpreter basis itself, without too much user code impact.

I'm totally +1 for the idea. I hope that I'll find the time (well, and
money) to work on PEP 489 in Cython soon, so that I can prove it right for
actual real-world code in Python 3.5. We'll then see about subinterpreter
support. That's certainly the next step.

Stefan