[Python-ideas] solving multi-core Python

[Antoine Pitrou]

On Sun, 21 Jun 2015 13:41:30 +0200
Sturla Molden <sturla.molden at gmail.com>
wrote:
>  From the perspective of software design, it would be good it the 
> CPython interpreter provided an environment instead of using global 
> objects. It would mean that all functions in the C API would need to 
> take the environment pointer as their first variable, which will be a 
> major rewrite. It would also allow the "one interpreter per thread" 
> design similar to tcl and .NET application domains.

From the point of view of API compatibility, it's unfortunately a no-no.

> The reason IPC in multiprocessing is slow is due to calling pickle, it 
> is not the IPC in itself.

No need to be pedantic :-) The "C" means communication, and pickling
objects is part of the communication between Python processes.

> All in all, I think we are better off finding a better way to share 
> Python objects between processes.

Sure. This is however a complex and experimental topic (how to share a
graph of garbage-collected objects between independant processes), with
no guarantees of showing any results at the end.

> P.S. Another thing to note is that with sub-interpreters, you can forget 
> about using ctypes or anything else that uses the simplified GIL API 
> (e.g. certain Cython generated extensions).

Indeed, the PyGILState API is still not subinterpreter-compatible.
There's a proposal on the tracker, IIRC, but the interested parties
never made any progress on it.

Regards

Antoine.