[Python-ideas] solving multi-core Python
solipsis at pitrou.net
Sun Jun 21 13:52:36 CEST 2015
On Sun, 21 Jun 2015 13:41:30 +0200
Sturla Molden <sturla.molden at gmail.com>
> From the perspective of software design, it would be good it the
> CPython interpreter provided an environment instead of using global
> objects. It would mean that all functions in the C API would need to
> take the environment pointer as their first variable, which will be a
> major rewrite. It would also allow the "one interpreter per thread"
> design similar to tcl and .NET application domains.
From the point of view of API compatibility, it's unfortunately a no-no.
> The reason IPC in multiprocessing is slow is due to calling pickle, it
> is not the IPC in itself.
No need to be pedantic :-) The "C" means communication, and pickling
objects is part of the communication between Python processes.
> All in all, I think we are better off finding a better way to share
> Python objects between processes.
Sure. This is however a complex and experimental topic (how to share a
graph of garbage-collected objects between independant processes), with
no guarantees of showing any results at the end.
> P.S. Another thing to note is that with sub-interpreters, you can forget
> about using ctypes or anything else that uses the simplified GIL API
> (e.g. certain Cython generated extensions).
Indeed, the PyGILState API is still not subinterpreter-compatible.
There's a proposal on the tracker, IIRC, but the interested parties
never made any progress on it.
More information about the Python-ideas