[Python-Dev] About the future of multi-process Python

Antoine Pitrou solipsis at pitrou.net
Wed Feb 6 12:06:06 EST 2019


For the record there are number of initiatives currently to boost the
usefulness and efficiency of multi-process computation in Python.

One of them is PEP 574 (zero-copy pickling with out-of-band buffers),
which I'm working on.

Another is Pierre Glaser's work on allowing pickling of dynamic
functions and classes with the C-accelerated _pickle module (rather than
the slow pure Python implementation):

Another is Davin's work on shared memory managers.

There are also emerging standards like Apache Arrow that provide a
shared, runtime-agnostic, compute-friendly representation for in-memory
tabular data, and third-party frameworks like Dask which are
potentially able to work on top of that and expose nice end-user APIs.

For maximum synergy between these initiatives and the resulting APIs,
it is better if things are done in the open ;-)



More information about the Python-Dev mailing list