[Python-Dev] About the future of multi-process Python

[Steve Dower]

On 06Feb2019 0906, Antoine Pitrou wrote:
> For the record there are number of initiatives currently to boost the
> usefulness and efficiency of multi-process computation in Python.
> 
> One of them is PEP 574 (zero-copy pickling with out-of-band buffers),
> which I'm working on.
> 
> Another is Pierre Glaser's work on allowing pickling of dynamic
> functions and classes with the C-accelerated _pickle module (rather than
> the slow pure Python implementation):
> https://bugs.python.org/issue35900
> https://bugs.python.org/issue35911
> 
> Another is Davin's work on shared memory managers.
> 
> There are also emerging standards like Apache Arrow that provide a
> shared, runtime-agnostic, compute-friendly representation for in-memory
> tabular data, and third-party frameworks like Dask which are
> potentially able to work on top of that and expose nice end-user APIs.
> 
> For maximum synergy between these initiatives and the resulting APIs,
> it is better if things are done in the open ;-)

Hopefully our steering council can determine (or delegate the 
determination of) the direction we should go here so we can all be 
pulling in the same direction :)

That said, there are certainly a number of interacting components and 
not a lot of information about how they interact and overlap. A good 
start would be to identify the likely overlap of this work to see where 
they can build upon each other rather than competing, as well as 
estimating the long-term burden of standardising.

Cheers,
Steve