[Numpy-discussion] PEP 574 - zero-copy pickling with out of band data

Andrew Nelson andyfaff at gmail.com
Mon Jul 2 20:41:04 EDT 2018


On Tue, 3 Jul 2018 at 09:31, Charles R Harris <charlesr.harris at gmail.com>

> ISTR that some parallel processing applications sent pickled arrays around
> to different processes, I don't know if that is still the case, but if so,
> no copy might be a big gain for them.

That is very much correct. One example is using MCMC, which is massively
parallel. I do parallelisation with mpi4py, and this requires distribution
of pickled data of a reasonable size to the entire MPI world. This pickling
introduces quite a bit of overhead.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180703/19ac1bc9/attachment-0001.html>

More information about the NumPy-Discussion mailing list