[Numpy-discussion] PEP 574 - zero-copy pickling with out of band data

[Nathan Goldbaum]

On Mon, Jul 2, 2018 at 7:42 PM Andrew Nelson <andyfaff at gmail.com> wrote:

> <snip>
>
> On Tue, 3 Jul 2018 at 09:31, Charles R Harris <charlesr.harris at gmail.com>
> wrote:
>
>>
>> ISTR that some parallel processing applications sent pickled arrays
>> around to different processes, I don't know if that is still the case, but
>> if so, no copy might be a big gain for them.
>>
>
> That is very much correct. One example is using MCMC, which is massively
> parallel. I do parallelisation with mpi4py, and this requires distribution
> of pickled data of a reasonable size to the entire MPI world. This pickling
> introduces quite a bit of overhead.
>

Doesn’t mpi4py have support for buffered low-level communication of numpy
arrays? See e.g.
https://mpi4py.scipy.org/docs/usrman/tutorial.html

Although I guess with Antoine’s proposal uses of the “lowercase” mpi4py API
where data might get pickled will see speedups.

_______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180702/ff571410/attachment.html>