[IPython-dev] IPython.parallel slow push

[Moritz Beber]

Hi Wes,

Thank you for the quick response.


On Mon, Aug 11, 2014 at 3:06 PM, Wes Turner <wes.turner at gmail.com> wrote:

> If I understand your use case correctly, each computation node needs a
> copy of all 2GB of the same data?
>
> For reference, how does the transfer rate compare to rsync-ing the
> same data over SSH?
>

I should have been more clear, it's actually a local cluster, i.e., same
host just multiple local kernels. So I have one process that loads the
data, does some pre-processing and then creates a dictionary that it pushes
to the namespace of the other kernels. I understand that the data needs to
go through ZMQ but since it's on the same machine I expected it to be
faster (with 64 kernels it takes about 15 min). If desired I can come up
with a minimal notebook.

Cheers,
Moritz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140811/f87095d7/attachment.html>