[IPython-dev] Parallel programming dependencies

MinRK benjaminrk at gmail.com
Tue Apr 29 20:20:35 EDT 2014

Passing output from one task to the input of another is not well supported.
As others have said, one approach is to persist the results to the
Another, assuming it’s okay to restrict dependent tasks to run on the same
engine as the dependency, is to persist the values in the engine’s
namespace and use parallel.Reference to get it as an argument to subsequent

Here’s an example <http://nbviewer.ipython.org/gist/minrk/11415238> of
doing this.


On Tue, Apr 29, 2014 at 4:25 PM, Andrea Zonca <zonca at sdsc.edu> wrote:

> Hi,
> On Tue, Apr 29, 2014 at 3:56 PM, Andrew Jaffe <a.h.jaffe at gmail.com> wrote:
> > the particular difference is passing the output of the
> > earlier tasks to the later ones -- this is a use-case that is very
> > specifically not addressed in those examples or the docs -- each of the
> > view.apply calls there just use no arguments, where I want to use the
> > value calculated by a previous view.apply.
> Not sure I understand your application,
> but isn't it an option to write the results to a shared filesystem
> from the early task and read it from the late task?
> This would work for tasks running on different nodes.
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140429/8a77635f/attachment.html>

More information about the IPython-dev mailing list