[IPython-dev] Parallel programming dependencies
MinRK
benjaminrk at gmail.com
Tue Apr 29 20:20:35 EDT 2014
Passing output from one task to the input of another is not well supported.
As others have said, one approach is to persist the results to the
filesystem.
Another, assuming it’s okay to restrict dependent tasks to run on the same
engine as the dependency, is to persist the values in the engine’s
namespace and use parallel.Reference to get it as an argument to subsequent
tasks.
Here’s an example <http://nbviewer.ipython.org/gist/minrk/11415238> of
doing this.
-MinRK
On Tue, Apr 29, 2014 at 4:25 PM, Andrea Zonca <zonca at sdsc.edu> wrote:
> Hi,
>
> On Tue, Apr 29, 2014 at 3:56 PM, Andrew Jaffe <a.h.jaffe at gmail.com> wrote:
> > the particular difference is passing the output of the
> > earlier tasks to the later ones -- this is a use-case that is very
> > specifically not addressed in those examples or the docs -- each of the
> > view.apply calls there just use no arguments, where I want to use the
> > value calculated by a previous view.apply.
>
> Not sure I understand your application,
> but isn't it an option to write the results to a shared filesystem
> from the early task and read it from the late task?
> This would work for tasks running on different nodes.
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140429/8a77635f/attachment.html>
More information about the IPython-dev
mailing list