[IPython-dev] Parallel programming dependencies

[Burkhard Ritter]

On Tue, Apr 29, 2014 at 4:56 PM, Andrew Jaffe <a.h.jaffe at gmail.com> wrote:
> On 29/04/2014 18:48, Matthias Bussonnier wrote:
>> and lots of stuff that look *a lot* like what you are trying to do.
>>
> Alas, this stuff is NOT really like what I'm trying to do. In
> particular, the particular difference is passing the output of the
> earlier tasks to the later ones -- this is a use-case that is very
> specifically not addressed in those examples or the docs -- each of the
> view.apply calls there just use no arguments, where I want to use the
> value calculated by a previous view.apply.

Hi Andrew,

I am not sure if I am missing something, but wouldn't something like
this work: Keep a list of all submitted jobs, then in a loop check
which jobs are done, and for each job that is done retrieve its result
and submit the next dependent job with the previous job's result as an
argument. If the dependencies are simple that should be relative
straightforward to implement. I don't use and need dependencies
myself, but use a similar loop to store job results on my controller
node immediately when the jobs are done.

Cheers,
Burkhard