[IPython-dev] Parallel programming dependencies

Cory Dolphin wcdolphin at gmail.com
Tue Apr 29 20:32:28 EDT 2014


Andrea, one solution is to poll from the results of the two async
operations:

e.g.

f1_queue = [apply_async(f1, args) for args in ...]
f2_queue = []

while True:
    check if any of the f1 calls are done
       if so, apply_async f2, and add the result to f2 queue
    check if f2 queue is not empty, and if not empty, are any elements done?
       if so, process result
    check if f1 and f2 are both empty
       exit



Not to derail the conversation, but what would you imagine the 'best case'
scenario for support would look like?

What would you like to see IPython.parallel provide? I could imagine
AsyncResults having more support for future-like operations, e.g. chaining
function calls.

E.G.

combined_async_result = apply_async(f1,args).then(lambda res:
apply_async(f2,res))

I wonder if something like this is possible to implement?


On Tue, Apr 29, 2014 at 8:20 PM, MinRK <benjaminrk at gmail.com> wrote:

> Passing output from one task to the input of another is not well
> supported. As others have said, one approach is to persist the results to
> the filesystem.
> Another, assuming it’s okay to restrict dependent tasks to run on the same
> engine as the dependency, is to persist the values in the engine’s
> namespace and use parallel.Reference to get it as an argument to subsequent
> tasks.
>
> Here’s an example <http://nbviewer.ipython.org/gist/minrk/11415238> of
> doing this.
>
> -MinRK
>
>
> On Tue, Apr 29, 2014 at 4:25 PM, Andrea Zonca <zonca at sdsc.edu> wrote:
>
>> Hi,
>>
>> On Tue, Apr 29, 2014 at 3:56 PM, Andrew Jaffe <a.h.jaffe at gmail.com>
>> wrote:
>> > the particular difference is passing the output of the
>> > earlier tasks to the later ones -- this is a use-case that is very
>> > specifically not addressed in those examples or the docs -- each of the
>> > view.apply calls there just use no arguments, where I want to use the
>> > value calculated by a previous view.apply.
>>
>> Not sure I understand your application,
>> but isn't it an option to write the results to a shared filesystem
>> from the early task and read it from the late task?
>> This would work for tasks running on different nodes.
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140429/0eaabc21/attachment.html>


More information about the IPython-dev mailing list