[IPython-dev] Parallel programming dependencies

John Gill jgill at tokiomillennium.com
Wed Apr 30 16:50:14 EDT 2014

Thanks for this.

I think the ‘follow’ temp flag might allow me to deal with the issue I raised the other day about having a non-homogeneous cluster (made up of engines on different OS’es) and have some tasks that can only run on a specific OS.   A similar use case would be having a non-homogeneous cluster and wanting some tasks to run on the engines with more memory or faster cpus.

So my plan is to run a task on each node to find out what OS it has and save the msg_ids into different sets based on the OS.

Now, when I have a task that has to run on a specific OS I can do something like:

with view.temp_flags(follow=linux, after=deps):

    msg_id = view.apply_async(task)

Will let you know how this works out – be very cool if this just works.


From: ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] On Behalf Of MinRK
Sent: Tuesday, April 29, 2014 9:21 PM
To: IPython developers list
Subject: Re: [IPython-dev] Parallel programming dependencies

Passing output from one task to the input of another is not well supported. As others have said, one approach is to persist the results to the filesystem.
Another, assuming it’s okay to restrict dependent tasks to run on the same engine as the dependency, is to persist the values in the engine’s namespace and use parallel.Reference to get it as an argument to subsequent tasks.

Here’s an example<http://nbviewer.ipython.org/gist/minrk/11415238> of doing this.


On Tue, Apr 29, 2014 at 4:25 PM, Andrea Zonca <zonca at sdsc.edu<mailto:zonca at sdsc.edu>> wrote:

On Tue, Apr 29, 2014 at 3:56 PM, Andrew Jaffe <a.h.jaffe at gmail.com<mailto:a.h.jaffe at gmail.com>> wrote:
> the particular difference is passing the output of the
> earlier tasks to the later ones -- this is a use-case that is very
> specifically not addressed in those examples or the docs -- each of the
> view.apply calls there just use no arguments, where I want to use the
> value calculated by a previous view.apply.
Not sure I understand your application,
but isn't it an option to write the results to a shared filesystem
from the early task and read it from the late task?
This would work for tasks running on different nodes.
IPython-dev mailing list
IPython-dev at scipy.org<mailto:IPython-dev at scipy.org>

This communication and any attachments contain information which is confidential and may also be legally privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying, printing or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please return it with the title "received in error" to postmaster at tokiomillennium.com and then permanently delete the email and any attachments from your system.

E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. It is the recipient's responsibility to ensure that e-mail transmissions and any attachments are virus free. We do not accept liability for any damages or other consequences caused by information that is intercepted, corrupted, amended, lost, destroyed, arrives late or incomplete or contains viruses.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140430/e0dcafab/attachment.html>

More information about the IPython-dev mailing list