[IPython-dev] automatically invoking ipython parallel workers

Drain, Theodore R (392P) theodore.r.drain at jpl.nasa.gov
Fri Jan 31 15:47:07 EST 2014


I'm in the process of finishing something very similar to this.  Here's an overview of what I did:

- helper functions start(profile), stop(profile), wait(numEngine,profile)
   - start() uses popen to run an executable script I wrote
      - exec script builds IPClusterStart object directly and sets some options (nice level, daemonize, etc) and then prints a JSON dict to stdout w/ the number of engines to expect.
      - start() reads the number of engines from popen and calls wait(profile,n) to wait until the controller and engines are read
      - wait builds a client and polls until all the engines are ready (see ipython-dev email 1/14/2014)

- custom Client class calls start(profile) in __init__ and stop(profile) in __del__

So the user does something like this to start the cluster and when c is deleted, the cluster stops.

c = Client( profile='cluster' )
#... parallel functions here


I've been using the IPython Client to do all communication rather than using process id.  You can build a client and then call client.shutdown( hub=True ) to kill the whole cluster.  I have run into some timing problems in that if I kill the main process (ctrl-c) before all the engines connect, then the controller is shut down but some of the engines are not.  Not sure what to do about that yet...

Ted



________________________________
From: ipython-dev-bounces at scipy.org [ipython-dev-bounces at scipy.org] on behalf of Stephen Bailey [stephenbailey at lbl.gov]
Sent: Friday, January 31, 2014 11:41 AM
To: ipython-dev at scipy.org
Subject: [IPython-dev] automatically invoking ipython parallel workers

Hi IPythoners,

I'd like to write an IPython-parallelized program that hides all the details from the user, i.e. doesn't require them to run ipcluster first.  If they have a set of workers running already, great, use that, but otherwise the program should spin up the workers automatically, use them, and then shut them down at the end.  Something like:
from IPython.parallel import Clienttry:    c = Client()    need_to_stop_workers = Falseexcept IOError:    # [Start workers as if "ipcluster start -n X" was run]    # ...    need_to_stop_workers = True    c = Client()
# ...
if need_to_stop_workers:    # [Stop workers]
Exploring what ipcluster is doing, it looks like it boils down to parsing sys.argv pretty deep in the call tree rather than at the top level and then passing arguments in to IPython.parallel.apps.ipclusterapp.launch_new_instance().   I found that I could modify sys.argv and then run launch_new_instance(), but then I didn't know how to kill the workers without resorting to unix "kill".  And I'd much rather just call some function with arguments rather than mucking with sys.argv as a global variable to achieve argument passing.
Is there a way to cleanly start and stop workers from within a python script, without having to muck with sys.argv, grepping PIDs, etc.?  i.e. something like launch_new_instance, but accepts arguments for how many workers, etc.?  This would be taking the "I" out of "IPython", but the basic parallelism constructs it provides are quite nice and could be useful in completely non-interactive contexts.

Full disclosure: I first pinged Fernando about this who suggested that I use subprocess.popen to spawn ipcluster and get the pid so that I could cleanly kill it at and the end, but he also suggested pinging the list for other cleaner ideas.

Thanks,

Stephen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140131/1acb6bd9/attachment.html>


More information about the IPython-dev mailing list