[IPython-dev] automatically invoking ipython parallel workers

Stephen Bailey stephenbailey at lbl.gov
Fri Jan 31 14:41:06 EST 2014


Hi IPythoners,

I'd like to write an IPython-parallelized program that hides all the
details from the user, i.e. doesn't require them to run ipcluster first.
 If they have a set of workers running already, great, use that, but
otherwise the program should spin up the workers automatically, use them,
and then shut them down at the end.  Something like:
from IPython.parallel import Clienttry:    c = Client()
need_to_stop_workers = Falseexcept IOError:    # [Start workers as if
"ipcluster start -n X" was run]    # ...    need_to_stop_workers = True
c = Client()
# ...
if need_to_stop_workers:    # [Stop workers]
Exploring what ipcluster is doing, it looks like it boils down to parsing
sys.argv pretty deep in the call tree rather than at the top level and then
passing arguments in to
IPython.parallel.apps.ipclusterapp.launch_new_instance().   I found that I
could modify sys.argv and then run launch_new_instance(), but then I didn't
know how to kill the workers without resorting to unix "kill".  And I'd
much rather just call some function with arguments rather than mucking with
sys.argv as a global variable to achieve argument passing.
Is there a way to cleanly start and stop workers from within a python
script, without having to muck with sys.argv, grepping PIDs, etc.?  i.e.
something like launch_new_instance, but accepts arguments for how many
workers, etc.?  This would be taking the "I" out of "IPython", but the
basic parallelism constructs it provides are quite nice and could be useful
in completely non-interactive contexts.

Full disclosure: I first pinged Fernando about this who suggested that I
use subprocess.popen to spawn ipcluster and get the pid so that I could
cleanly kill it at and the end, but he also suggested pinging the list for
other cleaner ideas.

Thanks,

Stephen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140131/33d8d29c/attachment.html>


More information about the IPython-dev mailing list