[IPython-dev] automatically invoking ipython parallel workers

MinRK benjaminrk at gmail.com
Fri Jan 31 14:47:16 EST 2014


I would recommend looking at
ipython-cluster-helper<https://github.com/roryk/ipython-cluster-helper>,
which aims to do this kind of thing.

-MinRK


On Fri, Jan 31, 2014 at 11:41 AM, Stephen Bailey <stephenbailey at lbl.gov>wrote:

> Hi IPythoners,
>
> I'd like to write an IPython-parallelized program that hides all the
> details from the user, i.e. doesn't require them to run ipcluster first.
>  If they have a set of workers running already, great, use that, but
> otherwise the program should spin up the workers automatically, use them,
> and then shut them down at the end.  Something like:
> from IPython.parallel import Clienttry:    c = Client()
> need_to_stop_workers = Falseexcept IOError:    # [Start workers as if
> "ipcluster start -n X" was run]    # ...    need_to_stop_workers = True
> c = Client()
> # ...
> if need_to_stop_workers:    # [Stop workers]
> Exploring what ipcluster is doing, it looks like it boils down to parsing
> sys.argv pretty deep in the call tree rather than at the top level and then
> passing arguments in to
> IPython.parallel.apps.ipclusterapp.launch_new_instance().   I found that I
> could modify sys.argv and then run launch_new_instance(), but then I didn't
> know how to kill the workers without resorting to unix "kill".  And I'd
> much rather just call some function with arguments rather than mucking with
> sys.argv as a global variable to achieve argument passing.
> Is there a way to cleanly start and stop workers from within a python
> script, without having to muck with sys.argv, grepping PIDs, etc.?  i.e.
> something like launch_new_instance, but accepts arguments for how many
> workers, etc.?  This would be taking the "I" out of "IPython", but the
> basic parallelism constructs it provides are quite nice and could be useful
> in completely non-interactive contexts.
>
> Full disclosure: I first pinged Fernando about this who suggested that I
> use subprocess.popen to spawn ipcluster and get the pid so that I could
> cleanly kill it at and the end, but he also suggested pinging the list for
> other cleaner ideas.
>
> Thanks,
>
> Stephen
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140131/04cbb615/attachment.html>


More information about the IPython-dev mailing list