[IPython-dev] ipcluster (LSF) timing (check if all engines are running)

Florian M. Wagner wagnerfl at student.ethz.ch
Mon Aug 19 09:34:48 EDT 2013


Hey all,

I am using IPython.parallel on a large cluster, where controller and 
engines are launched via LSF. My current workflow is as follows:

    #!/bin/bash
    python pre_processing.py
    ipcluster start --profile=cluster --n=128 > ipcluster.log 2>&1
    sleep 120
    python main_computation.py
    python post_processing.py


I am not entirely happy with this, since the 2 minutes are not always 
enough depending on the load of the cluster. I believe that there is a 
much more elegant way to launch the cluster and check if all the eninges 
are running, before proceeding with the main computation. I would highly 
appreciate any help.

Best regards
Florian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130819/681be7be/attachment.html>


More information about the IPython-dev mailing list