<div dir="ltr">That's a bug that sockets aren't properly cleaned up if the Client never finishes getting connected.  Should be fixed by [PR #4074](<a href="https://github.com/ipython/ipython/pull/4074">https://github.com/ipython/ipython/pull/4074</a>).</div>

<div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 20, 2013 at 3:20 PM, Florian M. Wagner <span dir="ltr"><<a href="mailto:wagnerfl@student.ethz.ch" target="_blank">wagnerfl@student.ethz.ch</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div>Hey MIN,<br>
      <br>
      thanks for the example. The first while statement waits for the
      json file as expected, but when I start the cluster and it finds
      it, a zeromq error occurs: Too many open files (signaler.cpp:330)<br>
      Do you have an idea?<br>
      <br>
      Am 19.08.2013 16:28, schrieb MinRK:<br>
    </div><div><div class="h5">
    <blockquote type="cite">
      
      <div dir="ltr">Something like this should work:
        <div><br>
        </div>
        <div>
          <div>from IPython import parallel</div>
          <div><br>
          </div>
          <div>def wait_for_cluster(engines=1, **kwargs):</div>
          <div>    """Wait for an IPython cluster to startup and
            register a minimum number of engines"""</div>
          <div>    # wait for the controller to come up</div>
          <div>    while True:</div>
          <div>        try:</div>
          <div>            client = parallel.Client(**kwargs)</div>
          <div>        except IOError:</div>
          <div>            print "No ipcontroller-client.json,
            waiting..."</div>
          <div>            time.sleep(10)</div>
          <div>        except TimeoutError:</div>
          <div>            print "No controller, waiting..."</div>
          <div>            time.sleep(10)</div>
          <div>    if not engines:</div>
          <div>        return</div>
          <div>    # wait for engines to register</div>
          <div>    print "waiting for %i engines" % engines,</div>
          <div>    running = len(client)</div>
          <div>    sys.stdout.write('.' * running)</div>
          <div>    while running < engines:</div>
          <div>        time.sleep(1)</div>
          <div>        previous = running</div>
          <div>        running = len(client)</div>
          <div>        sys.stdout.write('.' * (running - previous))</div>
        </div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra">
        <br>
        <br>
        <div class="gmail_quote">On Mon, Aug 19, 2013 at 6:34 AM,
          Florian M. Wagner <span dir="ltr"><<a href="mailto:wagnerfl@student.ethz.ch" target="_blank">wagnerfl@student.ethz.ch</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF"> Hey all,<br>
              <br>
              I am using IPython.parallel on a large cluster, where
              controller and engines are launched via LSF. My current
              workflow is as follows:<br>
              <br>
              <blockquote>#!/bin/bash<br>
                python pre_processing.py<br>
                ipcluster start --profile=cluster --n=128 >
                ipcluster.log 2>&1<br>
                sleep 120<br>
                python main_computation.py<br>
                python post_processing.py<br>
              </blockquote>
              <br>
              I am not entirely happy with this, since the 2 minutes are
              not always enough depending on the load of the cluster. I
              believe that there is a much more elegant way to launch
              the cluster and check if all the eninges are running,
              before proceeding with the main computation. I would
              highly appreciate any help.<br>
              <br>
              Best regards<span><font color="#888888"><br>
                  Florian<br>
                  <br>
                </font></span></div>
            <br>
            _______________________________________________<br>
            IPython-dev mailing list<br>
            <a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
            <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
            <br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
IPython-dev mailing list
<a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a>
</pre>
    </blockquote>
    <br>
  </div></div></div>

<br>_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
<br></blockquote></div><br></div>