[IPython-dev] ipcluster with ssh

Brian Granger ellisonbg at gmail.com
Sun Jan 30 22:41:36 EST 2011


On Mon, Jan 24, 2011 at 8:06 AM, Harald Schilly
<harald.schilly at gmail.com> wrote:
> Hi, I'm currently playing around with the ipcluster using ssh. I had a
> problem and I couldn't track it down, until I did some changes to the
> source. In file ipcluster.py in line 477 of version 10.1 (in the
> SSHEngineSet class/method start()) I change the parameter
> "consumeErrors" in "return gatherBoth(...)" from True to False. It
> revealed, that a warning of ssh triggered an error and the cluster
> just didn't do anything (no engine was started).
> The problem/warning is triggered using this ~/.ssh/config file:
> UserKnownHostsFile=/dev/null
> StrictHostKeyChecking=no
> NoHostAuthenticationForLocalhost=no
> So, each time a new ssh connection is set up, the hostname is unknown
> and this is reported. To get things running, I have to comment the
> first line, so that the usual known hostname file is used and after
> starting the cluster 2x all hosts are added and it works. The problem
> is that those hostnames change sometimes and then it's once again an
> error.
> My first proposal would be to enable reporting those errors by default!

Yes, I think that is a good idea.  But at this point we are making a
massive effort to push forward and completely rewrite of the parallel
stuff using pyzmq/zeromq.  What this means is that we are not spending
much time maintaining the older stuff you are using.  The new stuff is
in code review right now, but it will be a while before it is ready
for use.  It will be a huge improvement, but it does leave you in a
bit of a limbo.  I would say that minimally you should run with the
errors showing in your local version of the code.  If you want to
contribute a pull request on github with this fix for the 0.10 series,
that would also help us to get this in other people hands.

> Is there a recommended/better configuration for ssh? If yes, this
> should be part of the documentation. Also, I think the SSHEngineSet
> class could be more intelligent and ignore this specific warning which
> is indeed no problem.
> I'm using Ubuntu linux/openssh.
> The warning is:
> 2011-01-24 16:57:46+0100 [-] Unhandled error in Deferred:
> 2011-01-24 16:57:46+0100 [-] Unhandled Error
>        Traceback (most recent call last):
>        Failure: twisted.internet.utils._UnexpectedErrorOutput: got
> stderr: "Warning: Permanently added '...,...' (RSA) to the list of
> known hosts.\r\n"
> Another idea is to change the exec_engine command in _ssh_engine() to
> make it more intelligent. For some other project i added additional
> options directly to ssh so that it is more reliable. It looks like
> this:
>   p = Popen(['ssh','-x','-q',\
>              '-o','UserKnownHostsFile=/dev/null',\
>              '-o','ServerAliveInterval=60',\
>              '-o','StrictHostKeyChecking=no',\
>              '-o','ConnectTimeout=%s' % SSH_CONN_TIMEOUT,\
>              '-o','BatchMode=yes',\
>              '-n','%s@%s'%(user, machine),\
>              'ionice','-c','3','nice', cmd],\
>              stdout=PIPE, stdin=PIPE, stderr=PIPE)

Could you file a ticket on our github page about this.  That will be
very useful as we move forward.



> H
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the IPython-dev mailing list