[IPython-dev] iPython cluster on multiple Windows servers, without Windows HPC Server

[Jason Roberts]

I have a situation where I have to use MS Windows for a big parallel
processing job, due to Windows dependencies on some steps in the job. I have
successfully used iPython on a single 16-processor machine for this purpose.
Thank you very much for making this so easy to use! It has saved me a huge
amount of time.

 

Now, if possible, I would like to set up a cluster that has multiple Windows
servers (Windows Server 2008 R2 Standard). The iPython documentation
(http://ipython.org/ipython-doc/dev/parallel/parallel_process.html)
describes several options. The one that seems best oriented for Windows, at
least under the assumption that Microsoft technologies are the best choice
for Windows, is to use Microsoft HPC Pack 2008
(http://ipython.org/ipython-doc/dev/parallel/parallel_winhpc.html). I tried
this. Unfortunately HPC Pack appears to require Active Directory to be
deployed. My shop runs a mixture of different operating systems, and while
we have LDAP, we do not have a full-blown deployment of Active Directory.
This appears to rule out the HPC Pack option.

 

Are there other alternatives for running an iPython cluster composed of
multiple Windows servers, and which is best? Should I look at mpiexec with
Open MPI? Is there some way to do it with SSH, despite the iPython
documentation saying not?

 

Thanks for any advice you can provide, and thanks again for iPython's
parallel processing infrastructure. It truly is a time saver.

 

Jason

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140506/926e4afe/attachment.html>