<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.keyword
{mso-style-name:keyword;}
span.number
{mso-style-name:number;}
span.variable
{mso-style-name:variable;}
span.string
{mso-style-name:string;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:284164899;
mso-list-template-ids:456920722;}
@list l1
{mso-list-id:836458331;
mso-list-template-ids:799044886;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks MinRK – that is a very helpful explanation + suggestions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This question was pretty timely. I actually had ipython 1.x running fine with HPC, but was unable to get it to work under ipython2.0. I am working towards
using ssh to start the cluster up, but as you point out with a shared file system that is a whole lot easier + I think the HPC is a pretty big hammer to crack a tiny nut – and it tends to make a mess of the nut ;)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As mentioned in a previous post I am also interested in creating non-homogeneous clusters eg engines with different cpu and memory resources, on different OS’es
and with different software installed + have the scheduler deal with task dependencies whilst at the same time respecting any restrictions tasks have as to what kind of engine they need run on.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">John<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ipython-dev-bounces@scipy.org [mailto:ipython-dev-bounces@scipy.org]
<b>On Behalf Of </b>MinRK<br>
<b>Sent:</b> Tuesday, May 06, 2014 4:37 PM<br>
<b>To:</b> IPython developers list<br>
<b>Subject:</b> Re: [IPython-dev] iPython cluster on multiple Windows servers, without Windows HPC Server<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div id="markdown-here-wrapper-305491">
<p>An important thing to note about ipcluster is that it’s a very complicated way to do something that’s not very complicated. All it sets out to do is:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>start a controller with <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">ipcontroller</span></code>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>start 0-many engines with <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">ipengine</span></code><o:p></o:p></p>
<p style="margin:0px!important">All of the complexity comes from abstracting how processes actually start, including where machines are, batch systems, etc. But in the end, it’s just doing:
<o:p></o:p></p>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt;overflow:auto"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$> ipcluster<o:p></o:p></span></code></pre>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$> </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">for</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> i </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">in</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> {</span></code><span class="number"><span style="font-family:Consolas;color:#009999;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">1</span></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">..n}; </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">do</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> ipengine; </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">done</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"><o:p></o:p></span></code></pre>
<p style="margin:0px!important">ipcluster makes some simple cases easier, but if it doesn’t do what you want, you can always start the controller and engines yourself, with no loss of functionality. Plus, a tool that only deploys a cluster on your own system
is much simpler than one that tries to work in a wide variety of contexts like ipcluster.
<o:p></o:p></p>
<p style="margin:0px!important">The basic steps in getting a cluster up and running:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l1 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>configure the controller to listen on an IP visible to the other machines (<code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">c.HubFactory.ip = '1.2.3.4'</span></code>
in <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">ipcontroller_config.py</span></code> on the controller machine.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l1 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>start the controller with <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">ipcontroller</span></code><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l1 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>copy <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">.ipython/profile_default/security/ipcontroller-*.json</span></code> to all of the various machines on which you
plan to start engines.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:0in;text-indent:-.25in;mso-list:l1 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>start <code><span style="font-size:10.0pt;font-family:Consolas;border:solid #EAEAEA 1.0pt;padding:0in;background:#F8F8F8">ipengine</span></code> as many times as is appropriate on each machine.
<o:p></o:p></p>
<p style="margin:0px!important">Step 3. is unnecessary if your systems are on a shared filesystem.<o:p></o:p></p>
<p style="margin:0px!important">For instance, here is a simple version that starts a controller and engines with ssh on Linux or OS X machines, putting processes in the background with screen:
<o:p></o:p></p>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt;overflow:auto"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$> ssh controller_host screen -dmS ipcontroller<o:p></o:p></span></code></pre>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$> </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">for</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> host </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">in</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> host1 host2; </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">do</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"><o:p></o:p></span></code></pre>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">> scp ~/.ipython/profile_default/security/ipcontroller-*.json </span></code><span class="variable"><span style="font-family:Consolas;color:teal;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$host</span></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">:.ipython/profile_default/security/ <o:p></o:p></span></code></pre>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">> ssh </span></code><span class="variable"><span style="font-family:Consolas;color:teal;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">$host</span></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"> </span></code><span class="string"><span style="font-family:Consolas;color:#DD1144;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">'for n in {1..3}; do screen -dmS ipengine; done'</span></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"><o:p></o:p></span></code></pre>
<pre style="mso-margin-top-alt:0in;margin-right:1.8pt;margin-bottom:0in;margin-left:1.8pt;margin-bottom:.0001pt;line-height:14.4pt"><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">> </span></code><span class="keyword"><b><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite">done</span></b></span><code><span style="font-family:Consolas;color:#333333;border:solid #CCCCCC 1.0pt;padding:6.0pt;background:ghostwhite"><o:p></o:p></span></code></pre>
<p style="margin:0px!important">Which is <em>a lot</em> simpler than the hundreds of lines of ipcluster, and, frankly, better behaved than the SSH launchers that ship with IPython.
<o:p></o:p></p>
<p style="margin:0px!important">If you have Windows analogues for ‘tell machine X to run command Y,’ you can make a similar script, tailored to your use.<o:p></o:p></p>
<p style="margin:0px!important">-MinRK<o:p></o:p></p>
<p style="margin:0px!important">On Tue, May 6, 2014 at 11:35 AM, Jason Roberts <<a href="mailto:jason.roberts@duke.edu" target="_blank">jason.roberts@duke.edu</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif"">I have a situation where I have to use MS Windows for a big parallel processing job, due to Windows dependencies
on some steps in the job. I have successfully used iPython on a single 16-processor machine for this purpose. Thank you very much for making this so easy to use! It has saved me a huge amount of time.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif"">Now, if possible, I would like to set up a cluster that has multiple Windows servers (Windows Server 2008 R2 Standard).
The iPython documentation (<a href="http://ipython.org/ipython-doc/dev/parallel/parallel_process.html" target="_blank">http://ipython.org/ipython-doc/dev/parallel/parallel_process.html</a>) describes several options. The one that seems best oriented for Windows,
at least under the assumption that Microsoft technologies are the best choice for Windows, is to use Microsoft HPC Pack 2008 (<a href="http://ipython.org/ipython-doc/dev/parallel/parallel_winhpc.html" target="_blank">http://ipython.org/ipython-doc/dev/parallel/parallel_winhpc.html</a>).
I tried this. Unfortunately HPC Pack appears to require Active Directory to be deployed. My shop runs a mixture of different operating systems, and while we have LDAP, we do not have a full-blown deployment of Active Directory. This appears to rule out the
HPC Pack option.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif"">Are there other alternatives for running an iPython cluster composed of multiple Windows servers, and which is
best? Should I look at mpiexec with Open MPI? Is there some way to do it with SSH, despite the iPython documentation saying not?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif"">Thanks for any advice you can provide, and thanks again for iPython’s parallel processing infrastructure. It truly
is a time saver.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#888888"> </span><span style="color:#888888"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#888888">Jason</span><span style="color:#888888"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#888888"> </span><span style="color:#888888"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
<p style="MARGIN: 0in 0in 10pt" class="MsoNormal"><font face="Calibri">This communication and any attachments contain information which is confidential and may also be legally privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying, printing or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please return it with the title "received in error" to postmaster@tokiomillennium.com and then permanently delete the email and any attachments from your system.</font></p><p style="MARGIN: 0in 0in 10pt" class="MsoNormal"><p><font face="Calibri"> </font></p></p><p style="MARGIN: 0in 0in 10pt" class="MsoNormal"><font face="Calibri">E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. It is the recipient's responsibility to ensure that e-mail transmissions and any attachments are virus free. We do not accept liability for any damages or other consequences caused by information that is intercepted, corrupted, amended, lost, destroyed, arrives late or incomplete or contains viruses.</font></p><p style="MARGIN: 0in 0in 10pt" class="MsoNormal"><font face="Calibri">******************************************</font></p><p />
</body>
</html>