<div dir="ltr"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 28, 2014 at 5:04 PM, Patrick Fuller <span dir="ltr"><<a href="mailto:patrickfuller@gmail.com" target="_blank">patrickfuller@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">...the difference being that this would require starting a new engine on each segfault<div class="HOEnZb"><div class="h5">
<span></span><br><br>On Tuesday, January 28, 2014, Patrick Fuller <<a href="mailto:patrickfuller@gmail.com" target="_blank">patrickfuller@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I guess my question is more along the lines of: should the cluster continue on to complete the queued jobs (as it would if the segfaults were instead python exceptions)? </blockquote>
</div></div></blockquote><div><br></div><div>I see what you mean - the generator halts when it sees an exception, so it's inconvenient to get the successes, while ignoring the failures. I guess we could add separate methods that only iterate through just the successful results.</div>
<div><br></div><div>As far as task submission goes, it does indeed do what you seem to expect, so it's just viewing the results where there is an issue.</div><div><br></div><div>Here is <a href="http://nbviewer.ipython.org/gist/minrk/8680688">an example</a> of iterating through only the successful results of a map that segfaults.</div>
<div><br></div><div>-MinRK</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span></span><div>
<br>On Tuesday, January 28, 2014, MinRK <<a>benjaminrk@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I get an EngineError when an engine dies running a task:<div><br></div><div><a href="http://nbviewer.ipython.org/gist/minrk/8679553" target="_blank">http://nbviewer.ipython.org/gist/minrk/8679553</a><br>
</div><div><br></div>
<div>I think this is the desired behavior.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 28, 2014 at 2:18 PM, Patrick Fuller <span dir="ltr"><<a>patrickfuller@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="line-height:1.2;color:rgb(34,34,34);font-size:1em;font-family:Helvetica,arial,freesans,clean,sans-serif;border:none">
<p style="margin:1em 0px">Hi,</p>
<p style="margin:1em 0px">Has there been any discussion around how ipython parallel handles segfaulting?</p>
<p style="margin:1em 0px">To make this question more specific, the following code will cause some workers to crash. All results will become unreadable (or at least un-iterable), and future runs require a restart of the cluster. Is this behavior intended, or is it just something that hasn’t been discussed?</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;overflow:auto;margin:1em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:nowrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;display:inline;white-space:pre-wrap;border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block;padding:0.5em;color:rgb(51,51,51);background-color:rgb(248,248,255);background-repeat:initial initial"><span style="color:rgb(51,51,51);font-weight:bold">from</span> IPython.parallel <span style="color:rgb(51,51,51);font-weight:bold">import</span> Client
<span style="color:rgb(51,51,51);font-weight:bold">from</span> random <span style="color:rgb(51,51,51);font-weight:bold">import</span> random
<span><span style="color:rgb(51,51,51);font-weight:bold">def</span> <span style="color:rgb(153,0,0);font-weight:bold">segfaulty_function</span><span>(random_number, chance=<span style="color:rgb(0,153,153)">0.25</span>)</span>:</span>
<span style="color:rgb(51,51,51);font-weight:bold">if</span> random_number < chance:
<span style="color:rgb(51,51,51);font-weight:bold">import</span> ctypes
i = ctypes.c_char(<span style="color:rgb(221,17,68)">'a'</span>)
j = ctypes.pointer(i)
c = <span style="color:rgb(0,153,153)">0</span>
<span style="color:rgb(51,51,51);font-weight:bold">while</span> <span style="color:rgb(0,134,179)">True</span>:
j[c] = <span style="color:rgb(221,17,68)">'a'</span>
c += <span style="color:rgb(0,153,153)">1</span>
<span style="color:rgb(51,51,51);font-weight:bold">return</span> j
<span style="color:rgb(51,51,51);font-weight:bold">else</span>:
<span style="color:rgb(51,51,51);font-weight:bold">return</span> random_number
view = Client(profile=<span style="color:rgb(221,17,68)">"something-parallel-here"</span>).load_balanced_view()
results = view.map(segfaulty_function, [random() <span style="color:rgb(51,51,51);font-weight:bold">for</span> _ <span style="color:rgb(51,51,51);font-weight:bold">in</span> range(<span style="color:rgb(0,153,153)">100</span>)])
<span style="color:rgb(51,51,51);font-weight:bold">for</span> i, result <span style="color:rgb(51,51,51);font-weight:bold">in</span> enumerate(results):
<span style="color:rgb(51,51,51);font-weight:bold">print</span> i, result</code></pre>
<p style="margin:1em 0px">Backstory: Recently I’ve been working with a large monte carlo library that segfaults for, like, no reason at all. It’s due to some weird underlying random number issue and happens once every 5-10 thousand runs. I currently have each worker spin out a child process to isolate the occasional segfault, but this seems excessive. (I'm also trying to fix the source of the segfaults, but debugging is a slow process.)</p>
<p style="margin:1em 0px">Thanks,<br>Pat</p>
</div></div>
<br>_______________________________________________<br>
IPython-dev mailing list<br>
<a>IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
<br></blockquote></div><br></div>
</blockquote></div>
</blockquote>
</div></div><br>_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
<br></blockquote></div><br></div></div>