<div dir="ltr">Dear Min,<div><br></div><div>Thanks to the notebook I've found the problem: </div><div>the engines (at a certain point) execute the only startup script that I have (where I call `plt.ion()` see the notebook) but apparently ignores the ipython_configure.py.</div>
<div><br></div><div>Here is a notebook that show the problem with some more explanation: </div><div><a href="http://nbviewer.ipython.org/6592051">http://nbviewer.ipython.org/6592051</a></div><div><br></div><div>I did use the same config files and startup scripts in ipy v0.13, with the same parallel code and never had the issue. </div>
<div>When I start a console ipy is initialised correctly and `plt.ion()` called without problems. </div><div><br></div><div>If you need anything else let me know,</div><div><br></div><div>Fra</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/9/16 Francesco Montesano <span dir="ltr"><<a href="mailto:franz.bergesund@gmail.com" target="_blank">franz.bergesund@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">These codes of mine can be few hundred lines and depend upon other 2 or 3 modules that I have put together. <div><br></div><div>If the problem is where I think it is, I might be able to write a standalone program (maybe in a notebook). Stay tuned :D</div>
<div><br></div><div>Fra </div><div><div><br></div><div><div><br></div><div><br></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">2013/9/16 MinRK <span dir="ltr"><<a href="mailto:benjaminrk@gmail.com" target="_blank">benjaminrk@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Cn you please share your whole code?</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>
<div>On Mon, Sep 16, 2013 at 9:03 AM, Francesco Montesano <span dir="ltr"><<a href="mailto:franz.bergesund@gmail.com" target="_blank">franz.bergesund@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Dear all,<div><br></div><div>I'm having an issue with the parallel (v1.1.0, under python 2.7). </div>
<div><br></div><div>Some time ago I did build a number of python codes to manipulate catalogues. </div>
<div>I can have either thousands of small file or few possibly huge file.</div><div>So I've written my codes such that I can chose from command whether to use any of the two.</div><div><br></div><div>Typically my codes have the following structure:</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">import numpy as np<br>import pandas as pd<br>
def parse(...): #argparse<br> ....<br>def to_do(fname,...): #function(s) that do what I need<br> ....<br>if __name__=='__main__':<br> args = parse(...)<br> if args.paralell == False:<br> for fn in file_name_list:<br>
to_do(fn, ...)</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> else: #execute in parallel</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> parallel_env = Lbv() # custom class that init a load ballance view</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> imports = ['import numpy as np', 'import pandas as pd']</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> parallel_env.exec_on_engine(imports) # execute the above strings on all engines (direct view)</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> #execute in parallel </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
runs = [parallel_env.apply(to_do, os.path.abspath(fn), ...) for fn in file_name_list]</blockquote><div><br></div><div><br></div><div>I build the whole parallel dispatching against ipython 0.13 and everything worked fine.</div>
<div>But today I've tried to run one of my scripts enabling parallel and got the following error</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
File "code.py", line XXX, in <module><br> parallel_env.exec_on_engine(imports)<br> File "XXX/ipython_parallel.py", line 86, in exec_on_engine<br> e.raise_exception()<br> File "XXX.local/lib/python2.7/site-packages/IPython/parallel/error.py", line 199, in raise_exception<br>
raise RemoteError(en, ev, etb, ei)<br>RemoteError: NameError(name 'plt' is not defined)</blockquote><div><br></div><div>The only thing that uses matplotlib is pandas, and modifying <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
imports = ['import numpy as np', 'import matplotlib.pyplot as plt', 'import pandas as pd']</blockquote><div><br></div><div>seems to solve the problem (although at least in one case the first call of my code crashed with the error and the second went through).</div>
<div><br></div><div>If I run my code without requiring the ipy parallel I don't have any problem with 'plt'</div><div><br></div><div>I guess that this is a bug. But I still haven't understood how to debug what happens on the engines, so I can't give more details.</div>
<div>Any clues?</div><div><br></div><div>If needed I can load my 'ipython_parallel.py' module in gist/github</div><div><br></div><div>Cheers,</div><div><br></div><div>Fra</div></div>
<br></div></div>_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>