<div class="gmail_quote">On Fri, Jan 21, 2011 at 11:38, Satrajit Ghosh <span dir="ltr"><<a href="mailto:satra@mit.edu">satra@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
thanks min and brian. <br><br>we'll probably put this feature aside for the time being then. <br><br>two questions:<br><br>1. fernando had mentioned that there was an effort to allow creation of engines on the fly with shwazik (i'm sure i'm spelling her name wrong). any update on the status of this? and was it going to work with 0.10.x or was it primarily intended for 0.11.x<br>
</blockquote><div><br></div><div>Fernando and Soizic's work for starting engines was only for zmq.  It shouldn't be terribly difficult to start new engines in either one, depend on who you want to be starting them.  What are the conditions that you want to trigger new engines, and what environment (pbs, etc.)?  Since starting an engine is just a shell script, starting a new engine on the fly is just executing the same script, via subprocess.Popen, or some such.</div>
<meta charset="utf-8"><div>Â </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>2. when is the eta for a beta version of parallel operation with 0.11.x? we would love to move over to 0.11 but waiting for the parallel stuff currently.<br></blockquote><div><br></div><div>Hopefully it will be fairly soon. I was very busy with exams until recently, and Fernando is busy with work and travel now. Â If you want to play with the current state, which I would call 'alpha', because we have yet to do a final feature/API freeze, but the `apply` core at the client level is pretty static. it's at <a href="https://github.com/ipython/ipython/tree/newparallel">https://github.com/ipython/ipython/tree/newparallel</a>.</div>
<div><br></div><div>-MinRK</div><div>Â </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br clear="all">cheers,<br><br>satra<div><div></div><div class="h5"><br><br>
<br><div class="gmail_quote">
On Fri, Jan 21, 2011 at 1:08 PM, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Satra,<br>
<div><br>
> i've run into a related problem. i'm trying to run code on an engine that<br>
> can itself spawn new tasks, but i can't seem to execute function below on an<br>
> engine. it just doesn't return if you call this through the taskclient in<br>
> blocking mode.<br>
<br>
</div>This usage case of recursive tasks, is not supported with the current<br>
twisted based code. Â The reason has to do with subtle issues about how<br>
the Twisted event loop is run in the engines versus the client. Â For<br>
the client, we run the Twisted event loop in a thread, which is not<br>
compatible with the engine, which runs it in the main thread. Â This is<br>
probably why Min is seeing the odd GIL/thread related crash.<br>
<div><br>
> it seems to block in the request for the task client and multiengine client.<br>
<br>
</div>Yes, this is what you would observe in the current twisted version of<br>
the code. Â The new zeromq stuff Min has written doesn't have this<br>
limitation.<br>
<br>
Sorry about the hassle.<br>
<br>
Cheers,<br>
<br>
Brian<br>
<div><div></div><div><br>
> cheers,<br>
><br>
> satra<br>
><br>
> -----<br>
> def get_ipy_clients():<br>
> Â Â Â """Get ipython clients<br>
><br>
> Â Â Â Returns client, taskclient and multiengineclient<br>
> Â Â Â """<br>
> Â Â Â ipyclient = None<br>
> Â Â Â taskclient = None<br>
> Â Â Â mecclient = None<br>
> Â Â Â try:<br>
> Â Â Â Â Â Â Â name = 'IPython.kernel.client'<br>
> Â Â Â Â Â Â Â __import__(name)<br>
> Â Â Â Â Â Â Â ipyclient = sys.modules[name]<br>
> Â Â Â except ImportError:<br>
>        warn("Ipython kernel not found. Parallel execution will be" \<br>
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "unavailable", ImportWarning)<br>
> Â Â Â if ipyclient:<br>
> Â Â Â Â Â Â Â try:<br>
> Â Â Â Â Â Â Â Â Â Â Â taskclient = ipyclient.TaskClient()<br>
> Â Â Â Â Â Â Â Â Â Â Â mecclient = ipyclient.MultiEngineClient()<br>
> Â Â Â Â Â Â Â except Exception, e:<br>
> Â Â Â Â Â Â Â Â Â Â Â if isinstance(e, ConnectionRefusedError):<br>
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â warn("No clients found, running serially for now.")<br>
> Â Â Â Â Â Â Â Â Â Â Â if isinstance(e, ValueError):<br>
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â warn("Ipython kernel not installed")<br>
> Â Â Â return ipyclient, taskclient, mecclient<br>
> ------<br>
><br>
><br>
><br>
> On Wed, Jan 19, 2011 at 2:42 PM, MinRK <<a href="mailto:benjaminrk@gmail.com" target="_blank">benjaminrk@gmail.com</a>> wrote:<br>
>><br>
>> mec.queue_status() returns a list of the form:<br>
>> [<br>
>> Â Â (0, { 'pending' : "execute('a=5')", 'queue' : [ job1,job2,...] } ,<br>
>> Â Â (1, {'pending' : 'None', 'queue' : [] },<br>
>> Â Â ...<br>
>> ]<br>
>><br>
>> In this case, engine 1 is idle. Â I don't know why 1's pending is 'None'<br>
>> instead of None, that seems to be a bug.<br>
>> So you can see the idle engines with something like:<br>
>> def idle_engines(mec):<br>
>> Â Â Â """return list of engine_ids corresponding to idle engines."""<br>
>> Â Â Â qs = mec.queue_status()<br>
>> Â Â Â engines = []<br>
>> Â Â Â for e_id, status in qs:<br>
>> Â Â Â Â Â if status['queue']:<br>
>> Â Â Â Â Â Â Â continue<br>
>> Â Â Â Â Â if not status['pending'] or status['pending'] == 'None':<br>
>> Â Â Â Â Â Â Â engines.append(e_id)<br>
>> Â Â Â return engines<br>
>> Which would return a list of engine_ids that are idle, the length of which<br>
>> would of course be the number of idle engines.<br>
>> -MinRK<br>
>><br>
>><br>
>> On Wed, Jan 19, 2011 at 10:39, Satrajit Ghosh <<a href="mailto:satra@mit.edu" target="_blank">satra@mit.edu</a>> wrote:<br>
>>><br>
>>> hi brian and min,<br>
>>><br>
>>> i would like to do something like this:<br>
>>><br>
>>> if num_engines_available() > 2:<br>
>>> Â Â Â do_x<br>
>>> else:<br>
>>> Â Â Â do_y<br>
>>><br>
>>> in 0.10.1 series is there an easy way to query how many idle engines are<br>
>>> available?<br>
>>><br>
>>> cheers,<br>
>>><br>
>>> satra<br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> IPython-dev mailing list<br>
>>> <a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
>>> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
>>><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> IPython-dev mailing list<br>
> <a href="mailto:IPython-dev@scipy.org" target="_blank">IPython-dev@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
><br>
<br>
<br>
<br>
</div></div><font color="#888888">--<br>
Brian E. Granger, Ph.D.<br>
Assistant Professor of Physics<br>
Cal Poly State University, San Luis Obispo<br>
<a href="mailto:bgranger@calpoly.edu" target="_blank">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a><br>
</font></blockquote></div><br>
</div></div></blockquote></div><br>