[IPython-dev] finding the number of available engines
MinRK
benjaminrk at gmail.com
Wed Jan 19 18:49:30 EST 2011
It's peculiar - When I try it via a Task, it manages to cause a c-level
error in Twisted on the engine, bringing down the entire process:
Fatal Python error: PyEval_RestoreThread: NULL tstate
Strangely, I can create, connect, and use the clients without issue when I
use a MultiEngineClient.
I'm not sure why task submission would behave differently from execute.
I'll have the think about this.
-MinRK
On Wed, Jan 19, 2011 at 15:34, Satrajit Ghosh <satra at mit.edu> wrote:
> hi min,
>
> i've run into a related problem. i'm trying to run code on an engine that
> can itself spawn new tasks, but i can't seem to execute function below on an
> engine. it just doesn't return if you call this through the taskclient in
> blocking mode.
>
> it seems to block in the request for the task client and multiengine
> client.
>
> cheers,
>
> satra
>
> -----
> def get_ipy_clients():
> """Get ipython clients
>
> Returns client, taskclient and multiengineclient
> """
> ipyclient = None
> taskclient = None
> mecclient = None
> try:
> name = 'IPython.kernel.client'
> __import__(name)
> ipyclient = sys.modules[name]
> except ImportError:
> warn("Ipython kernel not found. Parallel execution will be" \
> "unavailable", ImportWarning)
> if ipyclient:
> try:
> taskclient = ipyclient.TaskClient()
> mecclient = ipyclient.MultiEngineClient()
> except Exception, e:
> if isinstance(e, ConnectionRefusedError):
> warn("No clients found, running serially for now.")
> if isinstance(e, ValueError):
> warn("Ipython kernel not installed")
> return ipyclient, taskclient, mecclient
> ------
>
>
>
> On Wed, Jan 19, 2011 at 2:42 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>> mec.queue_status() returns a list of the form:
>> [
>> (0, { 'pending' : "execute('a=5')", 'queue' : [ job1,job2,...] } ,
>> (1, {'pending' : 'None', 'queue' : [] },
>> ...
>> ]
>>
>> In this case, engine 1 is idle. I don't know why 1's pending is 'None'
>> instead of None, that seems to be a bug.
>>
>> So you can see the idle engines with something like:
>>
>> def idle_engines(mec):
>> """return list of engine_ids corresponding to idle engines."""
>> qs = mec.queue_status()
>> engines = []
>> for e_id, status in qs:
>> if status['queue']:
>> continue
>> if not status['pending'] or status['pending'] == 'None':
>> engines.append(e_id)
>> return engines
>>
>> Which would return a list of engine_ids that are idle, the length of which
>> would of course be the number of idle engines.
>>
>> -MinRK
>>
>>
>> On Wed, Jan 19, 2011 at 10:39, Satrajit Ghosh <satra at mit.edu> wrote:
>>
>>> hi brian and min,
>>>
>>> i would like to do something like this:
>>>
>>> if num_engines_available() > 2:
>>> do_x
>>> else:
>>> do_y
>>>
>>> in 0.10.1 series is there an easy way to query how many idle engines are
>>> available?
>>>
>>> cheers,
>>>
>>> satra
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110119/ca5f4f20/attachment.html>
More information about the IPython-dev
mailing list