[IPython-dev] finding the number of available engines

[MinRK]

It's peculiar - When I try it via a Task, it manages to cause a c-level
error in Twisted on the engine, bringing down the entire process:
Fatal Python error: PyEval_RestoreThread: NULL tstate

Strangely, I can create, connect, and use the clients without issue when I
use a MultiEngineClient.

I'm not sure why task submission would behave differently from execute.
 I'll have the think about this.

-MinRK


On Wed, Jan 19, 2011 at 15:34, Satrajit Ghosh <satra at mit.edu> wrote:

> hi min,
>
> i've run into a related problem. i'm trying to run code on an engine that
> can itself spawn new tasks, but i can't seem to execute function below on an
> engine. it just doesn't return if you call this through the taskclient in
> blocking mode.
>
> it seems to block in the request for the task client and multiengine
> client.
>
> cheers,
>
> satra
>
> -----
> def get_ipy_clients():
>     """Get ipython clients
>
>     Returns client, taskclient and multiengineclient
>     """
>     ipyclient = None
>     taskclient = None
>     mecclient = None
>     try:
>         name = 'IPython.kernel.client'
>         __import__(name)
>         ipyclient = sys.modules[name]
>     except ImportError:
>         warn("Ipython kernel not found.  Parallel execution will be" \
>                  "unavailable", ImportWarning)
>     if ipyclient:
>         try:
>             taskclient = ipyclient.TaskClient()
>             mecclient = ipyclient.MultiEngineClient()
>         except Exception, e:
>             if isinstance(e, ConnectionRefusedError):
>                 warn("No clients found, running serially for now.")
>             if isinstance(e, ValueError):
>                 warn("Ipython kernel not installed")
>     return ipyclient, taskclient, mecclient
> ------
>
>
>
> On Wed, Jan 19, 2011 at 2:42 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>> mec.queue_status() returns a list of the form:
>> [
>>   (0, { 'pending' : "execute('a=5')", 'queue' : [ job1,job2,...] } ,
>>   (1, {'pending' : 'None', 'queue' : [] },
>>   ...
>> ]
>>
>> In this case, engine 1 is idle.  I don't know why 1's pending is 'None'
>> instead of None, that seems to be a bug.
>>
>> So you can see the idle engines with something like:
>>
>> def idle_engines(mec):
>>     """return list of engine_ids corresponding to idle engines."""
>>     qs = mec.queue_status()
>>     engines = []
>>     for e_id, status in qs:
>>         if status['queue']:
>>             continue
>>         if not status['pending'] or status['pending'] == 'None':
>>             engines.append(e_id)
>>     return engines
>>
>> Which would return a list of engine_ids that are idle, the length of which
>> would of course be the number of idle engines.
>>
>> -MinRK
>>
>>
>> On Wed, Jan 19, 2011 at 10:39, Satrajit Ghosh <satra at mit.edu> wrote:
>>
>>> hi brian and min,
>>>
>>> i would like to do something like this:
>>>
>>> if num_engines_available() > 2:
>>>     do_x
>>> else:
>>>     do_y
>>>
>>> in 0.10.1 series is there an easy way to query how many idle engines are
>>> available?
>>>
>>> cheers,
>>>
>>> satra
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110119/ca5f4f20/attachment.html>