[IPython-dev] Named Engines

[Brian Granger]

On Wed, Jul 21, 2010 at 2:35 AM, MinRK <benjaminrk at gmail.com> wrote:
> I now have my MonitoredQueue object on git, which is the three socket Queue
> device that can be the core of the lightweight ME and Task models (depending
> on whether it is XREP on both sides for ME, or XREP/XREQ for load balanced
> tasks).

This sounds very cool.  What repos is this in?

> The biggest difference in terms of design between Python in the Controller
> picking the destination and this new device is that the client code actually
> needs to know the XREQ identity of each engine, and all the switching logic
> lives in the client code (if not the user exposed code) instead of the
> controller - if the client says 'do x in [1,2,3]' they actually issue 3
> sends, unlike before, when they issued 1 and the controller issued 3. This
> will increase traffic between the client and the controller, but
> dramatically reduce work done in the controller.

But because 0MQ has such low latency it might be a win.  Each request
the controller gets will be smaller and easier to handle.  The idea of
allowing clients to specify the names is something I have thought
about before.  One question though:  what does 0MQ do when you try to
send on an XREP socket to an identity that doesn't exist?  Will the
client be able to know that the client wasn't there?  That seems like
an important failure case.

> Since the engines' XREP IDs are known at the client level, and these are
> roughly any string, it brings up the question: should we have strictly
> integer ID engines, or should we allow engines to have names, like
> 'franklin1', corresponding directly to their XREP identity?

The idea of having names is pretty cool.  Maybe default to numbers,
but allow named prefixes as well as raw names?

> I think people might like using names, but I imagine it could get confusing.
>  It would be unambiguous in code, since we use integer IDs and XREP
> identities must be strings, so if someone keys on a string it must be the
> XREP id, and if they key on a number it must be by engine ID.

Right.  I will have a look at the code.

Cheers,

Brian

> -MinRK
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com