[IPython-dev] Named Engines

[MinRK]

On Wed, Jul 21, 2010 at 10:07, Brian Granger <ellisonbg at gmail.com> wrote:

> On Wed, Jul 21, 2010 at 2:35 AM, MinRK <benjaminrk at gmail.com> wrote:
> > I now have my MonitoredQueue object on git, which is the three socket
> Queue
> > device that can be the core of the lightweight ME and Task models
> (depending
> > on whether it is XREP on both sides for ME, or XREP/XREQ for load
> balanced
> > tasks).
>
> This sounds very cool.  What repos is this in?
>

all on my pyzmq master: github.com/minrk/pyzmq

The Devices are specified in the growing _zmq.pyx. Should I move them?  I
don't have enough Cython experience (this is my first nontrivial Cython
work) to know how to correctly move it to a new file still with all the
right zmq imports.


> > The biggest difference in terms of design between Python in the
> Controller
> > picking the destination and this new device is that the client code
> actually
> > needs to know the XREQ identity of each engine, and all the switching
> logic
> > lives in the client code (if not the user exposed code) instead of the
> > controller - if the client says 'do x in [1,2,3]' they actually issue 3
> > sends, unlike before, when they issued 1 and the controller issued 3.
> This
> > will increase traffic between the client and the controller, but
> > dramatically reduce work done in the controller.
>
> But because 0MQ has such low latency it might be a win.  Each request
> the controller gets will be smaller and easier to handle.  The idea of
> allowing clients to specify the names is something I have thought
> about before.  One question though:  what does 0MQ do when you try to
> send on an XREP socket to an identity that doesn't exist?  Will the
> client be able to know that the client wasn't there?  That seems like
> an important failure case.
>

As far as I can tell, the XREP socket sends messages out to XREQ ids, and
trusts that such an XREQ exists. If no such id is connected, the message is
silently lost to the aether.  However, with the controller monitoring the
queue, it knows when you have sent a message to an engine that is not
_registered_, and can tell you about it. This should be sufficient, since
presumably all the connected XREQ sockets should be registered engines.

To test:
a = ctx.socket(zmq.XREP)
a.bind('tcp://127.0.0.1:1234')
b = ctx.socket(zmq.XREQ)
b.setsockopt(zmq.IDENTITY, 'hello')
a.send_multipart(['hello', 'mr. b'])
time.sleep(.2)
b.connect('tcp://127.0.0.1:1234')
a.send_multipart(['hello', 'again'])
b.recv()
# 'again'



>
> > Since the engines' XREP IDs are known at the client level, and these are
> > roughly any string, it brings up the question: should we have strictly
> > integer ID engines, or should we allow engines to have names, like
> > 'franklin1', corresponding directly to their XREP identity?
>
> The idea of having names is pretty cool.  Maybe default to numbers,
> but allow named prefixes as well as raw names?
>

This part is purely up to our user-facing side of the client code. It
certainly doesn't affect how anything works inside. It's just a question of
what a valid `targets' argument (or key for a dictionary interface) would be
in the multiengine.


>
> > I think people might like using names, but I imagine it could get
> confusing.
> >  It would be unambiguous in code, since we use integer IDs and XREP
> > identities must be strings, so if someone keys on a string it must be the
> > XREP id, and if they key on a number it must be by engine ID.
>
> Right.  I will have a look at the code.
>
> Cheers,
>
> Brian
>
> > -MinRK
> >
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100721/48f83b80/attachment.html>