[IPython-dev] Load Balanced PyZMQ multikernel example

[Brian Granger]

Min,

This is a very nice demonstration of what you can do by hooking up
some 0mq sockets and devices.  I find that the possibilities are so
many that that it takes a while to really let it sink in.

Cheers,

Brian

On Mon, Jul 19, 2010 at 5:33 PM, MinRK <benjaminrk at gmail.com> wrote:
> I thought this might be of some interest to the zmq related IPython folks:
> pyzmq has a basic multiple client-one kernel remote process example called
> 'kernel'.  This morning, to explore zmq devices, I wrote a derived example
> that is multiple client - multiple kernel, and load balanced across kernels,
> and called it 'multikernel'. It took about an hour.
> The code is trivial, and uses the zmq XREQ socket's round robin load
> balancing.
> o The main addition is a relay process containing two zmq devices: a queue
> device for the XREQ/XREP connection, and a forwarder for PUB/SUB.
> o kernel.py had to change a little, since two socket IDs are contained in
> each message instead of just one, and its sockets connect instead of bind.
> o frontend.py and other code didn't have to change a letter.
> o Exactly zero work is done in Python in the relay process after the
> creation of the ØMQ devices.
> It does have some weird behavior, since even the tab-completion requests are
> load balanced, so if you have two kernels, and you do:
>>>>a=5
>>>>a='asdf'
>>>>a.<tab>
> ...
>>>>a.<tab>
> ...
> each press of the tab key will produce different results - which is fun to
> watch, if not especially useful.
> I even did a quick and dirty screencast to show 30 seconds of using it with
> 2 clients and 2 kernels.
> http://ptsg.berkeley.edu/~minrk/multikernel.m4v
> The example is pushed to my pyzmq fork on github, and depends on that fork
> for its implementation of ØMQ devices, not yet merged into Brian's trunk.
> http://github.com/minrk/pyzmq
> ØMQ really is spiffy.
> -MinRK



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com