[IPython-dev] Load Balanced PyZMQ multikernel example

[MinRK]

I thought this might be of some interest to the zmq related IPython folks:

pyzmq has a basic multiple client-one kernel remote process example called
'kernel'.  This morning, to explore zmq devices, I wrote a derived example
that is multiple client - multiple kernel, and load balanced across kernels,
and called it 'multikernel'. It took about an hour.

The code is trivial, and uses the zmq XREQ socket's round robin load
balancing.
o The main addition is a relay process containing two zmq devices: a queue
device for the XREQ/XREP connection, and a forwarder for PUB/SUB.
o kernel.py had to change a little, since two socket IDs are contained in
each message instead of just one, and its sockets connect instead of bind.
o frontend.py and other code didn't have to change a letter.
o Exactly zero work is done in Python in the relay process after the
creation of the ØMQ devices.

It does have some weird behavior, since even the tab-completion requests are
load balanced, so if you have two kernels, and you do:
>>>a=5
>>>a='asdf'
>>>a.<tab>
...
>>>a.<tab>
...
each press of the tab key will produce different results - which is fun to
watch, if not especially useful.

I even did a quick and dirty screencast to show 30 seconds of using it with
2 clients and 2 kernels.
http://ptsg.berkeley.edu/~minrk/multikernel.m4v

The example is pushed to my pyzmq fork on github, and depends on that fork
for its implementation of ØMQ devices, not yet merged into Brian's trunk.
http://github.com/minrk/pyzmq

ØMQ really is spiffy.
-MinRK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100719/3cf8aee5/attachment.html>