[IPython-dev] ZMQ Parallel IPython Performance preview

MinRK benjaminrk at gmail.com
Fri Oct 22 02:53:28 EDT 2010

I have my first performance numbers for throughput with the new parallel
code riding on ZeroMQ, and results are fairly promising.  Roundtrip time for
~512 tiny tasks submitted as fast as they can is ~100x faster than with

As a throughput test, I submitted a flood of many very small tasks that
should take ~no time:
def wait(t=0):
    import time
client.apply(wait, args=(t,))

task = StringTask("import time; time.sleep(%f)"%t)

Flooding the queue with these tasks with t=0, and then waiting for the
results, I tracked two times:
Sent: the time from the first submit until the last submit returns
Roundtrip: the time from the first submit to getting the last result

Plotting these times vs number of messages, we see some decent numbers:
* The pure ZMQ scheduler is fastest, 10-100 times faster than Twisted
* The Python scheduler is ~3x slower roundtrip than pure ZMQ, but no penalty
to the submission rate
* Twisted performance falls off very quickly as the number of tasks grows
* ZMQ performance is quite flat

zmq: the pure ZMQ Device is used for routing tasks
lru/weighted: the simplest/most complicated routing schemes respectively in
the Python ZMQ Scheduler (which supports dependencies)
twisted: the old IPython.kernel

[image: roundtrip.png]
[image: sent.png]
Test system:
Core-i7 930, 4x2 cores (ht), 4-engine cluster all over tcp/loopback, Ubuntu
10.04, Python 2.6.5

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: roundtrip.png
Type: image/png
Size: 30731 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sent.png
Type: image/png
Size: 31114 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment-0001.png>

More information about the IPython-dev mailing list