Min,<br><br><div class="gmail_quote"><div>Thanks for running these benchmarks, comments below...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Re-run for throughput with data:<div>
<br><div>submit 16 tasks of a given size, plot against size.</div><div>new-style:</div><div>def echo(a):</div><div> return a</div><div>old-style:</div><div>task = StringTask("b=a", push=dict(a=a), pull=['b'])<br>
<div><br></div></div></div></blockquote><div><br></div><div>I really like the style of the new API - echo is exactly what it does!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div><div><div class="gmail_quote">The input chosen was random numpy arrays (64b float, so len(A)/8 ~= size in B).</div><div class="gmail_quote"><br></div><div class="gmail_quote">Notable points:</div><div class="gmail_quote">
* ZMQ submission remains flat, independent of size, due to non-copying sends</div><div class="gmail_quote"></div></div></div></div></blockquote><div><br></div><div>We hoped that this would be the case, but this is really non-trivial and good to see.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div><div><div class="gmail_quote">* size doesn't come into account until ~100kB, and clearly dominates both after 1MB</div>
<div class="gmail_quote">
the turning point for Twisted is a little earlier than for ZMQ</div><div class="gmail_quote">* at 4MB, Twisted is submitting < 2 tasks per sec, while ZMQ is submitting ~90</div><div class="gmail_quote"></div></div>
</div></div></blockquote><div><br></div><div>This is a fantastic point of comparison. 4 MB is a non-trivial amount of data, and there is a huge difference between 0.5 second overhead (Twisted) and 0.01 sec overhead (zmq). It means that with zmq, users can get a parallel speedup on calculations that involve much less CPU cycles per byte of data sent.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div><div><div class="gmail_quote">* roundtrip, ZMQ is fairly consistently ~40x faster.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">memory usage:</div><div class="gmail_quote">* Peak memory for the engines is 20% higher with ZMQ, because more than one task can now be waiting in the queue on the engine at a time.</div>
<div class="gmail_quote"></div></div></div></div></blockquote><div><br></div><div>Right, but this is good news as it is offloading the data off the controller faster.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div><div><div class="gmail_quote">* Peak memory for the Controller including schedulers is 25% less than Twisted with pure ZMQ, and 20% less with the Python scheduler. Note that all results still reside in memory, since I haven't implemented the db backend yet.</div>
<div class="gmail_quote"></div></div></div></div></blockquote><div><br></div><div>I would think that is the biggest memory usage for the controller in the long run. But we know how to fix that.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div><div><div class="gmail_quote">* Peak memory for the Python scheduler is approximately the same as the engines</div><div class="gmail_quote"></div></div></div></div></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div><div><div class="gmail_quote">* Peak memory for the zmq scheduler is about half that.</div><div class="gmail_quote">
<br></div></div></div></div></blockquote><div><br></div><div>all very good news. I think these plots can definitely make it into a paper on this.</div><div><br></div><div>Cheers,</div><div><br></div><div>Brian</div><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div><div><div class="gmail_quote"></div><font color="#888888"><div class="gmail_quote">-MinRK</div></font><div>
<div></div><div class="h5"><div class="gmail_quote"><br></div><div class="gmail_quote">On Fri, Oct 22, 2010 at 09:52, MinRK <span dir="ltr"><<a href="mailto:benjaminrk@gmail.com" target="_blank">benjaminrk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'll get on the new tests, I already have a bandwidth one written, so I'm running it now. As for Twisted's throughput performance, it's at least partly our fault. Since the receiving is in Python, every time we try to send there are incoming results getting in the way. If we wrote it such that sending prevented the receipt of results, I'm sure the Twisted code would be faster for large numbers of messages. With ZMQ, though, we don't have to be receiving in Python to get the results to the client process, so they arrive in ZMQ and await simple memcpy/deserialization.<div>
<br></div><div><font color="#888888">-MinRK</font><div><div></div><div><br><br><div class="gmail_quote">On Fri, Oct 22, 2010 at 09:27, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Min,<div><br></div><div>Also, can you get memory consumption numbers for the controller and queues. I want to see how much worse Twisted is in that respect.</div><div><br></div><div>Cheers,</div><div><br></div><div><font color="#888888">Brian<br>
<br></font><div class="gmail_quote"><div>On Thu, Oct 21, 2010 at 11:53 PM, MinRK <span dir="ltr"><<a href="mailto:benjaminrk@gmail.com" target="_blank">benjaminrk@gmail.com</a>></span> wrote:<br></div><div>
<div></div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>I have my first performance numbers for throughput with the new parallel code riding on ZeroMQ, and results are fairly promising. Roundtrip time for ~512 tiny tasks submitted as fast as they can is ~100x faster than with Twisted.</div>
<div><br></div><div>As a throughput test, I submitted a flood of many very small tasks that should take ~no time:</div><div><div>new-style:</div><div>def wait(t=0):</div>
<div> import time</div><div> time.sleep(t)</div></div><div>submit:</div><div>client.apply(wait, args=(t,))</div><div><br></div><div>Twisted:</div><div>
task = StringTask("import time; time.sleep(%f)"%t)</div><div>submit:</div><div>client.run(task)</div><div><br></div><div>Flooding the queue with these tasks with t=0, and then waiting for the results, I tracked two times:</div>
<div>Sent: the time from the first submit until the last submit returns</div><div>Roundtrip: the time from the first submit to getting the last result</div><div><br></div><div>Plotting these times vs number of messages, we see some decent numbers:</div>
<div>* The pure ZMQ scheduler is fastest, 10-100 times faster than Twisted roundtrip</div><div>* The Python scheduler is ~3x slower roundtrip than pure ZMQ, but no penalty to the submission rate</div><div>* Twisted performance falls off very quickly as the number of tasks grows</div>
<div>* ZMQ performance is quite flat</div><div><br></div><div>Legend:</div><div>zmq: the pure ZMQ Device is used for routing tasks</div><div>lru/weighted: the simplest/most complicated routing schemes respectively in the Python ZMQ Scheduler (which supports dependencies)</div>
<div>twisted: the old IPython.kernel</div><div><br></div><div><img src="cid:ii_12bd2af6afc0eab2" alt="roundtrip.png" title="roundtrip.png"></div><div><img src="cid:ii_12bd2af38d3422df" alt="sent.png" title="sent.png"><br>
</div><div>Test system:</div><div>Core-i7 930, 4x2 cores (ht), 4-engine cluster all over tcp/loopback, Ubuntu 10.04, Python 2.6.5</div><div><br></div><font color="#888888"><div>-MinRK</div><div><a href="http://github.com/minrk" target="_blank">http://github.com/minrk</a></div>
</font></blockquote></div></div></div><br><br clear="all"><div><br>-- <br>Brian E. Granger, Ph.D.<br>Assistant Professor of Physics<br>Cal Poly State University, San Luis Obispo<br><a href="mailto:bgranger@calpoly.edu" target="_blank">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a><br>
</div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Brian E. Granger, Ph.D.<br>Assistant Professor of Physics<br>Cal Poly State University, San Luis Obispo<br><a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>