Hi Paolo,<br><br> I am aware of the problem with calculating benchmark means, but let me
explain my point of view.<br><br>You are correct in that it would be preferable to have absolute times. Well, you actually can, but see what it happens: <a href="http://speed.pypy.org/comparison/?hor=true&bas=none&chart=stacked+bars">http://speed.pypy.org/comparison/?hor=true&bas=none&chart=stacked+bars</a><br>
<br>Absolute values would only work if we had carefully chosen benchmaks runtimes to be very similar (for our cpython baseline). As it is, html5lib, spitfire and spitfire_cstringio completely dominate the cummulative time. And not because the interpreter is faster or slower but because the benchmark was arbitrarily designed to run that long. Any improvement in the long running benchmarks will carry much more weight than in the short running.<br>
<br>What is more useful is to have comparable slices of time so that the improvements can be seen relatively over time. Normalizing does that i think. It just says: we have 21 tasks which take 1 second to run each on interpreter X (cpython in the default case). Then we see how other executables compare to that. What would the geometric mean achieve here, exactly, for the end user?<br>
<br>I am not really calculating any mean. You can see that I carefully avoided to display any kind of total bar which would indeed incur in the problem you mention. That a stacked chart implicitly displays a total is something you can not avoid, and for that kind of chart I still think normalized results is visually the best option.<br>
<br>Still, i would very much like to read the paper you cite, but you need a login for it.<br><br>Cheers,<br>Miquel<br><br><br><div class="gmail_quote">2010/6/25 Paolo Giarrusso <span dir="ltr"><<a href="mailto:p.giarrusso@gmail.com">p.giarrusso@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hi!<br>
First, I want to restate the obvious, before pointing out what I think<br>
is a mistake: your work on this website is great and very useful!<br>
<br>
On Fri, Jun 25, 2010 at 13:08, Miquel Torres <<a href="mailto:tobami@googlemail.com">tobami@googlemail.com</a>> wrote:<br>
> - stacked bars<br>
Here you are summing up normalized times, which is more or less like<br>
taking their arithmetic average. And that doesn't work at all: in many<br>
cases you can "show" completely different results by normalizing<br>
relatively to another item. Even the simple question "who is faster?"<br>
can be answered in different ways<br>
So you should use the geometric mean, even if this is not so widely<br>
known. Or better, it is known by benchmarking experts, but it's<br>
difficult to become so.<br>
<br>
Please, have a look at the short paper:<br>
"How not to lie with statistics: the correct way to summarize benchmark results"<br>
<a href="http://scholar.google.com/scholar?cluster=1051144955483053492&hl=en&as_sdt=2000" target="_blank">http://scholar.google.com/scholar?cluster=1051144955483053492&hl=en&as_sdt=2000</a><br>
I downloaded it from the ACM library, please tell me if you can't find it.<br>
<div class="im"><br>
> horizontal(<a href="http://speed.pypy.org/comparison/?hor=true&bas=2%2B35&chart=stacked+bars" target="_blank">http://speed.pypy.org/comparison/?hor=true&bas=2%2B35&chart=stacked+bars</a>):<br>
> This is not meant to "demonstrate" that overall the jit is over two times<br>
> faster than cpython. It is just another way for a developer to picture how<br>
> long a programme would take to complete if it were composed of 21 such<br>
> tasks.<br>
<br>
</div>You are not summing up absolute times, so your claim is incorrect. And<br>
the error is significant, given the above paper.<br>
A sum of absolute times would provide what you claim.<br>
<div class="im"><br>
> You can see that cpython's (the normalization chosen) benchmarks all<br>
> take 1"relative" second.<br>
</div>Here, for instance, I see that CPython and pypy-c take more or less<br>
the same time, which surprises me (since the PyPy interpreter was<br>
known to be slower than CPython). But given that the result is<br>
invalid, it may well be an artifact of your statistics.<br>
<div class="im"><br>
> pypy-c needs more or less the same time, some<br>
> "tasks" being slower and some faster. Psyco shows an interesting picture:<br>
> From meteor-contest downwards (fortuitously) , all benchmarks are extremely<br>
> "compressed", which means they are speeded up by psyco quite a lot. But any<br>
> further speed up wouldn't make overall time much shorter because the first<br>
> group of benchmarks now takes most of the time to complete. pypy-c-jit is a<br>
> more extreme case of this: If the jit accelerated all "fast" benchmarks to 0<br>
> seconds (infinitely fast), it would only get about twice as fast as now<br>
> because ai, slowspitfire, spambayes and twisted_tcp now need half the entire<br>
> execution time. An good demonstration of "you are only as fast as your<br>
> slowest part". Of course the aggregate of all benchmarks is not a real app,<br>
> but it is still fun.<br>
<br>
</div>This could maybe be still true, at least in part, but you have to do<br>
this reasoning on absolute times.<br>
<br>
Best regards, and keep up the good work!<br>
<font color="#888888">--<br>
Paolo Giarrusso - Ph.D. Student<br>
<a href="http://www.informatik.uni-marburg.de/%7Epgiarrusso/" target="_blank">http://www.informatik.uni-marburg.de/~pgiarrusso/</a><br>
</font></blockquote></div><br>