[pypy-issue] Issue #2201: Benchmark results unintelligible (pypy/pypy)

Pete Vine issues-reply at bitbucket.org
Wed Dec 2 19:20:23 EST 2015

New issue 2201: Benchmark results unintelligible

Pete Vine:

Benchmarking two pypy binaries side by side (--changed option) prints the following example result to the console:

 ### ai ###
Min: 0.490526 -> 0.479404: 1.0232x faster
Avg: 0.494283 -> 0.486747: 1.0155x faster
Not significant
Stddev: 0.00369 -> 0.00644: 1.7443x larger

Provided the results are being presented in their respective running order, there's no clue as to whether **more** is better/worse meaning it's impossible to know which binary was actually faster!

More information about the pypy-issue mailing list