Hi,

I released pyperf 2.1.0: the compare_to command now computes the geometric mean of a whole benchmark suite and no longer displays percentages (display less number to not confuse readers).

If the benchmark suites contain more than one benchmark, the geometric mean is computed: normalize benchmark results means to the reference results means.

For comparisons, pyperf only displays numbers greater than or equal to 1.0 but mentions "faster" or "slower". speed.pypy.org displays the geometric *both* ways: "The geometric average of all benchmarks is 0.23 or 4.3 times faster than cpython." I prefer to only displays a single number, so I picked "4.3x faster" rather than "0.23 (faster)".

Example:

$ python3 -m pyperf compare_to --table mult_list_py36.json mult_list_py37.json mult_list_py38.json +----------------+----------------+-----------------------+-----------------------+ | Benchmark | mult_list_py36 | mult_list_py37 | mult_list_py38 | +================+================+=======================+=======================+ | [1]*1000 | 2.13 us | 2.09 us: 1.02x faster | not significant | +----------------+----------------+-----------------------+-----------------------+ | [1,2]*1000 | 3.70 us | 5.28 us: 1.42x slower | 3.18 us: 1.16x faster | +----------------+----------------+-----------------------+-----------------------+ | [1,2,3]*1000 | 4.61 us | 6.05 us: 1.31x slower | 4.17 us: 1.11x faster | +----------------+----------------+-----------------------+-----------------------+ | Geometric mean | (ref) | 1.22x slower | 1.09x faster | +----------------+----------------+-----------------------+-----------------------+

Here you can see that Python 3.7 is faster than Python 3.6 on one benchmark, but slower on two benchmarks. What does it mean? Is it slower or faster "overall"? The geometric mean "1.22x slower" helps to quickly see that Python 3.7 is overall slower.

Python 3.8 is faster than Python 3.6 on two benchmarks and one benchmark is not significant. The geometric mean "1.09x faster" confirms that it's faster, but it also tells how much "overall".

The geometric mean gives the same weight (1.0) to all benchmarks. It's up to you to carefully check the benchmarks that you consider as the most important, and not only rely on the geometric mean.

See also compare_to documentation for more details: https://pyperf.readthedocs.io/en/latest/cli.html#compare-to-cmd

Victor

Night gathers, and now my watch begins. It shall not end until my death.