Hi,
I implemented a new feature in the pyperf compare_to command: compute the geometric mean of the benchmarks values mean normalized to the reference benchmark suite.
Before making a release, I'm looking for testers and feedback to ensure that I implemented it properly and makes sure that it's useful and relevant. You can test using these commands:
git clone https://github.com/psf/pyperf/ cd pyperf python3 -m pyperf compare_to ref.json my_change.json
Example of the new feature:
$ python3 -m pyperf compare_to ./pyperf/tests/mult_list_py36.json ./pyperf/tests/mult_list_py37.json [1]*1000: Mean +- std dev: [mult_list_py36] 2.13 us +- 0.06 us -> [mult_list_py37] 2.09 us +- 0.04 us: 1.02x faster (-2%) [1,2]*1000: Mean +- std dev: [mult_list_py36] 3.70 us +- 0.05 us -> [mult_list_py37] 5.28 us +- 0.09 us: 1.42x slower (+42%) [1,2,3]*1000: Mean +- std dev: [mult_list_py36] 4.61 us +- 0.13 us -> [mult_list_py37] 6.05 us +- 0.11 us: 1.31x slower (+31%)
Geometric mean: 1.22 (slower)
$ python3 -m pyperf compare_to ./pyperf/tests/mult_list_py36.json ./pyperf/tests/mult_list_py37.json -G Slower (2):
- [1,2]*1000: 3.70 us +- 0.05 us -> 5.28 us +- 0.09 us: 1.42x slower (+42%)
- [1,2,3]*1000: 4.61 us +- 0.13 us -> 6.05 us +- 0.11 us: 1.31x slower (+31%)
Faster (1):
- [1]*1000: 2.13 us +- 0.06 us -> 2.09 us +- 0.04 us: 1.02x faster (-2%)
Geometric mean: 1.22 (slower)
$ python3 -m pyperf compare_to ./pyperf/tests/mult_list_py36.json ./pyperf/tests/mult_list_py37.json --table +----------------+----------------+------------------------------+ | Benchmark | mult_list_py36 | mult_list_py37 | +================+================+==============================+ | [1]*1000 | 2.13 us | 2.09 us: 1.02x faster (-2%) | +----------------+----------------+------------------------------+ | [1,2]*1000 | 3.70 us | 5.28 us: 1.42x slower (+42%) | +----------------+----------------+------------------------------+ | [1,2,3]*1000 | 4.61 us | 6.05 us: 1.31x slower (+31%) | +----------------+----------------+------------------------------+ | Geometric mean | (ref) | 1.22 (slower) | +----------------+----------------+------------------------------+
$ python3 -m pyperf compare_to ./pyperf/tests/mult_list_py36.json ./pyperf/tests/mult_list_py37.json --table -G +----------------+----------------+------------------------------+ | Benchmark | mult_list_py36 | mult_list_py37 | +================+================+==============================+ | [1]*1000 | 2.13 us | 2.09 us: 1.02x faster (-2%) | +----------------+----------------+------------------------------+ | [1,2,3]*1000 | 4.61 us | 6.05 us: 1.31x slower (+31%) | +----------------+----------------+------------------------------+ | [1,2]*1000 | 3.70 us | 5.28 us: 1.42x slower (+42%) | +----------------+----------------+------------------------------+ | Geometric mean | (ref) | 1.22 (slower) | +----------------+----------------+------------------------------+
Victor
Night gathers, and now my watch begins. It shall not end until my death.