Mailman 3 March 2017 - Speed

Re: [Speed] Median +- MAD or Mean +- std dev?
by Antoine Pitrou March 15, 2017

March 15, 2017

On Wed, 15 Mar 2017 17:59:07 +0100 Victor Stinner <victor.stinner(a)gmail.com> wrote: > > The perf module has already a function to emit warnings if a benchmark > is considered as "unstable". A warning is emitted if stdev/mean is > greater than 0.10. I chose this threshold arbitrary. It doesn't sound too bad :-) > Maybe we need another check to emit a warning when mean and median, or > std dev and MAD are too different? > > Maybe we need a new --median command line option to display > median/MAD, instead of mean/stdev displayed by default? I would say keep it simple. mean/stddev is informative enough, no need to add or maintain options of dubious utility. Regards Antoine.

2 1

Re: [Speed] Median +- MAD or Mean +- std dev?
by Antoine Pitrou March 15, 2017

March 15, 2017

On Tue, 7 Mar 2017 01:03:23 +0100 Victor Stinner <victor.stinner(a)gmail.com> wrote: > Another example on the same computer. It's interesting: > * MAD and std dev is the half of result 1 > * the benchmark is less unstable > * median is very close to result 1 > * mean changed much more than median > > Benchmark result 1: > > Median +- MAD: 276 ns +- 10 ns > Mean +- std dev: 371 ns +- 196 ns > > Benchmark result 2: > > Median +- MAD: 278 ns +- 5 ns > Mean +- std dev: 303 ns +- 103 ns > > If the goal is to get reproductible results, Median +- MAD seems better. Getting reproducible results is only half of the goal. Getting meaningful (i.e. informative) results is the other half. The mean approximates the expected performance over multiple runs (note "expected" is a rigorously defined term in statistics here: see https://en.wikipedia.org/wiki/Expected_value). The median doesn't tell you anything about the expected value (*). So the mean is more informative for the task at hand. Additionally, while mean and std dev are generally quite well understood, the properties of the median absolute deviation are generally little known. So my vote goes to mean +/- std dev. (*) Quick example: let's say your runtimes in seconds are [1, 1, 1, 1, 1, 1, 10, 10, 10, 10]. Evidently, there are four outliers (over 10 measurements) that indicate a huge performance regression occurring at random points. However, the median here is 1 and the median absolute deviation (the median of absolute deviations from the median, i.e. the median of [0, 0, 0, 0, 0, 0, 9, 9, 9, 9]) is 0: the information about possible performance regressions is entirely lost, and the numbers (median +/- MAD) make it look like the benchmark reliably takes 1 s. to run. Regards Antoine.

4 8

Re: [Speed] Median +- MAD or Mean +- std dev?
by Antoine Pitrou March 15, 2017

March 15, 2017

On Wed, 15 Mar 2017 00:42:07 +1000 Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > That would suggest that the implicit assumption of a measure-of-centrality > with a measure-of-symmetric-deviation may need to be challenged, as at > least some meaningful performance problems are going to show up as > non-normal distributions in the benchmark results. Well, the real issue here is that an important contributor to non-normality is not the benchmark itself, but measurement noise due to various issues (such as system noise, which has of course a highly skewed distribution). Victor is trying to eliminate the effects of system noise by using the median, but if that's the primary goal, using the minimum is arguably better, since the system noise is always a positive contributor (i.e. it can only increase the runtimes). The median is arguably a bastardized solution, which satisfies neither the requirement of eliminating system noise, nor the requirement of faithfully representing performance variations due to non-deterministic effects in the Python runtime and/or benchmark itself. Regards Antoine.

2 1

Re: [Speed] Median +- MAD or Mean +- std dev?
by Victor Stinner March 15, 2017

March 15, 2017

2017-03-14 15:42 GMT+01:00 Nick Coghlan <ncoghlan(a)gmail.com>: > That would suggest that the implicit assumption of a measure-of-centrality > with a measure-of-symmetric-deviation may need to be challenged, as at least > some meaningful performance problems are going to show up as non-normal > distributions in the benchmark results. > > Network services typically get around the "inherent variance" problem by > looking at a few key percentiles like 50%, 90% and 95%. Perhaps that would > be appropriate here as well? Right now, there is almost no visualisation tool for perf :-( It started to list projects that may be reused to visualize benchmark results, to "see" the distribution. A first step would be to add these "key percentiles like 50%, 90% and 95%" to the perf stats command. I don't know how to compute them. But my question is for the most important summary: the result of the "perf show" command, which is what most users see except if they use more advanced commands. Victor

1 0

Re: [Speed] Median +- MAD or Mean +- std dev?
by Serhiy Storchaka March 15, 2017

March 15, 2017

On 14.03.17 16:42, Nick Coghlan wrote: > That would suggest that the implicit assumption of a > measure-of-centrality with a measure-of-symmetric-deviation may need to > be challenged, as at least some meaningful performance problems are > going to show up as non-normal distributions in the benchmark results. > > Network services typically get around the "inherent variance" problem by > looking at a few key percentiles like 50%, 90% and 95%. Perhaps that > would be appropriate here as well? Yes, quantiles would be useful, but I suppose they are less stable. If you have have only 20 samples, it is not enough to determine the 95% percentile. But absolute values are not important for the purposes of our benchmarking. We need only know whether one build is faster or slower than others. I suggested to calculate the probability of one build be faster than the other when compare two builds. This is just one number and it doesn't depend on assumptions about the normality of distributions.

2 1

Re: [Speed] Median +- MAD or Mean +- std dev?
by Victor Stinner March 7, 2017

March 7, 2017

Another example on the same computer. It's interesting: * MAD and std dev is the half of result 1 * the benchmark is less unstable * median is very close to result 1 * mean changed much more than median Benchmark result 1: Median +- MAD: 276 ns +- 10 ns Mean +- std dev: 371 ns +- 196 ns Benchmark result 2: Median +- MAD: 278 ns +- 5 ns Mean +- std dev: 303 ns +- 103 ns If the goal is to get reproductible results, Median +- MAD seems better. --- haypo@selma$ PYTHONPATH=~/prog/GIT/perf ./python -m perf show --hist --stats bench2.json.gz 250 ns: 75 ########################################################### 278 ns: 73 ######################################################### 306 ns: 3 ## 333 ns: 0 | 361 ns: 0 | 389 ns: 0 | 417 ns: 0 | 445 ns: 0 | 472 ns: 1 # 500 ns: 1 # 528 ns: 0 | 556 ns: 0 | 584 ns: 0 | 611 ns: 1 # 639 ns: 0 | 667 ns: 0 | 695 ns: 1 # 722 ns: 0 | 750 ns: 1 # 778 ns: 1 # 806 ns: 3 ## Total duration: 1 min 4.0 sec Start date: 2017-03-07 00:39:03 End date: 2017-03-07 00:41:05 Raw sample minimum: 140 ms Raw sample maximum: 431 ms Number of runs: 42 Total number of samples: 160 Number of samples per run: 4 Number of warmups per run: 2 Loop iterations per sample: 2^19 (128 outer-loops x 4096 inner-loops) Minimum: 266 ns (-4%) Median +- MAD: 278 ns +- 5 ns Mean +- std dev: 303 ns +- 103 ns Maximum: 822 ns (+195%) ERROR: the benchmark is very unstable, the standard deviation is very high (stdev/mean: 34%)! Try to rerun the benchmark with more runs, samples and/or loops Median +- MAD: 278 ns +- 5 ns ---

1 0

Median +- MAD or Mean +- std dev?
by Victor Stinner March 6, 2017

March 6, 2017

Hi, Serhiy Storchaka opened a bug report in my perf module: perf displays Median +- std dev, whereas median absolute deviation (MAD) should be displayed instead: https://github.com/haypo/perf/issues/20 I just modified perf to display Median +- MAD, but I'm not sure that it's better than Mean +- std dev. The question is important when a benchmark is unstable (has a lot of outliers). There is good example below with "Median +- MAD: 276 ns +- 10 ns" and "Mean +- std dev: 371 ns +- 196 ns". The goal of perf is to get reproductible benchmark results. So the question is what should be displayed (median or mean?) to get the most reproductible output? Median +- MAD "hides" outliers. In my experience, outliers are not "reproductible", but caused by "noise" of the system and other applications. I feel that Median +- MAD is what I want, but I would feel more confortable if someone can confirm with his/her experience :-) ----------------- haypo@selma$ PYTHONPATH=~/prog/GIT/perf ./python -m perf show --hist --stats bench.json.gz 234 ns: 3 # 264 ns: 114 ################################################## 293 ns: 9 #### 322 ns: 2 # 351 ns: 0 | 381 ns: 0 | 410 ns: 0 | 439 ns: 1 | 469 ns: 0 | 498 ns: 1 | 527 ns: 1 | 557 ns: 0 | 586 ns: 1 | 615 ns: 1 | 644 ns: 1 | 674 ns: 2 # 703 ns: 1 | 732 ns: 1 | 762 ns: 2 # 791 ns: 15 ####### 820 ns: 5 ## Total duration: 1 min 14.5 sec Start date: 2017-03-06 23:30:49 End date: 2017-03-06 23:33:11 Raw sample minimum: 137 ms Raw sample maximum: 444 ms Number of runs: 42 Total number of samples: 160 Number of samples per run: 4 Number of warmups per run: 2 Loop iterations per sample: 2^19 (128 outer-loops x 4096 inner-loops) Minimum: 262 ns (-5%) Median +- MAD: 276 ns +- 10 ns Mean +- std dev: 371 ns +- 196 ns Maximum: 847 ns (+207%) ERROR: the benchmark is very unstable, the standard deviation is very high (stdev/mean: 53%)! Try to rerun the benchmark with more runs, samples and/or loops Median +- MAD: 276 ns +- 10 ns ----------------- See attached bench.json.gz for full data. Victor

1 0

Re: [Speed] perf 0.9.4 released
by Victor Stinner March 1, 2017

March 1, 2017

The perf API, command line interface (API) and JSON file format are now complete enough for *my* needs. I plan to use the version 1.0 for the next release and stabilize the API. I still have a long list of enhancement ideas (see the TODO.rst file in the Git repository), but none has a major impact on the API. If you see a major flaw in the API or CLI, please speak up! I know that the PyPy support is very limited, but again, fixing PyPy support shouldn't impact the CLI or API, and so can be done later. 2017-03-01 18:05 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>: > Hum, I should write an option to allow to specify the name of python > binaries, to replace [/home/haypo/prog/bench_python/master-ref/python] > just with [ref]. Ok, I just added a --python-names option ;-) I annoyed me to have to modify manually perf output when posting to bugs.python.org to replace long [...] with short [ref] and [patch]. Victor

1 0

perf 0.9.4 released
by Victor Stinner March 1, 2017

March 1, 2017

Hi, I released the version 0.9.4 of my Python perf module: * Add --compare-to option to the Runner CLI * compare_to command: Add --table option to render a table http://perf.readthedocs.io/en/latest/ I used the --table feature to write this FASTCALL microbenchmarks article: https://haypo.github.io/fastcall-microbenchmarks.html Example: +---------------------+---------+------------------------------+ | Benchmark | 3.5 | 3.7 | +=====================+=========+==============================+ | struct.pack("i", 1) | 105 ns | 77.6 ns: 1.36x faster (-26%) | +---------------------+---------+------------------------------+ | getattr(1, "real") | 79.4 ns | 64.4 ns: 1.23x faster (-19%) | +---------------------+---------+------------------------------+ Use --quiet for smaller table. The --compare-to command is the generalization to any perf script of the existing perf timeit --compare-to option, to quickly compare two Python binaries. Example with timeit (because I'm too lazy to write a perf script!): --- $ ./python -m perf timeit 'int(0)' --duplicate=100 --compare-to ../master-ref/python -p3 /home/haypo/prog/bench_python/master-ref/python: .... 112 ns +- 1 ns /home/haypo/prog/bench_python/master/python: .... 108 ns +- 1 ns Median +- std dev: [/home/haypo/prog/bench_python/master-ref/python] 112 ns +- 1 ns -> [/home/haypo/prog/bench_python/master/python] 108 ns +- 1 ns: 1.04x faster (-3%) ---- Hum, I should write an option to allow to specify the name of python binaries, to replace [/home/haypo/prog/bench_python/master-ref/python] just with [ref]. Victor

1 0