On 15.03.17 18:59, Victor Stinner wrote:
It's not easy to compare MAD to standard deviation. It seems like MAD can even be misleading when reading the "1 ms" part of "10 ms +- 1 ms".
Don't use the "+-" notation. It is misleading even for the stddev of normal distribution, because with the chance 1 against 2 the sample is out of the specified interval. Use "Mean: 10 ms Stddev: 1 ms" or "Median: 10 ms MAD: 1 ms" instead.
Maybe we need a new --median command line option to display median/MAD, instead of mean/stdev displayed by default?
Yes, make this configurable. And make median/MAD the default. ;)
2017-03-15 23:44 GMT+01:00 Serhiy Storchaka storchaka@gmail.com:
Don't use the "+-" notation. It is misleading even for the stddev of normal distribution, because with the chance 1 against 2 the sample is out of the specified interval. Use "Mean: 10 ms Stddev: 1 ms" or "Median: 10 ms MAD: 1 ms" instead.
I know that it's an abuse of "value +- range" notation.
Since I already changed the default formatting of a benchmark multiple times and it seems like Serhiy doesn't like the current format, a first action is to remove the public methods to format a benchmark :-) https://github.com/haypo/perf/commit/881a282cdac7969e3c759ff344ad766b3ae0f06...
So at least, I will not break the API if I change the format again in the future.
Victor
participants (2)
-
Serhiy Storchaka -
Victor Stinner