Re: [Speed] New CPython benchmark suite based on perf
On Tue, 5 Jul 2016 11:35:30 +0200 Victor Stinner <victor.stinner@gmail.com> wrote:
It looks like median helps to reduce the effect of outliers.
If you want to reduce the effect the outliers, you can just remove them: for example, ignore the 5% shortest samples and the 5% longest ones.
The median will not only reduce the effect of outliers but also completely ignore the value of most samples *except* the median sample.
In practice, it almost never occurs to have all samples with the same value. There is always a statistic distribution, usually as a gaussian curse.
If it's a gaussian curve (not a curse, probably :-)), then you can summarize it with two values: the mean and the stddev. But it's probably not a gaussian, because of system noise and other factors, so your assumption is wrong :-)
Regards
Antoine.
2016-07-05 12:08 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
If it's a gaussian curve (not a curse, probably :-)), then you can summarize it with two values: the mean and the stddev. But it's probably not a gaussian, because of system noise and other factors, so your assumption is wrong :-)
What do you propose? Revert to average (arithmeric mean) + std dev (centered on the average)?
Victor
On 5 July 2016 at 20:08, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 5 Jul 2016 11:35:30 +0200 Victor Stinner <victor.stinner@gmail.com> wrote:
In practice, it almost never occurs to have all samples with the same value. There is always a statistic distribution, usually as a gaussian curse.
If it's a gaussian curve (not a curse, probably :-)), then you can summarize it with two values: the mean and the stddev. But it's probably not a gaussian, because of system noise and other factors, so your assumption is wrong :-)
If you haven't already, I highly recommend reading the discussion in https://github.com/haypo/perf/issues/1 that led to Victor adopting the current median + stddev approach
As Mahmoud noted there, in terms of really understanding the benchmark results, there's no substitute for actually looking at the histograms with the result distributions. The numeric results are never going to be able to do more than provide a "flavour" for those results, since the distributions aren't Guassian, but trying to characterise and describe them properly would inevitably confuse folks that aren't already expert statisticians.
The median + stddev approach helps convey a "typical" result better than the minimum or mean do, while also providing an indication when the variation in results is too high for the median to really be meaningful.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Antoine Pitrou -
Nick Coghlan -
Victor Stinner