On Wed, 15 Mar 2017 00:42:07 +1000
Nick Coghlan <ncoghlan(a)gmail.com> wrote:
>
> That would suggest that the implicit assumption of a measure-of-centrality
> with a measure-of-symmetric-deviation may need to be challenged, as at
> least some meaningful performance problems are going to show up as
> non-normal distributions in the benchmark results.
Well, the real issue here is that an important contributor to
non-normality is not the benchmark itself, but measurement noise due to
various issues (such as system noise, which has of course a highly
skewed distribution).
Victor is trying to eliminate the effects of system noise by using the
median, but if that's the primary goal, using the minimum is arguably
better, since the system noise is always a positive contributor (i.e.
it can only increase the runtimes).
The median is arguably a bastardized solution, which satisfies neither
the requirement of eliminating system noise, nor the requirement of
faithfully representing performance variations due to non-deterministic
effects in the Python runtime and/or benchmark itself.
Regards
Antoine.