April 26, 2016
4:36 p.m.
On Tue, 26 Apr 2016 18:28:32 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
taking the minimum is a terrible idea anyway, none of the statistical discussion makes sense if you do that
The minimum is a reasonable metric for quick throwaway benchmarks as timeit is designed for, as it has a better hope of alleviating the impact of system load (as such throwaway benchmarks are often run on the developer's workstation).
For a persistent benchmarks suite, where we can afford longer benchmark runtimes and are able to keep system noise to a minimum, we might prefer another metric.
Regards
Antoine.