[Python-Dev] Stop using timeit, use perf.timeit!

[Steven D'Aprano]

On Fri, Jun 10, 2016 at 11:22:42PM +0200, Victor Stinner wrote:
> 2016-06-10 20:47 GMT+02:00 Meador Inge <meadori at gmail.com>:
> > Apologies in advance if this is answered in one of the links you posted, but
> > out of curiosity was geometric mean considered?
> >
> > In the compiler world this is a very common way of aggregating performance
> > results.
> 
> FYI I chose to store all timings in the JSON file. So later, you are
> free to recompute the average differently, compute other statistics,
> etc.
> 
> I saw that the CPython benchmark suite has an *option* to compute the
> geometric mean. I don't understand well the difference with the
> arithmeric mean.
> 
> Is the geometric mean recommended to aggregate results of different
> (unrelated) benchmarks, or also even for multuple runs of a single
> benchmark?

The Wikipedia article discusses this, but sits on the fence and can't 
decide whether using the gmean for performance results is a good or bad 
idea:

https://en.wikipedia.org/wiki/Geometric_mean#Properties

Geometric mean is usually used in finance for averaging rates of growth:

https://www.math.toronto.edu/mathnet/questionCorner/geomean.html

If you express your performances as speeds (as "calculations per 
second") then the harmonic mean is the right way to average them.


-- 
Steve