[Python-Dev] Stop using timeit, use perf.timeit!
dmalcolm at redhat.com
Fri Jun 10 10:34:26 EDT 2016
On Fri, 2016-06-10 at 23:20 +1000, Steven D'Aprano wrote:
> On Fri, Jun 10, 2016 at 01:13:10PM +0200, Victor Stinner wrote:
> > Hi,
> > Last weeks, I made researchs on how to get stable and reliable
> > benchmarks, especially for the corner case of microbenchmarks. The
> > first result is a serie of article, here are the first three:
> Thank you for this! I am very interested in benchmarking.
> > https://haypo.github.io/journey-to-stable-benchmark-system.html
> > https://haypo.github.io/journey-to-stable-benchmark-deadcode.html
> > https://haypo.github.io/journey-to-stable-benchmark-average.html
> I strongly question your statement in the third:
> But how can we compare performances if results are random?
> Take the minimum?
> No! You must never (ever again) use the minimum for
> benchmarking! Compute the average and some statistics like
> the standard deviation:
> [end quote]
> While I'm happy to see a real-world use for the statistics module, I
> disagree with your logic.
> The problem is that random noise can only ever slow the code down, it
> cannot speed it up.
Consider a workload being benchmarked running on one core, which has a
particular pattern of cache hits and misses. Now consider another
process running on a sibling core, sharing the same cache.
Isn't it possible that under some circumstances the 2nd process could
prefetch memory into the cache in such a way that the workload under
test actually gets faster than if the 2nd process wasn't running?
Hope this is constructive
More information about the Python-Dev