On Fri, 2016-06-10 at 23:20 +1000, Steven D'Aprano wrote:
On Fri, Jun 10, 2016 at 01:13:10PM +0200, Victor Stinner wrote:
Hi,
Last weeks, I made researchs on how to get stable and reliable benchmarks, especially for the corner case of microbenchmarks. The first result is a serie of article, here are the first three:
Thank you for this! I am very interested in benchmarking.
https://haypo.github.io/journey-to-stable-benchmark-system.html https://haypo.github.io/journey-to-stable-benchmark-deadcode.html https://haypo.github.io/journey-to-stable-benchmark-average.html
I strongly question your statement in the third:
[quote] But how can we compare performances if results are random? Take the minimum?
No! You must never (ever again) use the minimum for benchmarking! Compute the average and some statistics like the standard deviation: [end quote]
While I'm happy to see a real-world use for the statistics module, I disagree with your logic.
The problem is that random noise can only ever slow the code down, it cannot speed it up.
Consider a workload being benchmarked running on one core, which has a particular pattern of cache hits and misses. Now consider another process running on a sibling core, sharing the same cache. Isn't it possible that under some circumstances the 2nd process could prefetch memory into the cache in such a way that the workload under test actually gets faster than if the 2nd process wasn't running? [...snip...] Hope this is constructive Dave