On Sat, 24 Sep 2016 08:11:22 +0200 Armin Rigo <arigo@tunes.org> wrote:

Hi Victor,

On 24 September 2016 at 01:49, Victor Stinner <victor.stinner@gmail.com> wrote:

...

When running benchmarks, raw timings and CPU performance don't matter. Only comparisons between benchmark results and stable performances matter.

IMHO this is not a very good solution. With the CPU running at, say, a fifth of its nominal performance, you can't expect that it will behave in a remotely similar way. For example, it makes the RAM appear five times faster. I would guess (but I don't know) that even the on-core L2/L3 caches are not slowed down by nearly as much as five times. As a result, it is easy to introduce changes to the CPython core that appear beneficial, but are actually detrimental, or vice-versa. For example, replacing some computation by lookups in a table may look like a good idea, when it is not.

Agreed with Armin.

Regards

Antoine.