On Sat, 24 Sep 2016 08:11:22 +0200
Armin Rigo <arigo(a)tunes.org> wrote:
> Hi Victor,
> On 24 September 2016 at 01:49, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> > When running benchmarks, raw timings and CPU performance don't matter.
> > Only comparisons between benchmark results and stable performances
> > matter.
> IMHO this is not a very good solution. With the CPU running at, say,
> a fifth of its nominal performance, you can't expect that it will
> behave in a remotely similar way. For example, it makes the RAM
> appear five times faster. I would guess (but I don't know) that even
> the on-core L2/L3 caches are not slowed down by nearly as much as five
> times. As a result, it is easy to introduce changes to the CPython
> core that appear beneficial, but are actually detrimental, or
> vice-versa. For example, replacing some computation by lookups in a
> table may look like a good idea, when it is not.
Agreed with Armin.