On Sat, 24 Sep 2016 08:11:22 +0200 Armin Rigo email@example.com wrote:
On 24 September 2016 at 01:49, Victor Stinner firstname.lastname@example.org wrote:
When running benchmarks, raw timings and CPU performance don't matter. Only comparisons between benchmark results and stable performances matter.
IMHO this is not a very good solution. With the CPU running at, say, a fifth of its nominal performance, you can't expect that it will behave in a remotely similar way. For example, it makes the RAM appear five times faster. I would guess (but I don't know) that even the on-core L2/L3 caches are not slowed down by nearly as much as five times. As a result, it is easy to introduce changes to the CPython core that appear beneficial, but are actually detrimental, or vice-versa. For example, replacing some computation by lookups in a table may look like a good idea, when it is not.
Agreed with Armin.