Hi,
I always wanted to be able to compare the performance of two Python
versions using timeit *in a single command*. So I just implemented it!
I added --python and --compare-to options.
Real example to show the new "timeit --compared-to" feature:
---
$ export PYTHONPATH=~/prog/GIT/perf
$ ./python-resize -m perf timeit --inherit-environ=PYTHONPATH
--compare-to=./python-ref -s 'x = range(1000); d={}' 'for i in x:
d[i]=i; del d[i];' --rigorous
python-ref: ........................................ 77.6 us +- 1.8 us
python-resize: ........................................ 74.8 us +- 1.9 us
Median +- std dev: [python-ref] 77.6 us +- 1.8 us -> [python-resize]
74.8 us +- 1.9 us: 1.04x faster
---
http://bugs.python.org/issue28199#msg277755
Changes between 0.7.11 and 0.7.12:
* Add ``--python`` command line option
* ``timeit``: add ``--name``, ``--inner-loops`` and ``--compare-to``
options
* TextRunner don't set CPU affinity of the main process, only on worker
processes. It may help a little bit when using NOHZ_FULL.
* metadata: add ``boot_time`` and ``uptime`` on Linux
* metadata: add idle driver to ``cpu_config``
Victor
2016-09-29 17:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>:
> On my laptop, the speed ranges between 500MHz and 2300MHz.
Oh I see, on this computer the difference can be up to 5x slower!
>> * (Use NOHZ_FULL but) Force frequency to the maximum
>> * Don't use NOHZ_FULL
>
> I think we should force the frequency to a single number which is the
> maximum that can be reasonably sustained
In my experience, the CPU is fine when running at the nominal speed.
I'm talking about the value displayed in the CPU model in /cpu/cpuinfo
(2.9 GHz):
$ grep 'model name' /proc/cpuinfo
model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
> (I mean, not the overclocked speed done by some CPUs for short amounts of time).
This is the Turbo Mode. It can be disabled in the BIOS or by writing 1
into /sys/devices/system/cpu/intel_pstate/no_turbo
By the way, the most reliable tool that I found to read the CPU
frequency is turbostat. I used APERF and MPERF counters to compute the
"Busy MHz". You may also try "cpupower monitor" which is similar.
Victor
Hi Victor,
On 27 September 2016 at 15:40, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> 2016-09-24 8:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>:
>> IMHO this is not a very good solution. With the CPU running at, say,
>> a fifth of its nominal performance, you can't expect that it will
>> behave in a remotely similar way.
>
> The norminal speed is 3.4 GHz. The minimum speed is 1.6 GHz. Timings
> are just the double between nominal and minimum speed.
On my laptop, the speed ranges between 500MHz and 2300MHz.
> * (Use NOHZ_FULL but) Force frequency to the maximum
> * Don't use NOHZ_FULL
I think we should force the frequency to a single number which is the
maximum that can be reasonably sustained (I mean, not the overclocked
speed done by some CPUs for short amounts of time).
A bientôt,
Armin.
Hi,
I made further tests and I understood better the issue.
In short, the intel_pstate driver doesn't support NOHZ_FULL, and so
the frequency of CPUs using NOHZ_FULL depends on the workload of other
CPUs. This is especially true when using the powersave (default) cpu
frequency governor. At least, I tested on my CPU without HWP.
intel_pstate updates the Pstate of each CPU by writing into the MSR
199H. The purpose of NOHZ_FULL is to avoid any interruption, whereas
intel_pstate is based on interruptions to sample performances, pick
the right Pstate and write it into the MSR. To write into the MSR of
the CPU 7, the kernel must run on the CPU 7. If the benchmark is CPU
bound and never calls the kernel, there is no opportonity to run the
intel_pstate drive.
Antoine:
> Ah, well, I don't have HyperThreading on my CPU, sorry.
The bug can be reproduced without HyperThreading.
New much simpler scenario to reproduce the bug (and my analysis of the bug):
https://bugzilla.redhat.com/show_bug.cgi?id=1378529#c6
2016-09-24 8:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>:
> IMHO this is not a very good solution. With the CPU running at, say,
> a fifth of its nominal performance, you can't expect that it will
> behave in a remotely similar way.
The norminal speed is 3.4 GHz. The minimum speed is 1.6 GHz. Timings
are just the double between nominal and minimum speed.
> As a result, it is easy to introduce changes to the CPython
> core that appear beneficial, but are actually detrimental, or
> vice-versa. For example, replacing some computation by lookups in a
> table may look like a good idea, when it is not.
Yeah, maybe, I don't know.
Anyway, there are two solutions to run stable benchmarks at nominal speed:
* (Use NOHZ_FULL but) Force frequency to the maximum
* Don't use NOHZ_FULL
Victor
On Sat, 24 Sep 2016 08:11:22 +0200
Armin Rigo <arigo(a)tunes.org> wrote:
> Hi Victor,
>
> On 24 September 2016 at 01:49, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> > When running benchmarks, raw timings and CPU performance don't matter.
> > Only comparisons between benchmark results and stable performances
> > matter.
>
> IMHO this is not a very good solution. With the CPU running at, say,
> a fifth of its nominal performance, you can't expect that it will
> behave in a remotely similar way. For example, it makes the RAM
> appear five times faster. I would guess (but I don't know) that even
> the on-core L2/L3 caches are not slowed down by nearly as much as five
> times. As a result, it is easy to introduce changes to the CPython
> core that appear beneficial, but are actually detrimental, or
> vice-versa. For example, replacing some computation by lookups in a
> table may look like a good idea, when it is not.
Agreed with Armin.
Regards
Antoine.
Hi Victor,
On 24 September 2016 at 01:49, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> When running benchmarks, raw timings and CPU performance don't matter.
> Only comparisons between benchmark results and stable performances
> matter.
IMHO this is not a very good solution. With the CPU running at, say,
a fifth of its nominal performance, you can't expect that it will
behave in a remotely similar way. For example, it makes the RAM
appear five times faster. I would guess (but I don't know) that even
the on-core L2/L3 caches are not slowed down by nearly as much as five
times. As a result, it is easy to introduce changes to the CPython
core that appear beneficial, but are actually detrimental, or
vice-versa. For example, replacing some computation by lookups in a
table may look like a good idea, when it is not.
A bientôt,
Armin.
I wrote a long article explaing how I identified the bug by testing
Turbo Boost, CPU temperature, CPU frequency, etc.
https://haypo.github.io/intel-cpus-part2.html
Copy/paste of my conclusion:
To get stable benchmarks, the safest fix for all these issues is
probably to set the CPU frequency of the CPUs used by benchmarks to
the minimum. It seems like nothing can reduce the frequency of a CPU
below its minimum.
When running benchmarks, raw timings and CPU performance don't matter.
Only comparisons between benchmark results and stable performances
matter.
Victor
On Fri, 23 Sep 2016 11:44:12 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
> I guess that for some reasons, the CPU frequency was 1.6 GHz (min
> frequency) even if I configured the CPU frequency governor to
> performance.
Does that mean that all "perf"-reported benchmark results that you
reported from your machine are actually invalid?
> > Can you tell us how to "reproduce"?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1378529
>
> * Disable Turbo Boost
> * Enable HyperThreading
[...]
Ah, well, I don't have HyperThreading on my CPU, sorry.
Regards
Antoine.
On Fri, 23 Sep 2016 01:19:54 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
>
> While analyzing a performance regression (
> http://bugs.python.org/issue28243 ) , I had a major issue with my
> benchmark. Suddenly, for no reason, after 30 minutes of benchmarking,
> the benchmark became 2x FASTER...
Did the benchmark really become 2x faster, or did the clock become 2x
slower?
If you found a magic knob in your CPU that suddently makes it 2x
faster, many people would probably like to hear about it ;-)
> If you have an Intel CPU, use Linux, have a CPU with multiple physical
> cores and have 15 minutes to run a test, I would appreciate if you can
> try to reproduce the bug!
Can you tell us how to "reproduce"?
Regards
Antoine.
Hi,
While analyzing a performance regression (
http://bugs.python.org/issue28243 ) , I had a major issue with my
benchmark. Suddenly, for no reason, after 30 minutes of benchmarking,
the benchmark became 2x FASTER...
A similar issue occurred to me the last week when testing if PGO
compilation makes Python performance unstable.
It might be an issue related to the intel_pstate driver on Linux and
CPU isolation. I reported the bug in the Fedora bug tracker in the
kernel category:
https://bugzilla.redhat.com/show_bug.cgi?id=1378529
I don't know much about this issue yet, I contacted Intel engineers
who knows these things better than me :-)
If you have an Intel CPU, use Linux, have a CPU with multiple physical
cores and have 15 minutes to run a test, I would appreciate if you can
try to reproduce the bug!
Victor
