Mailman 3 September 2016 - Speed

perf 0.7.12: --python and --compare-to options
by Victor Stinner Sept. 30, 2016

Sept. 30, 2016

Hi, I always wanted to be able to compare the performance of two Python versions using timeit *in a single command*. So I just implemented it! I added --python and --compare-to options. Real example to show the new "timeit --compared-to" feature: --- $ export PYTHONPATH=~/prog/GIT/perf $ ./python-resize -m perf timeit --inherit-environ=PYTHONPATH --compare-to=./python-ref -s 'x = range(1000); d={}' 'for i in x: d[i]=i; del d[i];' --rigorous python-ref: ........................................ 77.6 us +- 1.8 us python-resize: ........................................ 74.8 us +- 1.9 us Median +- std dev: [python-ref] 77.6 us +- 1.8 us -> [python-resize] 74.8 us +- 1.9 us: 1.04x faster --- http://bugs.python.org/issue28199#msg277755 Changes between 0.7.11 and 0.7.12: * Add ``--python`` command line option * ``timeit``: add ``--name``, ``--inner-loops`` and ``--compare-to`` options * TextRunner don't set CPU affinity of the main process, only on worker processes. It may help a little bit when using NOHZ_FULL. * metadata: add ``boot_time`` and ``uptime`` on Linux * metadata: add idle driver to ``cpu_config`` Victor

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Victor Stinner Sept. 29, 2016

Sept. 29, 2016

2016-09-29 17:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > On my laptop, the speed ranges between 500MHz and 2300MHz. Oh I see, on this computer the difference can be up to 5x slower! >> * (Use NOHZ_FULL but) Force frequency to the maximum >> * Don't use NOHZ_FULL > > I think we should force the frequency to a single number which is the > maximum that can be reasonably sustained In my experience, the CPU is fine when running at the nominal speed. I'm talking about the value displayed in the CPU model in /cpu/cpuinfo (2.9 GHz): $ grep 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz > (I mean, not the overclocked speed done by some CPUs for short amounts of time). This is the Turbo Mode. It can be disabled in the BIOS or by writing 1 into /sys/devices/system/cpu/intel_pstate/no_turbo By the way, the most reliable tool that I found to read the CPU frequency is turbostat. I used APERF and MPERF counters to compute the "Busy MHz". You may also try "cpupower monitor" which is similar. Victor

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Armin Rigo Sept. 29, 2016

Sept. 29, 2016

Hi Victor, On 27 September 2016 at 15:40, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-09-24 8:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: >> IMHO this is not a very good solution. With the CPU running at, say, >> a fifth of its nominal performance, you can't expect that it will >> behave in a remotely similar way. > > The norminal speed is 3.4 GHz. The minimum speed is 1.6 GHz. Timings > are just the double between nominal and minimum speed. On my laptop, the speed ranges between 500MHz and 2300MHz. > * (Use NOHZ_FULL but) Force frequency to the maximum > * Don't use NOHZ_FULL I think we should force the frequency to a single number which is the maximum that can be reasonably sustained (I mean, not the overclocked speed done by some CPUs for short amounts of time). A bientôt, Armin.

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Victor Stinner Sept. 27, 2016

Sept. 27, 2016

Hi, I made further tests and I understood better the issue. In short, the intel_pstate driver doesn't support NOHZ_FULL, and so the frequency of CPUs using NOHZ_FULL depends on the workload of other CPUs. This is especially true when using the powersave (default) cpu frequency governor. At least, I tested on my CPU without HWP. intel_pstate updates the Pstate of each CPU by writing into the MSR 199H. The purpose of NOHZ_FULL is to avoid any interruption, whereas intel_pstate is based on interruptions to sample performances, pick the right Pstate and write it into the MSR. To write into the MSR of the CPU 7, the kernel must run on the CPU 7. If the benchmark is CPU bound and never calls the kernel, there is no opportonity to run the intel_pstate drive. Antoine: > Ah, well, I don't have HyperThreading on my CPU, sorry. The bug can be reproduced without HyperThreading. New much simpler scenario to reproduce the bug (and my analysis of the bug): https://bugzilla.redhat.com/show_bug.cgi?id=1378529#c6 2016-09-24 8:11 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > IMHO this is not a very good solution. With the CPU running at, say, > a fifth of its nominal performance, you can't expect that it will > behave in a remotely similar way. The norminal speed is 3.4 GHz. The minimum speed is 1.6 GHz. Timings are just the double between nominal and minimum speed. > As a result, it is easy to introduce changes to the CPython > core that appear beneficial, but are actually detrimental, or > vice-versa. For example, replacing some computation by lookups in a > table may look like a good idea, when it is not. Yeah, maybe, I don't know. Anyway, there are two solutions to run stable benchmarks at nominal speed: * (Use NOHZ_FULL but) Force frequency to the maximum * Don't use NOHZ_FULL Victor

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Antoine Pitrou Sept. 24, 2016

Sept. 24, 2016

On Sat, 24 Sep 2016 08:11:22 +0200 Armin Rigo <arigo(a)tunes.org> wrote: > Hi Victor, > > On 24 September 2016 at 01:49, Victor Stinner <victor.stinner(a)gmail.com> wrote: > > When running benchmarks, raw timings and CPU performance don't matter. > > Only comparisons between benchmark results and stable performances > > matter. > > IMHO this is not a very good solution. With the CPU running at, say, > a fifth of its nominal performance, you can't expect that it will > behave in a remotely similar way. For example, it makes the RAM > appear five times faster. I would guess (but I don't know) that even > the on-core L2/L3 caches are not slowed down by nearly as much as five > times. As a result, it is easy to introduce changes to the CPython > core that appear beneficial, but are actually detrimental, or > vice-versa. For example, replacing some computation by lookups in a > table may look like a good idea, when it is not. Agreed with Armin. Regards Antoine.

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Armin Rigo Sept. 24, 2016

Sept. 24, 2016

Hi Victor, On 24 September 2016 at 01:49, Victor Stinner <victor.stinner(a)gmail.com> wrote: > When running benchmarks, raw timings and CPU performance don't matter. > Only comparisons between benchmark results and stable performances > matter. IMHO this is not a very good solution. With the CPU running at, say, a fifth of its nominal performance, you can't expect that it will behave in a remotely similar way. For example, it makes the RAM appear five times faster. I would guess (but I don't know) that even the on-core L2/L3 caches are not slowed down by nearly as much as five times. As a result, it is easy to introduce changes to the CPython core that appear beneficial, but are actually detrimental, or vice-versa. For example, replacing some computation by lookups in a table may look like a good idea, when it is not. A bientôt, Armin.

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Victor Stinner Sept. 23, 2016

Sept. 23, 2016

I wrote a long article explaing how I identified the bug by testing Turbo Boost, CPU temperature, CPU frequency, etc. https://haypo.github.io/intel-cpus-part2.html Copy/paste of my conclusion: To get stable benchmarks, the safest fix for all these issues is probably to set the CPU frequency of the CPUs used by benchmarks to the minimum. It seems like nothing can reduce the frequency of a CPU below its minimum. When running benchmarks, raw timings and CPU performance don't matter. Only comparisons between benchmark results and stable performances matter. Victor

1 0

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Antoine Pitrou Sept. 23, 2016

Sept. 23, 2016

On Fri, 23 Sep 2016 11:44:12 +0200 Victor Stinner <victor.stinner(a)gmail.com> wrote: > I guess that for some reasons, the CPU frequency was 1.6 GHz (min > frequency) even if I configured the CPU frequency governor to > performance. Does that mean that all "perf"-reported benchmark results that you reported from your machine are actually invalid? > > Can you tell us how to "reproduce"? > > https://bugzilla.redhat.com/show_bug.cgi?id=1378529 > > * Disable Turbo Boost > * Enable HyperThreading [...] Ah, well, I don't have HyperThreading on my CPU, sorry. Regards Antoine.

2 1

Re: [Speed] intel_pstate C0 bug on isolated CPUs with the performance governor
by Antoine Pitrou Sept. 23, 2016

Sept. 23, 2016

On Fri, 23 Sep 2016 01:19:54 +0200 Victor Stinner <victor.stinner(a)gmail.com> wrote: > > While analyzing a performance regression ( > http://bugs.python.org/issue28243 ) , I had a major issue with my > benchmark. Suddenly, for no reason, after 30 minutes of benchmarking, > the benchmark became 2x FASTER... Did the benchmark really become 2x faster, or did the clock become 2x slower? If you found a magic knob in your CPU that suddently makes it 2x faster, many people would probably like to hear about it ;-) > If you have an Intel CPU, use Linux, have a CPU with multiple physical > cores and have 15 minutes to run a test, I would appreciate if you can > try to reproduce the bug! Can you tell us how to "reproduce"? Regards Antoine.

2 1

intel_pstate C0 bug on isolated CPUs with the performance governor
by Victor Stinner Sept. 22, 2016

Sept. 22, 2016

Hi, While analyzing a performance regression ( http://bugs.python.org/issue28243 ) , I had a major issue with my benchmark. Suddenly, for no reason, after 30 minutes of benchmarking, the benchmark became 2x FASTER... A similar issue occurred to me the last week when testing if PGO compilation makes Python performance unstable. It might be an issue related to the intel_pstate driver on Linux and CPU isolation. I reported the bug in the Fedora bug tracker in the kernel category: https://bugzilla.redhat.com/show_bug.cgi?id=1378529 I don't know much about this issue yet, I contacted Intel engineers who knows these things better than me :-) If you have an Intel CPU, use Linux, have a CPU with multiple physical cores and have 15 minutes to run a test, I would appreciate if you can try to reproduce the bug! Victor

1 0