Mailman 3 May 2016 - Speed

Re: [Speed] External sources of noise changing call_simple "performance"
by Armin Rigo May 18, 2016

May 18, 2016

Hi Victor, On 17 May 2016 at 23:11, Victor Stinner <victor.stinner(a)gmail.com> wrote: > with PYTHONHASHSEED=1 to test the same hash function. A more generic > solution is to use multiple processes to test multiple hash seeds to > get a better uniform distribution. What you say in the rest of the mail just shows that this "generic solution" should be applied not only to PYTHONHASHSEED, but also to other variables that seem to introduce deterministic noise. You've just found three more: the locale, the size of the command line, and the working directory. I guess the mere size of the environment also plays a role. So I guess, ideally, you'd run a large number of times with random values in all these parameters. (In practice it might be enough to run a smaller fixed number of times with known values in the parameters.) A bientôt, Armin.

1 0

Re: [Speed] External sources of noise changing call_simple "performance"
by Maciej Fijalkowski May 18, 2016

May 18, 2016

> => The performance of the benchmark depends on the usage of low-level > memory caches (L1, L2, L3). > > I understand that in some cases, more memory fits into the fatest > caches, and so the benchmark is faster. But sometimes, all memory > doesn't fit, and so the benchmark is slower. > > Maybe the problem is that memory is close to memory pages boundaries, > or doesn't fit into L1 cache lines, or something like that. > > Victor I think you misunderstand how caches work. The way caches work depends on the addresses of memory (their value) which even with ASLR disabled can differ between runs. Then you either do or don't have cache collisions. How about you just accept the fact that there is a statistical distribution of the results on not the concrete "right" result? I tried to explain to you before that even if you get the "right" result, it'll still be at best just one sample of the statistics.

1 0

Re: [Speed] CPU speed of one core changes for unknown reason
by Victor Stinner May 17, 2016

May 17, 2016

According to a friend, my CPU model "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz" has a "Turbo Mode" which is enabled by default. The CPU tries to use the Turbo Mode whener possible, but disables it when the CPU is too hot. The change should be visible with the exact CPU frequency (the change can be a single MHz: 3400 => 3401). I didn't notice such minor CPU frequency change, but I didn't check carefully. Anyway, I disabled the Turbo Mode and Hyperthreading in the EFI. It should avoid the strange performance "drop". Victor 2016-05-17 16:44 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > Hi, > > I'm still having fun with microbenchmarks. I disabled Power States > (pstate) of my Intel CPU and forced the frequency for 3.4 GHz. I > isolated 2 physical cores on a total of 4. Timings are very stable > *but* sometimes, I get impressive slowdown: like 60% or 80% slower, > but only for a short time. > > Do you know which CPU feature can explain such temporary slowdown? > > I tried cpupower & powertop tools to try to learn more about internal > CPU states, but I don't see anything obvious. I also noticed that > powertop has a major side effect: it changes the speed of my CPU > cores! Since the CPU cores used to run benchmarks are isolated, > powertop uses a low speed (like 1.6 GHz, half speed) while benchmarks > are running, probably because the kernel doesn't "see" the benchmark > processes. > > My CPU model is: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz > > I'm using "userspace" scaling governor for isolated CPU cores, but > "ondemand" for other CPU cores. > > I disabled pstate (kernel parameter: intel_pstate=disable), the CPU > scaling driver is "acpi-cpufreq". > > CPUs 2,3,6,7 are isolated. > > In the following examples, the same microbenchmark takes ~196 ms on > all cores, except of the core 3 on the first example. > > Example 1: > --- > $ for cpu in $(seq 0 7); do echo "=== CPU $cpu ==="; PYTHONHASHSEED=0 > taskset -c $cpu ../fastcall/pgo/python performance/bm_call_simple.py > -n 1 --timer perf_counter; done > === CPU 0 === > 0.19619656700160704 > === CPU 1 === > 0.19547197800056892 > === CPU 2 === > 0.19512042699716403 > === CPU 3 === > 0.35738898099953076 > === CPU 4 === > 0.19744606299718725 > === CPU 5 === > 0.195480646998476 > === CPU 6 === > 0.19495172200186062 > === CPU 7 === > 0.19495161599843414 > --- > > Example 2: > --- > $ for cpu in $(seq 0 7); do echo "=== CPU $cpu ==="; PYTHONHASHSEED=0 > taskset -c $cpu ../fastcall/pgo/python performance/bm_call_simple.py > -n 1 --timer perf_counter; done > === CPU 0 === > 0.19725238799946965 > === CPU 1 === > 0.19552089699936914 > === CPU 2 === > 0.19495758999983082 > === CPU 3 === > 0.19517506799820694 > === CPU 4 === > 0.1963375539999106 > === CPU 5 === > 0.19575440099652042 > === CPU 6 === > 0.19582506000006106 > === CPU 7 === > 0.19503543600148987 > --- > > If I repeat the same test, timings are always ~196 ms on all cores. > > It looks like some cores decide to sleep. > > Victor

1 0

External sources of noise changing call_simple "performance"
by Victor Stinner May 17, 2016

May 17, 2016

Hi, I'm still (!) investigating the reasons why the benchmark call_simple (ok, let's be honest: the *micro*benchmark) gets different results for unknown reasons. (*) Collisions in hash tables: perf.py already calls the benchmark with PYTHONHASHSEED=1 to test the same hash function. A more generic solution is to use multiple processes to test multiple hash seeds to get a better uniform distribution. (*) System load => CPU isolation, disable ASLR, set CPU affinity on IRQs, etc. work around this issue -- http://haypo-notes.readthedocs.io/microbenchmark.html (*) CPU heat => disable CPU Turbo Mode works around this issue (*) Locale, size of the command line and/or the current working directory => WTF?! Examples with a system tuned to get reliable benchmark. Example 1 using different locales: --- $ env -i PYTHONHASHSEED=1 LANG=$LANG taskset -c 3 ../fastcall/pgo/python performance/bm_call_simple.py -n 2 --timer perf_counter 0.1914542349995827 0.1914668690005783 $ env -i PYTHONHASHSEED=1 taskset -c 3 ../fastcall/pgo/python performance/bm_call_simple.py -n 2 --timer perf_counter 0.2037885540003117 0.20376207399931445 --- Example 2 using a different command line (the "xxx" is ignored but changes the benchmark result): -- $ env -i PYTHONHASHSEED=1 taskset -c 3 ../fastcall/pgo/python performance/bm_call_simple.py -n 2 --timer perf_counter 0.20377227199969639 0.20376165899961052 $ env -i PYTHONHASHSEED=1 taskset -c 3 ../fastcall/pgo/python performance/bm_call_simple.py -n 2 --timer perf_counter xxx 0.20814169400000537 0.20804374700037442 --- => My bet is that the locale, current working directory, command line, etc. impact how the heap memory is allocated, and this specific benchmark depends on the locality of memory allocated on the heap... For a microbenchmark, 191 ms, 203 ms or 208 ms are not the same numbers... Such very subtle difference impacts the final "NNNx slower" or "NNNx faster" line of perf.py. I tried different values of $LANG environment variable and differerent lengths of command lines. When the performance decreases, the stalled-cycles-frontend Linux perf event increases while the LLC-loads even increases. => The performance of the benchmark depends on the usage of low-level memory caches (L1, L2, L3). I understand that in some cases, more memory fits into the fatest caches, and so the benchmark is faster. But sometimes, all memory doesn't fit, and so the benchmark is slower. Maybe the problem is that memory is close to memory pages boundaries, or doesn't fit into L1 cache lines, or something like that. Victor

1 0

CPU speed of one core changes for unknown reason
by Victor Stinner May 17, 2016

May 17, 2016

Hi, I'm still having fun with microbenchmarks. I disabled Power States (pstate) of my Intel CPU and forced the frequency for 3.4 GHz. I isolated 2 physical cores on a total of 4. Timings are very stable *but* sometimes, I get impressive slowdown: like 60% or 80% slower, but only for a short time. Do you know which CPU feature can explain such temporary slowdown? I tried cpupower & powertop tools to try to learn more about internal CPU states, but I don't see anything obvious. I also noticed that powertop has a major side effect: it changes the speed of my CPU cores! Since the CPU cores used to run benchmarks are isolated, powertop uses a low speed (like 1.6 GHz, half speed) while benchmarks are running, probably because the kernel doesn't "see" the benchmark processes. My CPU model is: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz I'm using "userspace" scaling governor for isolated CPU cores, but "ondemand" for other CPU cores. I disabled pstate (kernel parameter: intel_pstate=disable), the CPU scaling driver is "acpi-cpufreq". CPUs 2,3,6,7 are isolated. In the following examples, the same microbenchmark takes ~196 ms on all cores, except of the core 3 on the first example. Example 1: --- $ for cpu in $(seq 0 7); do echo "=== CPU $cpu ==="; PYTHONHASHSEED=0 taskset -c $cpu ../fastcall/pgo/python performance/bm_call_simple.py -n 1 --timer perf_counter; done === CPU 0 === 0.19619656700160704 === CPU 1 === 0.19547197800056892 === CPU 2 === 0.19512042699716403 === CPU 3 === 0.35738898099953076 === CPU 4 === 0.19744606299718725 === CPU 5 === 0.195480646998476 === CPU 6 === 0.19495172200186062 === CPU 7 === 0.19495161599843414 --- Example 2: --- $ for cpu in $(seq 0 7); do echo "=== CPU $cpu ==="; PYTHONHASHSEED=0 taskset -c $cpu ../fastcall/pgo/python performance/bm_call_simple.py -n 1 --timer perf_counter; done === CPU 0 === 0.19725238799946965 === CPU 1 === 0.19552089699936914 === CPU 2 === 0.19495758999983082 === CPU 3 === 0.19517506799820694 === CPU 4 === 0.1963375539999106 === CPU 5 === 0.19575440099652042 === CPU 6 === 0.19582506000006106 === CPU 7 === 0.19503543600148987 --- If I repeat the same test, timings are always ~196 ms on all cores. It looks like some cores decide to sleep. Victor

1 0

Re: [Speed] Adapting pypy benchmark suite
by Maciej Fijalkowski May 3, 2016

May 3, 2016

Hi I'm willing to put some work after I'm back from holiday (mid-May) On Tue, May 3, 2016 at 12:24 AM, Brett Cannon <brett(a)python.org> wrote: > > > On Mon, 2 May 2016 at 15:18 Kevin Modzelewski <kmod(a)dropbox.com> wrote: >> >> I'm definitely interested and willing to clean up + contribute our >> benchmarks. >> >> On a side note, I'm a bit skeptical that there can be a single benchmark >> suite that satisfies everyone. I would imagine that there will still be >> projects with specific use-cases they prioritize (such as Pyston with >> webserver workloads), or that have some idea that their users will be >> "non-representative" in some way. One example of that is the emphasis on >> warmup vs steady-state performance, which can be reflected in different >> measurement methodologies -- I don't think there's a single right answer to >> the question "how much does warmup matter". > > > Totally agree. I think the general thinking is to at have a central > repository and a flexible enough benchmark runner that people can benchmark > whatever they find important to them. That way if e.g. Pyston adds nice web > server benchmarks other implementations can use them or users can decide > that's a workload they care about and make an informed decision of what > Python implementations may work for them (before testing their own workload > :). > > -Brett > >> >> >> But anyway, I'm still definitely +1 on the idea of merging all the >> benchmarks together, and I think that that will be better than the current >> situation. I'm imagining that we can at least have a common language for >> discussing these things ("Pyston prefers to use the flags `--webserver >> --include-warmup`"). I also see quite a few blog posts / academic papers on >> Python performance that seem to get led astray by the confusing benchmark >> situation, and I think having a blessed set of benchmarks (even if different >> people use them in different ways) would still be a huge step forward. >> >> kmod >> >> On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <brett(a)python.org> wrote: >>> >>> >>> >>> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall(a)gmail.com> wrote: >>>> >>>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis(a)pitrou.net> >>>> wrote: >>>> > On Wed, 13 Apr 2016 20:57:35 +0200 >>>> > Maciej Fijalkowski <fijall(a)gmail.com> >>>> > wrote: >>>> >> Hi >>>> >> >>>> >> I have a radical idea: to take a pypy benchmark suite, update the >>>> >> libraries to newer ones and replace python benchmarks with that. The >>>> >> main reason being that pypy has a much better coverage of things that >>>> >> are not microbenchmarks, the list (in json): >>>> > >>>> > So why not consolidate all benchmarks together, instead of throwing >>>> > away work already done? >>>> > >>>> > Regards >>>> > >>>> > Antoine. >>>> >>>> Yeah, you can call it that too. >>> >>> >>> I also reached out to Pyston at https://gitter.im/dropbox/pyston over the >>> weekend to see if they would want to participate as well. >>> >>> So are we actually going to try and make this happen? I guess we should >>> get people to vote on whether they like the idea enough before we hash out >>> how we want to structure the new repository and benchmark suite. >>> >>> I'm +1 on the idea, but I currently don't have the time to help beyond >>> helping drive the email conversation. >>> >>> _______________________________________________ >>> Speed mailing list >>> Speed(a)python.org >>> https://mail.python.org/mailman/listinfo/speed >>> >

1 0

Re: [Speed] Adapting pypy benchmark suite
by Brett Cannon May 2, 2016

May 2, 2016

On Mon, 2 May 2016 at 15:18 Kevin Modzelewski <kmod(a)dropbox.com> wrote: > I'm definitely interested and willing to clean up + contribute our > benchmarks. > > On a side note, I'm a bit skeptical that there can be a single benchmark > suite that satisfies everyone. I would imagine that there will still be > projects with specific use-cases they prioritize (such as Pyston with > webserver workloads), or that have some idea that their users will be > "non-representative" in some way. One example of that is the emphasis on > warmup vs steady-state performance, which can be reflected in different > measurement methodologies -- I don't think there's a single right answer to > the question "how much does warmup matter". > Totally agree. I think the general thinking is to at have a central repository and a flexible enough benchmark runner that people can benchmark whatever they find important to them. That way if e.g. Pyston adds nice web server benchmarks other implementations can use them or users can decide that's a workload they care about and make an informed decision of what Python implementations may work for them (before testing their own workload :). -Brett > > But anyway, I'm still definitely +1 on the idea of merging all the > benchmarks together, and I think that that will be better than the current > situation. I'm imagining that we can at least have a common language for > discussing these things ("Pyston prefers to use the flags `--webserver > --include-warmup`"). I also see quite a few blog posts / academic papers > on Python performance that seem to get led astray by the confusing > benchmark situation, and I think having a blessed set of benchmarks (even > if different people use them in different ways) would still be a huge step > forward. > > kmod > > On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <brett(a)python.org> wrote: > >> >> >> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall(a)gmail.com> wrote: >> >>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis(a)pitrou.net> >>> wrote: >>> > On Wed, 13 Apr 2016 20:57:35 +0200 >>> > Maciej Fijalkowski <fijall(a)gmail.com> >>> > wrote: >>> >> Hi >>> >> >>> >> I have a radical idea: to take a pypy benchmark suite, update the >>> >> libraries to newer ones and replace python benchmarks with that. The >>> >> main reason being that pypy has a much better coverage of things that >>> >> are not microbenchmarks, the list (in json): >>> > >>> > So why not consolidate all benchmarks together, instead of throwing >>> > away work already done? >>> > >>> > Regards >>> > >>> > Antoine. >>> >>> Yeah, you can call it that too. >>> >> >> I also reached out to Pyston at https://gitter.im/dropbox/pyston over >> the weekend to see if they would want to participate as well. >> >> So are we actually going to try and make this happen? I guess we should >> get people to vote on whether they like the idea enough before we hash out >> how we want to structure the new repository and benchmark suite. >> >> I'm +1 on the idea, but I currently don't have the time to help beyond >> helping drive the email conversation. >> >> _______________________________________________ >> Speed mailing list >> Speed(a)python.org >> https://mail.python.org/mailman/listinfo/speed >> >>

1 0

Re: [Speed] Adapting pypy benchmark suite
by Kevin Modzelewski May 2, 2016

May 2, 2016

I'm definitely interested and willing to clean up + contribute our benchmarks. On a side note, I'm a bit skeptical that there can be a single benchmark suite that satisfies everyone. I would imagine that there will still be projects with specific use-cases they prioritize (such as Pyston with webserver workloads), or that have some idea that their users will be "non-representative" in some way. One example of that is the emphasis on warmup vs steady-state performance, which can be reflected in different measurement methodologies -- I don't think there's a single right answer to the question "how much does warmup matter". But anyway, I'm still definitely +1 on the idea of merging all the benchmarks together, and I think that that will be better than the current situation. I'm imagining that we can at least have a common language for discussing these things ("Pyston prefers to use the flags `--webserver --include-warmup`"). I also see quite a few blog posts / academic papers on Python performance that seem to get led astray by the confusing benchmark situation, and I think having a blessed set of benchmarks (even if different people use them in different ways) would still be a huge step forward. kmod On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <brett(a)python.org> wrote: > > > On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall(a)gmail.com> wrote: > >> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis(a)pitrou.net> >> wrote: >> > On Wed, 13 Apr 2016 20:57:35 +0200 >> > Maciej Fijalkowski <fijall(a)gmail.com> >> > wrote: >> >> Hi >> >> >> >> I have a radical idea: to take a pypy benchmark suite, update the >> >> libraries to newer ones and replace python benchmarks with that. The >> >> main reason being that pypy has a much better coverage of things that >> >> are not microbenchmarks, the list (in json): >> > >> > So why not consolidate all benchmarks together, instead of throwing >> > away work already done? >> > >> > Regards >> > >> > Antoine. >> >> Yeah, you can call it that too. >> > > I also reached out to Pyston at https://gitter.im/dropbox/pyston over the > weekend to see if they would want to participate as well. > > So are we actually going to try and make this happen? I guess we should > get people to vote on whether they like the idea enough before we hash out > how we want to structure the new repository and benchmark suite. > > I'm +1 on the idea, but I currently don't have the time to help beyond > helping drive the email conversation. > > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed > >

1 0

Re: [Speed] Adapting pypy benchmark suite
by Brett Cannon May 2, 2016

May 2, 2016

On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis(a)pitrou.net> > wrote: > > On Wed, 13 Apr 2016 20:57:35 +0200 > > Maciej Fijalkowski <fijall(a)gmail.com> > > wrote: > >> Hi > >> > >> I have a radical idea: to take a pypy benchmark suite, update the > >> libraries to newer ones and replace python benchmarks with that. The > >> main reason being that pypy has a much better coverage of things that > >> are not microbenchmarks, the list (in json): > > > > So why not consolidate all benchmarks together, instead of throwing > > away work already done? > > > > Regards > > > > Antoine. > > Yeah, you can call it that too. > I also reached out to Pyston at https://gitter.im/dropbox/pyston over the weekend to see if they would want to participate as well. So are we actually going to try and make this happen? I guess we should get people to vote on whether they like the idea enough before we hash out how we want to structure the new repository and benchmark suite. I'm +1 on the idea, but I currently don't have the time to help beyond helping drive the email conversation.

1 0