Mailman 3 April 2016 - Speed

Re: [Speed] Disable hash randomization to get reliable benchmarks
by Antonio Cuni 26 Apr '16

26 Apr '16

Hello Victor, On Mon, Apr 25, 2016 at 12:49 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > Last months, I spent a lot of time on microbenchmarks. Probably too > much time :-) I found a great Linux config to get a much more stable > system to get reliable microbenchmarks: > https://haypo-notes.readthedocs.org/microbenchmark.html > > * isolate some CPU cores > you might be interested in cpusets and the cset utility: in theory, they allow you to isolate one CPU without having to reboot to change the kernel parameters: http://skebanga.blogspot.it/2012/06/cset-shield-easily-configure-cpusets.ht… https://github.com/lpechacek/cpuset However, I never did a scientific comparison between cpusets and isolcpu to see if the former behaves exactly like the latter. ciao, Anto

1 0

Re: [Speed] Disable hash randomization to get reliable benchmarks
by Armin Rigo 26 Apr '16

26 Apr '16

Hi, On 25 April 2016 at 08:25, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > The problem with disabled ASLR is that you change the measurment from > a statistical distribution, to one draw from a statistical > distribution repeatedly. There is no going around doing multiple runs > and doing an average on that. You should mention that it is usually enough to do the following: instead of running once with PYTHONHASHSEED=0, run five or ten times with PYTHONHASHSEED in range(5 or 10). In this way, you get all benefits: not-too-long benchmarking, no randomness, but still some statistically relevant sampling. A bientôt, Armin.

1 0

Re: [Speed] Disable hash randomization to get reliable benchmarks
by Victor Stinner 25 Apr '16

25 Apr '16

Hi, The problem with not cheating on the statistical distribution is that I would like to get a quick feedback on my changes to know if my change is faster or not. Having to wait 1 hour to check a single change is not really convenient. I prefer to get a feedback is less than 5 minutes. Tune Linux and disable random hash allows to run less iterations and so take less time. If you don't tune anything, you need *a lot* of iterations to reduce the "noise" of the benchmark. 2016-04-25 8:25 GMT+02:00 Maciej Fijalkowski <fijall(a)gmail.com>: > The problem with disabled ASLR is that you change the measurment from > a statistical distribution, to one draw from a statistical > distribution repeatedly. The problem is that perf.py only runs one process per benchmark and per Python binary. Let's that the binary A is run with no hash collision, all dict access succeed at the first iteration, whereas the binary B runs with many "hash collision" so get worse performance. Is it fair to compare only these two specific runs? What can we conclude from the result? Note: The dict type of CPython uses open addressing, so even if two keys get the different hash value, a dict lookup may need more than one iteration to retrieve a dict entry. Right now, using ASLR and randomized hash function with perf.py is not fair. I'm talking about only running perf.py once. I'm unable to combine/compare manually multiple runs of perf.py. Getting multiple different results is very confusing for me. If you want to use ASRL and hash randomization, perf.py must be modified to run multiple processes (sequentially) to get a better statistical distribution. No? I don't know how many processes do we have to run. Accoding to my quick analysis there are 3 different cases, the strict minimum would be to run 3 processes for my specific case. For bm_call_simple, the number of loops is 15 for fast, 150 by default, 300 using rigorous. Maybe we should only run one loop iteration per process? FYI right now, I'm not using perf.py to prove that my patch makes CPython faster, but more to analyze why my change makes CPython slower :-) It *looks* like Python function calls are between 2 and 7% slower, but it also looks that bm_call_simple is an unstable microbenchmark :-( > There is no going around doing multiple runs > and doing an average on that. Essentially for the same reason why > using min is much worse than using average, with ASLR say you get: > 2.0+-0.3 which we run 5 times and 1.8, 1.9, 2.2, 2.1, 2.1 now if you > disable ASLR, you get one draw repeated 5 times, which might be 2.0, > but also might be 1.8, 5 times. That just hides the problem, but does > not actually fix it (because if you touch something, stuff might be > allocated in a different order and then you get a different draw) My practical problem is to get reliable benchmark. If you don't tune Linux and don't disable hash randomization, the results look "random". Example of output without tuning. I ran "python3 perf.py ../default/python-revert ../default/python-commit -b call_simple -v --fast" 3 times. [ fast, 15 iterations ] ### call_simple ### Min: 0.235318 -> 0.247203: 1.05x slower Avg: 0.237601 -> 0.251384: 1.06x slower Significant (t=-6.32) Stddev: 0.00214 -> 0.00817: 3.8069x larger ### call_simple ### Min: 0.234191 -> 0.247109: 1.06x slower Avg: 0.234660 -> 0.247480: 1.05x slower Significant (t=-36.14) Stddev: 0.00102 -> 0.00093: 1.0967x smaller ### call_simple ### Min: 0.235790 -> 0.247089: 1.05x slower Avg: 0.238978 -> 0.247562: 1.04x slower Significant (t=-9.38) Stddev: 0.00342 -> 0.00094: 3.6504x smaller You ask to ignore the Min line. Ok. But the average line say 1.04, 1.05 and 1.06x slower. Which one is the "good" result? :-) Usually, the difference is much larger like between 1.02x slower and 1.07x slower. [ rigorous, 300 iterations ] The --fast option of perf.py is just a toy, right? Serious dev must use the super slow --rigorous mode! Ok, let's try it. ### call_simple ### Min: 0.234102 -> 0.248098: 1.06x slower Avg: 0.236218 -> 0.254318: 1.08x slower Significant (t=-30.32) Stddev: 0.00561 -> 0.00869: 1.5475x larger ### call_simple ### Min: 0.234109 -> 0.248024: 1.06x slower Avg: 0.240069 -> 0.255194: 1.06x slower Significant (t=-15.21) Stddev: 0.01126 -> 0.01304: 1.1584x larger ### call_simple ### Min: 0.235272 -> 0.248225: 1.06x slower Avg: 0.244106 -> 0.258349: 1.06x slower Significant (t=-13.27) Stddev: 0.00830 -> 0.01663: 2.0053x larger Again, I ignore the Min line. Average: hum... is it 1.06x or 1.08x slower? For me, it's not really the same :-/ In the reference Python, the average timing changes between 0.236 and 0.244 in the 3 runs. Since the difference between the reference and the patched Python is tiny, the stability of the microbenchmark matters. FYI I'm running benchmarks on my desktop PC. It's more convenient to run benchmarks locally than transfering changes to an hypotetical dedicated benchmark server (tuned to run more reliable (micro)benchmarks). Since benchmarks are slow, I'm doing something else while the benchmark is running. Victor

1 0

Re: [Speed] Disable hash randomization to get reliable benchmarks
by Maciej Fijalkowski 25 Apr '16

25 Apr '16

Hi Victor The problem with disabled ASLR is that you change the measurment from a statistical distribution, to one draw from a statistical distribution repeatedly. There is no going around doing multiple runs and doing an average on that. Essentially for the same reason why using min is much worse than using average, with ASLR say you get: 2.0+-0.3 which we run 5 times and 1.8, 1.9, 2.2, 2.1, 2.1 now if you disable ASLR, you get one draw repeated 5 times, which might be 2.0, but also might be 1.8, 5 times. That just hides the problem, but does not actually fix it (because if you touch something, stuff might be allocated in a different order and then you get a different draw) On Mon, Apr 25, 2016 at 12:49 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > Last months, I spent a lot of time on microbenchmarks. Probably too > much time :-) I found a great Linux config to get a much more stable > system to get reliable microbenchmarks: > https://haypo-notes.readthedocs.org/microbenchmark.html > > * isolate some CPU cores > * force CPU to performance > * disable ASLR > * block IRQ on isolated CPU cores > > With such Linux config, the system load doesn't impact benchmark results at all. > > Last days, I almost lost my mind trying to figure out why a very tiny > change in C code makes a difference up to 8% slower. > > My main issue was to get reliable benchmark since running the same > microbenchmark using perf.py gave me "random" results. > > I finished to run directly the underlying script bm_call_simple.py: > > taskset -c 7 ./python ../benchmarks/performance/bm_call_simple.py -n 5 > --timer perf_counter > > In a single run, timings of each loop iteration is very stable. Example: > > 0.22682707803323865 > 0.22741253697313368 > 0.227521265973337 > 0.22750743699725717 > 0.22752994997426867 > 0.22753606992773712 > 0.22742654103785753 > 0.22750875598285347 > 0.22752253606449813 > 0.22718404198531061 > > Problem: each new run gives a different result. Example: > > * run 1: 0.226... > * run 2: 0.255... > * run 3: 0.248... > * run 4: 0.258... > * etc. > > I saw 3 groups of values: ~0.226, ~0.248, ~0.255. > > I didn't understand how running the same program can give so different > result. The reply is the randomization of the Python hash function. > Aaaaaaah! The last source of entropy in my microbenchmark! > > The performance difference can be seen by forcing a specific hash function: > > PYTHONHASHSEED=2 => 0.254... > PYTHONHASHSEED=1 => 0.246... > PYTHONHASHSEED=5 => 0.228... > > Sadly, perf.py and timeit don't disable hash randomization for me. I > hacked perf.py to set PYTHONHASHSEED=0 and magically the result became > super stable! > > Multiple runs of the command: > > $ taskset_isolated.py python3 perf.py ../default/python-ref > ../default/python -b call_simple --fast > > Outputs: > > ### call_simple ### > Min: 0.232621 -> 0.247904: 1.07x slower > Avg: 0.232628 -> 0.247941: 1.07x slower > Significant (t=-591.78) > Stddev: 0.00001 -> 0.00010: 13.7450x larger > > ### call_simple ### > Min: 0.232619 -> 0.247904: 1.07x slower > Avg: 0.232703 -> 0.247955: 1.07x slower > Significant (t=-190.58) > Stddev: 0.00029 -> 0.00011: 2.6336x smaller > > ### call_simple ### > Min: 0.232621 -> 0.247903: 1.07x slower > Avg: 0.232629 -> 0.247918: 1.07x slower > Significant (t=-5896.14) > Stddev: 0.00001 -> 0.00001: 1.3350x larger > > Even with --fast, the result is *very* stable. See the very good > standard deviation. In 3 runs, I got exactly the same "1.07x". Average > timings are the same +/-1 up to 4 digits! > > No need to use the ultra slow --rigourous option. This option is > probably designed to hide the noise of a very unstable system. But > using my Linux config, it doesn't seem to be needed anymore, at least > on this very specific microbenchmark. > > Ok, now I can investigate why my change on the C code introduced a > performance regression :-D > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0

Disable hash randomization to get reliable benchmarks
by Victor Stinner 24 Apr '16

24 Apr '16

Hi, Last months, I spent a lot of time on microbenchmarks. Probably too much time :-) I found a great Linux config to get a much more stable system to get reliable microbenchmarks: https://haypo-notes.readthedocs.org/microbenchmark.html * isolate some CPU cores * force CPU to performance * disable ASLR * block IRQ on isolated CPU cores With such Linux config, the system load doesn't impact benchmark results at all. Last days, I almost lost my mind trying to figure out why a very tiny change in C code makes a difference up to 8% slower. My main issue was to get reliable benchmark since running the same microbenchmark using perf.py gave me "random" results. I finished to run directly the underlying script bm_call_simple.py: taskset -c 7 ./python ../benchmarks/performance/bm_call_simple.py -n 5 --timer perf_counter In a single run, timings of each loop iteration is very stable. Example: 0.22682707803323865 0.22741253697313368 0.227521265973337 0.22750743699725717 0.22752994997426867 0.22753606992773712 0.22742654103785753 0.22750875598285347 0.22752253606449813 0.22718404198531061 Problem: each new run gives a different result. Example: * run 1: 0.226... * run 2: 0.255... * run 3: 0.248... * run 4: 0.258... * etc. I saw 3 groups of values: ~0.226, ~0.248, ~0.255. I didn't understand how running the same program can give so different result. The reply is the randomization of the Python hash function. Aaaaaaah! The last source of entropy in my microbenchmark! The performance difference can be seen by forcing a specific hash function: PYTHONHASHSEED=2 => 0.254... PYTHONHASHSEED=1 => 0.246... PYTHONHASHSEED=5 => 0.228... Sadly, perf.py and timeit don't disable hash randomization for me. I hacked perf.py to set PYTHONHASHSEED=0 and magically the result became super stable! Multiple runs of the command: $ taskset_isolated.py python3 perf.py ../default/python-ref ../default/python -b call_simple --fast Outputs: ### call_simple ### Min: 0.232621 -> 0.247904: 1.07x slower Avg: 0.232628 -> 0.247941: 1.07x slower Significant (t=-591.78) Stddev: 0.00001 -> 0.00010: 13.7450x larger ### call_simple ### Min: 0.232619 -> 0.247904: 1.07x slower Avg: 0.232703 -> 0.247955: 1.07x slower Significant (t=-190.58) Stddev: 0.00029 -> 0.00011: 2.6336x smaller ### call_simple ### Min: 0.232621 -> 0.247903: 1.07x slower Avg: 0.232629 -> 0.247918: 1.07x slower Significant (t=-5896.14) Stddev: 0.00001 -> 0.00001: 1.3350x larger Even with --fast, the result is *very* stable. See the very good standard deviation. In 3 runs, I got exactly the same "1.07x". Average timings are the same +/-1 up to 4 digits! No need to use the ultra slow --rigourous option. This option is probably designed to hide the noise of a very unstable system. But using my Linux config, it doesn't seem to be needed anymore, at least on this very specific microbenchmark. Ok, now I can investigate why my change on the C code introduced a performance regression :-D Victor

1 0

Re: [Speed] Adapting pypy benchmark suite
by Antoine Pitrou 14 Apr '16

14 Apr '16

On Wed, 13 Apr 2016 20:57:35 +0200 Maciej Fijalkowski <fijall(a)gmail.com> wrote: > Hi > > I have a radical idea: to take a pypy benchmark suite, update the > libraries to newer ones and replace python benchmarks with that. The > main reason being that pypy has a much better coverage of things that > are not microbenchmarks, the list (in json): So why not consolidate all benchmarks together, instead of throwing away work already done? Regards Antoine.

2 1

Re: [Speed] Adapting pypy benchmark suite
by Maciej Fijalkowski 14 Apr '16

14 Apr '16

On Thu, Apr 14, 2016 at 6:51 PM, Brett Cannon <brett(a)python.org> wrote: > > > On Wed, 13 Apr 2016 at 13:33 Zachary Ware <zachary.ware+pydev(a)gmail.com> > wrote: >> >> On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fijall(a)gmail.com> >> wrote: >> > Hi >> > >> > I have a radical idea: to take a pypy benchmark suite, update the >> > libraries to newer ones and replace python benchmarks with that. The >> > main reason being that pypy has a much better coverage of things that >> > are not microbenchmarks, the list (in json): >> > >> > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ >> > >> > Which is much more extensive than this: >> > >> > https://hg.python.org/benchmarks/file/tip/performance >> > >> > I'm willing to put *some* effort, what do people think? >> >> I'm in favor. My support has two conditions, though: >> >> 1) at least a majority of the benchmarks must be Python3 compatible. >> Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are >> 2 compatible anyway. > > > Agreed (although I don't care about the 2/3 compatibility, just the 3 compat > ;) . > >> >> >> 2) and there should be an easy way to run the benchmarks against >> exactly 1 interpreter (for use with speed.python.org). I initially >> tried to set up speed.python.org using the PyPy benchmarks, but >> quickly ran into issues with trying to use 'nullpython.py' as the >> baseline Python. When I switched to using h.p.o/benchmarks, I added >> the '--raw' flag to perf.py which allows the benchmarks to be run on >> one interpreter instead of two. It was just a quick hack, though; I >> have no problems with that feature completely changing (even invoking >> it a different way is ok), so long as it exists. >> >> This project could probably start its life as >> github.com/python/benchmarks and save us from having to migrate >> h.p.o/benchmarks to GitHub. > > > Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org > if works starts on this idea and then not move the old repo at all if this > succeeds. We could then make the people who care about the benchmarks the > maintainers of the new repository and contribute to it directly (which means > interested people from CPython, PyPy, Pyston, IronPython, and Jython). That > way we all get exposure to everyone's benchmarks and there's no more > benchmark fragmentation because some people disagree with another's approach > (i.e. no more benchmark silos among the Python implementations). > > And just because we're talking a new repo, would it be worth considering > relying on pip to grab libraries instead of embedding them in the > repository, hence shrinking the overall size of the repo? > Both make sense - to benchmark against the latest lib X and to benchmark against a pinned lib X

1 0

Re: [Speed] Adapting pypy benchmark suite
by Brett Cannon 14 Apr '16

14 Apr '16

On Wed, 13 Apr 2016 at 13:33 Zachary Ware <zachary.ware+pydev(a)gmail.com> wrote: > On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: > > Hi > > > > I have a radical idea: to take a pypy benchmark suite, update the > > libraries to newer ones and replace python benchmarks with that. The > > main reason being that pypy has a much better coverage of things that > > are not microbenchmarks, the list (in json): > > > > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ > > > > Which is much more extensive than this: > > > > https://hg.python.org/benchmarks/file/tip/performance > > > > I'm willing to put *some* effort, what do people think? > > I'm in favor. My support has two conditions, though: > > 1) at least a majority of the benchmarks must be Python3 compatible. > Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are > 2 compatible anyway. > Agreed (although I don't care about the 2/3 compatibility, just the 3 compat ;) . > > 2) and there should be an easy way to run the benchmarks against > exactly 1 interpreter (for use with speed.python.org). I initially > tried to set up speed.python.org using the PyPy benchmarks, but > quickly ran into issues with trying to use 'nullpython.py' as the > baseline Python. When I switched to using h.p.o/benchmarks, I added > the '--raw' flag to perf.py which allows the benchmarks to be run on > one interpreter instead of two. It was just a quick hack, though; I > have no problems with that feature completely changing (even invoking > it a different way is ok), so long as it exists. > > This project could probably start its life as > github.com/python/benchmarks and save us from having to migrate > h.p.o/benchmarks to GitHub. > Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org if works starts on this idea and then not move the old repo at all if this succeeds. We could then make the people who care about the benchmarks the maintainers of the new repository and contribute to it directly (which means interested people from CPython, PyPy, Pyston, IronPython, and Jython). That way we all get exposure to everyone's benchmarks and there's no more benchmark fragmentation because some people disagree with another's approach (i.e. no more benchmark silos among the Python implementations). And just because we're talking a new repo, would it be worth considering relying on pip to grab libraries instead of embedding them in the repository, hence shrinking the overall size of the repo?

1 0

Re: [Speed] Adapting pypy benchmark suite
by Maciej Fijalkowski 13 Apr '16

13 Apr '16

On Wed, Apr 13, 2016 at 10:33 PM, Zachary Ware <zachary.ware+pydev(a)gmail.com> wrote: > On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >> Hi >> >> I have a radical idea: to take a pypy benchmark suite, update the >> libraries to newer ones and replace python benchmarks with that. The >> main reason being that pypy has a much better coverage of things that >> are not microbenchmarks, the list (in json): >> >> http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ >> >> Which is much more extensive than this: >> >> https://hg.python.org/benchmarks/file/tip/performance >> >> I'm willing to put *some* effort, what do people think? > > I'm in favor. My support has two conditions, though: > > 1) at least a majority of the benchmarks must be Python3 compatible. > Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are > 2 compatible anyway. The 3-compatible is likely about updating the libs > > 2) and there should be an easy way to run the benchmarks against > exactly 1 interpreter (for use with speed.python.org). I initially > tried to set up speed.python.org using the PyPy benchmarks, but > quickly ran into issues with trying to use 'nullpython.py' as the > baseline Python. When I switched to using h.p.o/benchmarks, I added > the '--raw' flag to perf.py which allows the benchmarks to be run on > one interpreter instead of two. It was just a quick hack, though; I > have no problems with that feature completely changing (even invoking > it a different way is ok), so long as it exists. That is something that we're tackling on "single-run" branch, check it out, I will finish it, maybe that's a good reason to finish it > > This project could probably start its life as > github.com/python/benchmarks and save us from having to migrate > h.p.o/benchmarks to GitHub. > > -- > Zach > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] Adapting pypy benchmark suite
by Zachary Ware 13 Apr '16

13 Apr '16

On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > Hi > > I have a radical idea: to take a pypy benchmark suite, update the > libraries to newer ones and replace python benchmarks with that. The > main reason being that pypy has a much better coverage of things that > are not microbenchmarks, the list (in json): > > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ > > Which is much more extensive than this: > > https://hg.python.org/benchmarks/file/tip/performance > > I'm willing to put *some* effort, what do people think? I'm in favor. My support has two conditions, though: 1) at least a majority of the benchmarks must be Python3 compatible. Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are 2 compatible anyway. 2) and there should be an easy way to run the benchmarks against exactly 1 interpreter (for use with speed.python.org). I initially tried to set up speed.python.org using the PyPy benchmarks, but quickly ran into issues with trying to use 'nullpython.py' as the baseline Python. When I switched to using h.p.o/benchmarks, I added the '--raw' flag to perf.py which allows the benchmarks to be run on one interpreter instead of two. It was just a quick hack, though; I have no problems with that feature completely changing (even invoking it a different way is ok), so long as it exists. This project could probably start its life as github.com/python/benchmarks and save us from having to migrate h.p.o/benchmarks to GitHub. -- Zach

1 0