Mailman 3 February 2016 - Speed

Re: [Speed] Should we change what benchmarks we have?
by Brett Cannon 23 Feb '16

23 Feb '16

On Sun, 14 Feb 2016 at 08:57 Brett Cannon <brett(a)python.org> wrote: > > > On Fri, Feb 12, 2016, 07:49 Antoine Pitrou <solipsis(a)pitrou.net> wrote: > >> On Fri, 12 Feb 2016 15:48:01 +0100 >> Maciej Fijalkowski <fijall(a)gmail.com> >> wrote: >> > I presume you looked at the pypy benchmark suite, which contains a >> > large collection of library-based benchmarks. >> >> Not in a long time, I admit... >> > > So it sounds like: > > * we should drop regex_v8, telco, and spectral_norm > Created http://bugs.python.org/issue26416 to track this. > *Having an explanation as to what a benchmark is meant to exercise > wouldn't go amiss > This can wait until we migrate to GitHub. > * Pyston and PyPy have potential benchmarks to steal (although they need > to work with at least Python 3.5 to be considered) > No one stepped forward for this on either the PyPy/Pyston or CPython side. -Brett > > Anyone want the satisfaction of deprecating those benchmarks? How about > writing a README file for what each of the benchmarks are for (which will > become the README for the future GitHub repo)? And do we want the Pyston > and PyPy folks to nominate benchmarks they think we really should add (with > a wild hope of finally having a single suite that everyone at least starts > from), or should some cpython devs look s at what PyPy and Pyston have and > raid their benchmarks? > > Brett > > > >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Speed mailing list >> Speed(a)python.org >> https://mail.python.org/mailman/listinfo/speed >> >

1 0

Re: [Speed] Should we change what benchmarks we have?
by Antoine Pitrou 14 Feb '16

14 Feb '16

On Fri, 12 Feb 2016 15:48:01 +0100 Maciej Fijalkowski <fijall(a)gmail.com> wrote: > I presume you looked at the pypy benchmark suite, which contains a > large collection of library-based benchmarks. Not in a long time, I admit... Regards Antoine.

2 1

Re: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark
by Patrascu, Alecsandru 14 Feb '16

14 Feb '16

Hello, The existence of variance across machines is true even with ASLR on. My point was regarding doing repetitive measurements on the same machine. Nevertheless, even if small variations may appear in certain circumstances, you can minimize them if you use identical machines, with identical software, settings, etc. And the most important, the deltas remain comparable. For the dedicated Python CodeSpeed machine that does daily measurements, among others, this can be a good setting for a bit more reliable results. Thank you, Alecsandru > -----Original Message----- > From: Maciej Fijalkowski [mailto:fijall@gmail.com] > Sent: Sunday, February 14, 2016 2:20 PM > To: Patrascu, Alecsandru <alecsandru.patrascu(a)intel.com> > Cc: Victor Stinner <victor.stinner(a)gmail.com>; speed(a)python.org > Subject: Re: [Speed] Linux tip: use isolcpus to have (more) reliable > benchmark > > Hi. > > Disabling ASLR means you get more repeatable benchmarks, of course, but > also means that on another identical machine (or a bit different > circumstances), you can get different results, hence you moved statistical > error to a more systematic one. I don't think that's a win > > On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru > <alecsandru.patrascu(a)intel.com> wrote: > > Hi, > > > > Some of the things we do here at Intel, in our Languages Performance Lab > [1,2], is to disable ASLR as you get more reliable results. This can be > achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. > Also, setting the CPU frequency at a fixed frequency, disabling Turbo > Boost and Hyper Threading, also helps for benchmark stability. > > > > From my experience, the isolcpus feature is useful when you have a lot > of cores on your machine because the kernel will have other cores on which > it can schedule its work; furthermore, it is a best effort situation and > it is not an absolute guarantee that the kernel will not use the cores > specified if you have a lot of processes running (for example, if you > benchmark on a machine with 2 physical cores and you isolate one of the > cores, there is a big chance that the kernel will schedule processes on > this core also, even it is for a small amount of time). Nevertheless, for > machines with more physical cores, it can be good to have dedicated > core(s) on which we do benchmarking. > > > > [1] http://languagesperformance.intel.com/ > > [2] https://lists.01.org/pipermail/langperf/ > > > > Thank you, > > Alecsandru > > > >> -----Original Message----- > >> From: Speed [mailto:speed- > >> bounces+alecsandru.patrascu=intel.com(a)python.org] On Behalf Of Victor > >> Stinner > >> Sent: Friday, February 12, 2016 12:54 AM > >> To: speed(a)python.org > >> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable > >> benchmark > >> > >> Hi, > >> > >> I'm sharing with you my notes (tricks) to get more reliable > >> benchmarks on Linux if your CPU have multiple cores: > >> > >> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micr > >> o- > >> benchmarks > >> > >> FYI perf.py recently got a new --affinity= optional parameter. I plan > >> to send a patch to automatically use /sys/devices/system/cpu/isolated > >> if it's not empty. > >> > >> What are your "tricks" to get reliable benchmarks? > >> > >> Victor > >> _______________________________________________ > >> Speed mailing list > >> Speed(a)python.org > >> https://mail.python.org/mailman/listinfo/speed > > _______________________________________________ > > Speed mailing list > > Speed(a)python.org > > https://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark
by Maciej Fijalkowski 14 Feb '16

14 Feb '16

Hi. Disabling ASLR means you get more repeatable benchmarks, of course, but also means that on another identical machine (or a bit different circumstances), you can get different results, hence you moved statistical error to a more systematic one. I don't think that's a win On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru <alecsandru.patrascu(a)intel.com> wrote: > Hi, > > Some of the things we do here at Intel, in our Languages Performance Lab [1,2], is to disable ASLR as you get more reliable results. This can be achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. Also, setting the CPU frequency at a fixed frequency, disabling Turbo Boost and Hyper Threading, also helps for benchmark stability. > > From my experience, the isolcpus feature is useful when you have a lot of cores on your machine because the kernel will have other cores on which it can schedule its work; furthermore, it is a best effort situation and it is not an absolute guarantee that the kernel will not use the cores specified if you have a lot of processes running (for example, if you benchmark on a machine with 2 physical cores and you isolate one of the cores, there is a big chance that the kernel will schedule processes on this core also, even it is for a small amount of time). Nevertheless, for machines with more physical cores, it can be good to have dedicated core(s) on which we do benchmarking. > > [1] http://languagesperformance.intel.com/ > [2] https://lists.01.org/pipermail/langperf/ > > Thank you, > Alecsandru > >> -----Original Message----- >> From: Speed [mailto:speed- >> bounces+alecsandru.patrascu=intel.com(a)python.org] On Behalf Of Victor >> Stinner >> Sent: Friday, February 12, 2016 12:54 AM >> To: speed(a)python.org >> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark >> >> Hi, >> >> I'm sharing with you my notes (tricks) to get more reliable benchmarks on >> Linux if your CPU have multiple cores: >> >> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micro- >> benchmarks >> >> FYI perf.py recently got a new --affinity= optional parameter. I plan to >> send a patch to automatically use /sys/devices/system/cpu/isolated if it's >> not empty. >> >> What are your "tricks" to get reliable benchmarks? >> >> Victor >> _______________________________________________ >> Speed mailing list >> Speed(a)python.org >> https://mail.python.org/mailman/listinfo/speed > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] Do we want to stop vendoring source of third-party libraries with the benchmarks?
by Antoine Pitrou 13 Feb '16

13 Feb '16

On Thu, 11 Feb 2016 18:35:29 +0000 Brett Cannon <brett(a)python.org> wrote: > Maybe we should just have a requirements.txt file for Python 2 and another > for Python 3 that are pegged to specific versions? We could even install > things into a venv for isolation. How does this impact interaction with the benchmarks suite? E.g. does it increase the time of running a couple of benchmarks? Does it make it easier or harder to benchmark a work-in-progress patch for whatever interpreter? > If we go this route then we could make > the benchmark suite a package on PyPI and have people install the benchmark > suite and then have instructions to run pip on the requirements files that > we embed in the package. I'm not fond of encouraging random users to run the benchmarks suite without understanding what they're doing, and starting throwing around pointless numbers and misconceptions about performance (which are then very hard to fight since people tend to be irrationally captivated by "performance numbers"). The benchmarks suite is mostly a tool for developers of Python implementations, not the greater public. Having the benchmarks suite only available through hg or git kind of discourages those tendencies. Regards Antoine.

2 1

Re: [Speed] Experiences with Microbenchmarking
by Paul 13 Feb '16

13 Feb '16

On 12 Feb 2016 07:10, Armin Rigo <arigo(a)tunes.org> wrote: > > Hi Edd, > > On Fri, Feb 12, 2016 at 12:18 PM, Edd Barrett <edd(a)theunixzoo.co.uk> wrote: > > JITted VMs (currently PyPy, HotSpot, Graal, LuaJIT, HHVM, JRubyTruffle > > and V8) using microbenchmarks. For each microbenchmark/VM pairing we > > sequentially run a number of processes (currently 10), and within each > > process we run 2000 iterations of the microbenchmark. We then plot the > > results and make observations. > > PyPy typically needs more than 2000 iterations to be warmed up. > Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance. Paul.

3 3

Re: [Speed] Experiences with Microbenchmarking
by Maciej Fijalkowski 13 Feb '16

13 Feb '16

On Fri, Feb 12, 2016 at 7:00 PM, Armin Rigo <arigo(a)tunes.org> wrote: > Hi Paul, > > On Fri, Feb 12, 2016 at 5:00 PM, Paul <paul(a)paulgraydon.co.uk> wrote: >>> PyPy typically needs more than 2000 iterations to be warmed up. >> >> Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance. > > Ew, it's even longer than PyPy :-) > > In the PyPy case, the number 2000 is particularly bad, because the JIT > starts after 1039 iterations. It also adds a few extra paths > afterwards, starting maybe around ~400-500 extra iterations (as a mean > value). Each time, the JIT produces more machine code > and there is a relatively important pause. So 2000 is close to the > worst case: even running 2000 purely-interpreted iterations would be > faster. > > > A bientôt, > > Armin. > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed Armin, those are "2000 iterations of a benchmark" and not "2000 iterations of a loop". A lot of those are pypy benchmarks, just run longer

1 0

Re: [Speed] Tool to run Python microbenchmarks
by Maciej Fijalkowski 12 Feb '16

12 Feb '16

On Fri, Feb 12, 2016 at 4:58 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > 2016-02-12 16:42 GMT+01:00 Maciej Fijalkowski <fijall(a)gmail.com>: >> timeit does two really terrible things - uses min(time) and disables >> the garbage collector, which makes it completely unreliable. > > Can you please elaborate why using min(times) is a bad idea? > > I'm also using min() in my tool, I expect that it helps to ignore the > sporadic peeks when the system is unstable. > > Victor Yes, it also helps to ignore systematic peaks that will happen randomly (due to cache alignment, memory ordering, dicts etc.). Some operations are really random that you should not ignore. E.g. if you have: l.append('a') in a loop, you gonna ignore all the places that resize loop. I'll look for a reference

1 0

Re: [Speed] Tool to run Python microbenchmarks
by Victor Stinner 12 Feb '16

12 Feb '16

Hi, 2016-02-12 16:42 GMT+01:00 Maciej Fijalkowski <fijall(a)gmail.com>: > timeit does two really terrible things - uses min(time) and disables > the garbage collector, which makes it completely unreliable. Can you please elaborate why using min(times) is a bad idea? I'm also using min() in my tool, I expect that it helps to ignore the sporadic peeks when the system is unstable. Victor

1 0

Re: [Speed] Tool to run Python microbenchmarks
by Maciej Fijalkowski 12 Feb '16

12 Feb '16

Hi Victor timeit does two really terrible things - uses min(time) and disables the garbage collector, which makes it completely unreliable. On Thu, Feb 11, 2016 at 11:39 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > To run "micro"-benchmarks on "micro"-optimizations, I started to use > timeit, but in my experience timeit it far from reliable. > > When I say micro: I'm talking about a test which takes less than 1000 > ns, sometimes even a few nanoseconds! > > You always have to run the same micro-benchmark when timeit *at least* > 5 times to find the "real" "minimum" runtime. > > That's why I wrote my own tool to run microbenchmarks: > https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py > > Yury suggested me to add this tool to the Python benchmark project. > I'm ok with that, but only if we rename it to "microbench.py" :-) I > wrote this tool to compare micro-optimizations with a long list of > very simple tests. The result is written into a file. Then you can > compare two files and compare more files, and maybe even compare > multiple files to a "reference". It "hides" difference smaller than 5% > to ignore the noise. > > The main feature is benchmark.py is that it calibrates the benchmark > using time to choose the number of runs and number of loops. I > proposed a similar idea for perf.py: > https://bugs.python.org/issue26275 > > What do you think? Would this tool be useful? > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0