Mailman 3 February 2016 - Speed

Experiences with Microbenchmarking
by Edd Barrett Feb. 12, 2016

Feb. 12, 2016

Hi, A colleague has just pointed me to the discussions on this list regarding benchmarking methodology. Over the past few months we have been devising an "as rigorous as possible" micro-benchmarking experiment. It seems there's a lot of crossover in our work and your discussions. In short, our experiment is investigating the warmup behaviours of JITted VMs (currently PyPy, HotSpot, Graal, LuaJIT, HHVM, JRubyTruffle and V8) using microbenchmarks. For each microbenchmark/VM pairing we sequentially run a number of processes (currently 10), and within each process we run 2000 iterations of the microbenchmark. We then plot the results and make observations. The experiments were run under our own "paranoid" benchmark runner (Krun), which aims to control as many confounding variables as are practically possible. Amongst others, it checks that all benchmarks are run with the system at a similar starting temperature, disables ASLR, uses a monotonic system clock (in some cases we had to patch VMs) and it reboots the system before each benchmark. We did not isolate CPUs, since we found that this creates artificial contention on multi-threaded VMs, however, we did use (and Krun checks for) a tickless Linux kernel. We expected to see typical warmup behaviours (with distinct phases for profiling, compilation, and peak performance), but in reality we saw all kinds of crazy behaviours and even slowdowns. We've published a draft paper showing our preliminary findings here: http://arxiv.org/abs/1602.00602 The draft shows a subset of our results. Run-sequence plots for all process executions can be found here: https://archive.org/download/softdev_warmup_experiment_artefacts/v0.1/all_g… For the final version of the paper we are trying to devise statistical methods to automatically classify the strange warmup behaviours we encountered. We will also run CPython in our final experiment, which may interest you guys :) If this interests anyone, I'd be happy to discuss further. Cheers -- Best Regards Edd Barrett http://www.theunixzoo.co.uk

2 1

Re: [Speed] Should we change what benchmarks we have?
by Antoine Pitrou Feb. 12, 2016

Feb. 12, 2016

On Thu, 11 Feb 2016 18:36:33 +0000 Brett Cannon <brett(a)python.org> wrote: > Are we happy with the current benchmarks? Are there some we want to drop? > How about add? Do we want to have explanations as to why each benchmark is > included? There are no real explanations except the provenance of said benchmarks: - the benchmarks suite was originally developed for Unladen Swallow - some benchmarks were taken and adapted from the "Great Computer Language Shootout" (which I think is a poor source of benchmarks) - some benchmarks have been added for specific concerns that may not be of enough interest in general (for example micro-benchmarks of methods calls, or benchmarks of json / pickle performance) > A better balance of micro vs. macro benchmarks (and probably > matching groups)? Easier said than done :-) Macro-benchmarks are harder to write, especially with the constraints that 1) runtimes should be short enough for convenient use 2) performance numbers should be stable enough accross runs. Regards Antoine.

2 2

Re: [Speed] Do we want to stop vendoring source of third-party libraries with the benchmarks?
by Donald Stufft Feb. 12, 2016

Feb. 12, 2016

> On Feb 12, 2016, at 12:03 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > On 12 February 2016 at 04:35, Brett Cannon <brett(a)python.org> wrote: >> Maybe we should just have a requirements.txt file for Python 2 and another >> for Python 3 that are pegged to specific versions? We could even install >> things into a venv for isolation. If we go this route then we could make the >> benchmark suite a package on PyPI and have people install the benchmark >> suite and then have instructions to run pip on the requirements files that >> we embed in the package. This also gets us around any potential licensing >> issues with embedding third-party libraries. > > +1, especially if you use peep to update the requirements list with > the sdist hashes > pip 8 has peep functionality built in. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

Re: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark
by Patrascu, Alecsandru Feb. 12, 2016

Feb. 12, 2016

Hi, Some of the things we do here at Intel, in our Languages Performance Lab [1,2], is to disable ASLR as you get more reliable results. This can be achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. Also, setting the CPU frequency at a fixed frequency, disabling Turbo Boost and Hyper Threading, also helps for benchmark stability. >From my experience, the isolcpus feature is useful when you have a lot of cores on your machine because the kernel will have other cores on which it can schedule its work; furthermore, it is a best effort situation and it is not an absolute guarantee that the kernel will not use the cores specified if you have a lot of processes running (for example, if you benchmark on a machine with 2 physical cores and you isolate one of the cores, there is a big chance that the kernel will schedule processes on this core also, even it is for a small amount of time). Nevertheless, for machines with more physical cores, it can be good to have dedicated core(s) on which we do benchmarking. [1] http://languagesperformance.intel.com/ [2] https://lists.01.org/pipermail/langperf/ Thank you, Alecsandru > -----Original Message----- > From: Speed [mailto:speed- > bounces+alecsandru.patrascu=intel.com(a)python.org] On Behalf Of Victor > Stinner > Sent: Friday, February 12, 2016 12:54 AM > To: speed(a)python.org > Subject: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark > > Hi, > > I'm sharing with you my notes (tricks) to get more reliable benchmarks on > Linux if your CPU have multiple cores: > > https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micro- > benchmarks > > FYI perf.py recently got a new --affinity= optional parameter. I plan to > send a patch to automatically use /sys/devices/system/cpu/isolated if it's > not empty. > > What are your "tricks" to get reliable benchmarks? > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] Should we change what benchmarks we have?
by Nick Coghlan Feb. 12, 2016

Feb. 12, 2016

On 12 February 2016 at 15:17, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > It's probably best to consider telco as a microbenchmark of decimal > module performance rather than as a general macrobenchmark, though - > that's why the integration of cdecimal improved it so dramatically. Ah, I had misread the rest of the thread - if telco in its current form isn't useful as a decimal microbenchmark, then yes, updating it to improve its stability is more important than preserving it as is. Its original use case was to optimise the decimal implementation itself by figuring out where the hotspots were and optimising those, rather than as a general benchmark for other changes to the interpreter implementation. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Speed] Should we change what benchmarks we have?
by Yury Selivanov Feb. 12, 2016

Feb. 12, 2016

On 2016-02-11 5:37 PM, Victor Stinner wrote: > 2016-02-11 19:36 GMT+01:00 Brett Cannon <brett(a)python.org>: >> Are we happy with the current benchmarks? > bm_regex8 looks unstable, but I don't know if it's an issue of the > benchmark itself or perf.py (see the other thread "[Speed] Any changes > we want to make to perf.py?"). It's super unstable. As well as telco -- I don't trust those benchmarks. > > I spent a lot of time (probably too much!) last months trying to > micro-optimize some parts of Python, specially operations on Python > int. See for example this long issue: > https://bugs.python.org/issue21955 > > At the end, the discussed patched only makes two benchmarks faster: > nbody & spectral_norm. > > I'm disappointed because I don't know if it's worth to take these > micro-optimizations only to run two *benchmarks* faster. Are they > representative of "regular" Python code and "real-world applications"? > Or are they typical maths benchmark? > > For maths, we all know that pure Python sucks and that maybe better > options are available: PyPy, numba, Cython, etc. For example, PyPy is > around 10x faster, whereas discussed micro-optimizations are 1.18x > faster in the best case (in one very specific micro-benchmark). 18% is a pretty serious improvement. I consider issue 21955 as an attempt to fix a performance regression in Python 3. int+int operations in Py2 have a fast path in Python2, so they should have it in Python 3. Right now, spectral_norm is 50% faster on python 2 (when compared to 3.5). With patches from: - #26288 (fast PyLong_AsDouble, committed), - #26289 (faster floor division for longs, committed), - #24165 (free list for longs, will be committed) and - #21955 (fast path for longs in ceval, not committed) we can make 3.6 as fast as 2.7 for numeric code. Yes, spectral_norm is micro-benchmark, but still, there is a lot of python code out there that does some calculation in pure Python not involving numpy or pypy. I think it's important to fix py3 for that kind of code. That said, I'd like to find a better alternative to spectral-norm, something real, that stresses ints/floats and not using numpy. We also need a numpy benchmark, to make sure that we don't make numpy code slower by optimizing CPython. Yury

4 7

Re: [Speed] Do we want to stop vendoring source of third-party libraries with the benchmarks?
by Nick Coghlan Feb. 12, 2016

Feb. 12, 2016

On 12 February 2016 at 04:35, Brett Cannon <brett(a)python.org> wrote: > Maybe we should just have a requirements.txt file for Python 2 and another > for Python 3 that are pegged to specific versions? We could even install > things into a venv for isolation. If we go this route then we could make the > benchmark suite a package on PyPI and have people install the benchmark > suite and then have instructions to run pip on the requirements files that > we embed in the package. This also gets us around any potential licensing > issues with embedding third-party libraries. +1, especially if you use peep to update the requirements list with the sdist hashes Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Speed] Should we change what benchmarks we have?
by Yury Selivanov Feb. 11, 2016

Feb. 11, 2016

On 2016-02-11 6:24 PM, Victor Stinner wrote: > 2016-02-12 0:16 GMT+01:00 Yury Selivanov <yselivanov.ml(a)gmail.com>: >> Unless 21955 makes numpy code slower, I'm not sure why we're discussing >> this. > Stefan Krah wrote that it makes the decimal module 6% slower: > https://bugs.python.org/issue21955#msg259571 > > Again in another message, "big slowdown for _decimal": > https://bugs.python.org/issue21955#msg259793 > > Victor Yes, we need a good benchmark for decimals or numpy. Both use operator overloading extensively. Then I guess we can talk if there is an actual slowdown. Yury

1 0

Linux tip: use isolcpus to have (more) reliable benchmark
by Victor Stinner Feb. 11, 2016

Feb. 11, 2016

Hi, I'm sharing with you my notes (tricks) to get more reliable benchmarks on Linux if your CPU have multiple cores: https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micro-benc… FYI perf.py recently got a new --affinity= optional parameter. I plan to send a patch to automatically use /sys/devices/system/cpu/isolated if it's not empty. What are your "tricks" to get reliable benchmarks? Victor

1 0

Tool to run Python microbenchmarks
by Victor Stinner Feb. 11, 2016

Feb. 11, 2016

Hi, To run "micro"-benchmarks on "micro"-optimizations, I started to use timeit, but in my experience timeit it far from reliable. When I say micro: I'm talking about a test which takes less than 1000 ns, sometimes even a few nanoseconds! You always have to run the same micro-benchmark when timeit *at least* 5 times to find the "real" "minimum" runtime. That's why I wrote my own tool to run microbenchmarks: https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py Yury suggested me to add this tool to the Python benchmark project. I'm ok with that, but only if we rename it to "microbench.py" :-) I wrote this tool to compare micro-optimizations with a long list of very simple tests. The result is written into a file. Then you can compare two files and compare more files, and maybe even compare multiple files to a "reference". It "hides" difference smaller than 5% to ignore the noise. The main feature is benchmark.py is that it calibrates the benchmark using time to choose the number of runs and number of loops. I proposed a similar idea for perf.py: https://bugs.python.org/issue26275 What do you think? Would this tool be useful? Victor

1 0