Re: [Speed] Experiences with Microbenchmarking
On 12 Feb 2016 07:10, Armin Rigo <arigo@tunes.org> wrote:
Hi Edd,
On Fri, Feb 12, 2016 at 12:18 PM, Edd Barrett <edd@theunixzoo.co.uk> wrote:
JITted VMs (currently PyPy, HotSpot, Graal, LuaJIT, HHVM, JRubyTruffle and V8) using microbenchmarks. For each microbenchmark/VM pairing we sequentially run a number of processes (currently 10), and within each process we run 2000 iterations of the microbenchmark. We then plot the results and make observations.
PyPy typically needs more than 2000 iterations to be warmed up.
Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance.
Paul.
Hi Paul,
On Fri, Feb 12, 2016 at 5:00 PM, Paul <paul@paulgraydon.co.uk> wrote:
PyPy typically needs more than 2000 iterations to be warmed up.
Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance.
Ew, it's even longer than PyPy :-)
In the PyPy case, the number 2000 is particularly bad, because the JIT starts after 1039 iterations. It also adds a few extra paths afterwards, starting maybe around ~400-500 extra iterations (as a mean value). Each time, the JIT produces more machine code and there is a relatively important pause. So 2000 is close to the worst case: even running 2000 purely-interpreted iterations would be faster.
A bientôt,
Armin.
On Fri, Feb 12, 2016 at 08:00:23AM -0800, Paul wrote:
PyPy typically needs more than 2000 iterations to be warmed up.
Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance.
To be clear, what I called an "iteration" is one in-process run of an entire benchmark. Each benchmark will invoke tons of methods and execute tons of user loops.
2000 in-process iterations should be plenty enough to warm up the VMs. Most benchmarking experiments take only around 30 post-warmup in-process iterations (enough to compute a confidence interval).
The well-behaved benchmark/vm pairs in our experiment warmup in less than ten in-process iterations.
Cheers
-- Best Regards Edd Barrett
Hi Edd,
On Fri, Feb 12, 2016 at 7:44 PM, Edd Barrett <edd@theunixzoo.co.uk> wrote:
To be clear, what I called an "iteration" is one in-process run of an entire benchmark.
Oops, sorry. The subject of this thread is "Experiences with Microbenchmarking". I naturally assumed that a microbenchmark is doing one simple thing not in a loop, in which case "iterations" is simply repeating that simple thing. If you have in mind benchmarks that are not as micro as that, then I stand corrected.
A bientôt,
Armin.
participants (3)
-
Armin Rigo -
Edd Barrett -
Paul