On Fri, Feb 12, 2016 at 08:00:23AM -0800, Paul wrote:
PyPy typically needs more than 2000 iterations to be warmed up.
Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance.
To be clear, what I called an "iteration" is one in-process run of an entire benchmark. Each benchmark will invoke tons of methods and execute tons of user loops.
2000 in-process iterations should be plenty enough to warm up the VMs. Most benchmarking experiments take only around 30 post-warmup in-process iterations (enough to compute a confidence interval).
The well-behaved benchmark/vm pairs in our experiment warmup in less than ten in-process iterations.
Cheers
-- Best Regards Edd Barrett