[Speed] Experiences with Microbenchmarking

[Edd Barrett]

On Fri, Feb 12, 2016 at 08:00:23AM -0800, Paul wrote:
> > PyPy typically needs more than 2000 iterations to be warmed up.
> >
> 
> Same goes for the JVM. Off the top of my head it doesn't even start marking a method as hot until around 10,000 iterations (at which point it'll start to do the first stage of optimisations). If you're below that threshold you're dealing with pure interpreter performance.

To be clear, what I called an "iteration" is one in-process run of an
entire benchmark. Each benchmark will invoke tons of methods and execute
tons of user loops.

2000 in-process iterations should be plenty enough to warm up the VMs.
Most benchmarking experiments take only around 30 post-warmup in-process
iterations (enough to compute a confidence interval).

The well-behaved benchmark/vm pairs in our experiment warmup in less
than ten in-process iterations.

Cheers

-- 
Best Regards
Edd Barrett

http://www.theunixzoo.co.uk