[Speed] Linux tip: use isolcpus to have (more) reliable benchmark

[Patrascu, Alecsandru]

Hello,

The existence of variance across machines is true even with ASLR on. My point was regarding doing repetitive measurements on the same machine. Nevertheless, even if small variations may appear in certain circumstances, you can minimize them if you use identical machines, with identical software, settings, etc. And the most important, the deltas remain comparable.

For the dedicated Python CodeSpeed machine that does daily measurements, among others, this can be a good setting for a bit more reliable results.

Thank you,
Alecsandru

> -----Original Message-----
> From: Maciej Fijalkowski [mailto:fijall at gmail.com]
> Sent: Sunday, February 14, 2016 2:20 PM
> To: Patrascu, Alecsandru <alecsandru.patrascu at intel.com>
> Cc: Victor Stinner <victor.stinner at gmail.com>; speed at python.org
> Subject: Re: [Speed] Linux tip: use isolcpus to have (more) reliable
> benchmark
> 
> Hi.
> 
> Disabling ASLR means you get more repeatable benchmarks, of course, but
> also means that on another identical machine (or a bit different
> circumstances), you can get different results, hence you moved statistical
> error to a more systematic one. I don't think that's a win
> 
> On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru
> <alecsandru.patrascu at intel.com> wrote:
> > Hi,
> >
> > Some of the things we do here at Intel, in our Languages Performance Lab
> [1,2], is to disable ASLR as you get more reliable results. This can be
> achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space.
> Also, setting the CPU frequency at a fixed frequency, disabling Turbo
> Boost and Hyper Threading, also helps for benchmark stability.
> >
> > From my experience, the isolcpus feature is useful when you have a lot
> of cores on your machine because the kernel will have other cores on which
> it can schedule its work; furthermore, it is a best effort situation and
> it is not an absolute guarantee that the kernel will not use the cores
> specified if you have a lot of processes running (for example, if you
> benchmark on a machine with 2 physical cores and you isolate one of the
> cores, there is a big chance that the kernel will schedule processes on
> this core also, even it is for a small amount of time). Nevertheless, for
> machines with more physical cores, it can be good to have dedicated
> core(s) on which we do benchmarking.
> >
> > [1] http://languagesperformance.intel.com/
> > [2] https://lists.01.org/pipermail/langperf/
> >
> > Thank you,
> > Alecsandru
> >
> >> -----Original Message-----
> >> From: Speed [mailto:speed-
> >> bounces+alecsandru.patrascu=intel.com at python.org] On Behalf Of Victor
> >> Stinner
> >> Sent: Friday, February 12, 2016 12:54 AM
> >> To: speed at python.org
> >> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable
> >> benchmark
> >>
> >> Hi,
> >>
> >> I'm sharing with you my notes (tricks) to get more reliable
> >> benchmarks on Linux if your CPU have multiple cores:
> >>
> >> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micr
> >> o-
> >> benchmarks
> >>
> >> FYI perf.py recently got a new --affinity= optional parameter. I plan
> >> to send a patch to automatically use /sys/devices/system/cpu/isolated
> >> if it's not empty.
> >>
> >> What are your "tricks" to get reliable benchmarks?
> >>
> >> Victor
> >> _______________________________________________
> >> Speed mailing list
> >> Speed at python.org
> >> https://mail.python.org/mailman/listinfo/speed
> > _______________________________________________
> > Speed mailing list
> > Speed at python.org
> > https://mail.python.org/mailman/listinfo/speed