Hello,

The existence of variance across machines is true even with ASLR on. My point was regarding doing repetitive measurements on the same machine. Nevertheless, even if small variations may appear in certain circumstances, you can minimize them if you use identical machines, with identical software, settings, etc. And the most important, the deltas remain comparable.

For the dedicated Python CodeSpeed machine that does daily measurements, among others, this can be a good setting for a bit more reliable results.

Thank you, Alecsandru

-----Original Message----- From: Maciej Fijalkowski [mailto:fijall@gmail.com] Sent: Sunday, February 14, 2016 2:20 PM To: Patrascu, Alecsandru alecsandru.patrascu@intel.com Cc: Victor Stinner victor.stinner@gmail.com; speed@python.org Subject: Re: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark

Hi.

Disabling ASLR means you get more repeatable benchmarks, of course, but also means that on another identical machine (or a bit different circumstances), you can get different results, hence you moved statistical error to a more systematic one. I don't think that's a win

...

Hi,

Some of the things we do here at Intel, in our Languages Performance Lab [1,2], is to disable ASLR as you get more reliable results. This can be achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. Also, setting the CPU frequency at a fixed frequency, disabling Turbo Boost and Hyper Threading, also helps for benchmark stability.

From my experience, the isolcpus feature is useful when you have a lot of cores on your machine because the kernel will have other cores on which it can schedule its work; furthermore, it is a best effort situation and it is not an absolute guarantee that the kernel will not use the cores specified if you have a lot of processes running (for example, if you benchmark on a machine with 2 physical cores and you isolate one of the cores, there is a big chance that the kernel will schedule processes on

On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru alecsandru.patrascu@intel.com wrote: this core also, even it is for a small amount of time). Nevertheless, for machines with more physical cores, it can be good to have dedicated core(s) on which we do benchmarking.

...

[1] http://languagesperformance.intel.com/ [2] https://lists.01.org/pipermail/langperf/

Thank you, Alecsandru

...

-----Original Message----- From: Speed [mailto:speed- bounces+alecsandru.patrascu=intel.com@python.org] On Behalf Of Victor Stinner Sent: Friday, February 12, 2016 12:54 AM To: speed@python.org Subject: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark

Hi,

I'm sharing with you my notes (tricks) to get more reliable benchmarks on Linux if your CPU have multiple cores:

https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micr o- benchmarks

FYI perf.py recently got a new --affinity= optional parameter. I plan to send a patch to automatically use /sys/devices/system/cpu/isolated if it's not empty.

What are your "tricks" to get reliable benchmarks?

Victor

---

Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed

---

Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed