Mailman 3 March 2017 - Speed

perf 1.0 released: with a stable API
by Victor Stinner 16 Mar '17

16 Mar '17

Hi, After 9 months of development, the perf API became stable with the awaited "1.0" version. The perf module has now a complete API to write, run and analyze benchmarks and a nice documentation explaining traps of benchmarking and how to avoid, or even, fix them. http://perf.readthedocs.io/ Last days, I rewrote the documentation, hid a few more functions to prevent API changes after the 1.0 release, and I made last backward incompatible changes to fix old design issues. I don't expect the module to be perfect. It's more a milestone to freeze the API and focus on features instead ;-) Changes between 0.9.6 and 1.0: Enhancements: * ``stats`` command now displays percentiles * ``hist`` command now also checks the benchmark stability by default * dump command now displays raw value of calibration runs. * Add ``Benchmark.percentile()`` method Backward incompatible changes: * Remove the ``compare`` command to only keep the ``compare_to`` command which is better defined * Run warmup values must now be normalized per loop iteration. * Remove ``format()`` and ``__str__()`` methods from Benchmark. These methods were too opiniated. * Rename ``--name=NAME`` option to ``--benchmark=NAME`` * Remove ``perf.monotonic_clock()`` since it wasn't monotonic on Python 2.7. * Remove ``is_significant()`` from the public API Other changes: * check command now only complains if min/max is 50% smaller/larger than the mean, instead of 25%. Note: I already updated the performance project to perf 1.0. Victor

1 0

Re: [Speed] ASLR
by Antoine Pitrou 16 Mar '17

16 Mar '17

On Thu, 16 Mar 2017 01:50:39 +0100 Victor Stinner <victor.stinner(a)gmail.com> wrote: > > I made my own experiment on the impact on temperature on performance, > and above 100°C, I didn't notice anything: > https://haypo.github.io/intel-cpus-part2.html > "Impact of the CPU temperature on benchmarks" I suspect temperature can have an impact on performance if Turbo is enabled (or, as you noticed, if CPU cooling is deficient). Note that tweaking a system for benchmarking (disabling Turbo, disabling ASLR, etc.) may make the results more reproducible, but it may also make them less representative of real-world conditions (because few people disable Turbo or ASLR, except precisely on benchmarking machines :-)). It's a delicate balancing act! Regards Antoine.

3 2

Re: [Speed] Median +- MAD or Mean +- std dev?
by Serhiy Storchaka 16 Mar '17

16 Mar '17

On 15.03.17 18:59, Victor Stinner wrote: > It's not easy to compare MAD to standard deviation. It seems like MAD > can even be misleading when reading the "1 ms" part of "10 ms +- 1 > ms". Don't use the "+-" notation. It is misleading even for the stddev of normal distribution, because with the chance 1 against 2 the sample is out of the specified interval. Use "Mean: 10 ms Stddev: 1 ms" or "Median: 10 ms MAD: 1 ms" instead. > Maybe we need a new --median command line option to display > median/MAD, instead of mean/stdev displayed by default? Yes, make this configurable. And make median/MAD the default. ;)

2 1

Re: [Speed] ASLR
by Brett Cannon 16 Mar '17

16 Mar '17

On Wed, 15 Mar 2017 at 17:54 Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2017-03-16 1:38 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com > >: > > Hi All, > > > > I am attaching an image with comparison running the CALL_METHOD in the > old Grand Unified Python Benchmark (GUPB) suite ( > https://hg.python.org/benchmarks), with and without ASLR disabled. > > This benchmark suite is now deprecated, please update to the new > 'performance' benchmark suite: > https://github.com/python/performance > > The old benchmark suite didn't spawn multiple processes and so was > less reliable. > > By the way, maybe I should commit a change in hg.python.org/benchmarks > to remove the code and only keep a README.txt? Code will still be > accessible in Mercurial history. > Since we might not shut down hg.python.org for a long time I say go ahead and commit such a change.

1 0

Re: [Speed] perf 0.9.6 released
by Victor Stinner 15 Mar '17

15 Mar '17

I updated performance for perf 0.9.6. I patched python_startup and hg_startup benchmarks to use the new bench_command() method. This new method uses the following Python script to measure the time to execute a command: https://github.com/haypo/perf/blob/master/perf/_process_time.py I wrote the _process_time.py script to be small and simple to reduce the overhead of the benchmark itself. It's similar to the "real time" line of UNIX 'time' command, but it works on Windows too. I chose to use time.perf_counter(), wall clock, instead of using getrusage() which provides CPU time. It's easy for me to understand wall clock time rather than CPU time, and it's more consistent with other perf methods. Victor 2017-03-16 1:59 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>: > Hi, > > I released perf 0.9.6 with many changes. First, "Mean +- std dev" is > now displayed, instead of "Median +- std dev", as a result of the > previous thread on this list. The median is still accessible via the > stats command. By the way, the "stats" command now displays "Median +- > MAD" instead of "Median +- std dev". > > I broke the API to fix an old mistake. I used the term "sample" for a > single value, whereas a "sample" in statistics is a set of values (one > or more), and so the term is misused. I replace "sample" with "value" > and "samples" with "values" everywhere in perf. > > http://perf.readthedocs.io/en/latest/changelog.html#version-0-9-6-2017-03-15 > > Version 0.9.6 (2017-03-15) > -------------------------- > > Major change: > > * Display ``Mean +- std dev`` instead of ``Median +- std dev`` > > Enhancements: > > * Add a new ``Runner.bench_command()`` method to measure the execution time of > a command. > * Add ``mean()``, ``median_abs_dev()`` and ``stdev()`` methods to ``Benchmark`` > * ``check`` command: test also minimum and maximum compared to the mean > > Major API change, rename "sample" to "value": > > * Rename attributes and methods: > > - ``Benchmark.bench_sample_func()`` => ``Benchmark.bench_time_func()``. > - ``Run.samples`` => ``Run.values`` > - ``Benchmark.get_samples()`` => ``Benchmark.get_values()`` > - ``get_nsample()`` => ``get_nvalue()`` > - ``Benchmark.format_sample()`` => ``Benchmark.format_value()`` > - ``Benchmark.format_samples()`` => ``Benchmark.format_values()`` > > * Rename Runner command line options: > > - ``--samples`` => ``--values`` > - ``--debug-single-sample`` => ``--debug-single-value`` > > Changes: > > * ``convert``: Remove ``--remove-outliers`` option > * ``check`` command now tests stdev/mean, instead of testing stdev/median > * setup.py: statistics dependency is now installed using ``extras_require`` to > support setuptools 18 and newer > * Add setup.cfg to enable universal builds: same wheel package for Python 2 > and Python 3 > * Add ``perf.VERSION`` constant: tuple of int > * JSON version 6: write metadata common to all benchmarks (common to all runs > of all benchmarks) at the root; rename 'samples' to 'values' in runs. > > Victor

1 0

Re: [Speed] ASLR
by Victor Stinner 15 Mar '17

15 Mar '17

2017-03-16 2:04 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com>: > Understood on the obsolete benchmark part. This was the work done before the new benchmark was created on github. I strongly advice you to move to performance. It also has a nice a API. It now produces a JSON file with *all* data, instead of just writing into summaries into stdout. > I thought this is related, and thus didn't open a new thread. The other thread was a discussion about statistics, how to summarize all timing into two numbers :-) > Maybe you could point me to one single micro-benchmark for the time being, and then we could compare result across? The "new" performance project is a fork of the old "benchmark" project. Benchmark names are very close or even the same for many benchmarks. If you would like to validate that your benchmark runner is stable: run call_method and call_simple microbenchmarks on different revisions of CPython, reboot sometimes the computer used to run benchmarks, and make sure that results are stable. Compare them with results of speed.python.org. call_method: https://speed.python.org/timeline/#/?exe=5&ben=call_method&env=1&revs=50&eq… call_simple: https://speed.python.org/timeline/#/?exe=5&ben=call_simple&env=1&revs=50&eq… Around november and december 2016, you should notice a significant speedup on call_method. The best is to be able to avoid "temporary spikes" like this one: https://haypo.github.io/analysis-python-performance-issue.html The API of the perf project, PGO and LTO compilation, new performance using perf, "perf system tune" for system tuning, etc. helped to get more stable results. Victor

1 0

Re: [Speed] ASLR
by Wang, Peter Xihong 15 Mar '17

15 Mar '17

Hi Victor, Understood on the obsolete benchmark part. This was the work done before the new benchmark was created on github. I thought this is related, and thus didn't open a new thread. Maybe you could point me to one single micro-benchmark for the time being, and then we could compare result across? Regards, Peter -----Original Message----- From: Victor Stinner [mailto:victor.stinner@gmail.com] Sent: Wednesday, March 15, 2017 5:51 PM To: speed(a)python.org; Wang, Peter Xihong <peter.xihong.wang(a)intel.com> Subject: ASLR 2017-03-16 1:38 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com>: > Hi All, > > I am attaching an image with comparison running the CALL_METHOD in the old Grand Unified Python Benchmark (GUPB) suite (https://hg.python.org/benchmarks), with and without ASLR disabled. This benchmark suite is now deprecated, please update to the new 'performance' benchmark suite: https://github.com/python/performance The old benchmark suite didn't spawn multiple processes and so was less reliable. By the way, maybe I should commit a change in hg.python.org/benchmarks to remove the code and only keep a README.txt? Code will still be accessible in Mercurial history. > You could see the run2run variation was reduced significantly, from data scattering all over the place, to just one single outlier, out of 30 repeated runs. > This effectively eliminated most of the variations for this micro-benchmark. > > On a Linux system, you could do this by: > as root > echo 0 > /proc/sys/kernel/randomize_va_space # to disable > echo 2 > /proc/sys/kernel/randomize_va_space # to enable > > If anyone still experiences run2run variation, I'd suggest to read on: > Based on my observation in our labs, a lot of factors could impact > performance, including environment (yes, even a room temperature), I made my own experiment on the impact on temperature on performance, and above 100°C, I didn't notice anything: https://haypo.github.io/intel-cpus-part2.html "Impact of the CPU temperature on benchmarks" I tested a desktop and a laptop PC with an Intel CPU. > HW components or related such as platforms, chipset, memory DIMMs, CPU generations and stepping, BIOS version, kernels, the list goes on and on. > > Being said that, would it be helpful we work together, to identify the root cause, be it due to SW, or anything else? We could start with a specific micro-benchmark, with specific goal as what to measure. > After that, or in parallel after some baseline work is done, then focus on measurement process/methodology? > > Is this helpful? > > Thanks, > > Peter Note: Please open a new thread instead of replying to an email of an existing thread. Victor

1 0

perf 0.9.6 released
by Victor Stinner 15 Mar '17

15 Mar '17

Hi, I released perf 0.9.6 with many changes. First, "Mean +- std dev" is now displayed, instead of "Median +- std dev", as a result of the previous thread on this list. The median is still accessible via the stats command. By the way, the "stats" command now displays "Median +- MAD" instead of "Median +- std dev". I broke the API to fix an old mistake. I used the term "sample" for a single value, whereas a "sample" in statistics is a set of values (one or more), and so the term is misused. I replace "sample" with "value" and "samples" with "values" everywhere in perf. http://perf.readthedocs.io/en/latest/changelog.html#version-0-9-6-2017-03-15 Version 0.9.6 (2017-03-15) -------------------------- Major change: * Display ``Mean +- std dev`` instead of ``Median +- std dev`` Enhancements: * Add a new ``Runner.bench_command()`` method to measure the execution time of a command. * Add ``mean()``, ``median_abs_dev()`` and ``stdev()`` methods to ``Benchmark`` * ``check`` command: test also minimum and maximum compared to the mean Major API change, rename "sample" to "value": * Rename attributes and methods: - ``Benchmark.bench_sample_func()`` => ``Benchmark.bench_time_func()``. - ``Run.samples`` => ``Run.values`` - ``Benchmark.get_samples()`` => ``Benchmark.get_values()`` - ``get_nsample()`` => ``get_nvalue()`` - ``Benchmark.format_sample()`` => ``Benchmark.format_value()`` - ``Benchmark.format_samples()`` => ``Benchmark.format_values()`` * Rename Runner command line options: - ``--samples`` => ``--values`` - ``--debug-single-sample`` => ``--debug-single-value`` Changes: * ``convert``: Remove ``--remove-outliers`` option * ``check`` command now tests stdev/mean, instead of testing stdev/median * setup.py: statistics dependency is now installed using ``extras_require`` to support setuptools 18 and newer * Add setup.cfg to enable universal builds: same wheel package for Python 2 and Python 3 * Add ``perf.VERSION`` constant: tuple of int * JSON version 6: write metadata common to all benchmarks (common to all runs of all benchmarks) at the root; rename 'samples' to 'values' in runs. Victor

1 0

ASLR
by Victor Stinner 15 Mar '17

15 Mar '17

2017-03-16 1:38 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com>: > Hi All, > > I am attaching an image with comparison running the CALL_METHOD in the old Grand Unified Python Benchmark (GUPB) suite (https://hg.python.org/benchmarks), with and without ASLR disabled. This benchmark suite is now deprecated, please update to the new 'performance' benchmark suite: https://github.com/python/performance The old benchmark suite didn't spawn multiple processes and so was less reliable. By the way, maybe I should commit a change in hg.python.org/benchmarks to remove the code and only keep a README.txt? Code will still be accessible in Mercurial history. > You could see the run2run variation was reduced significantly, from data scattering all over the place, to just one single outlier, out of 30 repeated runs. > This effectively eliminated most of the variations for this micro-benchmark. > > On a Linux system, you could do this by: > as root > echo 0 > /proc/sys/kernel/randomize_va_space # to disable > echo 2 > /proc/sys/kernel/randomize_va_space # to enable > > If anyone still experiences run2run variation, I'd suggest to read on: > Based on my observation in our labs, a lot of factors could impact performance, including environment (yes, even a room temperature), I made my own experiment on the impact on temperature on performance, and above 100°C, I didn't notice anything: https://haypo.github.io/intel-cpus-part2.html "Impact of the CPU temperature on benchmarks" I tested a desktop and a laptop PC with an Intel CPU. > HW components or related such as platforms, chipset, memory DIMMs, CPU generations and stepping, BIOS version, kernels, the list goes on and on. > > Being said that, would it be helpful we work together, to identify the root cause, be it due to SW, or anything else? We could start with a specific micro-benchmark, with specific goal as what to measure. > After that, or in parallel after some baseline work is done, then focus on measurement process/methodology? > > Is this helpful? > > Thanks, > > Peter Note: Please open a new thread instead of replying to an email of an existing thread. Victor

1 0

Re: [Speed] Median +- MAD or Mean +- std dev?
by Wang, Peter Xihong 15 Mar '17

15 Mar '17

Hi All, I am attaching an image with comparison running the CALL_METHOD in the old Grand Unified Python Benchmark (GUPB) suite (https://hg.python.org/benchmarks), with and without ASLR disabled. You could see the run2run variation was reduced significantly, from data scattering all over the place, to just one single outlier, out of 30 repeated runs. This effectively eliminated most of the variations for this micro-benchmark. On a Linux system, you could do this by: as root echo 0 > /proc/sys/kernel/randomize_va_space # to disable echo 2 > /proc/sys/kernel/randomize_va_space # to enable If anyone still experiences run2run variation, I'd suggest to read on: Based on my observation in our labs, a lot of factors could impact performance, including environment (yes, even a room temperature), HW components or related such as platforms, chipset, memory DIMMs, CPU generations and stepping, BIOS version, kernels, the list goes on and on. Being said that, would it be helpful we work together, to identify the root cause, be it due to SW, or anything else? We could start with a specific micro-benchmark, with specific goal as what to measure. After that, or in parallel after some baseline work is done, then focus on measurement process/methodology? Is this helpful? Thanks, Peter -----Original Message----- From: Speed [mailto:speed-bounces+peter.xihong.wang=intel.com@python.org] On Behalf Of Victor Stinner Sent: Wednesday, March 15, 2017 11:11 AM To: Antoine Pitrou <solipsis(a)pitrou.net> Cc: speed(a)python.org Subject: Re: [Speed] Median +- MAD or Mean +- std dev? 2017-03-15 18:11 GMT+01:00 Antoine Pitrou <solipsis(a)pitrou.net>: > I would say keep it simple. mean/stddev is informative enough, no > need to add or maintain options of dubious utility. Ok. I added a message to suggest to use perf stats to analyze results. Example of warnings with a benchmark result considered as unstable, python startup time measured by the new bench_command() function: --- $ python3 -m perf show startup1.json WARNING: the benchmark result may be unstable * the standard deviation (6.08 ms) is 16% of the mean (39.1 ms) * the minimum (23.6 ms) is 40% smaller than the mean (39.1 ms) Try to rerun the benchmark with more runs, values and/or loops. Run 'python3 -m perf system tune' command to reduce the system jitter. Use perf stats to analyze results, or --quiet to hide warnings. Median +- MAD: 40.7 ms +- 3.9 ms ---- Statistics of this result: --- $ python3 -m perf stats startup1.json -q Total duration: 37.2 sec Start date: 2017-03-15 18:02:46 End date: 2017-03-15 18:03:27 Raw value minimum: 189 ms Raw value maximum: 390 ms Number of runs: 25 Total number of values: 75 Number of values per run: 3 Number of warmups per run: 1 Loop iterations per value: 8 Minimum: 23.6 ms (-42% of the median) Median +- MAD: 40.7 ms +- 3.9 ms Mean +- std dev: 39.1 ms +- 6.1 ms Maximum: 48.7 ms (+20% of the median) --- Victor _______________________________________________ Speed mailing list Speed(a)python.org https://mail.python.org/mailman/listinfo/speed

1 0