Speed July 2016

speed@python.org

6 participants
24 discussions

Re: [Speed] New CPython benchmark suite based on perf
by Victor Stinner 04 Jul '16

04 Jul '16

2016-07-04 16:17 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > I modified the CPython benchmark suite to use my perf module: > https://hg.python.org/sandbox/benchmarks_perf Hum, you need the development version of perf to test it: git clone https://github.com/haypo/perf.git > Changes: > > * replace explicit warmups with perf automatic warmup > (...) > * avoid nested loops, prefer a single level of loop: perf is > responsible to call the sample function enough times to collect enough > samples Concrete example with performance/bm_go.py. Before: ------------------------- def main(n, timer): times = [] for i in range(5): versus_cpu() # warmup for i in range(n): t1 = timer() versus_cpu() t2 = timer() times.append(t2 - t1) return times ------------------------- After: ------------------------- def main(loops): t0 = perf.perf_counter() for _ in xrange(loops): versus_cpu() return perf.perf_counter() - t0 ------------------------- Example of go benchmark output: --- $ python3 benchmarks_perf/performance/bm_go.py -v calibration: 1 loop: 599 ms calibration: use 1 loop Run 1/25: warmup (1): 601 ms; raw samples (3): 593 ms, 593 ms, 593 ms Run 2/25: warmup (1): 609 ms; raw samples (3): 609 ms, 610 ms, 608 ms Run 3/25: warmup (1): 599 ms; raw samples (3): 598 ms, 606 ms, 598 ms (...) Run 25/25: warmup (1): 606 ms; raw samples (3): 591 ms, 590 ms, 591 ms Median +- std dev: 598 ms +- 8 ms --- The warmup samples ("warmup (1): ... ms") are not used to compute median or std dev. Another example to show fancy features of perf: --- $ python3 benchmarks_perf/performance/bm_telco.py -v --hist --stats --metadata -n5 -p50 calibration: 1 loop: 34.6 ms calibration: 2 loops: 57.8 ms calibration: 4 loops: 105 ms calibration: use 4 loops Run 1/50: warmup (1): 116 ms; raw samples (5): 106 ms, 106 ms, 105 ms, 106 ms, 106 ms Run 2/50: warmup (1): 107 ms; raw samples (5): 107 ms, 107 ms, 106 ms, 106 ms, 106 ms Run 3/50: warmup (1): 107 ms; raw samples (5): 106 ms, 106 ms, 106 ms, 106 ms, 106 ms (...) Run 50/50: warmup (1): 106 ms; raw samples (5): 104 ms, 105 ms, 105 ms, 106 ms, 105 ms Metadata: - aslr: enabled - cpu_count: 4 - cpu_model_name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz - date: 2016-07-04T17:00:33 - description: Test the performance of the Telco decimal benchmark - duration: 35.6 sec - hostname: smithers - name: telco - perf_version: 0.6 - platform: Linux-4.5.7-300.fc24.x86_64-x86_64-with-fedora-24-Twenty_Four - python_executable: /usr/bin/python3 - python_implementation: cpython - python_version: 3.5.1 (64bit) - timer: clock_gettime(CLOCK_MONOTONIC), resolution: 1.00 ns 25.8 ms: 1 ## 25.9 ms: 2 ##### 26.0 ms: 4 ########## 26.0 ms: 13 ############################### 26.1 ms: 27 ################################################################# 26.2 ms: 28 ################################################################### 26.3 ms: 21 ################################################## 26.3 ms: 25 ############################################################ 26.4 ms: 32 ############################################################################# 26.5 ms: 33 ############################################################################### 26.6 ms: 18 ########################################### 26.6 ms: 13 ############################### 26.7 ms: 8 ################### 26.8 ms: 8 ################### 26.8 ms: 7 ################# 26.9 ms: 4 ########## 27.0 ms: 4 ########## 27.1 ms: 1 ## 27.1 ms: 0 | 27.2 ms: 0 | 27.3 ms: 1 ## Number of samples: 250 (50 runs x 5 samples; 1 warmup) Standard deviation / median: 1% Shortest raw sample: 103 ms (4 loops) Minimum: 25.9 ms (-2.1%) Median +- std dev: 26.4 ms +- 0.2 ms Maximum: 27.3 ms (+3.4%) Median +- std dev: 26.4 ms +- 0.2 ms --- I used " -n5 -p50" to compute 5 samples per process and use 50 processes. It helps to get a nicer histogram :-) (to have a better uniform distribution) For histogram, I like using telco because it generates a regular gaussian curve :-) Victor

1 0

New CPython benchmark suite based on perf
by Victor Stinner 04 Jul '16

04 Jul '16

Hi, I modified the CPython benchmark suite to use my perf module: https://hg.python.org/sandbox/benchmarks_perf Changes: * use statistics.median() rather than mean() to compute of "average" of samples. Example: Median +- Std dev: 256 ms +- 3 ms -> 262 ms +- 4 ms: 1.03x slower * replace compat.py with external six dependency * replace util.py with perf * replace explicit warmups with perf automatic warmup * add name metadata * for benchmark taking parameters, save parameters in metadata * avoid nested loops, prefer a single level of loop: perf is responsible to call the sample function enough times to collect enough samples * store django and mako version in metadata * use JSON format to exchange timings between benchmarks and runner.py perf adds more features: * run each benchmark in multiple processes (25 by default, 50 in rigorous mode) * calibrate each benchmark to compute the number of loops to get a sample between 100 ms and 1 second TODO: * Right now the calibration in done twice: in the reference python and in the changed python. It should only be once in the reference python * runner.py should write results in a JSON file. Currently, data are not written on disk (a pipe is used with child processes) * Drop external dependencies and create a virtual environment per python * Port more Python 2-only benchmarks to Python 3 * Add more benchmarks from PyPy, Pyston and Pyjion benchmark suites: unify again the benchmark suites :-) perf has builtin tools to analyze the distribution of samples: * add --hist option to a benchmark to display an histogram in text mode * add --stats option to a benchmark to display statistics: number of samples, shortest raw sample, min, max, etc. * "python3 -m perf" CLI allows has many commands to analyze a benchmark: http://perf.readthedocs.io/en/latest/cli.html Right now, perf JSON format is only able to store one benchmark. I will extend the format to be able to store a list of benchmarks. So it will be possible to store all results of a python version into a single file. By the way, I also want to change runner.py CLI to be able to run the benchmarks on a single python version and then use a second command to compare two files. Rather than always running each benchmark twice (reference python, changed python). PyPy runner also works like that if I recall correctly. Victor

1 0

Re: [Speed] bm_pickle: why testing protocol 2?
by Antoine Pitrou 04 Jul '16

04 Jul '16

On Mon, 4 Jul 2016 10:53:25 +0200 Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > performance/bm_pickle.py of the CPython benchmark suite uses the > pickle protocol 2 by default. Why not always testing the highest > protocol? I think this comes from the Unladen Swallow benchmark suite, and Unladen Swallow was Python 2-only, so protocol 2 *was* the highest protocol in those circumstances. Regards Antoine.

1 0

bm_pickle: why testing protocol 2?
by Victor Stinner 04 Jul '16

04 Jul '16

Hi, performance/bm_pickle.py of the CPython benchmark suite uses the pickle protocol 2 by default. Why not always testing the highest protocol? In Python 3.5, the highest protocol is 4 which is more efficient than the protocol 2. Is it a deliberate choice to test exactly the same thing between Python 2 and Python 3? Victor

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed July 2016