Speed November 2016

speed@python.org

10 participants
28 discussions

Re: [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

by Victor Stinner

2016-11-04 21:58 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>: > 2016-11-04 20:21 GMT+01:00 Yury Selivanov <yselivanov.ml(a)gmail.com>: >> I'm curious why call_* benchmarks became slower on 3.x? > > It's almost the same between 2.7 and default. For 3.6, it looks like > an issue on the benchmark runner, not on Python itself: (...) Aha, it seems to be a compiler performance issue. I proposed a patch to fix the issue: http://bugs.python.org/issue28618 Victor

3 years, 8 months

Re: [Speed] Performance difference in call_method()

by Victor Stinner

I proposed a patch which fixes the issue: http://bugs.python.org/issue28618 "Decorate hot functions using __attribute__((hot)) to optimize Python" Victor

3 years, 8 months

Re: [Speed] Performance difference in call_method()

by Victor Stinner

I found some interesting differences using the Linux perf tool. # perf stat -e L1-icache-loads,L1-icache-load-misses ./python performance/benchmarks/bm_call_method.py --inherit=PYTHONPATH -v --worker -l1 -n 25 -w0 2016-11-04 23:35 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>: > (1) ce85a1f129e3: 17 ms => 83877018ef97 (Oct 18): 25 ms > > https://hg.python.org/cpython/rev/83877018ef97 Comparison of metrics of rev ce85a1f129e3 (fast) => rev 83877018ef97 (slow): L1-icache-load-misses: 0.06% => 8.41% of all L1-icache hits Instructions per cycle: 2.38 => 1.41 stalled-cycles-frontend: 12.99% => 42.85% frontend cycles idle stalled-cycles-backend: 2.28% => 21.36% backend cycles idle So it confirms what I expected: call_method() is highly impacted by the CPU L1 instruction cache. I don't know exactly why the revision 83877018ef97 has an impact on the CPU L1 cache. Victor

3 years, 8 months

Performance difference in call_method()

by Victor Stinner

Hi, I noticed a temporary performance peak in the call_method: https://speed.python.org/timeline/#/?exe=4&ben=call_method&env=1&revs=50&eq… The difference is major: 17 ms => 29 ms, 70% slower! I expected a temporary issue on the server used to run benchmarks, but... I reproduced the result on the server. Recently, the performance of call_method() changed in CPython default from 17 ms to 28 ms (well, the exact value is variable: 25 ms, 28 ms, 29 ms, ...) and then back to 17 ms: (1) ce85a1f129e3: 17 ms => 83877018ef97 (Oct 18): 25 ms https://hg.python.org/cpython/rev/83877018ef97 (2) 3e073e7b4460: 28 ms => 204a43c452cc (Oct 22): 17 ms https://hg.python.org/cpython/rev/204a43c452cc None of these revisions modify code used in the call_method() benchmark, so I guess that it's yet another compiler joke. On my laptop and my desktop PC, I'm unable to reproduce the issue: the performance is the same (I tested ce85a1f129e3, 83877018ef97, 204a43c452cc). These PC uses Fedora 24, GCC 6.2.1. CPUs: * laptop: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz * desktop: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz The speed-python runs Ubuntu 14.04, GCC 4.8.4-2ubuntu1~14.04. CPU: "Intel(R) Xeon(R) CPU X5680 @ 3.33GHz". call_method() benchmark is a microbenchmark which seems to depend a lot of very low level stuff like CPU L1 cache. Maybe the impact on the compiler is more important on speed-python which has an older CPU, than my more recent hardware. Maybe GCC 6.2 produces more efficient machine code than GCC 4.8. I expect that PGO would "fix" the call_method() performance issue, but PGO compilation fails on Ubuntu 14.04 with a compiler error :-p A solution would be to upgrade the OS of this server. Victor

3 years, 8 months

Re: [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

by Yury Selivanov

I'm curious why call_* benchmarks became slower on 3.x? Yury On 2016-11-04 8:28 AM, Victor Stinner wrote: > Hi, > > You may know that I'm working on benchmarks. I regenerated all > benchmark results of speed.python.org using performance 0.3.2 > (benchmark suite). I started to analyze results. > > All results are available online on the website: > > https://speed.python.org/ > > > To communicate on my work on benchmarks, I tweeted two pictures: > > "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python > 2.7 #python #benchmark": > https://twitter.com/VictorStinner/status/794289596683210760 > > "Python 3.6 is between 25% and 54% slower than Python 2.7 in the > following benchmarks": > https://twitter.com/VictorStinner/status/794305065708376069 > > > Many people were disappointed that Python 3.6 can be up to 54% slower > than Python 2.7. In fact, I know many reasons which explain that, but > it's hard to summarize them in 140 characters ;-) > > For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark > pycrypto_aes. This benchmark tests a pure Python implementation of the > crypto cipher AES. You may know that CPython is slow for CPU intensive > functions, especially on integer and floatting point numbers. > > "int" in Python 3 is now "long integers" by default, which is known to > be a little bit slower than "short int" of Python 2. On a more > realistic benchmark (see other benchmarks), the overhead of Python 3 > "long int" is negligible. > > AES is a typical example stressing integers. For me, it's a dummy > benchmark: it doesn't make sense to use Python for AES: modern CPUs > have an *hardware* implemention which is super fast. > > > Well, I didn't have time to analyze in depth individual benchmarks. If > you want to help me, here is the source code of benchmarks: > https://github.com/python/performance/blob/master/performance/benchmarks/ > > > Raw results of Python 3.6 compared to Python 2.7: > ------------------- > $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz > 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 > Slower (40): > - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower > - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower > - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower > - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower > - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower > - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower > - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower > - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower > - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower > - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower > - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower > - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower > - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower > - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower > - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower > - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower > - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower > - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower > - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower > - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower > - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower > - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower > - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower > - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower > - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower > - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower > - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower > - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower > - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower > - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower > - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower > - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower > - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower > - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower > - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower > - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower > - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower > - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower > - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: > 1.07x slower > - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower > > Faster (15): > - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster > - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster > - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster > - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster > - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster > - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster > - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster > - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster > - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster > - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster > - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster > - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster > - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster > - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster > - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster > > Benchmark hidden because not significant (8): 2to3, dulwich_log, > nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle > Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: > hg_startup, pyflate, spambayes > ------------------- > > Please ignore call_method, call_method_slots, call_method_unknown > benchmarks: it seems like I had an issue on the benchmark server. I > was unable to reproduce he 70% slowdown on my laptop. > > I attached the two compressed JSON files to this email if you want to > analyze them yourself. > > I hope that my work on benchmarks will motive some developers to look > closer at Python 3 performance to find interesting optimizations ;-) > > Victor > > > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

3 years, 8 months

Re: [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

by Victor Stinner

2016-11-04 20:18 GMT+01:00 Miquel Torres <tobami(a)gmail.com>: > Nice! For the record, I'll be giving a talk in PyCon Ireland about > Codespeed. Would you mind me citing those tweets and screenshots, to > highlight usage on speed.python.org? Sure. Keep me in touch in you publish your slides later. > You mentioned new more reliable vs old results. How close are we to have an > stable setup that gives us benchmarks numbers regularly? My plan for the short term is to analyze last (latest?) benchmarks hiccups and try to fix them. The fully automated script to run benchmarks is already written: https://github.com/python/performance/tree/master/scripts Then, the plan we decided with Zachary Ware is to run a script in a loop which compiles the default branch of CPython. Later, we may also do the same for 2.7 and 3.6 branches. And then add PyPy (and PyPy 3). Victor

3 years, 8 months

Re: [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

by Miquel Torres

Nice! For the record, I'll be giving a talk in PyCon Ireland about Codespeed. Would you mind me citing those tweets and screenshots, to highlight usage on speed.python.org? You mentioned new more reliable vs old results. How close are we to have an stable setup that gives us benchmarks numbers regularly? Cheers, Miquel El El vie, 4 nov 2016 a las 12:30, Victor Stinner <victor.stinner(a)gmail.com> escribió: > Hi, > > You may know that I'm working on benchmarks. I regenerated all > benchmark results of speed.python.org using performance 0.3.2 > (benchmark suite). I started to analyze results. > > All results are available online on the website: > > https://speed.python.org/ > > > To communicate on my work on benchmarks, I tweeted two pictures: > > "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python > 2.7 #python #benchmark": > https://twitter.com/VictorStinner/status/794289596683210760 > > "Python 3.6 is between 25% and 54% slower than Python 2.7 in the > following benchmarks": > https://twitter.com/VictorStinner/status/794305065708376069 > > > Many people were disappointed that Python 3.6 can be up to 54% slower > than Python 2.7. In fact, I know many reasons which explain that, but > it's hard to summarize them in 140 characters ;-) > > For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark > pycrypto_aes. This benchmark tests a pure Python implementation of the > crypto cipher AES. You may know that CPython is slow for CPU intensive > functions, especially on integer and floatting point numbers. > > "int" in Python 3 is now "long integers" by default, which is known to > be a little bit slower than "short int" of Python 2. On a more > realistic benchmark (see other benchmarks), the overhead of Python 3 > "long int" is negligible. > > AES is a typical example stressing integers. For me, it's a dummy > benchmark: it doesn't make sense to use Python for AES: modern CPUs > have an *hardware* implemention which is super fast. > > > Well, I didn't have time to analyze in depth individual benchmarks. If > you want to help me, here is the source code of benchmarks: > https://github.com/python/performance/blob/master/performance/benchmarks/ > > > Raw results of Python 3.6 compared to Python 2.7: > ------------------- > $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz > 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 > Slower (40): > - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower > - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x > slower > - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower > - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower > - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower > - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower > - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower > - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower > - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower > - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower > - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower > - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower > - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower > - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower > - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower > - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower > - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower > - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower > - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower > - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower > - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower > - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower > - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower > - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower > - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower > - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower > - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower > - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower > - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower > - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower > - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower > - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower > - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower > - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower > - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower > - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower > - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower > - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower > - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: > 1.07x slower > - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower > > Faster (15): > - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster > - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster > - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster > - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster > - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster > - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster > - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster > - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster > - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster > - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster > - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster > - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster > - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster > - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster > - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster > > Benchmark hidden because not significant (8): 2to3, dulwich_log, > nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle > Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: > hg_startup, pyflate, spambayes > ------------------- > > Please ignore call_method, call_method_slots, call_method_unknown > benchmarks: it seems like I had an issue on the benchmark server. I > was unable to reproduce he 70% slowdown on my laptop. > > I attached the two compressed JSON files to this email if you want to > analyze them yourself. > > I hope that my work on benchmarks will motive some developers to look > closer at Python 3 performance to find interesting optimizations ;-) > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

3 years, 8 months

Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

by Victor Stinner

Hi, You may know that I'm working on benchmarks. I regenerated all benchmark results of speed.python.org using performance 0.3.2 (benchmark suite). I started to analyze results. All results are available online on the website: https://speed.python.org/ To communicate on my work on benchmarks, I tweeted two pictures: "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python 2.7 #python #benchmark": https://twitter.com/VictorStinner/status/794289596683210760 "Python 3.6 is between 25% and 54% slower than Python 2.7 in the following benchmarks": https://twitter.com/VictorStinner/status/794305065708376069 Many people were disappointed that Python 3.6 can be up to 54% slower than Python 2.7. In fact, I know many reasons which explain that, but it's hard to summarize them in 140 characters ;-) For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark pycrypto_aes. This benchmark tests a pure Python implementation of the crypto cipher AES. You may know that CPython is slow for CPU intensive functions, especially on integer and floatting point numbers. "int" in Python 3 is now "long integers" by default, which is known to be a little bit slower than "short int" of Python 2. On a more realistic benchmark (see other benchmarks), the overhead of Python 3 "long int" is negligible. AES is a typical example stressing integers. For me, it's a dummy benchmark: it doesn't make sense to use Python for AES: modern CPUs have an *hardware* implemention which is super fast. Well, I didn't have time to analyze in depth individual benchmarks. If you want to help me, here is the source code of benchmarks: https://github.com/python/performance/blob/master/performance/benchmarks/ Raw results of Python 3.6 compared to Python 2.7: ------------------- $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 Slower (40): - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: 1.07x slower - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower Faster (15): - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster Benchmark hidden because not significant (8): 2to3, dulwich_log, nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: hg_startup, pyflate, spambayes ------------------- Please ignore call_method, call_method_slots, call_method_unknown benchmarks: it seems like I had an issue on the benchmark server. I was unable to reproduce he 70% slowdown on my laptop. I attached the two compressed JSON files to this email if you want to analyze them yourself. I hope that my work on benchmarks will motive some developers to look closer at Python 3 performance to find interesting optimizations ;-) Victor

3 years, 8 months

New benchmarks results on speed.python.org

by Victor Stinner

Hi, Good news, I regenerated all benchmark results of CPython using the latest versions of perf and perfomance and the results look much more reliable. Sadly, I didn't kept a screenshot of old benchmarks, so you should trust me, I cannot show you the old unstable timeline. -- I regenerated all benchmark results of speed.python.org using performance 0.3.2. I now have an (almost) fully automated script to run benchmarks (compile python, run benchmarks, etc.) using a list of Python revisions and/or branches. Only the last step, upload the JSON, is still manual, but it's nothing to automate this part ;-) https://github.com/python/performance/tree/master/scripts Python is compiled using LTO, but not PGO. The compilation with PGO fails with an internal GCC bug, speed-python uses Ubuntu 14.04, the GCC bug seems to be known (and fixed upstream...). Because of various bugs (including a bug in the Linux kernel ;-) NOHZ_FULL+intel_pstate), I didn't have time to analyze the impact of compilation options (-O2, -O3, LTO, PGO, etc.) on the stability of benchmark results. I isolated all CPUs of the NUMA node 1 (the CPU has two NUMA nodes): I added the following parameters to the the Linux kernel command line of the speed-python server: isolcpus=1,3,5,7,9,11,13,15,17,19,21,23 rcu_nocbs=1,3,5,7,9,11,13,15,17,19,21,23 Before running the benchmarks, I used the "python3 -m perf system tune" command (of the development version of perf) to tune the server. Results of the tuning: ------------------------- $ sudo python3 -m perf system System state ============ ASLR: Full randomization Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 CPU Frequency: 0,2,4,6,8,10,12,14,16,18,20,22=min=1600 MHz, max=3333 MHz; 1,3,5,7,9,11,13,15,17,19,21,23=min=max=3333 MHz Turbo Boost (MSR): CPU 0,2,4,6,8,10,12,14,16,18,20,22: enabled, CPU 1,3,5,7,9,11,13,15,17,19,21,23: disabled IRQ affinity: irqbalance service: inactive IRQ affinity: Default IRQ affinity: CPU 0,2,4,6,8,10,12,14,16,18,20,22 IRQ affinity: IRQ affinity: 0,2=0-23, 1,3-15,17,20,22-23,67-82=0,2,4,6,8,10,12,14,16,18,20,22 ------------------------- I don't well yet the hardware of the speed-python server. The CPU is a "Intel(R) Xeon(R) CPU X5680 @ 3.33GHz": * I only disabled Turbo Boost on the CPUs used to run benchmarks. Maybe I should disable Turbo Boost on all CPUs? On my computers using intel_pstate, Turbo Boost is disabled globally (for all CPUs) using an option of the intel_pstate driver. * I didn't tune the CPU scaling governor yet: all CPUs use "ondemand" * Maybe I should use a fixed CPU frequency on all CPUs and use the "userland" scaling governor? Results seem more stable, but it's still not perfect yet (see below). See [Timeline] (x) Display all in grid: https://speed.python.org/timeline/#/?exe=4&ben=grid&env=1&revs=50&equid=off… There are still some hiccups: (*) call_method: temporary peak of 29 ms for October 19, whereas all other revisions are around 17 ms: https://speed.python.org/timeline/#/?exe=4&ben=call_method&env=1&revs=50&eq… (*) python_startup increased from 21 ms to 27.5 ms between Sept 9 and Sept 15... The problem is that this one is not a temporary hiccup, but seems like a real performance regression: there are 4 points at 21 ms (Sept 4-Sept 9) and 7 points at 27.5 ms (Sept 15-Nov 3). But I was unable yet to reproduce the slowndown on my laptop. https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50… Victor

3 years, 8 months

Tune the system the run benchmarks

by Victor Stinner

Hi, Last months, I used various shell and Python scripts to tune a system to run benchmarks. There are many parameters to be set to "fully" configure a Linux: * Turbo Boost of Intel CPU * CPU scaling governor * CPU speed * Isolate CPU * Disable kernel RCU on isolated CPUs * etc. I added a new "sytem tune" command to the newly released perf 0.8.4. I implemented many operations: http://perf.readthedocs.io/en/latest/cli.html#system Right now, intel_pstate is better supported. I'm not sure about the CPU scaling governor when intel_pstate is not used, so this is one is not implemented yet. In my old Python script, I used the "userland" governor and a fixed speed for the CPUs. My old Python script also disabled interruptions (IRQ) on isolated CPUs. I will also implement that later. I don't know if setting the default CPU mask for IRQ is enough, or if it's better to set the CPU mask of all invididual IRQs. Example on the speed.python.org server: ----- haypo@speed-python$ sudo python3 -m perf system tune CPU Frequency: Minimum frequency of CPU 1 set to the maximum frequency CPU Frequency: Minimum frequency of CPU 3 set to the maximum frequency ... CPU Frequency: Minimum frequency of CPU 23 set to the maximum frequency Turbo Boost (MSR): Turbo Boost disabled on CPU 0: MSR 0x1a0 set to 0x4000850089 Turbo Boost (MSR): Turbo Boost disabled on CPU 1: MSR 0x1a0 set to 0x4000850089 ASLR: Full randomization Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 CPU Frequency: 0,2,4,6,8,10,12,14,16,18,20,22=min=1600 MHz, max=3333 MHz; 1,3,5,7,9,11,13,15,17,19,21,23=min=3333 MHz, max=3333 MHz Turbo Boost (MSR): CPU 0-23: disabled ----- "Reset" the config: ----- haypo@speed-python$ sudo python3 -m perf system reset CPU Frequency: Minimum frequency of CPU 1 reset to the minimum frequency CPU Frequency: Minimum frequency of CPU 3 reset to the minimum frequency ... CPU Frequency: Minimum frequency of CPU 23 reset to the minimum frequency Turbo Boost (MSR): Turbo Boost enabled on CPU 0: MSR 0x1a0 set to 0x850089 Turbo Boost (MSR): Turbo Boost enabled on CPU 1: MSR 0x1a0 set to 0x850089 ASLR: Full randomization Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23 CPU Frequency: 0-23=min=1600 MHz, max=3333 MHz Turbo Boost (MSR): CPU 0-23: enabled ----- Victor

3 years, 8 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed November 2016