The nice thing with benchmarks is that you can always take a subset of numbers to prove something :-) Another subset where Python 3.6 is faster: "Python 3.6 is between 12% (1.14x) and 97% (32x) FASTER than Python 2.7 in the following benchmarks:" https://twitter.com/VictorStinner/status/794525289623719937 Victor 2016-11-04 13:53 GMT+01:00 Victor Stinner :

(Re-post without the two attached files of 100 KB each.)

Hi,

You may know that I'm working on benchmarks. I regenerated all benchmark results of speed.python.org using performance 0.3.2 (benchmark suite). I started to analyze results.

All results are available online on the website:

https://speed.python.org/

To communicate on my work on benchmarks, I tweeted two pictures:

"sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python 2.7 #python #benchmark": https://twitter.com/VictorStinner/status/794289596683210760

"Python 3.6 is between 25% and 54% slower than Python 2.7 in the following benchmarks": https://twitter.com/VictorStinner/status/794305065708376069

Many people were disappointed that Python 3.6 can be up to 54% slower than Python 2.7. In fact, I know many reasons which explain that, but it's hard to summarize them in 140 characters ;-)

For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark pycrypto_aes. This benchmark tests a pure Python implementation of the crypto cipher AES. You may know that CPython is slow for CPU intensive functions, especially on integer and floatting point numbers.

"int" in Python 3 is now "long integers" by default, which is known to be a little bit slower than "short int" of Python 2. On a more realistic benchmark (see other benchmarks), the overhead of Python 3 "long int" is negligible.

AES is a typical example stressing integers. For me, it's a dummy benchmark: it doesn't make sense to use Python for AES: modern CPUs have an *hardware* implemention which is super fast.

Well, I didn't have time to analyze in depth individual benchmarks. If you want to help me, here is the source code of benchmarks: https://github.com/python/performance/blob/master/performance/benchmarks/

Raw results of Python 3.6 compared to Python 2.7: ------------------- $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 Slower (40): - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: 1.07x slower - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower

Faster (15): - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster

Benchmark hidden because not significant (8): 2to3, dulwich_log, nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: hg_startup, pyflate, spambayes -------------------

Please ignore call_method, call_method_slots, call_method_unknown benchmarks: it seems like I had an issue on the benchmark server. I was unable to reproduce he 70% slowdown on my laptop.

JSON files if you want to analyze them yourself:

http://www.haypocalc.com/tmp/2016-11-03_15-36-2.7-91f024fc9b3a.json.gz http://www.haypocalc.com/tmp/2016-11-03_15-38-3.6-c4319c0d0131.json.gz

I hope that my work on benchmarks will motive some developers to look closer at Python 3 performance to find interesting optimizations ;-)

Victor

On 05.11.2016 10:56, Antoine Pitrou wrote:

Hi Victor,

On Fri, 4 Nov 2016 13:53:10 +0100 Victor Stinner wrote:

...

Raw results of Python 3.6 compared to Python 2.7:

That's interesting, but I would be personally more interested in a performance comparison of 3.5 and 3.6, to know if anything interesting (or worrying :-)) has happened there.

You can get this as well from https://speed.python.org/comparison/ and https://speed.python.org/timeline and looking at this, I think there is something worrying indeed: Startup time has increased by ~ 30 % between 3.5 and 3.6 again. More specifically, all this increase happened between Sep 09 and Sep 15. I have no clue why that is, but it is definitely the biggest effect far and wide.

The performance differences between 2.7 and 3.x are quite well-known by now, and none of them are really dramatic except for the increase in startup time.

Regards

Antoine.

2016-11-07 20:20 GMT+01:00 Guido van Rossum :

Re: https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on

That's suspiciously close to the core sprint. Since the -S time stayed roughly the same I suspect that either a new module was added to the startup sequence or one of the (too many) modules already involved grew a lot. My money is on a new module. Using `python -v -c pass` on a Python built from the Sept. 9 tree and one built from Sept. 15, it shouldn't be too hard to figure out which new module(s).

I identified the regression, I created: http://bugs.python.org/issue28637 It's a regression caused by: https://hg.python.org/cpython/rev/223731925d06/ http://bugs.python.org/issue28082 The change added "import enum" in Lib/re.py. I propose to revert this change for now, and discuss later a solution which doesn't impact performances. It's nice to see that my work of performance was useful to catch a performance regression. Victor