[Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

Fri Nov 4 15:21:31 EDT 2016

I'm curious why call_* benchmarks became slower on 3.x?

Yury

On 2016-11-04 8:28 AM, Victor Stinner wrote:
> Hi,
>
> You may know that I'm working on benchmarks. I regenerated all
> benchmark results of speed.python.org using performance 0.3.2
> (benchmark suite). I started to analyze results.
>
> All results are available online on the website:
>
>     https://speed.python.org/
>
>
> To communicate on my work on benchmarks, I tweeted two pictures:
>
> "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
> 2.7 #python #benchmark":
> https://twitter.com/VictorStinner/status/794289596683210760
>
> "Python 3.6 is between 25% and 54% slower than Python 2.7 in the
> following benchmarks":
> https://twitter.com/VictorStinner/status/794305065708376069
>
>
> Many people were disappointed that Python 3.6 can be up to 54% slower
> than Python 2.7. In fact, I know many reasons which explain that, but
> it's hard to summarize them in 140 characters ;-)
>
> For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
> pycrypto_aes. This benchmark tests a pure Python implementation of the
> crypto cipher AES. You may know that CPython is slow for CPU intensive
> functions, especially on integer and floatting point numbers.
>
> "int" in Python 3 is now "long integers" by default, which is known to
> be a little bit slower than "short int" of Python 2. On a more
> realistic benchmark (see other benchmarks), the overhead of Python 3
> "long int" is negligible.
>
> AES is a typical example stressing integers. For me, it's a dummy
> benchmark: it doesn't make sense to use Python for AES: modern CPUs
> have an *hardware* implemention which is super fast.
>
>
> Well, I didn't have time to analyze in depth individual benchmarks. If
> you want to help me, here is the source code of benchmarks:
> https://github.com/python/performance/blob/master/performance/benchmarks/
>
>
> Raw results of Python 3.6 compared to Python 2.7:
> -------------------
> $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
> 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
> Slower (40):
> - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
> - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower
> - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
> - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
> - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
> - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
> - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
> - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
> - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
> - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
> - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
> - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
> - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
> - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
> - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
> - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
> - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
> - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
> - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
> - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
> - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
> - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
> - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
> - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
> - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
> - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
> - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
> - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
> - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
> - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
> - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
> - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
> - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
> - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
> - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
> - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
> - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
> - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
> - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
> 1.07x slower
> - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower
>
> Faster (15):
> - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
> - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
> - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
> - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
> - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster
> - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster
> - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster
> - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster
> - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster
> - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster
> - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster
> - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster
> - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster
> - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster
> - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster
>
> Benchmark hidden because not significant (8): 2to3, dulwich_log,
> nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle
> Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json:
> hg_startup, pyflate, spambayes
> -------------------
>
> Please ignore call_method, call_method_slots, call_method_unknown
> benchmarks: it seems like I had an issue on the benchmark server. I
> was unable to reproduce he 70% slowdown on my laptop.
>
> I attached the two compressed JSON files to this email if you want to
> analyze them yourself.
>
> I hope that my work on benchmarks will motive some developers to look
> closer at Python 3 performance to find interesting optimizations ;-)
>
> Victor
>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed