2016-11-04 21:58 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>:
> 2016-11-04 20:21 GMT+01:00 Yury Selivanov <yselivanov.ml(a)gmail.com>:
>> I'm curious why call_* benchmarks became slower on 3.x?
>
> It's almost the same between 2.7 and default. For 3.6, it looks like
> an issue on the benchmark runner, not on Python itself: (...)
Aha, it seems to be a compiler performance issue. I proposed a patch
to fix the issue:
http://bugs.python.org/issue28618
Victor
I proposed a patch which fixes the issue:
http://bugs.python.org/issue28618
"Decorate hot functions using __attribute__((hot)) to optimize Python"
Victor
I found some interesting differences using the Linux perf tool.
# perf stat -e L1-icache-loads,L1-icache-load-misses ./python
performance/benchmarks/bm_call_method.py --inherit=PYTHONPATH -v
--worker -l1 -n 25 -w0
2016-11-04 23:35 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>:
> (1) ce85a1f129e3: 17 ms => 83877018ef97 (Oct 18): 25 ms
>
> https://hg.python.org/cpython/rev/83877018ef97
Comparison of metrics of rev ce85a1f129e3 (fast) => rev 83877018ef97 (slow):
L1-icache-load-misses: 0.06% => 8.41% of all L1-icache hits
Instructions per cycle: 2.38 => 1.41
stalled-cycles-frontend: 12.99% => 42.85% frontend cycles idle
stalled-cycles-backend: 2.28% => 21.36% backend cycles idle
So it confirms what I expected: call_method() is highly impacted by
the CPU L1 instruction cache.
I don't know exactly why the revision 83877018ef97 has an impact on
the CPU L1 cache.
Victor
Hi,
I noticed a temporary performance peak in the call_method:
https://speed.python.org/timeline/#/?exe=4&ben=call_method&env=1&revs=50&...
The difference is major: 17 ms => 29 ms, 70% slower!
I expected a temporary issue on the server used to run benchmarks,
but... I reproduced the result on the server.
Recently, the performance of call_method() changed in CPython default
from 17 ms to 28 ms (well, the exact value is variable: 25 ms, 28 ms,
29 ms, ...) and then back to 17 ms:
(1) ce85a1f129e3: 17 ms => 83877018ef97 (Oct 18): 25 ms
https://hg.python.org/cpython/rev/83877018ef97
(2) 3e073e7b4460: 28 ms => 204a43c452cc (Oct 22): 17 ms
https://hg.python.org/cpython/rev/204a43c452cc
None of these revisions modify code used in the call_method()
benchmark, so I guess that it's yet another compiler joke.
On my laptop and my desktop PC, I'm unable to reproduce the issue: the
performance is the same (I tested ce85a1f129e3, 83877018ef97,
204a43c452cc). These PC uses Fedora 24, GCC 6.2.1. CPUs:
* laptop: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
* desktop: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
The speed-python runs Ubuntu 14.04, GCC 4.8.4-2ubuntu1~14.04. CPU:
"Intel(R) Xeon(R) CPU X5680 @ 3.33GHz".
call_method() benchmark is a microbenchmark which seems to depend a
lot of very low level stuff like CPU L1 cache. Maybe the impact on the
compiler is more important on speed-python which has an older CPU,
than my more recent hardware. Maybe GCC 6.2 produces more efficient
machine code than GCC 4.8.
I expect that PGO would "fix" the call_method() performance issue, but
PGO compilation fails on Ubuntu 14.04 with a compiler error :-p A
solution would be to upgrade the OS of this server.
Victor
I'm curious why call_* benchmarks became slower on 3.x?
Yury
On 2016-11-04 8:28 AM, Victor Stinner wrote:
> Hi,
>
> You may know that I'm working on benchmarks. I regenerated all
> benchmark results of speed.python.org using performance 0.3.2
> (benchmark suite). I started to analyze results.
>
> All results are available online on the website:
>
> https://speed.python.org/
>
>
> To communicate on my work on benchmarks, I tweeted two pictures:
>
> "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
> 2.7 #python #benchmark":
> https://twitter.com/VictorStinner/status/794289596683210760
>
> "Python 3.6 is between 25% and 54% slower than Python 2.7 in the
> following benchmarks":
> https://twitter.com/VictorStinner/status/794305065708376069
>
>
> Many people were disappointed that Python 3.6 can be up to 54% slower
> than Python 2.7. In fact, I know many reasons which explain that, but
> it's hard to summarize them in 140 characters ;-)
>
> For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
> pycrypto_aes. This benchmark tests a pure Python implementation of the
> crypto cipher AES. You may know that CPython is slow for CPU intensive
> functions, especially on integer and floatting point numbers.
>
> "int" in Python 3 is now "long integers" by default, which is known to
> be a little bit slower than "short int" of Python 2. On a more
> realistic benchmark (see other benchmarks), the overhead of Python 3
> "long int" is negligible.
>
> AES is a typical example stressing integers. For me, it's a dummy
> benchmark: it doesn't make sense to use Python for AES: modern CPUs
> have an *hardware* implemention which is super fast.
>
>
> Well, I didn't have time to analyze in depth individual benchmarks. If
> you want to help me, here is the source code of benchmarks:
> https://github.com/python/performance/blob/master/performance/benchmarks/
>
>
> Raw results of Python 3.6 compared to Python 2.7:
> -------------------
> $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
> 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
> Slower (40):
> - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
> - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower
> - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
> - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
> - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
> - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
> - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
> - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
> - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
> - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
> - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
> - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
> - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
> - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
> - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
> - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
> - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
> - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
> - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
> - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
> - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
> - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
> - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
> - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
> - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
> - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
> - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
> - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
> - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
> - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
> - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
> - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
> - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
> - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
> - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
> - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
> - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
> - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
> - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
> 1.07x slower
> - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower
>
> Faster (15):
> - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
> - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
> - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
> - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
> - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster
> - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster
> - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster
> - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster
> - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster
> - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster
> - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster
> - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster
> - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster
> - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster
> - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster
>
> Benchmark hidden because not significant (8): 2to3, dulwich_log,
> nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle
> Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json:
> hg_startup, pyflate, spambayes
> -------------------
>
> Please ignore call_method, call_method_slots, call_method_unknown
> benchmarks: it seems like I had an issue on the benchmark server. I
> was unable to reproduce he 70% slowdown on my laptop.
>
> I attached the two compressed JSON files to this email if you want to
> analyze them yourself.
>
> I hope that my work on benchmarks will motive some developers to look
> closer at Python 3 performance to find interesting optimizations ;-)
>
> Victor
>
>
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
2016-11-04 20:18 GMT+01:00 Miquel Torres <tobami(a)gmail.com>:
> Nice! For the record, I'll be giving a talk in PyCon Ireland about
> Codespeed. Would you mind me citing those tweets and screenshots, to
> highlight usage on speed.python.org?
Sure. Keep me in touch in you publish your slides later.
> You mentioned new more reliable vs old results. How close are we to have an
> stable setup that gives us benchmarks numbers regularly?
My plan for the short term is to analyze last (latest?) benchmarks
hiccups and try to fix them.
The fully automated script to run benchmarks is already written:
https://github.com/python/performance/tree/master/scripts
Then, the plan we decided with Zachary Ware is to run a script in a
loop which compiles the default branch of CPython. Later, we may also
do the same for 2.7 and 3.6 branches. And then add PyPy (and PyPy 3).
Victor
Nice! For the record, I'll be giving a talk in PyCon Ireland about
Codespeed. Would you mind me citing those tweets and screenshots, to
highlight usage on speed.python.org?
You mentioned new more reliable vs old results. How close are we to have an
stable setup that gives us benchmarks numbers regularly?
Cheers,
Miquel
El El vie, 4 nov 2016 a las 12:30, Victor Stinner <victor.stinner(a)gmail.com>
escribió:
> Hi,
>
> You may know that I'm working on benchmarks. I regenerated all
> benchmark results of speed.python.org using performance 0.3.2
> (benchmark suite). I started to analyze results.
>
> All results are available online on the website:
>
> https://speed.python.org/
>
>
> To communicate on my work on benchmarks, I tweeted two pictures:
>
> "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
> 2.7 #python #benchmark":
> https://twitter.com/VictorStinner/status/794289596683210760
>
> "Python 3.6 is between 25% and 54% slower than Python 2.7 in the
> following benchmarks":
> https://twitter.com/VictorStinner/status/794305065708376069
>
>
> Many people were disappointed that Python 3.6 can be up to 54% slower
> than Python 2.7. In fact, I know many reasons which explain that, but
> it's hard to summarize them in 140 characters ;-)
>
> For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
> pycrypto_aes. This benchmark tests a pure Python implementation of the
> crypto cipher AES. You may know that CPython is slow for CPU intensive
> functions, especially on integer and floatting point numbers.
>
> "int" in Python 3 is now "long integers" by default, which is known to
> be a little bit slower than "short int" of Python 2. On a more
> realistic benchmark (see other benchmarks), the overhead of Python 3
> "long int" is negligible.
>
> AES is a typical example stressing integers. For me, it's a dummy
> benchmark: it doesn't make sense to use Python for AES: modern CPUs
> have an *hardware* implemention which is super fast.
>
>
> Well, I didn't have time to analyze in depth individual benchmarks. If
> you want to help me, here is the source code of benchmarks:
> https://github.com/python/performance/blob/master/performance/benchmarks/
>
>
> Raw results of Python 3.6 compared to Python 2.7:
> -------------------
> $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
> 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
> Slower (40):
> - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
> - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x
> slower
> - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
> - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
> - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
> - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
> - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
> - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
> - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
> - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
> - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
> - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
> - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
> - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
> - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
> - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
> - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
> - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
> - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
> - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
> - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
> - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
> - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
> - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
> - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
> - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
> - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
> - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
> - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
> - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
> - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
> - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
> - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
> - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
> - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
> - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
> - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
> - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
> - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
> 1.07x slower
> - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower
>
> Faster (15):
> - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
> - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
> - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
> - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
> - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster
> - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster
> - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster
> - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster
> - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster
> - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster
> - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster
> - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster
> - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster
> - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster
> - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster
>
> Benchmark hidden because not significant (8): 2to3, dulwich_log,
> nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle
> Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json:
> hg_startup, pyflate, spambayes
> -------------------
>
> Please ignore call_method, call_method_slots, call_method_unknown
> benchmarks: it seems like I had an issue on the benchmark server. I
> was unable to reproduce he 70% slowdown on my laptop.
>
> I attached the two compressed JSON files to this email if you want to
> analyze them yourself.
>
> I hope that my work on benchmarks will motive some developers to look
> closer at Python 3 performance to find interesting optimizations ;-)
>
> Victor
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
>
Hi,
You may know that I'm working on benchmarks. I regenerated all
benchmark results of speed.python.org using performance 0.3.2
(benchmark suite). I started to analyze results.
All results are available online on the website:
https://speed.python.org/
To communicate on my work on benchmarks, I tweeted two pictures:
"sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
2.7 #python #benchmark":
https://twitter.com/VictorStinner/status/794289596683210760
"Python 3.6 is between 25% and 54% slower than Python 2.7 in the
following benchmarks":
https://twitter.com/VictorStinner/status/794305065708376069
Many people were disappointed that Python 3.6 can be up to 54% slower
than Python 2.7. In fact, I know many reasons which explain that, but
it's hard to summarize them in 140 characters ;-)
For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
pycrypto_aes. This benchmark tests a pure Python implementation of the
crypto cipher AES. You may know that CPython is slow for CPU intensive
functions, especially on integer and floatting point numbers.
"int" in Python 3 is now "long integers" by default, which is known to
be a little bit slower than "short int" of Python 2. On a more
realistic benchmark (see other benchmarks), the overhead of Python 3
"long int" is negligible.
AES is a typical example stressing integers. For me, it's a dummy
benchmark: it doesn't make sense to use Python for AES: modern CPUs
have an *hardware* implemention which is super fast.
Well, I didn't have time to analyze in depth individual benchmarks. If
you want to help me, here is the source code of benchmarks:
https://github.com/python/performance/blob/master/performance/benchmarks/
Raw results of Python 3.6 compared to Python 2.7:
-------------------
$ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
Slower (40):
- python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
- python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower
- unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
- call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
- call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
- call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
- crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
- xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
- logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
- logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
- pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
- spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
- logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
- chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
- go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
- xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
- sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
- xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
- django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
- fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
- hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
- chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
- regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
- json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
- nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
- genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
- raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
- scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
- scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
- deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
- sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
- call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
- scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
- meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
- pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
- richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
- genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
- float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
- scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
1.07x slower
- xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower
Faster (15):
- telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
- unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
- pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
- json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
- pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster
- sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster
- sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster
- regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster
- sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster
- regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster
- mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster
- html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster
- sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster
- pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster
- scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster
Benchmark hidden because not significant (8): 2to3, dulwich_log,
nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle
Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json:
hg_startup, pyflate, spambayes
-------------------
Please ignore call_method, call_method_slots, call_method_unknown
benchmarks: it seems like I had an issue on the benchmark server. I
was unable to reproduce he 70% slowdown on my laptop.
I attached the two compressed JSON files to this email if you want to
analyze them yourself.
I hope that my work on benchmarks will motive some developers to look
closer at Python 3 performance to find interesting optimizations ;-)
Victor
Hi,
Good news, I regenerated all benchmark results of CPython using the
latest versions of perf and perfomance and the results look much more
reliable. Sadly, I didn't kept a screenshot of old benchmarks, so you
should trust me, I cannot show you the old unstable timeline.
--
I regenerated all benchmark results of speed.python.org using
performance 0.3.2. I now have an (almost) fully automated script to
run benchmarks (compile python, run benchmarks, etc.) using a list of
Python revisions and/or branches. Only the last step, upload the JSON,
is still manual, but it's nothing to automate this part ;-)
https://github.com/python/performance/tree/master/scripts
Python is compiled using LTO, but not PGO. The compilation with PGO
fails with an internal GCC bug, speed-python uses Ubuntu 14.04, the
GCC bug seems to be known (and fixed upstream...).
Because of various bugs (including a bug in the Linux kernel ;-)
NOHZ_FULL+intel_pstate), I didn't have time to analyze the impact of
compilation options (-O2, -O3, LTO, PGO, etc.) on the stability of
benchmark results.
I isolated all CPUs of the NUMA node 1 (the CPU has two NUMA nodes): I
added the following parameters to the the Linux kernel command line of
the speed-python server:
isolcpus=1,3,5,7,9,11,13,15,17,19,21,23
rcu_nocbs=1,3,5,7,9,11,13,15,17,19,21,23
Before running the benchmarks, I used the "python3 -m perf system
tune" command (of the development version of perf) to tune the server.
Results of the tuning:
-------------------------
$ sudo python3 -m perf system
System state
============
ASLR: Full randomization
Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
CPU Frequency: 0,2,4,6,8,10,12,14,16,18,20,22=min=1600 MHz, max=3333
MHz; 1,3,5,7,9,11,13,15,17,19,21,23=min=max=3333 MHz
Turbo Boost (MSR): CPU 0,2,4,6,8,10,12,14,16,18,20,22: enabled, CPU
1,3,5,7,9,11,13,15,17,19,21,23: disabled
IRQ affinity: irqbalance service: inactive
IRQ affinity: Default IRQ affinity: CPU 0,2,4,6,8,10,12,14,16,18,20,22
IRQ affinity: IRQ affinity: 0,2=0-23,
1,3-15,17,20,22-23,67-82=0,2,4,6,8,10,12,14,16,18,20,22
-------------------------
I don't well yet the hardware of the speed-python server. The CPU is a
"Intel(R) Xeon(R) CPU X5680 @ 3.33GHz":
* I only disabled Turbo Boost on the CPUs used to run benchmarks.
Maybe I should disable Turbo Boost on all CPUs? On my computers using
intel_pstate, Turbo Boost is disabled globally (for all CPUs) using an
option of the intel_pstate driver.
* I didn't tune the CPU scaling governor yet: all CPUs use "ondemand"
* Maybe I should use a fixed CPU frequency on all CPUs and use the
"userland" scaling governor?
Results seem more stable, but it's still not perfect yet (see below).
See [Timeline] (x) Display all in grid:
https://speed.python.org/timeline/#/?exe=4&ben=grid&env=1&revs=50&equid=o...
There are still some hiccups:
(*) call_method: temporary peak of 29 ms for October 19, whereas all
other revisions are around 17 ms:
https://speed.python.org/timeline/#/?exe=4&ben=call_method&env=1&revs=50&...
(*) python_startup increased from 21 ms to 27.5 ms between Sept 9 and
Sept 15... The problem is that this one is not a temporary hiccup, but
seems like a real performance regression: there are 4 points at 21 ms
(Sept 4-Sept 9) and 7 points at 27.5 ms (Sept 15-Nov 3). But I was
unable yet to reproduce the slowndown on my laptop.
https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=...
Victor
Hi,
Last months, I used various shell and Python scripts to tune a system
to run benchmarks.
There are many parameters to be set to "fully" configure a Linux:
* Turbo Boost of Intel CPU
* CPU scaling governor
* CPU speed
* Isolate CPU
* Disable kernel RCU on isolated CPUs
* etc.
I added a new "sytem tune" command to the newly released perf 0.8.4. I
implemented many operations:
http://perf.readthedocs.io/en/latest/cli.html#system
Right now, intel_pstate is better supported.
I'm not sure about the CPU scaling governor when intel_pstate is not
used, so this is one is not implemented yet. In my old Python script,
I used the "userland" governor and a fixed speed for the CPUs.
My old Python script also disabled interruptions (IRQ) on isolated
CPUs. I will also implement that later. I don't know if setting the
default CPU mask for IRQ is enough, or if it's better to set the CPU
mask of all invididual IRQs.
Example on the speed.python.org server:
-----
haypo@speed-python$ sudo python3 -m perf system tune
CPU Frequency: Minimum frequency of CPU 1 set to the maximum frequency
CPU Frequency: Minimum frequency of CPU 3 set to the maximum frequency
...
CPU Frequency: Minimum frequency of CPU 23 set to the maximum frequency
Turbo Boost (MSR): Turbo Boost disabled on CPU 0: MSR 0x1a0 set to 0x4000850089
Turbo Boost (MSR): Turbo Boost disabled on CPU 1: MSR 0x1a0 set to 0x4000850089
ASLR: Full randomization
Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
CPU Frequency: 0,2,4,6,8,10,12,14,16,18,20,22=min=1600 MHz, max=3333
MHz; 1,3,5,7,9,11,13,15,17,19,21,23=min=3333 MHz, max=3333 MHz
Turbo Boost (MSR): CPU 0-23: disabled
-----
"Reset" the config:
-----
haypo@speed-python$ sudo python3 -m perf system reset
CPU Frequency: Minimum frequency of CPU 1 reset to the minimum frequency
CPU Frequency: Minimum frequency of CPU 3 reset to the minimum frequency
...
CPU Frequency: Minimum frequency of CPU 23 reset to the minimum frequency
Turbo Boost (MSR): Turbo Boost enabled on CPU 0: MSR 0x1a0 set to 0x850089
Turbo Boost (MSR): Turbo Boost enabled on CPU 1: MSR 0x1a0 set to 0x850089
ASLR: Full randomization
Linux scheduler: Isolated CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
Linux scheduler: RCU disabled on CPUs (12/24): 1,3,5,7,9,11,13,15,17,19,21,23
CPU Frequency: 0-23=min=1600 MHz, max=3333 MHz
Turbo Boost (MSR): CPU 0-23: enabled
-----
Victor
