Speed February 2017

speed@python.org

5 participants
8 discussions

Re: [Speed] CPython 3.7 is now faster than CPython 2.7 on most benchmarks

by Ludovic Gasc

Awesome ;-) -- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc 2017-02-19 7:45 GMT+01:00 INADA Naoki <songofacandy(a)gmail.com>: > On Wed, Feb 8, 2017 at 12:59 AM, Victor Stinner > <victor.stinner(a)gmail.com> wrote: > > Hi, > > > > FYI CPython 3.7 is now faster than CPython 2.7 on most benchmarks. It > > would be interesting to investigate why 12 benchmarks are still slower > > to check if we missed something obvious. > > > > Comparison, ignoring differences smaller than 10%: > > > > haypo@speed-python$ python3 -m perf compare_to > > 2017-01-03_11-17-2.7-5988caffbff9.json > > 2017-02-06_07-15-default-e06af4027546.json -G --min-speed=10 > > Slower (12): > > - python_startup_no_site: 3.11 ms +- 0.02 ms -> 8.33 ms +- 0.04 ms: > > 2.68x slower (+168%) > > - python_startup: 6.42 ms +- 0.04 ms -> 14.1 ms +- 0.1 ms: 2.20x slower > (+120%) > > > The top two slower benchmark is startup. There are some unavoidable > reasons (io, importlib, > and more rich types based on abc). > But I sent two pull requests reduce cost of site.py. > > * http://bugs.python.org/issue29585 > * http://bugs.python.org/issue29592 > > If they are merged, (python_startup - python_startup_no_site) will be > very close to Python 2.7. > > For more optimization, I think we need to write some function in C for > importlib. > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

3 years, 9 months

Re: [Speed] Benchmark tooling broken by the migration to Git

by Miquel Torres

Hi Victor, Codespeed supports both git and Github. When you configure a project in the admin, you should see a dropdown for the type of revision control. Github is probably best so there is no need for locally cloning the git repository. Anything else that might be a blocker? Miquel On Mon, Feb 20, 2017 at 1:58 PM Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > The website speed.python.org is made of different tools which were > written for Mercurial, but CPython moved to Git. These tools should be > updated. > > Tools: > > * The Django application "codespeed" > * scripts/bench_cpython.py and scripts/bench_revisions.py of > https://github.com/python/performance/ > * my private tool to benchmark a patch -- short bash script, sorry I > should put it in python/performance/scripts/ as well > * maybe others > > Moreover, all benchmark results was generated using Mercurial commit > identifiers. We must remove all of them and recompute all results > using Git commit identifiers. It's not a big deal. It's just that > someone has to do it. > > I spent almost a year to fix all these minor things to get a working > speed.python.org, so I'm not ready to do it right now. I may do the > required changes later ;-) > > If codespeed doesn't support Git, I suggest to drop the list of > changes between two commits on the https://speed.python.org/changes/ > tab. I'm now trying to only compute benchmarks manually something like > once per week, because the UI doesn't handle well too many points, and > because performance don't change every day ;-) The changes/ tab is > very slow when there is a long list of commits, whereas it's very easy > to get this list using Git directly, GitHub, or whatever. I don't > think that it's very useful on the website. > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

3 years, 9 months

Benchmark tooling broken by the migration to Git

by Victor Stinner

Hi, The website speed.python.org is made of different tools which were written for Mercurial, but CPython moved to Git. These tools should be updated. Tools: * The Django application "codespeed" * scripts/bench_cpython.py and scripts/bench_revisions.py of https://github.com/python/performance/ * my private tool to benchmark a patch -- short bash script, sorry I should put it in python/performance/scripts/ as well * maybe others Moreover, all benchmark results was generated using Mercurial commit identifiers. We must remove all of them and recompute all results using Git commit identifiers. It's not a big deal. It's just that someone has to do it. I spent almost a year to fix all these minor things to get a working speed.python.org, so I'm not ready to do it right now. I may do the required changes later ;-) If codespeed doesn't support Git, I suggest to drop the list of changes between two commits on the https://speed.python.org/changes/ tab. I'm now trying to only compute benchmarks manually something like once per week, because the UI doesn't handle well too many points, and because performance don't change every day ;-) The changes/ tab is very slow when there is a long list of commits, whereas it's very easy to get this list using Git directly, GitHub, or whatever. I don't think that it's very useful on the website. Victor

3 years, 9 months

Re: [Speed] CPython 3.7 is now faster than CPython 2.7 on most benchmarks

by INADA Naoki

On Wed, Feb 8, 2017 at 12:59 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > FYI CPython 3.7 is now faster than CPython 2.7 on most benchmarks. It > would be interesting to investigate why 12 benchmarks are still slower > to check if we missed something obvious. > > Comparison, ignoring differences smaller than 10%: > > haypo@speed-python$ python3 -m perf compare_to > 2017-01-03_11-17-2.7-5988caffbff9.json > 2017-02-06_07-15-default-e06af4027546.json -G --min-speed=10 > Slower (12): > - python_startup_no_site: 3.11 ms +- 0.02 ms -> 8.33 ms +- 0.04 ms: > 2.68x slower (+168%) > - python_startup: 6.42 ms +- 0.04 ms -> 14.1 ms +- 0.1 ms: 2.20x slower (+120%) The top two slower benchmark is startup. There are some unavoidable reasons (io, importlib, and more rich types based on abc). But I sent two pull requests reduce cost of site.py. * http://bugs.python.org/issue29585 * http://bugs.python.org/issue29592 If they are merged, (python_startup - python_startup_no_site) will be very close to Python 2.7. For more optimization, I think we need to write some function in C for importlib.

3 years, 9 months

Re: [Speed] CPython 3.7 is now faster than CPython 2.7 on most benchmarks

by Ludovic Gasc

Indeed, it's pretty cool: one of latest argument to not migrate to Python 3 is almost ready to fall. Maybe in one moment (2018 ?), it might be interesting that to have a leaflet with the list of bias about Python 3 migration (No library support, slower...) to help to change minds about that ? Or maybe it already exist ? -- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc 2017-02-07 16:59 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>: > Hi, > > FYI CPython 3.7 is now faster than CPython 2.7 on most benchmarks. It > would be interesting to investigate why 12 benchmarks are still slower > to check if we missed something obvious. > > Comparison, ignoring differences smaller than 10%: > > haypo@speed-python$ python3 -m perf compare_to > 2017-01-03_11-17-2.7-5988caffbff9.json > 2017-02-06_07-15-default-e06af4027546.json -G --min-speed=10 > Slower (12): > - python_startup_no_site: 3.11 ms +- 0.02 ms -> 8.33 ms +- 0.04 ms: > 2.68x slower (+168%) > - python_startup: 6.42 ms +- 0.04 ms -> 14.1 ms +- 0.1 ms: 2.20x slower > (+120%) > - unpickle_pure_python: 384 us +- 8 us -> 670 us +- 11 us: 1.74x slower > (+74%) > - spectral_norm: 194 ms +- 1 ms -> 259 ms +- 4 ms: 1.33x slower (+33%) > - sqlite_synth: 6.70 us +- 0.11 us -> 8.49 us +- 0.16 us: 1.27x slower > (+27%) > - crypto_pyaes: 158 ms +- 1 ms -> 199 ms +- 2 ms: 1.26x slower (+26%) > - xml_etree_parse: 193 ms +- 4 ms -> 242 ms +- 5 ms: 1.25x slower (+25%) > - logging_format: 28.9 us +- 0.2 us -> 33.9 us +- 0.5 us: 1.17x slower > (+17%) > - xml_etree_iterparse: 154 ms +- 2 ms -> 179 ms +- 4 ms: 1.16x slower > (+16%) > - pickle_pure_python: 847 us +- 14 us -> 974 us +- 14 us: 1.15x slower > (+15%) > - logging_simple: 24.4 us +- 0.3 us -> 27.9 us +- 0.4 us: 1.14x slower > (+14%) > - go: 439 ms +- 3 ms -> 493 ms +- 7 ms: 1.12x slower (+12%) > > Faster (23): > - telco: 634 ms +- 5 ms -> 14.7 ms +- 0.3 ms: 43.23x faster (-98%) > - unpickle_list: 15.7 us +- 0.6 us -> 6.36 us +- 0.11 us: 2.47x faster > (-59%) > - pickle_list: 13.7 us +- 0.2 us -> 7.53 us +- 0.41 us: 1.81x faster (-45%) > - pickle: 38.8 us +- 1.5 us -> 22.1 us +- 0.3 us: 1.76x faster (-43%) > - mako: 56.0 ms +- 0.9 ms -> 32.0 ms +- 0.3 ms: 1.75x faster (-43%) > - sympy_sum: 335 ms +- 11 ms -> 192 ms +- 6 ms: 1.74x faster (-43%) > - sympy_expand: 1.65 sec +- 0.02 sec -> 949 ms +- 12 ms: 1.74x faster > (-43%) > - sympy_str: 678 ms +- 5 ms -> 425 ms +- 5 ms: 1.59x faster (-37%) > - json_loads: 89.8 us +- 1.0 us -> 56.6 us +- 0.8 us: 1.59x faster (-37%) > - scimark_lu: 483 ms +- 10 ms -> 352 ms +- 11 ms: 1.37x faster (-27%) > - xml_etree_generate: 282 ms +- 3 ms -> 208 ms +- 2 ms: 1.36x faster (-26%) > - call_method_slots: 15.3 ms +- 0.4 ms -> 11.2 ms +- 0.3 ms: 1.36x faster > (-26%) > - html5lib: 269 ms +- 7 ms -> 201 ms +- 5 ms: 1.34x faster (-25%) > - call_method_unknown: 17.5 ms +- 0.3 ms -> 13.1 ms +- 0.1 ms: 1.34x > faster (-25%) > - call_method: 15.1 ms +- 0.3 ms -> 11.7 ms +- 0.2 ms: 1.29x faster (-23%) > - sympy_integrate: 51.7 ms +- 0.6 ms -> 40.8 ms +- 0.3 ms: 1.27x faster > (-21%) > - unpack_sequence: 140 ns +- 3 ns -> 111 ns +- 1 ns: 1.26x faster (-21%) > - regex_v8: 53.7 ms +- 0.9 ms -> 43.7 ms +- 0.6 ms: 1.23x faster (-19%) > - 2to3: 763 ms +- 4 ms -> 622 ms +- 4 ms: 1.23x faster (-19%) > - pickle_dict: 69.4 us +- 1.1 us -> 56.6 us +- 4.3 us: 1.23x faster (-18%) > - call_simple: 12.7 ms +- 0.3 ms -> 10.3 ms +- 0.4 ms: 1.23x faster (-18%) > - richards: 162 ms +- 2 ms -> 145 ms +- 5 ms: 1.12x faster (-11%) > - float: 266 ms +- 2 ms -> 238 ms +- 3 ms: 1.12x faster (-10%) > > Benchmark hidden because not significant (29): (...) > Ignored benchmarks (3) of 2017-01-03_11-17-2.7-5988caffbff9.json: > hg_startup, pyflate, spambayes > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

3 years, 9 months

Re: [Speed] CPython 3.7 is now faster than CPython 2.7 on most benchmarks

by Yury Selivanov

Nice! spectral_norm can be further sped up by optimizing 1+1 in ceval (no kidding) which is optimized in py2 but not py3. Since we decided no to port this particular optimization, it will always be slightly slower. Yury On 2017-02-07 10:59 AM, Victor Stinner wrote: > Hi, > > FYI CPython 3.7 is now faster than CPython 2.7 on most benchmarks. It > would be interesting to investigate why 12 benchmarks are still slower > to check if we missed something obvious. > > Comparison, ignoring differences smaller than 10%: > > haypo@speed-python$ python3 -m perf compare_to > 2017-01-03_11-17-2.7-5988caffbff9.json > 2017-02-06_07-15-default-e06af4027546.json -G --min-speed=10 > Slower (12): > - python_startup_no_site: 3.11 ms +- 0.02 ms -> 8.33 ms +- 0.04 ms: > 2.68x slower (+168%) > - python_startup: 6.42 ms +- 0.04 ms -> 14.1 ms +- 0.1 ms: 2.20x slower (+120%) > - unpickle_pure_python: 384 us +- 8 us -> 670 us +- 11 us: 1.74x slower (+74%) > - spectral_norm: 194 ms +- 1 ms -> 259 ms +- 4 ms: 1.33x slower (+33%) > - sqlite_synth: 6.70 us +- 0.11 us -> 8.49 us +- 0.16 us: 1.27x slower (+27%) > - crypto_pyaes: 158 ms +- 1 ms -> 199 ms +- 2 ms: 1.26x slower (+26%) > - xml_etree_parse: 193 ms +- 4 ms -> 242 ms +- 5 ms: 1.25x slower (+25%) > - logging_format: 28.9 us +- 0.2 us -> 33.9 us +- 0.5 us: 1.17x slower (+17%) > - xml_etree_iterparse: 154 ms +- 2 ms -> 179 ms +- 4 ms: 1.16x slower (+16%) > - pickle_pure_python: 847 us +- 14 us -> 974 us +- 14 us: 1.15x slower (+15%) > - logging_simple: 24.4 us +- 0.3 us -> 27.9 us +- 0.4 us: 1.14x slower (+14%) > - go: 439 ms +- 3 ms -> 493 ms +- 7 ms: 1.12x slower (+12%) > > Faster (23): > - telco: 634 ms +- 5 ms -> 14.7 ms +- 0.3 ms: 43.23x faster (-98%) > - unpickle_list: 15.7 us +- 0.6 us -> 6.36 us +- 0.11 us: 2.47x faster (-59%) > - pickle_list: 13.7 us +- 0.2 us -> 7.53 us +- 0.41 us: 1.81x faster (-45%) > - pickle: 38.8 us +- 1.5 us -> 22.1 us +- 0.3 us: 1.76x faster (-43%) > - mako: 56.0 ms +- 0.9 ms -> 32.0 ms +- 0.3 ms: 1.75x faster (-43%) > - sympy_sum: 335 ms +- 11 ms -> 192 ms +- 6 ms: 1.74x faster (-43%) > - sympy_expand: 1.65 sec +- 0.02 sec -> 949 ms +- 12 ms: 1.74x faster (-43%) > - sympy_str: 678 ms +- 5 ms -> 425 ms +- 5 ms: 1.59x faster (-37%) > - json_loads: 89.8 us +- 1.0 us -> 56.6 us +- 0.8 us: 1.59x faster (-37%) > - scimark_lu: 483 ms +- 10 ms -> 352 ms +- 11 ms: 1.37x faster (-27%) > - xml_etree_generate: 282 ms +- 3 ms -> 208 ms +- 2 ms: 1.36x faster (-26%) > - call_method_slots: 15.3 ms +- 0.4 ms -> 11.2 ms +- 0.3 ms: 1.36x faster (-26%) > - html5lib: 269 ms +- 7 ms -> 201 ms +- 5 ms: 1.34x faster (-25%) > - call_method_unknown: 17.5 ms +- 0.3 ms -> 13.1 ms +- 0.1 ms: 1.34x > faster (-25%) > - call_method: 15.1 ms +- 0.3 ms -> 11.7 ms +- 0.2 ms: 1.29x faster (-23%) > - sympy_integrate: 51.7 ms +- 0.6 ms -> 40.8 ms +- 0.3 ms: 1.27x faster (-21%) > - unpack_sequence: 140 ns +- 3 ns -> 111 ns +- 1 ns: 1.26x faster (-21%) > - regex_v8: 53.7 ms +- 0.9 ms -> 43.7 ms +- 0.6 ms: 1.23x faster (-19%) > - 2to3: 763 ms +- 4 ms -> 622 ms +- 4 ms: 1.23x faster (-19%) > - pickle_dict: 69.4 us +- 1.1 us -> 56.6 us +- 4.3 us: 1.23x faster (-18%) > - call_simple: 12.7 ms +- 0.3 ms -> 10.3 ms +- 0.4 ms: 1.23x faster (-18%) > - richards: 162 ms +- 2 ms -> 145 ms +- 5 ms: 1.12x faster (-11%) > - float: 266 ms +- 2 ms -> 238 ms +- 3 ms: 1.12x faster (-10%) > > Benchmark hidden because not significant (29): (...) > Ignored benchmarks (3) of 2017-01-03_11-17-2.7-5988caffbff9.json: > hg_startup, pyflate, spambayes > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

3 years, 9 months

Slides and video of my "How to run a stable benchmark" talk at FOSDEM

by Victor Stinner

Hi, I annoyed many of you on this mailing list with all my WTF performance issues. I consider that I now succeeded to get benchmarks stable enough to become "usable" (reliable). Two days ago, I gave a talk "How to run a stable benchmark" at FOSDEM (Brussels, Belgium) on my findings. Slides and video of my talk are now online, enjoy! https://fosdem.org/2017/schedule/event/python_stable_benchmark/ "Takeaway": * Tune system to run benchmarks: python3 -m perf system tune * Stop using timeit!: python3 -m timeit STMT ⇒ python3 -m perf timeit STMT * Use perf and its documentation! http://perf.rtfd.io/ Victor

3 years, 9 months

CPython 3.7 is now faster than CPython 2.7 on most benchmarks

by Victor Stinner

Hi, FYI CPython 3.7 is now faster than CPython 2.7 on most benchmarks. It would be interesting to investigate why 12 benchmarks are still slower to check if we missed something obvious. Comparison, ignoring differences smaller than 10%: haypo@speed-python$ python3 -m perf compare_to 2017-01-03_11-17-2.7-5988caffbff9.json 2017-02-06_07-15-default-e06af4027546.json -G --min-speed=10 Slower (12): - python_startup_no_site: 3.11 ms +- 0.02 ms -> 8.33 ms +- 0.04 ms: 2.68x slower (+168%) - python_startup: 6.42 ms +- 0.04 ms -> 14.1 ms +- 0.1 ms: 2.20x slower (+120%) - unpickle_pure_python: 384 us +- 8 us -> 670 us +- 11 us: 1.74x slower (+74%) - spectral_norm: 194 ms +- 1 ms -> 259 ms +- 4 ms: 1.33x slower (+33%) - sqlite_synth: 6.70 us +- 0.11 us -> 8.49 us +- 0.16 us: 1.27x slower (+27%) - crypto_pyaes: 158 ms +- 1 ms -> 199 ms +- 2 ms: 1.26x slower (+26%) - xml_etree_parse: 193 ms +- 4 ms -> 242 ms +- 5 ms: 1.25x slower (+25%) - logging_format: 28.9 us +- 0.2 us -> 33.9 us +- 0.5 us: 1.17x slower (+17%) - xml_etree_iterparse: 154 ms +- 2 ms -> 179 ms +- 4 ms: 1.16x slower (+16%) - pickle_pure_python: 847 us +- 14 us -> 974 us +- 14 us: 1.15x slower (+15%) - logging_simple: 24.4 us +- 0.3 us -> 27.9 us +- 0.4 us: 1.14x slower (+14%) - go: 439 ms +- 3 ms -> 493 ms +- 7 ms: 1.12x slower (+12%) Faster (23): - telco: 634 ms +- 5 ms -> 14.7 ms +- 0.3 ms: 43.23x faster (-98%) - unpickle_list: 15.7 us +- 0.6 us -> 6.36 us +- 0.11 us: 2.47x faster (-59%) - pickle_list: 13.7 us +- 0.2 us -> 7.53 us +- 0.41 us: 1.81x faster (-45%) - pickle: 38.8 us +- 1.5 us -> 22.1 us +- 0.3 us: 1.76x faster (-43%) - mako: 56.0 ms +- 0.9 ms -> 32.0 ms +- 0.3 ms: 1.75x faster (-43%) - sympy_sum: 335 ms +- 11 ms -> 192 ms +- 6 ms: 1.74x faster (-43%) - sympy_expand: 1.65 sec +- 0.02 sec -> 949 ms +- 12 ms: 1.74x faster (-43%) - sympy_str: 678 ms +- 5 ms -> 425 ms +- 5 ms: 1.59x faster (-37%) - json_loads: 89.8 us +- 1.0 us -> 56.6 us +- 0.8 us: 1.59x faster (-37%) - scimark_lu: 483 ms +- 10 ms -> 352 ms +- 11 ms: 1.37x faster (-27%) - xml_etree_generate: 282 ms +- 3 ms -> 208 ms +- 2 ms: 1.36x faster (-26%) - call_method_slots: 15.3 ms +- 0.4 ms -> 11.2 ms +- 0.3 ms: 1.36x faster (-26%) - html5lib: 269 ms +- 7 ms -> 201 ms +- 5 ms: 1.34x faster (-25%) - call_method_unknown: 17.5 ms +- 0.3 ms -> 13.1 ms +- 0.1 ms: 1.34x faster (-25%) - call_method: 15.1 ms +- 0.3 ms -> 11.7 ms +- 0.2 ms: 1.29x faster (-23%) - sympy_integrate: 51.7 ms +- 0.6 ms -> 40.8 ms +- 0.3 ms: 1.27x faster (-21%) - unpack_sequence: 140 ns +- 3 ns -> 111 ns +- 1 ns: 1.26x faster (-21%) - regex_v8: 53.7 ms +- 0.9 ms -> 43.7 ms +- 0.6 ms: 1.23x faster (-19%) - 2to3: 763 ms +- 4 ms -> 622 ms +- 4 ms: 1.23x faster (-19%) - pickle_dict: 69.4 us +- 1.1 us -> 56.6 us +- 4.3 us: 1.23x faster (-18%) - call_simple: 12.7 ms +- 0.3 ms -> 10.3 ms +- 0.4 ms: 1.23x faster (-18%) - richards: 162 ms +- 2 ms -> 145 ms +- 5 ms: 1.12x faster (-11%) - float: 266 ms +- 2 ms -> 238 ms +- 3 ms: 1.12x faster (-10%) Benchmark hidden because not significant (29): (...) Ignored benchmarks (3) of 2017-01-03_11-17-2.7-5988caffbff9.json: hg_startup, pyflate, spambayes Victor

3 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed February 2017