Speed March 2017

speed@python.org

7 participants
29 discussions

Issues to run benchmarks on Python before 2015-04-01

by Victor Stinner

Hi, I'm trying to run benchmarks on revisions between 2014-01-01 and today, but I got two different issues: see below. I'm now looking for workarounds :-/ Because of these bugs, I'm unable to get benchmarks results before 2015-04-01 (at 2015-04-01, benchmarks work again). (1) 2014-01-01: "python3 -m pip install performance" fails with a TypeError: "charset argument must be specified when non-ASCII characters are used in the payload" It's a regression of Python 3.4 beta: http://bugs.python.org/issue20531 (2) 2014-04-01, 2014-07-01, 2014-10-01, 2015-01-01: "venv/bin/python -m pip install" fails in extract_stack() of pyparsing --- haypo@selma$ /home/haypo/prog/bench_python/tmpdir/prefix/bin/python3 Python 3.5.0a0 (default, Apr 1 2017, 00:01:30) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pip Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/__init__.py", line 26, in <module> from pip.utils import get_installed_distributions, get_prog File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/utils/__init__.py", line 27, in <module> from pip._vendor import pkg_resources File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 74, in <module> __import__('pip._vendor.packaging.requirements') File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 9, in <module> from pip._vendor.pyparsing import ( File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 4715, in <module> _escapedPunc = Word( _bslash, r"\[]-*.$+^?()~ ", exact=2 ).setParseAction(lambda s,l,t:t[0][1]) File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1261, in setParseAction self.parseAction = list(map(_trim_arity, list(fns))) File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1043, in _trim_arity this_line = extract_stack(limit=2)[-1] File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in extract_stack return [(frame_summary.filename, frame_summary.lineno)] AttributeError: 'tuple' object has no attribute 'filename' --- Note: same error using the pip program (ex: " prefix/bin/pip --version"). Victor

3 years

Results of CPython benchmarks on 2016

by Victor Stinner

Hi, Before removing everything from speed.python.org database, I took screenshots on interesting pages: https://haypo.github.io/speed-python-org-march-2017.html * Benchmarks where Python 3.7 is faster than Python 2.7 * Benchmarks where Python 3.7 is slower than Python 2.7 * Significant optimizations * etc. CPython became faster on many benchmarks in 2016: * call_method * float * hexiom * nqueens * pickle_list * richards * scimark_lu * scimark_sor * sympy_sum * telco * unpickle_list. I now have to analyze what made these benchmarks faster for my future Pycon US talk "Optimizations which made Python 3.6 faster than Python 3.5" ;-) I also kept many screenshots to see that benchmarks are now stable! Victor

3 years

Interesting Ruby pull request

by Victor Stinner

Hi, It seems like Urabe, Shyouhei succeeded to write an efficient optimizer for Ruby: https://github.com/ruby/ruby/pull/1419 Since Ruby and CPython design are similar, maybe we can pick some ideas. It seems like the optimizer is not done yet, the PR is not merged yet. I don't understand how the optimizer works. An interesting commit: https://github.com/ruby/ruby/pull/1419/commits/d7b376949eb1626b9e5088f907db… --- basic optimization infrastructure This commit adds on-the-fly ISeq analyzer. It detects an ISeq's purity, i.e. if that ISeq has side-effect or not. Purity is the key concept of whole optimization techniques in general, but in Ruby it is yet more important because there is a method called eval. A pure ISeq is free from eval, while those not pure are stuck in the limbo where any of its side effects _could_ result in (possibly aliased) call to eval. So an optimization tend not be possible against them. Note however, that the analyzer cannot statically say if the ISeq in question is pure or not. It categorizes an ISeq into 3 states namely pure, not pure, or "unpredictable". The last category is used when for instance there are branches yet to be analyzed, or method calls to another unpredictable ISeq. An ISeq's purity changes over time, not only by redefinition of methods, but by other optimizations, like, by entering a rarely-taken branch of a formerly-unpredictable ISeq to kick analyzer to fix its purity. Such change propagates to its callers. * optimize.c: new file. * optimize.h: new file. * common.mk (COMMONOBJS): dependencies for new files. * iseq.h (ISEQ_NEEDS_ANALYZE): new flag to denote the iseq in question might need (re)analyzing. --- I had this link in my bookmark for months, but I forgot it. This email is not to forget it again ;-) Someone may find it useful! Victor

3 years

Re: [Speed] speed.python.org: move to Git, remove old previous results

by Victor Stinner

2017-03-28 9:36 GMT+02:00 Miquel Torres <tobami(a)gmail.com>: > I can have a look into increasing the number of points displayed. There is a "Show the last [50] results" widget, but it's disabled if you select "(o) Display all in a grid". Maybe we should enable the first widget but limit the maximum number of results when this specific view is selected? Just keep 50 by default ;-) Victor

3 years

Re: [Speed] speed.python.org: move to Git, remove old previous results

by Miquel Torres

I can have a look into increasing the number of points displayed. El El lun, 27 mar 2017 a las 15:44, Victor Stinner <victor.stinner(a)gmail.com> escribió: > Zachary Ware told me on IRC that it's ok for him to drop old data. > > If nobody else complains, I will remove old data tomorrow (tuesday). > > I already validated that the patched scripts work on Git. I released > new versions of perf and performance to make sure that the latest > version of the code is released and used. By the way, the newly > released perf 1.1 gets a new "perf command" command to measure the > time of a command, it's like the Unix "time" command. > > http://perf.readthedocs.io/en/latest/cli.html#command-cmd > > $ python3 -m perf command -- python2 -c pass > ..................... > command: Mean +- std dev: 21.2 ms +- 3.2 ms > > Victor > > 2017-03-27 0:12 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > > Hi, > > > > I'm going to remove old previous benchmark results from > > speed.python.org. As we discussed previously, there is no plan to keep > > old results when we need to change something. In this case, CPython > > moved from Mercurial to Git, and I'm too lazy to upgrade the revisions > > in database. I prefer to run again benchmarks :-) > > > > My plan: > > > > * Remove all previous benchmark results > > * Run benchmarks on master, 2.7, 3.6 and 3.5 branches > > * Run benchmarks on one revision per year quarter on the last 2 years > > * Then see if we should run benchmarks on even older revisions and/or > > if we need more than one plot per quarter. > > * Maybe one point per month at least? The problem is that the UI is > > limited to 50 points on the "Display all in a grid" view of the > > Timeline. I would like to be able to render 2 years on this view. > > > > For each year quarter, I will use the first commit of the master > > branch on this period. > > > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

3 years

Re: [Speed] ASLR

by Victor Stinner

2017-03-16 17:19 GMT+01:00 Brett Cannon <brett(a)python.org>: >> By the way, maybe I should commit a change in hg.python.org/benchmarks >> to remove the code and only keep a README.txt? Code will still be >> accessible in Mercurial history. > > Since we might not shut down hg.python.org for a long time I say go ahead > and commit such a change. Ok, done! https://hg.python.org/benchmarks/file/tip/README.txt https://hg.python.org/benchmarks/file/tip Victor

3 years

Re: [Speed] speed.python.org: move to Git, remove old previous results

by Victor Stinner

Zachary Ware told me on IRC that it's ok for him to drop old data. If nobody else complains, I will remove old data tomorrow (tuesday). I already validated that the patched scripts work on Git. I released new versions of perf and performance to make sure that the latest version of the code is released and used. By the way, the newly released perf 1.1 gets a new "perf command" command to measure the time of a command, it's like the Unix "time" command. http://perf.readthedocs.io/en/latest/cli.html#command-cmd $ python3 -m perf command -- python2 -c pass ..................... command: Mean +- std dev: 21.2 ms +- 3.2 ms Victor 2017-03-27 0:12 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > Hi, > > I'm going to remove old previous benchmark results from > speed.python.org. As we discussed previously, there is no plan to keep > old results when we need to change something. In this case, CPython > moved from Mercurial to Git, and I'm too lazy to upgrade the revisions > in database. I prefer to run again benchmarks :-) > > My plan: > > * Remove all previous benchmark results > * Run benchmarks on master, 2.7, 3.6 and 3.5 branches > * Run benchmarks on one revision per year quarter on the last 2 years > * Then see if we should run benchmarks on even older revisions and/or > if we need more than one plot per quarter. > * Maybe one point per month at least? The problem is that the UI is > limited to 50 points on the "Display all in a grid" view of the > Timeline. I would like to be able to render 2 years on this view. > > For each year quarter, I will use the first commit of the master > branch on this period. > > Victor

3 years

speed.python.org: move to Git, remove old previous results

by Victor Stinner

Hi, I'm going to remove old previous benchmark results from speed.python.org. As we discussed previously, there is no plan to keep old results when we need to change something. In this case, CPython moved from Mercurial to Git, and I'm too lazy to upgrade the revisions in database. I prefer to run again benchmarks :-) My plan: * Remove all previous benchmark results * Run benchmarks on master, 2.7, 3.6 and 3.5 branches * Run benchmarks on one revision per year quarter on the last 2 years * Then see if we should run benchmarks on even older revisions and/or if we need more than one plot per quarter. * Maybe one point per month at least? The problem is that the UI is limited to 50 points on the "Display all in a grid" view of the Timeline. I would like to be able to render 2 years on this view. For each year quarter, I will use the first commit of the master branch on this period. Victor

3 years

pymicrobench: collection of CPython microbenchmarks

by Victor Stinner

Hi, I started to create a collection of microbenchmarks for CPython from scripts found on the bug tracker: https://github.com/haypo/pymicrobench I'm not sure that this collection is used yet, but some of you may want to take a look :-) I know that some people have random microbenchmarks in a local directory. Maybe you want to share them? I don't really care to sort them or group them. My plan is first to populate the repository, and later see what to do with it :-) Victor

3 years

Re: [Speed] ASLR

by Victor Stinner

2017-03-17 0:00 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com>: > In addition to turbo boost, I also turned off hyperthreading, and c-state, p-state, on Intel CPUs. My "python3 -m perf system tune" command sets the minimum frequency of CPUs used for benchmarks to the maximum frequency. I expect that it reduces or even avoid changes on P-state and C-state. See my documentation on How to tune a system for benchmarking: http://perf.readthedocs.io/en/latest/system.html Victor

3 years

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed March 2017