Hi,
I'm trying to run benchmarks on revisions between 2014-01-01 and
today, but I got two different issues: see below. I'm now looking for
workarounds :-/ Because of these bugs, I'm unable to get benchmarks
results before 2015-04-01 (at 2015-04-01, benchmarks work again).
(1) 2014-01-01: "python3 -m pip install performance" fails with a
TypeError: "charset argument must be specified when non-ASCII
characters are used in the payload"
It's a regression of Python 3.4 beta: http://bugs.python.org/issue20531
(2) 2014-04-01, 2014-07-01, 2014-10-01, 2015-01-01: "venv/bin/python
-m pip install" fails in extract_stack() of pyparsing
---
haypo@selma$ /home/haypo/prog/bench_python/tmpdir/prefix/bin/python3
Python 3.5.0a0 (default, Apr 1 2017, 00:01:30)
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/__init__.py",
line 26, in <module>
from pip.utils import get_installed_distributions, get_prog
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/utils/__init__.py",
line 27, in <module>
from pip._vendor import pkg_resources
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py",
line 74, in <module>
__import__('pip._vendor.packaging.requirements')
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py",
line 9, in <module>
from pip._vendor.pyparsing import (
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py",
line 4715, in <module>
_escapedPunc = Word( _bslash, r"\[]-*.$+^?()~ ", exact=2
).setParseAction(lambda s,l,t:t[0][1])
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py",
line 1261, in setParseAction
self.parseAction = list(map(_trim_arity, list(fns)))
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py",
line 1043, in _trim_arity
this_line = extract_stack(limit=2)[-1]
File "/home/haypo/prog/bench_python/tmpdir/prefix/lib/python3.5/site-packages/pip/_vendor/pyparsing.py",
line 1028, in extract_stack
return [(frame_summary.filename, frame_summary.lineno)]
AttributeError: 'tuple' object has no attribute 'filename'
---
Note: same error using the pip program (ex: " prefix/bin/pip --version").
Victor
Hi,
Before removing everything from speed.python.org database, I took
screenshots on interesting pages:
https://haypo.github.io/speed-python-org-march-2017.html
* Benchmarks where Python 3.7 is faster than Python 2.7
* Benchmarks where Python 3.7 is slower than Python 2.7
* Significant optimizations
* etc.
CPython became faster on many benchmarks in 2016:
* call_method
* float
* hexiom
* nqueens
* pickle_list
* richards
* scimark_lu
* scimark_sor
* sympy_sum
* telco
* unpickle_list.
I now have to analyze what made these benchmarks faster for my future
Pycon US talk "Optimizations which made Python 3.6 faster than Python
3.5" ;-)
I also kept many screenshots to see that benchmarks are now stable!
Victor
Hi,
It seems like Urabe, Shyouhei succeeded to write an efficient
optimizer for Ruby:
https://github.com/ruby/ruby/pull/1419
Since Ruby and CPython design are similar, maybe we can pick some
ideas. It seems like the optimizer is not done yet, the PR is not
merged yet.
I don't understand how the optimizer works. An interesting commit:
https://github.com/ruby/ruby/pull/1419/commits/d7b376949eb1626b9e5088f907db…
---
basic optimization infrastructure
This commit adds on-the-fly ISeq analyzer. It detects an ISeq's
purity, i.e. if that ISeq has side-effect or not. Purity is the key
concept of whole optimization techniques in general, but in Ruby it is
yet more important because there is a method called eval. A pure ISeq
is free from eval, while those not pure are stuck in the limbo where
any of its side effects _could_ result in (possibly aliased) call to
eval. So an optimization tend not be possible against them.
Note however, that the analyzer cannot statically say if the ISeq in
question is pure or not. It categorizes an ISeq into 3 states namely
pure, not pure, or "unpredictable". The last category is used when
for instance there are branches yet to be analyzed, or method calls to
another unpredictable ISeq.
An ISeq's purity changes over time, not only by redefinition of
methods, but by other optimizations, like, by entering a rarely-taken
branch of a formerly-unpredictable ISeq to kick analyzer to fix its
purity. Such change propagates to its callers.
* optimize.c: new file.
* optimize.h: new file.
* common.mk (COMMONOBJS): dependencies for new files.
* iseq.h (ISEQ_NEEDS_ANALYZE): new flag to denote the iseq in
question might need (re)analyzing.
---
I had this link in my bookmark for months, but I forgot it. This email
is not to forget it again ;-) Someone may find it useful!
Victor
2017-03-28 9:36 GMT+02:00 Miquel Torres <tobami(a)gmail.com>:
> I can have a look into increasing the number of points displayed.
There is a "Show the last [50] results" widget, but it's disabled if
you select "(o) Display all in a grid". Maybe we should enable the
first widget but limit the maximum number of results when this
specific view is selected? Just keep 50 by default ;-)
Victor
I can have a look into increasing the number of points displayed.
El El lun, 27 mar 2017 a las 15:44, Victor Stinner <victor.stinner(a)gmail.com>
escribió:
> Zachary Ware told me on IRC that it's ok for him to drop old data.
>
> If nobody else complains, I will remove old data tomorrow (tuesday).
>
> I already validated that the patched scripts work on Git. I released
> new versions of perf and performance to make sure that the latest
> version of the code is released and used. By the way, the newly
> released perf 1.1 gets a new "perf command" command to measure the
> time of a command, it's like the Unix "time" command.
>
> http://perf.readthedocs.io/en/latest/cli.html#command-cmd
>
> $ python3 -m perf command -- python2 -c pass
> .....................
> command: Mean +- std dev: 21.2 ms +- 3.2 ms
>
> Victor
>
> 2017-03-27 0:12 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>:
> > Hi,
> >
> > I'm going to remove old previous benchmark results from
> > speed.python.org. As we discussed previously, there is no plan to keep
> > old results when we need to change something. In this case, CPython
> > moved from Mercurial to Git, and I'm too lazy to upgrade the revisions
> > in database. I prefer to run again benchmarks :-)
> >
> > My plan:
> >
> > * Remove all previous benchmark results
> > * Run benchmarks on master, 2.7, 3.6 and 3.5 branches
> > * Run benchmarks on one revision per year quarter on the last 2 years
> > * Then see if we should run benchmarks on even older revisions and/or
> > if we need more than one plot per quarter.
> > * Maybe one point per month at least? The problem is that the UI is
> > limited to 50 points on the "Display all in a grid" view of the
> > Timeline. I would like to be able to render 2 years on this view.
> >
> > For each year quarter, I will use the first commit of the master
> > branch on this period.
> >
> > Victor
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
>
Zachary Ware told me on IRC that it's ok for him to drop old data.
If nobody else complains, I will remove old data tomorrow (tuesday).
I already validated that the patched scripts work on Git. I released
new versions of perf and performance to make sure that the latest
version of the code is released and used. By the way, the newly
released perf 1.1 gets a new "perf command" command to measure the
time of a command, it's like the Unix "time" command.
http://perf.readthedocs.io/en/latest/cli.html#command-cmd
$ python3 -m perf command -- python2 -c pass
.....................
command: Mean +- std dev: 21.2 ms +- 3.2 ms
Victor
2017-03-27 0:12 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>:
> Hi,
>
> I'm going to remove old previous benchmark results from
> speed.python.org. As we discussed previously, there is no plan to keep
> old results when we need to change something. In this case, CPython
> moved from Mercurial to Git, and I'm too lazy to upgrade the revisions
> in database. I prefer to run again benchmarks :-)
>
> My plan:
>
> * Remove all previous benchmark results
> * Run benchmarks on master, 2.7, 3.6 and 3.5 branches
> * Run benchmarks on one revision per year quarter on the last 2 years
> * Then see if we should run benchmarks on even older revisions and/or
> if we need more than one plot per quarter.
> * Maybe one point per month at least? The problem is that the UI is
> limited to 50 points on the "Display all in a grid" view of the
> Timeline. I would like to be able to render 2 years on this view.
>
> For each year quarter, I will use the first commit of the master
> branch on this period.
>
> Victor
Hi,
I'm going to remove old previous benchmark results from
speed.python.org. As we discussed previously, there is no plan to keep
old results when we need to change something. In this case, CPython
moved from Mercurial to Git, and I'm too lazy to upgrade the revisions
in database. I prefer to run again benchmarks :-)
My plan:
* Remove all previous benchmark results
* Run benchmarks on master, 2.7, 3.6 and 3.5 branches
* Run benchmarks on one revision per year quarter on the last 2 years
* Then see if we should run benchmarks on even older revisions and/or
if we need more than one plot per quarter.
* Maybe one point per month at least? The problem is that the UI is
limited to 50 points on the "Display all in a grid" view of the
Timeline. I would like to be able to render 2 years on this view.
For each year quarter, I will use the first commit of the master
branch on this period.
Victor
Hi,
I started to create a collection of microbenchmarks for CPython from
scripts found on the bug tracker:
https://github.com/haypo/pymicrobench
I'm not sure that this collection is used yet, but some of you may
want to take a look :-)
I know that some people have random microbenchmarks in a local
directory. Maybe you want to share them?
I don't really care to sort them or group them. My plan is first to
populate the repository, and later see what to do with it :-)
Victor
2017-03-17 0:00 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang(a)intel.com>:
> In addition to turbo boost, I also turned off hyperthreading, and c-state, p-state, on Intel CPUs.
My "python3 -m perf system tune" command sets the minimum frequency of
CPUs used for benchmarks to the maximum frequency. I expect that it
reduces or even avoid changes on P-state and C-state.
See my documentation on How to tune a system for benchmarking:
http://perf.readthedocs.io/en/latest/system.html
Victor
