[Python-Dev] Benchmarking Python and micro-optimizations

[Maciej Fijalkowski]

Hi Victor

Despite the fact that I was not able to find time to run your stuff
yet, thanks for all the awesome work!

On Thu, Oct 20, 2016 at 12:56 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.
>
> * I wrote a new perf module which runs benchmarks in a reliable way
> and contains a LOT of features: collect metadata, JSON file format,
> commands to compare, render an histogram, etc.
>
> * I rewrote the Python benchmark suite: the old benchmarks Mercurial
> repository moved to a new performance GitHub project which uses my
> perf module and contains more benchmarks.
>
> * I also made minor enhancements to timeit in Python 3.7 -- some dev
> don't want major changes to not "break the backward compatibility".
>
> For timeit, I suggest to use my perf tool which includes a reliable
> timeit command and has much more features like --duplicate (repeat the
> statements to reduce the cost of the outer loop) and --compare-to
> (compare two versions of Python), but also all builtin perf features
> (JSON output, statistics, histogram, etc.).
>
> I added benchmarks from PyPy and Pyston benchmark suites to
> performance: performance 0.3.1 contains 51 benchmark scripts which run
> a total of 121 benchmarks. Example of tested Python modules:
>
> * SQLAlchemy
> * Dulwich (full Git implementation in Python)
> * Mercurial (currently only the startup time)
> * html5lib
> * pyaes (AES crypto cipher in pure Python)
> * sympy
> * Tornado (HTTP client and server)
> * Django (sadly, only the template engine right now, Pyston contains
> HTTP benchmarks)
> * pathlib
> * spambayes
>
> More benchmarks will be added later. It would be nice to add
> benchmarks on numpy for example, numpy is important for a large part
> of our community.
>
> All these (new or updated) tools can now be used to take smarter
> decisions on optimizations. Please don't push any optimization anymore
> without providing reliable benchmark results!
>
>
> My first major action was to close the latest attempt to
> micro-optimize int+int in Python/ceval.c,
> http://bugs.python.org/issue21955 : I closed the issue as rejected,
> because there is no significant speedup on benchmarks other than two
> (tiny) microbenchmarks. To make sure that no one looses its time on
> trying to micro-optimize int+int, I even added a comment to
> Python/ceval.c :-)
>
>    https://hg.python.org/cpython/rev/61fcb12a9873
>    "Please don't try to micro-optimize int+int"
>
>
> The perf and performance are now well tested: Travis CI runs tests on
> the new commits and pull requests, and the "tox" command can be used
> locally to test different Python versions, pep8, doc, ... in a single
> command.
>
>
> Next steps:
>
> * Run performance 0.3.1 on speed.python.org: the benchmark runner is
> currently stopped (and still uses the old benchmarks project). The
> website part may be updated to allow to download full JSON files which
> includes *all* information (all timings, metadata and more).
>
> * I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy
> 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much
> resources.
>
> * Later, we can consider adding more implementations of Python:
> Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks
> should be run on the same hardware to be comparable.
>
> * Later, we might also allow other projects to upload their own
> benchmark results, but we should find a solution to groups benchmark
> results per benchmark runner (ex: at least by the hostname, perf JSON
> contains the hostname) to not compare two results from two different
> hardware
>
> * We should continue to add more benchmarks to the performance
> benchmark suite, especially benchmarks more representative of real
> applications (we have enough microbenchmarks!)
>
>
> Links:
>
> * perf: http://perf.readthedocs.io/
> * performance: https://github.com/python/performance
> * Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed
> * https://speed.python.org/ (currently outdated, and don't use performance yet)
>
> See https://pypi.python.org/pypi/performance which contains even more
> links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com