CPython benchmark status, April 2017
Hi, I'm still working on analyzing past optimizations to guide future optimizations. I succeeded to identify multiple significant optimizations over the last 3 years. At least for me, some were unexpected like "Use the test suite for profile data" which made pickle 1.28x faster and pidigts 1.16x faster. Here is a report of my benchmark work of last weeks. I succeeded to compute benchmarks on CPython master on the period April, 2014-April,2017: we now have have a timeline over 3 years of CPython performance! https://speed.python.org/timeline/ I started to take notes on significant performance changes (speedup and slowdown) of this timeline: http://pyperformance.readthedocs.io/cpython_results_2017.html To identify the change which introduced a significant performance change, I wrote a Python script running a Git bisection: compile CPython, run benchmark, repeat. https://github.com/haypo/misc/blob/master/misc/bisect_cpython_perf.py It uses a configuration file which looks like: --- [config] work_dir = ~/prog/bench_python/bisect-pickle src_dir = ~/prog/bench_python/master old_commit = 133138a284be1985ebd9ec9014f1306b9a42 new_commit = 10427f44852b6e872034061421a8890902b8f benchmark = ~/prog/bench_python/performance/performance/benchmarks/bm_pickle.py pickle benchmark_opts = --inherit-environ=PYTHONPATH -p5 -v configure_args = --- I succeeded to identify many significant optimizations, examples: * PyMem_Malloc() now uses the fast pymalloc allocator * Add a C implementation of collections.OrderedDict * Use the test suite for profile data * Speedup method calls 1.2x * Added C implementation of functools.lru_cache() * Optimized ElementTree.iterparse(); it is now 2x faster perf, performance, server configuration, etc. evolve quicker than expected, so I created a Git project to keep a copy of JSON files: https://github.com/haypo/performance_results I lost data of my first milestone (november-december 2016), but you have data from the second (december 2016-february 2017) and third (march 2016-today) milestones. Victor
On 4/7/2017 12:27 PM, Victor Stinner wrote:
I succeeded to compute benchmarks on CPython master on the period April, 2014-April,2017: we now have have a timeline over 3 years of CPython performance!
The boxes are un-scaled. Is vertical always 'time for fixed work', so lower is always better? The vertical variation is a small fraction of vertical space, so most vertical space is wasted. Box says 'last 50', but only 13 results. These cover last 3 years? -- Terry Jan Reedy
2017-04-07 20:50 GMT+02:00 Terry Reedy <tjreedy@udel.edu>:
The boxes are un-scaled.
Which page? "Display All in a grid" has no scale, but if you select a single benchmark, there are scales. It's a number of seconds: "(less is better)" as written on the Y scale.
Is vertical always 'time for fixed work', so lower is always better?
Yes.
The vertical variation is a small fraction of vertical space, so most vertical space is wasted.
I don't well the website part, please see the https://github.com/tobami/codespeed/ project.
Box says 'last 50', but only 13 results. These cover last 3 years?
Currently, the oldest result is the commit 0cf3ed60d089cf2a2671fa66cdaa18498091: April 1, 2014. I'm unable to count the exact number of dots :-D On the special "Display All in a grid" view, you cannot select the number of results. I don't know how many results are displayed: 50 max I think. If you select a benchmark, 50 results max are displayed. Currently, it's enough to over 3 years. Victor
participants (2)
-
Terry Reedy -
Victor Stinner