Hi,
After a few months of work, I created a "new" benchmark suite for Python:
https://github.com/python/benchmarks
It's based on: https://hg.python.org/sandbox/benchmarks_perf which is my fork the CPython benchmark suite: https://hg.python.org/benchmarks which is based on Unladen Swallow's benchmark suite (if I understood correctly).
Major differences:
- Use the perf module to run benchmarks in multiple processes and store results as JSON
- Create virtual environments using requirements.txt to download dependencies from PyPI (rather than using old copies of libraries)
- Many libraries have been upgraded: see requirements.txt
The project works on Python 2 and Python 3 (I tested 2.7 and 3.6).
Known regressions:
- Memory tracking is broken
- run_compare command is currently broken: use run (store result into a file) + compare manually
- Some benchmarks have been removed: rietveld, spitfire (not on PyPI), pystone, gcbench, tuple_gc_hell
- I only tested Linux, I expect issues on Windows. (I didn't try my perf module on Windows yet.)
I already allowed all Python core developers to push to the GitHub project. We can create a new "benchmarks" (or "Performance" maybe?) team if we want to allow more contributors who are not core developers.
PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to take a look at my project and test it ;-) Tell me what is broken, what is missing, and I will try to help you to move your project to this new benchmark suite!
As requested (suggested?) by Brett Canon, the Git repository has no history, it only contains 1 commit! I'm really sorry of loosing all the history and all authors, but it allows to start with a much smaller repository: around 2 MB. The current benchmark repository is more around 200 MB!
TODO:
- continue to upgrade libraries in requirements.txt. I failed to upgrade Django to 1.10, it complains about a missing template engine config setting.
- convert more code to the perf module, like "startup" tests
- run benchmarks and analyze results ;-)
- write more documentation explaining how to run reliable benchmarks
- ...
Victor