Re: [Speed] Adapting pypy benchmark suite
On Thu, Apr 14, 2016 at 6:51 PM, Brett Cannon brett@python.org wrote:
On Wed, 13 Apr 2016 at 13:33 Zachary Ware zachary.ware+pydev@gmail.com wrote:
On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski fijall@gmail.com wrote:
Hi
I have a radical idea: to take a pypy benchmark suite, update the libraries to newer ones and replace python benchmarks with that. The main reason being that pypy has a much better coverage of things that are not microbenchmarks, the list (in json):
http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/
Which is much more extensive than this:
https://hg.python.org/benchmarks/file/tip/performance
I'm willing to put *some* effort, what do people think?
I'm in favor. My support has two conditions, though:
- at least a majority of the benchmarks must be Python3 compatible. Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are 2 compatible anyway.
Agreed (although I don't care about the 2/3 compatibility, just the 3 compat ;) .
- and there should be an easy way to run the benchmarks against exactly 1 interpreter (for use with speed.python.org). I initially tried to set up speed.python.org using the PyPy benchmarks, but quickly ran into issues with trying to use 'nullpython.py' as the baseline Python. When I switched to using h.p.o/benchmarks, I added the '--raw' flag to perf.py which allows the benchmarks to be run on one interpreter instead of two. It was just a quick hack, though; I have no problems with that feature completely changing (even invoking it a different way is ok), so long as it exists.
This project could probably start its life as github.com/python/benchmarks and save us from having to migrate h.p.o/benchmarks to GitHub.
Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org if works starts on this idea and then not move the old repo at all if this succeeds. We could then make the people who care about the benchmarks the maintainers of the new repository and contribute to it directly (which means interested people from CPython, PyPy, Pyston, IronPython, and Jython). That way we all get exposure to everyone's benchmarks and there's no more benchmark fragmentation because some people disagree with another's approach (i.e. no more benchmark silos among the Python implementations).
And just because we're talking a new repo, would it be worth considering relying on pip to grab libraries instead of embedding them in the repository, hence shrinking the overall size of the repo?
Both make sense - to benchmark against the latest lib X and to benchmark against a pinned lib X
participants (1)
-
Maciej Fijalkowski