On Thu, 28 Jul 2016 at 10:25 Victor Stinner firstname.lastname@example.org wrote:
I updated all benchmarks of the CPython Benchmark Suite to use my perf module. So you get timings of all individual run all *all* benchmarks and store them in JSON to analyze them in detail. Each benchmark has a full CLI, for example it gets a new --output option to store result as JSON directly. But it also gets nice funtions like --hist for histogram, --stats for statistics, etc.
The two remaining questions are:
- Should it support --track_memory? it doesn't support it correctly
right now, it's less precise than before (memory is tracked directly in worker processes, no more by the main process) => discuss this point in my previous email
I don't have an opinion as I have never gotten to use the old feature.
- Should we remove vendor copies of libraries and work with virtual
environments? Not all libraries are available on PyPI :-/ See the requirements.txt file and TODO.
If they are not on PyPI then we should just drop the benchmark. And I say we do use virtual environments to keep the repo size down.
My repository: https://hg.python.org/sandbox/benchmarks_perf
I would like to push my work as a single giant commit.
Brett also proposed me to move the benchmarks repository to GitHub (and so convert it to Git). I don't know if it's appropriate to do all these things at once? What do you think?
I say just start a new repo from scratch. There isn't a ton of magical history in the hg repo that I think we need to have carried around in the git repo. Plus if we stop shipping project source with the repo then it will be immensely smaller if we start from scratch.
Reminder: My final goal is to merge again all benchmarks suites (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!
I hope this happens!