Hi,

I updated all benchmarks of the CPython Benchmark Suite to use my perf module. So you get timings of all individual run all *all* benchmarks and store them in JSON to analyze them in detail. Each benchmark has a full CLI, for example it gets a new --output option to store result as JSON directly. But it also gets nice funtions like --hist for histogram, --stats for statistics, etc.

The two remaining questions are:

• Should it support --track_memory? it doesn't support it correctly right now, it's less precise than before (memory is tracked directly in worker processes, no more by the main process) => discuss this point in my previous email

• Should we remove vendor copies of libraries and work with virtual environments? Not all libraries are available on PyPI :-/ See the requirements.txt file and TODO.

My repository: https://hg.python.org/sandbox/benchmarks_perf

I would like to push my work as a single giant commit.

Brett also proposed me to move the benchmarks repository to GitHub (and so convert it to Git). I don't know if it's appropriate to do all these things at once? What do you think?

Reminder: My final goal is to merge again all benchmarks suites (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!

Victor