[Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

Brett Cannon brett at python.org
Fri Jul 29 13:03:10 EDT 2016

On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner at gmail.com>

> Hi,
> I updated all benchmarks of the CPython Benchmark Suite to use my perf
> module. So you get timings of all individual run all *all* benchmarks
> and store them in JSON to analyze them in detail. Each benchmark has a
> full CLI, for example it gets a new --output option to store result as
> JSON directly. But it also gets nice funtions like --hist for
> histogram, --stats for statistics, etc.
> The two remaining questions are:
> * Should it support --track_memory? it doesn't support it correctly
> right now, it's less precise than before (memory is tracked directly
> in worker processes, no more by the main process) => discuss this
> point in my previous email

I don't have an opinion as I have never gotten to use the old feature.

> * Should we remove vendor copies of libraries and work with virtual
> environments? Not all libraries are available on PyPI :-/ See the
> requirements.txt file and TODO.

If they are not on PyPI then we should just drop the benchmark. And I say
we do use virtual environments to keep the repo size down.

> My repository:
> https://hg.python.org/sandbox/benchmarks_perf
> I would like to push my work as a single giant commit.
> Brett also proposed me to move the benchmarks repository to GitHub
> (and so convert it to Git). I don't know if it's appropriate to do all
> these things at once? What do you think?

I say just start a new repo from scratch. There isn't a ton of magical
history in the  hg repo that I think we need to have carried around in the
git repo. Plus if we stop shipping project source with the repo then it
will be immensely smaller if we start from scratch.

> Reminder: My final goal is to merge again all benchmarks suites
> (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!

I hope this happens!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20160729/619d6631/attachment.html>

More information about the Speed mailing list