[pypy-dev] New speed.pypy.org version

Paolo Giarrusso p.giarrusso at gmail.com
Fri Jun 25 22:48:59 CEST 2010


On Fri, Jun 25, 2010 at 17:53, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hey Paolo.
>
> While in general I agree with you, this is not exactly science, I
> still think it's giving somewhat an impression what's going on to
> outsiders.
As long as no important scholars look at that part, and it's just me,
yes. When that happen, you probably lose some credibility. No matter
if speed.pypy.org is made by people external to the team.

> Inside, we still look mostly at particular benchmarks.

But if a change improves one benchmark and worsens another, you need
some summary.

> I'm
> not sure having any convoluted (at least to normal people) metric
> while summarizing would help, maybe.
You're talking to programmers, not to street people. Even before
knowing why I should use the geomean here, I never felt too confused
by people using it, even without explanation.

The geometric mean is just a mean. And it's the only way to get an
average performance ratio: how much faster is PyPy than CPython, on
these benchmarks, considered all with equal weight? If you want, put
just this on the graph, and the term "geometric mean" in some note.

> Speaking a bit on Miguel's
> behalf, feel free to implement this as a feature on codespeed (it's an
> open source project after all), you can fork it on github
> http://github.com/tobami/codespeed.

Of course I can, so your answer is valid, but that's not my plan,
sorry - the difference of effort needed to me and to him is huge. If I
had time to spend, I'd hack PyPy itself instead.

Best regards
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/



More information about the Pypy-dev mailing list