[pypy-dev] speed.pypy.org launched

René Dudfield renesd at gmail.com
Mon Mar 1 11:15:22 CET 2010

On Fri, Feb 26, 2010 at 9:59 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Carl Friedrich Bolz, 25.02.2010 18:38:
>> On 02/25/2010 04:10 PM, Miquel Torres wrote:
>>> As some of you already know, there is a new performance site online:
>>> speed.pypy.org.
>> [...]
>> I'm quite impressed, this is very cool work and a good improvement over
>> the current plots. Thanks for doing this!
>> One thing that I would really find useful are error bars in the timeline
>> view. This would help us judge whether up-and-down movements are within
>> the margin of randomness, or whether it is a real effect. I don't know
>> how annoying they are to implement though, no clue whether the plot lib
>> supports that. There should be enough information about errors in the
>> json files, as far as I remember.
> The standard deviation is commonly considered meaningless for benchmark
> results. All that really matters is the fastest run, everything else is
> just fog.
> Read the docs on timeit, for example.
> Stefan


For some sets of problems, the first run is very important.  For
example, where you only want to process the particular data once.

For others the Worst performance is the most important.  For example
'real time' like programs which require the runtime only takes at
maximum N where N is measured.

Running something 10 times and removing outliers fails for these two
important cases.


More information about the Pypy-dev mailing list