[Python-Dev] Python Benchmarks

[Andrew Dalke]

On 6/2/06, Terry Reedy <tjreedy at udel.edu> wrote:
> Hardly a setting in which to run comparison tests, seems to me.

The point though was to show that the time distribution is non-Gaussian,
so intuition based on that doesn't help.

> > Using the minimum looks like the way to go for calibration.
>
> Or possibly the median.

Why?  I can't think of why that's more useful than the minimum time.

Given an large number of samples the difference between the
minimum and the median/average/whatever is mostly providing
information about the background noise, which is pretty irrelevent
to most benchmarks.

> But even better, the way to go to run comparison timings is to use a system
> with as little other stuff going on as possible.  For Windows, this means
> rebooting in safe mode, waiting until the system is quiescent, and then run
> the timing test with *nothing* else active that can be avoided.

A reason I program in Python is because I want to get work done and not
deal with stoic purity.  I'm not going to waste all that time (or money to buy
a new machine) just to run a benchmark.

Just how much more accurate would that be over the numbers we get
now.  Have you tried it?  What additional sensitivity did you get and was
the extra effort worthwhile?

> Even then, I would look at the distribution of times for a given test to
> check for anomalously high values that should be tossed.  (This can be
> automated somewhat.)

I say it can be automated completely.  Toss all but the lowest.
It's the one with the least noise overhead.

I think fitting the smaller data points to a gamma distribution might
yield better (more reproducible and useful) numbers but I know my
stats ability is woefully decayed so I'm not going to try.  My observation
is that the shape factor is usually small so in a few dozen to a hundred
samples there's a decent chance of getting a time with minimal noise
overhead.

                                Andrew
                                dalke at dalkescientific.com