[Python-checkins] r46146 - sandbox/trunk/rjsh-pybench/pybench.py

[Steve Holden]

M.-A. Lemburg wrote:
> steve.holden wrote:
> 
>>Author: steve.holden
>>Date: Tue May 23 21:21:00 2006
>>New Revision: 46146
>>
>>Modified:
>>   sandbox/trunk/rjsh-pybench/pybench.py
>>Log:
>>Use the appropriate clock for the platform.
>>Default the number of calibration runs to 0 (can set with -C)
> 
> 
> Defaulting to 0 is a bad idea, since you basically turn
> off calibration altogether.
> 
> Note that calibration is *very* important so
> that only the operation itself is timed, not the setup,
> loop and other logic not related to the test run.
> 
I understand the purpose of the calibration and agree that omitting 
calibration could falsely skew the operation times. I don't feel the 
operation times are a particularly valuable performance parameter anyway 
- nobody is going to say "ah, well I'll use dict lookup because it's 
faster than instance creation".

It may not be appropriate to check back into the trunk a version with 
the default number of calibration runs at zero. However, repeated 
testing has shown that there is sufficient variability in calibration 
times to throw some results into confusion on some platforms. Comparing 
a run with calibration to a run without, for example, does not (on 
Windows, at least) yield the uniform speedup I should have expected.

I therefore think we should at least retain the option to ignore the 
operation times and omit calibration as it will give a more reproducible 
measure of absolute execution speed.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Love me, love my blog  http://holdenweb.blogspot.com
Recent Ramblings     http://del.icio.us/steve.holden