[New-bugs-announce] [issue7029] Improve pybench

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson <kristjan at ccpgames.com>:

The attached patch contains suggested fixes to pybench.py:
1) add a processtime timer for windows
2) fix a bug in timer selection: timer wasn't put in 'self'
3) Collect 'gross' times for each round
4) show statistics for per-round sums, both for sum of "adjusted" times 
for each round, and 'gross' time per round.

The last one is important.  Traditionally, the focus would be on 
individual tests.  The series of individual timings for each test would 
be  examined, min and afverage found.  These minima and averages would 
then be summed up to get a total time.  These results are very noisy.  
In addition, the sum of individual minima is not a linear operator that 
has any meaning and is thus extremely noisy.

Looking at the minimum and average of the sum is a much stabler 
indication of total benchmark performance.

Another thing that I found when working with this, is that using 
"calibration" significantly adds to noise and can produce incorrect 
results.  It adds a level of non-linearity to the results that can 
appear very strange.  Typically the 'overhead' is so low that we should 
consider skipping the calibration.  The purpose of the benchmark must be 
to measure the performance of python in context.  The meaning of 
individual operations in isolation are pretty meaningless.  Although I 
didn't change the default number of calibration runs from 10, I would 
suggest documenting that it may be useful to turn it to 0 to increase 
predictability and get numbers indicative of real performance.

----------
components: Demos and Tools
files: pybench.patch
keywords: patch
messages: 93416
nosy: krisvale
severity: normal
status: open
title: Improve pybench
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file15016/pybench.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7029>
_______________________________________