[Python-Dev] pystone rant [was] proposal+patch: sys.gettickeraccumulation()

[Raymond Hettinger]

(Old message)
> > =)  Parrotbench and PyBench would be enough for me to sign off on
any
> > performance hit.  There is also pystone.

pystone is good a predicting the relative performance of python apps on
difference hardware/software environments.

As a tool for evaluating proposed language changes, it completely sucks
and should *never* be used for anything other than an extra
informational datapoint.

The foremost issue is that it exercises only a tiny portion of the
language.  

Its design makes it worse than totally useless for evaluating eval-loop
overhead.  It runs a loop twice, once with benchmarking code and once
without.  Only the difference is reported.  In theory, that means that
all eval-loop speedups and slowdowns are netted out of the result.  In
practice, the reported result may indicate exactly the opposite of
reality because the empty loop has vastly different cache effects than
the benchmarked loop.

For useful timings, run timeit on the specific feature in question.
Then check for overall impact using pybench, parrotbench, and
test_decimal.


Raymond