[Python-Dev] Python Benchmarks

[Fredrik Lundh]

M.-A. Lemburg wrote:

> AFAIK, there were no real issues with pybench, only with the
> fact that time.clock() (the timer used by pybench) is wall-time
> on Windows and thus an MP3-player running in the background
> will cause some serious noise in the measurements 

oh, please; as I mentioned back then, PyBench reported massive slowdowns 
and huge speedups in code that wasn't touched, gave unrepeatable results 
on most platforms, and caused us to waste quite some time investigating 
potential regressions from 2.4 that simply didn't exist.

of about a dozen claimed slowdowns when comparing 2.4 to 2.5a2 on 
several platforms, only *one* slowdown could be independently confirmed 
with other tools.

and when we fixed that, and ended up with an implementation that was 
*faster* than in 2.4, PyBench didn't even notice the speedup.

the fact is that the results for individual tests in PyBench are 100% 
unreliable.  I have no idea why.

the accumulated result may be somewhat useful (at least after the "use 
minimum time instead of average" changes), but I wouldn't use it for any 
serious purpose.  at least PyStone is unusable in a well-defined way ;-)

</F>