[Python-Dev] Python Benchmarks
Fredrik Lundh
fredrik at pythonware.com
Wed May 31 19:47:20 CEST 2006
M.-A. Lemburg wrote:
> AFAIK, there were no real issues with pybench, only with the
> fact that time.clock() (the timer used by pybench) is wall-time
> on Windows and thus an MP3-player running in the background
> will cause some serious noise in the measurements
oh, please; as I mentioned back then, PyBench reported massive slowdowns
and huge speedups in code that wasn't touched, gave unrepeatable results
on most platforms, and caused us to waste quite some time investigating
potential regressions from 2.4 that simply didn't exist.
of about a dozen claimed slowdowns when comparing 2.4 to 2.5a2 on
several platforms, only *one* slowdown could be independently confirmed
with other tools.
and when we fixed that, and ended up with an implementation that was
*faster* than in 2.4, PyBench didn't even notice the speedup.
the fact is that the results for individual tests in PyBench are 100%
unreliable. I have no idea why.
the accumulated result may be somewhat useful (at least after the "use
minimum time instead of average" changes), but I wouldn't use it for any
serious purpose. at least PyStone is unusable in a well-defined way ;-)
</F>
More information about the Python-Dev
mailing list