[Python-Dev] Re: Are we collecting benchmark results across machines

[Guido van Rossum]

> On this box (3.05G P4, 512M, Fedora Core 1):
> 
> Using Python2.3.3:
> real    0m16.027s
> user    0m14.940s
> sys     0m0.280s
> 
> Pystone(1.1) time for 50000 passes = 1.43
> This machine benchmarks at 34965 pystones/second
> 
> 
> Using current-CVS python (and the bytecode compiled with 2.3.3):
> real    0m14.556s
> user    0m14.130s
> sys     0m0.280s
> 
> Pystone(1.1) time for 50000 passes = 1.41
> This machine benchmarks at 35461 pystones/second
> 
> (Hm: should the default number of passes for pystone be raised? One and
> a half seconds seems rather a short time for a benchmark...)

Probably.  Maybe you can try the new pystone from CVS which has a
command line option.  (If I were to write it over I'd use the strategy
from timeit, which has a self-adjusting strategy to pick an
appropriate number of loops.)

I multiplied the two numbers (pystones/sec and parrotbench seconds)
for your two runs and found the product to be much higher for your
first run than for the second.  This is suspicious (perhaps points at
too much variation in the pystone timing); for contrast, Skip's two
runs before and after rebooting give a product within 0.05 percent of
each other).

Hm, of course this could also have to do with Python versions.  Let me
try...

Yes, it looks like for me, on one machine I have handy here, Python
2.3.3 gives a higher product than current CVS (though not by as much as
you measured).

So what does this product mean?  It's higher if pystone is faster, or
parrotbench is slower.  Any factor that makes both faster (or slower)
by the same amount has no effect.  Ah (thinking aloud here) the unit
is "pystones".  Not pystones per second, just pystones.  And this
leads to an interpretation: it is how many pystone loops execute in
the time needed to complete the parrotbench benchmark (so the unit is
really "pystones per parrotbench").

So what makes a machine run more pystones in a parrotbench?

I don't know enough about CPUs and caches to draw a conclusion, and
I've got to start doing real work today...

--Guido van Rossum (home page: http://www.python.org/~guido/)