[Python-Dev] Re: Are we collecting benchmark results across machines

Kurt B. Kaiser kbk at shore.net
Fri Jan 2 19:31:07 EST 2004

Guido van Rossum <guido at python.org> writes:

> Hm...  My IBM T40 with 1.4 GHz P(M) reports 15.608.  I bet the caches
> are more similar, and affect performance more than CPU speed...

You have a 32K L1 and a 1024K (!) L2.  What a great machine!

As an example of the other end of the spectrum, I'm running current,
but low end hardware: a 2.2 GHz Celeron, 400 MHz FSB.  256MB DDR SDRAM.
Verified working as expected by Intel Processor Speed Test.

My L1 is 12K trace, 8K data.
My L2 is only 128K, and when there's a miss there's no L3 to fall back on.
Cost for processor and mobo, new, $120.

I find this setup pretty snappy for what I do on it: development and
home server.  It's definitely not my game machine :-)

Python 2.2.1 (#1, Oct 17 2003, 16:36:36) 
[GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3
[best of 3]:
Pystone(1.1) time for 10000 passes = 0.93
This machine benchmarks at 10752.7 pystones/second

Python 2.3.3 (#15, Jan  2 2004, 14:39:36) 
[best of 3]:
Pystone(1.1) time for 50000 passes = 3.46
This machine benchmarks at 14450.9 pystones/second

Python 2.4a0 (#40, Jan  1 2004, 22:22:45) [current cvs]
[best of 3]:
Pystone(1.1) time for 50000 passes = 2.91
This machine benchmarks at 17182.1 pystones/second

(but see the p.s. below)

Now the parrotbench, version 1.04.  [make extra passes to get .pyo

First, python 2.3.3:
best 3: 31.1/31.8/32.3

Next, python 2.4a0, current cvs:
best 3: 31.8/31.9/32.1

Since I noticed quite different ratios between the individual tests
compared to what was posted by Seo Sanghyeon on the pypy list, here's
my numbers (2.4a0):

hydra /home/kbk/PYTHON/python/nondist/sandbox/parrotbench$ make times
for i in 0 1 2 3 4 5 6; do  echo b$i.py;  time /home/kbk/PYSRC/python b$i.py >@out$i;  cmp @out$i out$i;  done
    5.48s real     5.30s user     0.05s system
    1.36s real     1.22s user     0.10s system
    0.44s real     0.42s user     0.04s system
    2.01s real     1.94s user     0.04s system
    1.69s real     1.63s user     0.05s system
    4.80s real     4.73s user     0.02s system
    1.84s real     1.56s user     0.26s system

I notice that some of these tests are a little faster on 2.3.3 while
others are faster on 2.4, resulting in the overall time being about
the same on both releases.

N.B. compiling Python w/o the stack protector doesn't make a
noticeable difference ;-)

There may be some other problem with this box that I haven't yet
discovered, but right now I'm blaming the tiny cache for performance
being 2 - 3 x lower than expected from the clock rate, compared
to what others are getting.


p.s. I saw quite a large outlier on 2.4 pystone when I first tried
it.  I didn't believe it, but was able to scroll back and clip it:

Python 2.4a0 (#40, Jan  1 2004, 22:22:45) 
[GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3
Type "help", "copyright", "credits" or "license" for more information.
>>> from test.pystone import main
>>> main(); main(); main()
Pystone(1.1) time for 50000 passes = 4.22
This machine benchmarks at 11848.3 pystones/second
Pystone(1.1) time for 50000 passes = 4.21
This machine benchmarks at 11876.5 pystones/second
Pystone(1.1) time for 50000 passes = 4.21
This machine benchmarks at 11876.5 pystones/second

This is 30% lower than the rate quoted above.  I haven't been able
to duplicate it.  Maybe the OS or X was doing something which
tied up the cache.  This is a fairly lightly loaded machine 
running X, Ion, and emacs.

I've also seen 20% variations in the 2.2.1 pystone benchmark.

It seems to me that this benchmark is pretty cache sensitive and
should be done on an unloaded system, preferable x/o X, and with
the results averaged over many random trials if comparisions are
desired, especially if the cache is small.

I don't see the same variation in the parrotbench.  It's just
consistently low for this box.

More information about the Python-Dev mailing list