[Python-Dev] Benchmarking Python 3.3 against Python 2.7 (wide build)

Mon Oct 1 01:12:47 CEST 2012

I am presenting the talk "Python 3.3: Trust Me, It's Better Than 2.7" as
PyCon Argentina and Brasil (and US if they accept the talk). As part of
that talk I need to be able to benchmark Python 3.3 against 2.7 (both from
tip) using the unladen benchmarks (which now include benchmarks from PyPy
that can be relatively easily ported to Python 3).

To make sure the unladen benchmarks run fine against Python 3.3, I did a
fast run of the benchmarks. I figured people might be interested in the
quick-and-dirty results on my 2 GHz Intel Core i7 MacBook Pro w/ 8 GB RAM
and no attempt to control for performance beyond not actively browsing the
web. As I said, quick-and-dirty and not authoritative; all done just to
make sure all the benchmarks could run to completion (including the django,
html5lib, and genshi benchmarks which are only on my laptop ATM until those
projects cut a release with official Python 3 support).

One thing to keep in mind is that many benchmarks use a raw str for things,
so the benchmarks often compare Python 2.7 str vs. Python 3.3 str (i.e. str
vs. unicode). While this might seem unfair, this is what real-world
comparisons in performance will be from users so it's an (somewhat unfair)
comparison that we just have to live with. I might take the time to try to
make some tests run under both raw strings and unicode so both comparisons
are available.

If you care about helping out with the benchmarks (e.g. helping spot where
the iteration counts should be higher, etc.) then head over to the
speed at mailing list.

> python3 perf.py -T --basedir ../benchmarks -f -b py3k
../cpython/builds/2.7-wide/bin/python ../cpython/builds/3.3/bin/python3.3

... output about the command line for the benchmarks ...

### 2to3 ###
0.785234 -> 0.722169: 1.09x faster

### call_method ###
Min: 0.491433 -> 0.414841: 1.18x faster
Avg: 0.493640 -> 0.416564: 1.19x faster
Significant (t=127.21)
Stddev: 0.00170 -> 0.00162: 1.0513x smaller

### call_method_slots ###
Min: 0.492749 -> 0.416280: 1.18x faster
Avg: 0.497888 -> 0.419275: 1.19x faster
Significant (t=61.72)
Stddev: 0.00433 -> 0.00237: 1.8304x smaller

### call_method_unknown ###
Min: 0.575536 -> 0.427234: 1.35x faster
Avg: 0.577286 -> 0.433428: 1.33x faster
Significant (t=66.09)
Stddev: 0.00117 -> 0.00835: 7.1621x larger

### call_simple ###
Min: 0.413011 -> 0.338923: 1.22x faster
Avg: 0.415862 -> 0.340699: 1.22x faster
Significant (t=111.94)
Stddev: 0.00223 -> 0.00134: 1.6616x smaller

### chaos ###
Min: 0.375286 -> 0.435456: 1.16x slower
Avg: 0.382798 -> 0.459515: 1.20x slower
Significant (t=-5.01)
Stddev: 0.01116 -> 0.03234: 2.8980x larger

### fastpickle ###
Min: 0.853560 -> 0.770580: 1.11x faster
Avg: 0.879498 -> 0.776249: 1.13x faster
Significant (t=8.24)
Stddev: 0.02771 -> 0.00407: 6.7995x smaller

### float ###
Min: 0.476596 -> 0.391101: 1.22x faster
Avg: 0.486164 -> 0.411553: 1.18x faster
Significant (t=9.07)
Stddev: 0.01049 -> 0.01511: 1.4411x larger

### formatted_logging ###
Min: 0.346703 -> 0.451643: 1.30x slower
Avg: 0.351218 -> 0.454626: 1.29x slower
Significant (t=-51.50)
Stddev: 0.00376 -> 0.00246: 1.5265x smaller

### genshi ###
Min: 0.275107 -> 0.294309: 1.07x slower
Avg: 0.287433 -> 0.299026: 1.04x slower
Significant (t=-3.82)
Stddev: 0.01077 -> 0.00467: 2.3044x smaller

### go ###
Min: 0.719160 -> 0.781042: 1.09x slower
Avg: 0.729322 -> 0.798135: 1.09x slower
Significant (t=-8.54)
Stddev: 0.01300 -> 0.01248: 1.0415x smaller

### hexiom2 ###
203.842661 -> 187.107363: 1.09x faster

### iterative_count ###
Min: 0.145088 -> 0.153285: 1.06x slower
Avg: 0.146369 -> 0.154425: 1.06x slower
Significant (t=-9.21)
Stddev: 0.00134 -> 0.00142: 1.0569x larger

### json_dump_v2 ###
Min: 3.512367 -> 4.040813: 1.15x slower
Avg: 3.521879 -> 4.057966: 1.15x slower
Significant (t=-64.29)
Stddev: 0.01071 -> 0.01526: 1.4247x larger

### json_load ###
Min: 1.024560 -> 0.642353: 1.60x faster
Avg: 1.025255 -> 0.644000: 1.59x faster
Significant (t=426.59)
Stddev: 0.00049 -> 0.00194: 3.9240x larger

### mako_v2 ###
Min: 0.137584 -> 0.287701: 2.09x slower
Avg: 0.140620 -> 0.293204: 2.09x slower
Significant (t=-296.14)
Stddev: 0.00243 -> 0.00272: 1.1195x larger

### meteor_contest ###
Min: 0.284739 -> 0.254285: 1.12x faster
Avg: 0.286174 -> 0.255323: 1.12x faster
Significant (t=38.02)
Stddev: 0.00124 -> 0.00133: 1.0725x larger

### nbody ###
Min: 0.491416 -> 0.336127: 1.46x faster
Avg: 0.493339 -> 0.337467: 1.46x faster
Significant (t=185.50)
Stddev: 0.00164 -> 0.00092: 1.7927x smaller

### normal_startup ###
Min: 0.639285 -> 0.898157: 1.40x slower
Avg: 0.645513 -> 0.901586: 1.40x slower
Significant (t=-90.10)
Stddev: 0.00575 -> 0.00270: 2.1309x smaller

### nqueens ###
Min: 0.399351 -> 0.429575: 1.08x slower
Avg: 0.403643 -> 0.430284: 1.07x slower
Significant (t=-9.83)
Stddev: 0.00603 -> 0.00053: 11.3092x smaller

### pathlib ###
Min: 0.137462 -> 0.170506: 1.24x slower
Avg: 0.145370 -> 0.172849: 1.19x slower
Significant (t=-11.09)
Stddev: 0.01232 -> 0.00128: 9.6403x smaller

### pidigits ###
Min: 0.400265 -> 0.379307: 1.06x faster
Avg: 0.401755 -> 0.381171: 1.05x faster
Significant (t=14.65)
Stddev: 0.00259 -> 0.00178: 1.4496x smaller

### raytrace ###
Min: 1.770596 -> 1.958350: 1.11x slower
Avg: 1.773719 -> 1.968401: 1.11x slower
Significant (t=-44.19)
Stddev: 0.00439 -> 0.00882: 2.0099x larger

### regex_effbot ###
Min: 0.076566 -> 0.098124: 1.28x slower
Avg: 0.077491 -> 0.098696: 1.27x slower
Significant (t=-54.47)
Stddev: 0.00052 -> 0.00069: 1.3227x larger

### regex_v8 ###
Min: 0.091530 -> 0.109116: 1.19x slower
Avg: 0.092308 -> 0.113627: 1.23x slower
Significant (t=-5.72)
Stddev: 0.00088 -> 0.00829: 9.4271x larger

### richards ###
Min: 0.257974 -> 0.232134: 1.11x faster
Avg: 0.259248 -> 0.234325: 1.11x faster
Significant (t=23.80)
Stddev: 0.00144 -> 0.00185: 1.2823x larger

### simple_logging ###
Min: 0.326569 -> 0.416797: 1.28x slower
Avg: 0.331694 -> 0.418844: 1.26x slower
Significant (t=-36.32)
Stddev: 0.00523 -> 0.00122: 4.3004x smaller

### spectral_norm ###
Min: 0.483011 -> 0.741558: 1.54x slower
Avg: 0.487128 -> 0.749741: 1.54x slower
Significant (t=-57.40)
Stddev: 0.00512 -> 0.00886: 1.7299x larger

### startup_nosite ###
Min: 0.220444 -> 0.374521: 1.70x slower
Avg: 0.222773 -> 0.376785: 1.69x slower
Significant (t=-176.17)
Stddev: 0.00166 -> 0.00221: 1.3331x larger

### threaded_count ###
Min: 0.171352 -> 0.151892: 1.13x faster
Avg: 0.183180 -> 0.153634: 1.19x faster
Significant (t=8.12)
Stddev: 0.00801 -> 0.00140: 5.7241x smaller

### unpack_sequence ###
Min: 0.000075 -> 0.000061: 1.23x faster
Avg: 0.000101 -> 0.000065: 1.54x faster
Significant (t=206.90)
Stddev: 0.00001 -> 0.00000: 3.2374x smaller

The following not significant results are hidden, use -v to show them:
chameleon, fannkuch, fastunpickle, regex_compile, silent_logging

### django ###
Min: 0.868956 -> 0.894571: 1.03x slower
Avg: 0.873620 -> 0.905274: 1.04x slower
Significant (t=-6.97)
Stddev: 0.00313 -> 0.00966: 3.0912x larger

### genshi ###
Min: 0.269615 -> 0.286348: 1.06x slower
Avg: 0.272206 -> 0.290708: 1.07x slower
Significant (t=-12.29)
Stddev: 0.00253 -> 0.00526: 2.0793x larger

### html5lib ###
12.279808 -> 11.862586: 1.04x faster
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120930/ea3e95d3/attachment.html>