Thanks a lot for the benchmarks. So we still have to wait , i guess. Would be nice a blogpost with PyPy 2 vs PyPy3 vs CPython 2 and 3. On Wed, Oct 29, 2014 at 12:53 AM, Philip Jenvey <pjenvey@underboss.org> wrote:
On Oct 24, 2014, at 3:02 PM, Phyo Arkar <phyo.arkarlwin@gmail.com> wrote:
Congrats! I am using Pypy 2.4 for tornado based web development , performance improvement is the real deal there.
i am waiting to switch to python3 and its becoming near! Its when pypy3 3.4 is ready (ie , Python3 is only starting to interesting after 3.4.x due to asyncio module) .
I have a question: In CPython , python 2 vs python 3 is around 10-30% performance difference. In PyPy world is there such difference? Same performance between pypy2 and pypy3 ?
Hi there,
PyPy3 performance isn’t quite as good as PyPy2’s yet. Our main focus has been on compatibility, as you said yourself, you probably won’t switch until a later version of Python 3’s supported.
The benchmark infrastructure for PyPy3 is in need of help. Pypy's own benchmark suite used by http://speed.pypy.org isn't Python 3 compatible yet, nor is the site really setup to run PyPy3 benchmark runs.
A few years ago there were plans for the creation of a shared between VMs/the PSF version of speed.pypy.org (to be http://speed.python.org), unfortunately these efforts seemed to have failed or at the very least stalled indefinitely.
However the benchmark suite intended for this new site supports Python 3. It’s similar to pypy’s own suite, albeit with a few different benchmarks and unfortunately separately maintained. Here’s output from a run of it with CPython 3.2.5 vs PyPy3 2.4.0: though this was ran on a laptop which isn’t going to produce the most stable numbers but it'll give you some idea:
### 2to3 ### 8.652382 -> 9.627286: 1.11x slower
### call_method ### Min: 0.385220 -> 0.011149: 34.55x faster Avg: 0.412039 -> 0.012474: 33.03x faster Significant (t=293.52) Stddev: 0.01644 -> 0.00279: 5.8934x smaller
### call_method_slots ### Min: 0.394147 -> 0.011669: 33.78x faster Avg: 0.419456 -> 0.013503: 31.06x faster Significant (t=185.89) Stddev: 0.02655 -> 0.00326: 8.1550x smaller
### call_method_unknown ### Min: 0.416863 -> 0.016139: 25.83x faster Avg: 0.444017 -> 0.019560: 22.70x faster Significant (t=187.44) Stddev: 0.02645 -> 0.00835: 3.1669x smaller
### call_simple ### Min: 0.324141 -> 0.011182: 28.99x faster Avg: 0.343808 -> 0.013248: 25.95x faster Significant (t=235.96) Stddev: 0.01667 -> 0.00406: 4.1072x smaller
### chameleon ### Min: 0.061195 -> 0.010301: 5.94x faster Avg: 0.066450 -> 0.015595: 4.26x faster Significant (t=52.46) Stddev: 0.00668 -> 0.00982: 1.4710x larger
### chaos ### Min: 0.340240 -> 0.011240: 30.27x faster Avg: 0.366752 -> 0.018671: 19.64x faster Significant (t=60.31) Stddev: 0.01713 -> 0.03704: 2.1620x larger
### django_v2 ### Min: 0.692480 -> 0.029994: 23.09x faster Avg: 0.731315 -> 0.037337: 19.59x faster Significant (t=173.98) Stddev: 0.02693 -> 0.00837: 3.2160x smaller
### etree_generate ### Min: 0.599923 -> 1.087517: 1.81x slower Avg: 0.636505 -> 1.149231: 1.81x slower Significant (t=-77.16) Stddev: 0.02478 -> 0.03993: 1.6114x larger
### etree_parse ### Min: 0.296519 -> 0.931630: 3.14x slower Avg: 0.331876 -> 1.010314: 3.04x slower Significant (t=-78.30) Stddev: 0.03799 -> 0.04807: 1.2652x larger
### etree_process ### Min: 0.421074 -> 0.739647: 1.76x slower Avg: 0.454918 -> 0.798387: 1.76x slower Significant (t=-55.61) Stddev: 0.03127 -> 0.03048: 1.0260x smaller
### fannkuch ### Min: 1.744042 -> 0.179821: 9.70x faster Avg: 1.815167 -> 0.193547: 9.38x faster Significant (t=293.69) Stddev: 0.03619 -> 0.01465: 2.4707x smaller
### fastpickle ### Min: 0.671650 -> 109.014578: 162.31x slower Avg: 0.708033 -> 115.379135: 162.96x slower Significant (t=-299.42) Stddev: 0.02026 -> 2.70799: 133.6909x larger
### fastunpickle ### Min: 0.586665 -> 55.736337: 95.01x slower Avg: 0.625024 -> 56.833369: 90.93x slower Significant (t=-357.75) Stddev: 0.02488 -> 1.11069: 44.6438x larger
### float ### Min: 0.358756 -> 0.026437: 13.57x faster Avg: 0.387649 -> 0.041561: 9.33x faster Significant (t=106.89) Stddev: 0.02159 -> 0.00762: 2.8351x smaller
### formatted_logging ### Min: 0.306111 -> 0.086534: 3.54x faster Avg: 0.331045 -> 0.116844: 2.83x faster Significant (t=32.06) Stddev: 0.02016 -> 0.04273: 2.1198x larger
### go ### Min: 0.640692 -> 0.110813: 5.78x faster Avg: 0.686035 -> 0.153473: 4.47x faster Significant (t=81.23) Stddev: 0.02366 -> 0.03987: 1.6853x larger
### hexiom2 ### Min: 168.499830 -> 13.845624: 12.17x faster Avg: 170.731699 -> 14.320609: 11.92x faster Significant (t=68.55) Stddev: 3.15634 -> 0.67173: 4.6988x smaller
### iterative_count ### Min: 0.155776 -> 0.001828: 85.22x faster Avg: 0.169719 -> 0.001959: 86.63x faster Significant (t=66.83) Stddev: 0.01775 -> 0.00017: 104.6217x smaller
### json_dump ### Min: 0.563337 -> 2.088108: 3.71x slower Avg: 0.600209 -> 2.212642: 3.69x slower Significant (t=-92.38) Stddev: 0.01963 -> 0.12185: 6.2066x larger
### json_dump_v2 ### Min: 3.703154 -> 12.685391: 3.43x slower Avg: 3.906239 -> 13.462007: 3.45x slower Significant (t=-73.61) Stddev: 0.16381 -> 0.90320: 5.5139x larger
### json_load ### Min: 0.637112 -> 0.284530: 2.24x faster Avg: 0.691114 -> 0.340403: 2.03x faster Significant (t=48.13) Stddev: 0.03443 -> 0.03834: 1.1134x larger
### mako ### Min: 0.068892 -> 0.023433: 2.94x faster Avg: 0.073817 -> 0.034535: 2.14x faster Significant (t=45.74) Stddev: 0.01044 -> 0.00868: 1.2032x smaller
### mako_v2 ### Min: 0.055735 -> 0.023094: 2.41x faster Avg: 0.060464 -> 0.034531: 1.75x faster Significant (t=39.36) Stddev: 0.01003 -> 0.01079: 1.0764x larger
### meteor_contest ### Min: 0.234554 -> 0.101157: 2.32x faster Avg: 0.253522 -> 0.112062: 2.26x faster Significant (t=45.33) Stddev: 0.01735 -> 0.01364: 1.2717x smaller
### nbody ### Min: 0.332543 -> 0.024746: 13.44x faster Avg: 0.352898 -> 0.026261: 13.44x faster Significant (t=111.01) Stddev: 0.02059 -> 0.00298: 6.9071x smaller
### normal_startup ### Min: 0.974583 -> 1.578062: 1.62x slower Avg: 1.011902 -> 1.640190: 1.62x slower Significant (t=-144.15) Stddev: 0.01753 -> 0.02535: 1.4458x larger
### nqueens ### Min: 0.357568 -> 0.052418: 6.82x faster Avg: 0.386644 -> 0.055879: 6.92x faster Significant (t=98.78) Stddev: 0.02223 -> 0.00816: 2.7244x smaller
### pathlib ### Min: 0.119940 -> 0.384461: 3.21x slower Avg: 0.130969 -> 0.504688: 3.85x slower Significant (t=-142.33) Stddev: 0.01389 -> 0.03912: 2.8161x larger
### pickle_dict ### Min: 0.496007 -> 8.778371: 17.70x slower Avg: 0.522927 -> 9.051905: 17.31x slower Significant (t=-306.08) Stddev: 0.01866 -> 0.19615: 10.5126x larger
### pickle_list ### Min: 0.295601 -> 6.132281: 20.75x slower Avg: 0.315390 -> 6.327789: 20.06x slower Significant (t=-215.11) Stddev: 0.01909 -> 0.19672: 10.3048x larger
### pidigits ### Min: 0.345745 -> 0.297060: 1.16x faster Avg: 0.362924 -> 0.316708: 1.15x faster Significant (t=13.14) Stddev: 0.01719 -> 0.01798: 1.0456x larger
### raytrace ### Min: 1.527975 -> 0.046997: 32.51x faster Avg: 1.575889 -> 0.052239: 30.17x faster Significant (t=344.42) Stddev: 0.02992 -> 0.00911: 3.2830x smaller
### regex_compile ### Min: 0.462712 -> 0.076845: 6.02x faster Avg: 0.489184 -> 0.159849: 3.06x faster Significant (t=11.67) Stddev: 0.01954 -> 0.19853: 10.1609x larger
### regex_effbot ### Min: 0.061423 -> 0.034389: 1.79x faster Avg: 0.065657 -> 0.041867: 1.57x faster Significant (t=16.14) Stddev: 0.00658 -> 0.00808: 1.2285x larger
### richards ### Min: 0.201109 -> 0.004319: 46.56x faster Avg: 0.215034 -> 0.004968: 43.28x faster Significant (t=91.42) Stddev: 0.01619 -> 0.00140: 11.5878x smaller
### silent_logging ### Min: 0.080098 -> 0.007613: 10.52x faster Avg: 0.085028 -> 0.009930: 8.56x faster Significant (t=43.10) Stddev: 0.00980 -> 0.00747: 1.3113x smaller
### simple_logging ### Min: 0.296057 -> 0.081953: 3.61x faster Avg: 0.315012 -> 0.110826: 2.84x faster Significant (t=33.22) Stddev: 0.01866 -> 0.03925: 2.1031x larger
### spectral_norm ### Min: 0.365987 -> 0.016084: 22.75x faster Avg: 0.393062 -> 0.018651: 21.07x faster Significant (t=124.58) Stddev: 0.02014 -> 0.00679: 2.9647x smaller
### startup_nosite ### Min: 0.485740 -> 0.613260: 1.26x slower Avg: 0.518384 -> 0.654354: 1.26x slower Significant (t=-49.49) Stddev: 0.01842 -> 0.02038: 1.1066x larger
### telco ### Min: 0.773569 -> 0.043440: 17.81x faster Avg: 0.800002 -> 0.049813: 16.06x faster Significant (t=276.68) Stddev: 0.01300 -> 0.01409: 1.0844x larger
### threaded_count ### Min: 0.167072 -> 0.005613: 29.76x faster Avg: 0.193020 -> 0.005944: 32.47x faster Significant (t=88.19) Stddev: 0.01499 -> 0.00036: 41.4676x smaller
### tornado_http ### Min: 0.307076 -> 0.390125: 1.27x slower Avg: 0.378676 -> 0.459946: 1.21x slower Significant (t=-8.21) Stddev: 0.04109 -> 0.09005: 2.1916x larger
### unpack_sequence ### Min: 0.000052 -> 0.000057: 1.10x slower Avg: 0.000077 -> 0.000404: 5.21x slower Significant (t=-16.30) Stddev: 0.00011 -> 0.00447: 41.4670x larger
### unpickle_list ### Min: 0.455635 -> 0.963103: 2.11x slower Avg: 0.514364 -> 1.017570: 1.98x slower Significant (t=-79.98) Stddev: 0.02640 -> 0.03581: 1.3564x larger
So PyPy3’s generally faster, except on startup and tornado_http, where it’s only slightly slower.
Then it’s quite a bit slower on “mini interpreters” such as etree/json/Pickle. These modules also happen to be pure/mostly pure python versions on PyPy vs C versions used by CPython. Pickle looks particularly bad, I have a guess that it’s due to removal of an optimization in marshal that happened during py3k compatibility work, see:
https://bitbucket.org/pypy/pypy/src/a245775c1e3687662879fe57964a5ece689cd781...
and it’s still slower than PyPy2 on many things.
Anyway, to reiterate we’ve pretty busy with compatibility work. We could use some help with benchmarking infrastructure =]
-- Philip Jenvey
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev