[pypy-dev] PyPy3 2.4.0 released

Thu Oct 30 09:23:34 CET 2014

Thanks a lot for the benchmarks.

So we still have to wait , i guess.
Would be nice a blogpost with PyPy 2 vs PyPy3 vs CPython 2 and 3.

On Wed, Oct 29, 2014 at 12:53 AM, Philip Jenvey <pjenvey at underboss.org> wrote:
>
> On Oct 24, 2014, at 3:02 PM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
>
>> Congrats!
>> I am using Pypy 2.4 for tornado based web development , performance
>> improvement is the real deal there.
>>
>> i am waiting to switch to python3 and its becoming near! Its when
>> pypy3 3.4 is ready (ie , Python3 is only starting to interesting after
>> 3.4.x due to asyncio module) .
>>
>> I have a question:
>> In CPython , python 2 vs python 3 is around 10-30% performance
>> difference. In PyPy world is there such difference? Same performance
>> between pypy2 and pypy3 ?
>
> Hi there,
>
> PyPy3 performance isn’t quite as good as PyPy2’s yet. Our main focus has been on compatibility, as you said yourself, you probably won’t switch until a later version of Python 3’s supported.
>
> The benchmark infrastructure for PyPy3 is in need of help. Pypy's own benchmark suite used by http://speed.pypy.org isn't Python 3 compatible yet, nor is the site really setup to run PyPy3 benchmark runs.
>
> A few years ago there were plans for the creation of a shared between VMs/the PSF version of speed.pypy.org (to be http://speed.python.org), unfortunately these efforts seemed to have failed or at the very least stalled indefinitely.
>
> However the benchmark suite intended for this new site supports Python 3. It’s similar to pypy’s own suite, albeit with a few different benchmarks and unfortunately separately maintained. Here’s output from a run of it with CPython 3.2.5 vs PyPy3 2.4.0: though this was ran on a laptop which isn’t going to produce the most stable numbers but it'll give you some idea:
>
>
> ### 2to3 ###
> 8.652382 -> 9.627286: 1.11x slower
>
> ### call_method ###
> Min: 0.385220 -> 0.011149: 34.55x faster
> Avg: 0.412039 -> 0.012474: 33.03x faster
> Significant (t=293.52)
> Stddev: 0.01644 -> 0.00279: 5.8934x smaller
>
> ### call_method_slots ###
> Min: 0.394147 -> 0.011669: 33.78x faster
> Avg: 0.419456 -> 0.013503: 31.06x faster
> Significant (t=185.89)
> Stddev: 0.02655 -> 0.00326: 8.1550x smaller
>
> ### call_method_unknown ###
> Min: 0.416863 -> 0.016139: 25.83x faster
> Avg: 0.444017 -> 0.019560: 22.70x faster
> Significant (t=187.44)
> Stddev: 0.02645 -> 0.00835: 3.1669x smaller
>
> ### call_simple ###
> Min: 0.324141 -> 0.011182: 28.99x faster
> Avg: 0.343808 -> 0.013248: 25.95x faster
> Significant (t=235.96)
> Stddev: 0.01667 -> 0.00406: 4.1072x smaller
>
> ### chameleon ###
> Min: 0.061195 -> 0.010301: 5.94x faster
> Avg: 0.066450 -> 0.015595: 4.26x faster
> Significant (t=52.46)
> Stddev: 0.00668 -> 0.00982: 1.4710x larger
>
> ### chaos ###
> Min: 0.340240 -> 0.011240: 30.27x faster
> Avg: 0.366752 -> 0.018671: 19.64x faster
> Significant (t=60.31)
> Stddev: 0.01713 -> 0.03704: 2.1620x larger
>
> ### django_v2 ###
> Min: 0.692480 -> 0.029994: 23.09x faster
> Avg: 0.731315 -> 0.037337: 19.59x faster
> Significant (t=173.98)
> Stddev: 0.02693 -> 0.00837: 3.2160x smaller
>
> ### etree_generate ###
> Min: 0.599923 -> 1.087517: 1.81x slower
> Avg: 0.636505 -> 1.149231: 1.81x slower
> Significant (t=-77.16)
> Stddev: 0.02478 -> 0.03993: 1.6114x larger
>
> ### etree_parse ###
> Min: 0.296519 -> 0.931630: 3.14x slower
> Avg: 0.331876 -> 1.010314: 3.04x slower
> Significant (t=-78.30)
> Stddev: 0.03799 -> 0.04807: 1.2652x larger
>
> ### etree_process ###
> Min: 0.421074 -> 0.739647: 1.76x slower
> Avg: 0.454918 -> 0.798387: 1.76x slower
> Significant (t=-55.61)
> Stddev: 0.03127 -> 0.03048: 1.0260x smaller
>
> ### fannkuch ###
> Min: 1.744042 -> 0.179821: 9.70x faster
> Avg: 1.815167 -> 0.193547: 9.38x faster
> Significant (t=293.69)
> Stddev: 0.03619 -> 0.01465: 2.4707x smaller
>
> ### fastpickle ###
> Min: 0.671650 -> 109.014578: 162.31x slower
> Avg: 0.708033 -> 115.379135: 162.96x slower
> Significant (t=-299.42)
> Stddev: 0.02026 -> 2.70799: 133.6909x larger
>
> ### fastunpickle ###
> Min: 0.586665 -> 55.736337: 95.01x slower
> Avg: 0.625024 -> 56.833369: 90.93x slower
> Significant (t=-357.75)
> Stddev: 0.02488 -> 1.11069: 44.6438x larger
>
> ### float ###
> Min: 0.358756 -> 0.026437: 13.57x faster
> Avg: 0.387649 -> 0.041561: 9.33x faster
> Significant (t=106.89)
> Stddev: 0.02159 -> 0.00762: 2.8351x smaller
>
> ### formatted_logging ###
> Min: 0.306111 -> 0.086534: 3.54x faster
> Avg: 0.331045 -> 0.116844: 2.83x faster
> Significant (t=32.06)
> Stddev: 0.02016 -> 0.04273: 2.1198x larger
>
> ### go ###
> Min: 0.640692 -> 0.110813: 5.78x faster
> Avg: 0.686035 -> 0.153473: 4.47x faster
> Significant (t=81.23)
> Stddev: 0.02366 -> 0.03987: 1.6853x larger
>
> ### hexiom2 ###
> Min: 168.499830 -> 13.845624: 12.17x faster
> Avg: 170.731699 -> 14.320609: 11.92x faster
> Significant (t=68.55)
> Stddev: 3.15634 -> 0.67173: 4.6988x smaller
>
> ### iterative_count ###
> Min: 0.155776 -> 0.001828: 85.22x faster
> Avg: 0.169719 -> 0.001959: 86.63x faster
> Significant (t=66.83)
> Stddev: 0.01775 -> 0.00017: 104.6217x smaller
>
> ### json_dump ###
> Min: 0.563337 -> 2.088108: 3.71x slower
> Avg: 0.600209 -> 2.212642: 3.69x slower
> Significant (t=-92.38)
> Stddev: 0.01963 -> 0.12185: 6.2066x larger
>
> ### json_dump_v2 ###
> Min: 3.703154 -> 12.685391: 3.43x slower
> Avg: 3.906239 -> 13.462007: 3.45x slower
> Significant (t=-73.61)
> Stddev: 0.16381 -> 0.90320: 5.5139x larger
>
> ### json_load ###
> Min: 0.637112 -> 0.284530: 2.24x faster
> Avg: 0.691114 -> 0.340403: 2.03x faster
> Significant (t=48.13)
> Stddev: 0.03443 -> 0.03834: 1.1134x larger
>
> ### mako ###
> Min: 0.068892 -> 0.023433: 2.94x faster
> Avg: 0.073817 -> 0.034535: 2.14x faster
> Significant (t=45.74)
> Stddev: 0.01044 -> 0.00868: 1.2032x smaller
>
> ### mako_v2 ###
> Min: 0.055735 -> 0.023094: 2.41x faster
> Avg: 0.060464 -> 0.034531: 1.75x faster
> Significant (t=39.36)
> Stddev: 0.01003 -> 0.01079: 1.0764x larger
>
> ### meteor_contest ###
> Min: 0.234554 -> 0.101157: 2.32x faster
> Avg: 0.253522 -> 0.112062: 2.26x faster
> Significant (t=45.33)
> Stddev: 0.01735 -> 0.01364: 1.2717x smaller
>
> ### nbody ###
> Min: 0.332543 -> 0.024746: 13.44x faster
> Avg: 0.352898 -> 0.026261: 13.44x faster
> Significant (t=111.01)
> Stddev: 0.02059 -> 0.00298: 6.9071x smaller
>
> ### normal_startup ###
> Min: 0.974583 -> 1.578062: 1.62x slower
> Avg: 1.011902 -> 1.640190: 1.62x slower
> Significant (t=-144.15)
> Stddev: 0.01753 -> 0.02535: 1.4458x larger
>
> ### nqueens ###
> Min: 0.357568 -> 0.052418: 6.82x faster
> Avg: 0.386644 -> 0.055879: 6.92x faster
> Significant (t=98.78)
> Stddev: 0.02223 -> 0.00816: 2.7244x smaller
>
> ### pathlib ###
> Min: 0.119940 -> 0.384461: 3.21x slower
> Avg: 0.130969 -> 0.504688: 3.85x slower
> Significant (t=-142.33)
> Stddev: 0.01389 -> 0.03912: 2.8161x larger
>
> ### pickle_dict ###
> Min: 0.496007 -> 8.778371: 17.70x slower
> Avg: 0.522927 -> 9.051905: 17.31x slower
> Significant (t=-306.08)
> Stddev: 0.01866 -> 0.19615: 10.5126x larger
>
> ### pickle_list ###
> Min: 0.295601 -> 6.132281: 20.75x slower
> Avg: 0.315390 -> 6.327789: 20.06x slower
> Significant (t=-215.11)
> Stddev: 0.01909 -> 0.19672: 10.3048x larger
>
> ### pidigits ###
> Min: 0.345745 -> 0.297060: 1.16x faster
> Avg: 0.362924 -> 0.316708: 1.15x faster
> Significant (t=13.14)
> Stddev: 0.01719 -> 0.01798: 1.0456x larger
>
> ### raytrace ###
> Min: 1.527975 -> 0.046997: 32.51x faster
> Avg: 1.575889 -> 0.052239: 30.17x faster
> Significant (t=344.42)
> Stddev: 0.02992 -> 0.00911: 3.2830x smaller
>
> ### regex_compile ###
> Min: 0.462712 -> 0.076845: 6.02x faster
> Avg: 0.489184 -> 0.159849: 3.06x faster
> Significant (t=11.67)
> Stddev: 0.01954 -> 0.19853: 10.1609x larger
>
> ### regex_effbot ###
> Min: 0.061423 -> 0.034389: 1.79x faster
> Avg: 0.065657 -> 0.041867: 1.57x faster
> Significant (t=16.14)
> Stddev: 0.00658 -> 0.00808: 1.2285x larger
>
> ### richards ###
> Min: 0.201109 -> 0.004319: 46.56x faster
> Avg: 0.215034 -> 0.004968: 43.28x faster
> Significant (t=91.42)
> Stddev: 0.01619 -> 0.00140: 11.5878x smaller
>
> ### silent_logging ###
> Min: 0.080098 -> 0.007613: 10.52x faster
> Avg: 0.085028 -> 0.009930: 8.56x faster
> Significant (t=43.10)
> Stddev: 0.00980 -> 0.00747: 1.3113x smaller
>
> ### simple_logging ###
> Min: 0.296057 -> 0.081953: 3.61x faster
> Avg: 0.315012 -> 0.110826: 2.84x faster
> Significant (t=33.22)
> Stddev: 0.01866 -> 0.03925: 2.1031x larger
>
> ### spectral_norm ###
> Min: 0.365987 -> 0.016084: 22.75x faster
> Avg: 0.393062 -> 0.018651: 21.07x faster
> Significant (t=124.58)
> Stddev: 0.02014 -> 0.00679: 2.9647x smaller
>
> ### startup_nosite ###
> Min: 0.485740 -> 0.613260: 1.26x slower
> Avg: 0.518384 -> 0.654354: 1.26x slower
> Significant (t=-49.49)
> Stddev: 0.01842 -> 0.02038: 1.1066x larger
>
> ### telco ###
> Min: 0.773569 -> 0.043440: 17.81x faster
> Avg: 0.800002 -> 0.049813: 16.06x faster
> Significant (t=276.68)
> Stddev: 0.01300 -> 0.01409: 1.0844x larger
>
> ### threaded_count ###
> Min: 0.167072 -> 0.005613: 29.76x faster
> Avg: 0.193020 -> 0.005944: 32.47x faster
> Significant (t=88.19)
> Stddev: 0.01499 -> 0.00036: 41.4676x smaller
>
> ### tornado_http ###
> Min: 0.307076 -> 0.390125: 1.27x slower
> Avg: 0.378676 -> 0.459946: 1.21x slower
> Significant (t=-8.21)
> Stddev: 0.04109 -> 0.09005: 2.1916x larger
>
> ### unpack_sequence ###
> Min: 0.000052 -> 0.000057: 1.10x slower
> Avg: 0.000077 -> 0.000404: 5.21x slower
> Significant (t=-16.30)
> Stddev: 0.00011 -> 0.00447: 41.4670x larger
>
> ### unpickle_list ###
> Min: 0.455635 -> 0.963103: 2.11x slower
> Avg: 0.514364 -> 1.017570: 1.98x slower
> Significant (t=-79.98)
> Stddev: 0.02640 -> 0.03581: 1.3564x larger
>
>
> So PyPy3’s generally faster, except on startup and tornado_http, where it’s only slightly slower.
>
> Then it’s quite a bit slower on “mini interpreters” such as etree/json/Pickle. These modules also happen to be pure/mostly pure python versions on PyPy vs C versions used by CPython. Pickle looks particularly bad, I have a guess that it’s due to removal of an optimization in marshal that happened during py3k compatibility work, see:
>
> https://bitbucket.org/pypy/pypy/src/a245775c1e3687662879fe57964a5ece689cd781/pypy/module/marshal/interp_marshal.py?at=py3k#cl-13
>
> and it’s still slower than PyPy2 on many things.
>
> Anyway, to reiterate we’ve pretty busy with compatibility work. We could use some help with benchmarking infrastructure =]
>
> --
> Philip Jenvey
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev