He says, currently, PyPy's threading does not scale properly. More below. Maybe we want to use his benchmark? Laura ------- Forwarded Message Return-Path: russel@russel.org.uk Delivery-Date: Thu Sep 29 13:53:50 2011 Subject: PyPy and multiprocessing From: Russel Winder To: Laura Creighton - --=-I2STZOatYEgK/vXAGHYd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Laura, I have a collection of various versions (using various features of various languages) of the embarrassingly parallel problem of calculating Pi using quadrature. It is a micro-benchmark and so suffers from all the issues they suffer from (especially on the JVM). The code is a Bazaar branch http://www.russel.org.uk/Bazaar/Pi_Quadrature. I am writing as there appears to be an interesting feature using PyPy and the microprocessing package in pool mode. This is a twin-Xeon machine so has 8 cores =E2=80=94 a 32 thread run should= only go as fast as an 8 thread run. Scaling should be linear in the number of cores. Using CPython 2.7, I get: |> python2.7 pi_python2_multiprocessing_pool.py =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 3.5378549099 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 1.97133994102 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.515691041946 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.521239995956 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 Using PyPy 1.6 I get: |> pypy pi_python2_multiprocessing_pool.py =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.249331951141 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.104065895081 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.0764398574829 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.124751091003 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 There is no statistical significance to these one off numbers but I am fairly confident that there are no large variations should a proper collection of data be taken. The point here is that whereas CPython shows the expected scaling, PyPy does not give the expected scaling for larger nubmers of cores. Indeed having more threads than cores is detrimental to PyPy but not to CPython. Hopefully we will soon be seeing PyPy be Python 3.2 compliant! - --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel@russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder - --=-I2STZOatYEgK/vXAGHYd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit - -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAk6EWpIACgkQr2EGkixYSbrjHQCeJCcoOamtzwY3rPSFofuXACYK 7RgAnjuWAuGHI0mSUzQ/BhS/clZbZaUy =kVgo - -----END PGP SIGNATURE----- - --=-I2STZOatYEgK/vXAGHYd-- ------- End of Forwarded Message