[pypy-dev] I was talking with Russel Winder at PyCON UK.

[Laura Creighton]

He says, currently, PyPy's threading does not scale properly.  More below.  Maybe we
want to use his benchmark?

Laura


------- Forwarded Message

Return-Path: russel at russel.org.uk
Delivery-Date: Thu Sep 29 13:53:50 2011
Subject: PyPy and multiprocessing
From: Russel Winder <russel at russel.org.uk>
To: Laura Creighton <lac at openend.se>

- --=-I2STZOatYEgK/vXAGHYd
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Laura,

I have a collection of various versions (using various features of
various languages) of the embarrassingly parallel problem of calculating
Pi using quadrature.  It is a micro-benchmark and so suffers from all
the issues they suffer from (especially on the JVM).  The code is a
Bazaar branch http://www.russel.org.uk/Bazaar/Pi_Quadrature.

I am writing as there appears to be an interesting feature using PyPy
and the microprocessing package in pool mode.

This is a twin-Xeon machine so has 8 cores =E2=80=94 a 32 thread run should=
 only
go as fast as an 8 thread run.  Scaling should be linear in the number
of cores.

Using CPython 2.7, I get:

|> python2.7 pi_python2_multiprocessing_pool.py
=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 3.5378549099
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 1.97133994102
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.515691041946
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.521239995956
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

Using PyPy 1.6 I get:

|> pypy pi_python2_multiprocessing_pool.py
=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.249331951141
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.104065895081
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.0764398574829
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

=3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
=3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
=3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.124751091003
=3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32
=3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8

There is no statistical significance to these one off numbers but I am
fairly confident that there are no large variations should a proper
collection of data be taken.

The point here is that whereas CPython shows the expected scaling, PyPy
does not give the expected scaling for larger nubmers of cores.  Indeed
having more threads than cores is detrimental to PyPy but not to
CPython.

Hopefully we will soon be seeing PyPy be Python 3.2 compliant!

- --=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

- --=-I2STZOatYEgK/vXAGHYd
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

- -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAk6EWpIACgkQr2EGkixYSbrjHQCeJCcoOamtzwY3rPSFofuXACYK
7RgAnjuWAuGHI0mSUzQ/BhS/clZbZaUy
=kVgo
- -----END PGP SIGNATURE-----

- --=-I2STZOatYEgK/vXAGHYd--

------- End of Forwarded Message