[pypy-dev] I was talking with Russel Winder at PyCON UK.

[Romain Guillebert]

Hi

David Beazley noticed that PyPy's GIL isn't very good compared to
CPython's :

https://twitter.com/#!/dabeaz/status/118889721358327808
https://twitter.com/#!/dabeaz/status/118888789136523264
https://twitter.com/#!/dabeaz/status/118864260175634433

IMO it's the same issue

Cheers
Romain




On Thu, Sep 29, 2011 at 07:17:45PM +0200, Laura Creighton wrote:
> 
> He says, currently, PyPy's threading does not scale properly.  More below.  Maybe we
> want to use his benchmark?
> 
> Laura
> 
> 
> ------- Forwarded Message
> 
> Return-Path: russel at russel.org.uk
> Delivery-Date: Thu Sep 29 13:53:50 2011
> Subject: PyPy and multiprocessing
> From: Russel Winder <russel at russel.org.uk>
> To: Laura Creighton <lac at openend.se>
> 
> - --=-I2STZOatYEgK/vXAGHYd
> Content-Type: text/plain; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
> 
> Laura,
> 
> I have a collection of various versions (using various features of
> various languages) of the embarrassingly parallel problem of calculating
> Pi using quadrature.  It is a micro-benchmark and so suffers from all
> the issues they suffer from (especially on the JVM).  The code is a
> Bazaar branch http://www.russel.org.uk/Bazaar/Pi_Quadrature.
> 
> I am writing as there appears to be an interesting feature using PyPy
> and the microprocessing package in pool mode.
> 
> This is a twin-Xeon machine so has 8 cores =E2=80=94 a 32 thread run should=
>  only
> go as fast as an 8 thread run.  Scaling should be linear in the number
> of cores.
> 
> Using CPython 2.7, I get:
> 
> |> python2.7 pi_python2_multiprocessing_pool.py
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 3.5378549099
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 1.97133994102
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.515691041946
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.521239995956
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> Using PyPy 1.6 I get:
> 
> |> pypy pi_python2_multiprocessing_pool.py
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.249331951141
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.104065895081
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.0764398574829
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359
> =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000
> =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.124751091003
> =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32
> =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8
> 
> There is no statistical significance to these one off numbers but I am
> fairly confident that there are no large variations should a proper
> collection of data be taken.
> 
> The point here is that whereas CPython shows the expected scaling, PyPy
> does not give the expected scaling for larger nubmers of cores.  Indeed
> having more threads than cores is detrimental to PyPy but not to
> CPython.
> 
> Hopefully we will soon be seeing PyPy be Python 3.2 compliant!
> 
> - --=20
> Russel.
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D
> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.n=
> et
> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
> 
> - --=-I2STZOatYEgK/vXAGHYd
> Content-Type: application/pgp-signature; name="signature.asc"
> Content-Description: This is a digitally signed message part
> Content-Transfer-Encoding: 7bit
> 
> - -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> 
> iEYEABECAAYFAk6EWpIACgkQr2EGkixYSbrjHQCeJCcoOamtzwY3rPSFofuXACYK
> 7RgAnjuWAuGHI0mSUzQ/BhS/clZbZaUy
> =kVgo
> - -----END PGP SIGNATURE-----
> 
> - --=-I2STZOatYEgK/vXAGHYd--
> 
> ------- End of Forwarded Message
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev