Mailman 3 September 2011 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

Updated 'High Performance Python' tutorial (the one from EuroPython 2011)
by Ian Ozsvald 16 Nov '11

16 Nov '11

Dear all, I've published v0.2 of my High Performance Python tutorial write-up from the session I ran at EuroPython: http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from… Antonio - you asked earlier if the 'expanded math' version of the Mandelbrot solver (using doubles rather than complex numbers) would be faster - I've timed it and it is a bit faster with a nightly build of PyPy, but nowhere near as fast at ShedSkin's generated C output (details below). Maciej - thanks for pointing me at the numpy module. I've added a tiny section showing numpy in PyPy but I haven't converted the Mandelbrot solver to use it (even finishing v0.2 took longer than I'd thought). I'm hoping that some more exposure in the report might bring in more volunteers from outside. Here's a clip from the report in the PyPy section: "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. (Ian - the only difference with pure_python_2.py is that local dereferences in the tight loop are moved outside the loop, causing fewer dereference operations) As an additional test (not shown in the graphs) I ran pypy shedskin2.py 1000 1000 which runs the expanded math version of the shedskin variant below (this replaces complex numbers with floats and expands abs to avoid the square root). The shedskin2.py result takes 3.2 seconds (which is still much slower than the 0.4s version compiled using shedskin)." The pure_python src is here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… shedskin2.py is available here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… I haven't tested whether the warm-up periods for PyPy are significant, possibly they account for much of the difference between ShedSkin and PyPy? I want to revisit this but for the next few weeks I have to go back to other projects. I hope the report brings in some new folk for PyPy, Ian. -- Ian Ozsvald (A.I. researcher, screencaster) ian(a)IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald

5 8

Anyone interested in a MIPS port...
by Vishal 16 Oct '11

16 Oct '11

Hello, I would like to collect volunteers for a MIPS port of PyPy JIT. MIPS is used across the embedded scene universally and a PyPy support for MIPS would be wonderful. So I have 2 questions (may seem like very nuby like...pardon me for that): a) Does it make sense to have a MIPS port of the PyPy JIT. b) How much hardware dependent is a JIT port? Thanks and best regards, Vishal Sapre

5 5

Gothenburg Sprint Dates
by Carl Friedrich Bolz 13 Oct '11

13 Oct '11

Hi all, Laura asked me to post the following, since she has problems with her mail: ---------------------------------------------------------------------- Some of us need to be in Stockholm Oct 24 and 28. Anto needs to be with his family Nov 1. Fscons starts Friday Nov 11 in Göteborg, and we're giving a talk. Thus I propose that we hold a Sprint at my house Wednesday Nov 2nd through Thursday Nov 10. fscons: http://fscons.org/ Nov 11-13 Not sure what day we will be speaking yet. What do the rest of you think of this idea? Laura

8 16

Re: [pypy-dev] [pypy-commit] pypy default: Hack to ensure that ll_arraycopy gets a proper effectinfo.write_descrs_arrays
by Hakan Ardo 03 Oct '11

03 Oct '11

Hi, is there a better way to fix this? The same kind of issue might arise elsewhere? > Author: Hakan Ardo <hakan at debian.org> > Branch: > Changeset: r47722:bf3f65e2b1c2 > Date: 2011-09-30 19:48 +0200 > http://bitbucket.org/pypy/pypy/changeset/bf3f65e2b1c2/ > > Log: Hack to ensure that ll_arraycopy gets a proper > effectinfo.write_descrs_arrays > > diff --git a/pypy/rlib/rgc.py b/pypy/rlib/rgc.py > --- a/pypy/rlib/rgc.py > +++ b/pypy/rlib/rgc.py > @@ -143,6 +143,10 @@ > from pypy.rpython.lltypesystem.lloperation import llop > from pypy.rlib.objectmodel import keepalive_until_here > > + # XXX: Hack to ensure that we get a proper effectinfo.write_descrs_arrays > + if length > 0: > + dest[dest_start] = source[source_start] > + > # supports non-overlapping copies only > if not we_are_translated(): > if source == dest: -- Håkan Ardö

2 5

pypy with virtualenv?
by John Anderson 01 Oct '11

01 Oct '11

I read that this should just work with the latest versions, here is what I'm getting: sontek@beast$ virtualenv --python=pypy ~/code/pypyenv Running virtualenv with interpreter /usr/bin/pypy New pypy executable in /home/sontek/code/pypyenv/bin/pypy ERROR: The executable /home/sontek/code/pypyenv/bin/pypy is not functioning ERROR: It thinks sys.prefix is u'/usr/lib64/pypy-1.5' (should be '/home/sontek/code/pypyenv') ERROR: virtualenv is not compatible with this system or executable sontek@beast$ pypy --version Python 2.7.1 (?, May 02 2011, 19:05:35) [PyPy 1.5.0-alpha0 with GCC 4.6.0] ~ sontek@beast$ virtualenv --version 1.6.4

5 9

PyPy packaging help needed
by Maciej Fijalkowski 30 Sep '11

30 Sep '11

Hi. Does anyone feel helping with PyPy's PPA? The packages are super-outdated and I think it's one of the most requested pypy feature. Cheers, fijal

4 4

Compile to executable program running in sandboxed environment?
by Galfy Pundee 30 Sep '11

30 Sep '11

Hi Pypy gurus, Is it possible to create an executable package, using PyPy, that is running the python code in a sandboxed environment? Also when I run in a sandbox environment is it possible to code the logic of the external process handling the policy in python? Thanks in advance for your answers. Regards, Gal

2 3

I was talking with Russel Winder at PyCON UK.
by Laura Creighton 30 Sep '11

30 Sep '11

He says, currently, PyPy's threading does not scale properly. More below. Maybe we want to use his benchmark? Laura ------- Forwarded Message Return-Path: russel(a)russel.org.uk Delivery-Date: Thu Sep 29 13:53:50 2011 Subject: PyPy and multiprocessing From: Russel Winder <russel(a)russel.org.uk> To: Laura Creighton <lac(a)openend.se> - --=-I2STZOatYEgK/vXAGHYd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Laura, I have a collection of various versions (using various features of various languages) of the embarrassingly parallel problem of calculating Pi using quadrature. It is a micro-benchmark and so suffers from all the issues they suffer from (especially on the JVM). The code is a Bazaar branch http://www.russel.org.uk/Bazaar/Pi_Quadrature. I am writing as there appears to be an interesting feature using PyPy and the microprocessing package in pool mode. This is a twin-Xeon machine so has 8 cores =E2=80=94 a 32 thread run should= only go as fast as an 8 thread run. Scaling should be linear in the number of cores. Using CPython 2.7, I get: |> python2.7 pi_python2_multiprocessing_pool.py =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 3.5378549099 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 1.97133994102 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.515691041946 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.521239995956 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 Using PyPy 1.6 I get: |> pypy pi_python2_multiprocessing_pool.py =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.249331951141 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 1 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.104065895081 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 2 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.0764398574829 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 =3D=3D=3D=3D Python Multiprocessing Pool pi =3D 3.14159265359 =3D=3D=3D=3D Python Multiprocessing Pool iteration count =3D 10000000 =3D=3D=3D=3D Python Multiprocessing Pool elapse =3D 0.124751091003 =3D=3D=3D=3D Python Multiprocessing Pool process count =3D 32 =3D=3D=3D=3D Python Multiprocessing Pool processor count =3D 8 There is no statistical significance to these one off numbers but I am fairly confident that there are no large variations should a proper collection of data be taken. The point here is that whereas CPython shows the expected scaling, PyPy does not give the expected scaling for larger nubmers of cores. Indeed having more threads than cores is detrimental to PyPy but not to CPython. Hopefully we will soon be seeing PyPy be Python 3.2 compliant! - --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel(a)russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder - --=-I2STZOatYEgK/vXAGHYd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit - -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAk6EWpIACgkQr2EGkixYSbrjHQCeJCcoOamtzwY3rPSFofuXACYK 7RgAnjuWAuGHI0mSUzQ/BhS/clZbZaUy =kVgo - -----END PGP SIGNATURE----- - --=-I2STZOatYEgK/vXAGHYd-- ------- End of Forwarded Message

5 7

pypy-dev September 2011