Mailman 3 October 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus 22 Dec '13

22 Dec '13

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

Any publications regarding PyPy / trace-based JIT compiler?
by Michael Sioutis 11 Nov '12

11 Nov '12

Dear PyPy team, I am in the process of writing a paper that will target some AI conference, and I would like to ask if there are any relevant publications of yours or in general that showcase the possible advantages of trace-based JIT compilation over method-based JIT compilation or static compilation. I just want to use 2 or 3 of them as references when I explain why my implementation is more scalable and robust when compared to some C/C++/Java implementations (appart from different data structures and algorithms). Thank you :) Mike

7 12

Need help in concluding Multi-Thread based model thoughts between Python and PyPy
by Sasikanth Eda 31 Oct '12

31 Oct '12

Hi All, I have tried to perform a CPU intensive operation ( which typically involved calculation of prime numbers from 1-100000 ) using Multi-process and Multi-Thread models provided by C, Python-2.6, PyPy-1.9. Here are results that are obtained from this exercise; *C Program results ( Number of processes = 500 )* CPU utilization = 99.81% (peak) Memory utilization = 14.77 GB (peak) Execution time = *2m16s* (real) ( time taken between invocation to termination ) *C Program results ( Number of Threads = 500 )* CPU utilization = 99.94% (peak) Memory utilization = 14.79 GB (peak) Execution time = *2m10s* (real) *{ Here I came to a conclusion that C gave a comparatively equal good results in terms of both Multi-Threaded as well as Multi-Process }* *Python Program results ( Number of processes = 500 )* CPU utilization = 99.62% (peak) Memory utilization = 17.03 GB (peak) Execution time = *87m42.785s* (real) *Python Program results ( Number of Threads = 500 )* CPU utilization = 18.23% (peak) Memory utilization = 33.074 GB (peak) Execution time = *239m10.210s* (real) *{ Here I came to a conclusion that Python is limited by the Global Interpreter Look (GIL) when used in Multi-Thread mode (that is why it gave poor results when compared with process model }* *PyPy Program results ( Number of processes = 500 )* CPU utilization = 99.19% (peak) Memory utilization = 22.92 GB (peak) Execution time = *6m4.970s* (real) *PyPy Program results ( Number of Threads = 500 )* CPU utilization = 38.88% (peak) Memory utilization = 21.06 GB (peak) Execution time = *59m29s* (real) *{ Here I came to a conclusion that PyPy is better than Python }* * * Now what am I struggling to understand is ; * * 1. Has PyPy optimized / reduced the GIL limitation ? ( what is the progress in PyPy version 1.9 in that when compared with Python's progress ) 2. If PyPy is also suffering from the same GIL limitations, what made the program run faster than Python, is it because of more warm-up time, optimization of loops ? 3. What are your suggestions for me if I wanted to go for Multi-Thread application design ( in-terms of Python / PyPy ) I request your inputs/suggestion techniques towards optimizing Multi threaded load loads. Kindly share your experiences and feedback in this scenario. -- Sasikanth

5 6

Problems in compiling libpypy-c.so
by Srinivasan Balasubramanian 31 Oct '12

31 Oct '12

Folks, I am trying to run a python uwsgi container using pypy. For doing that i had to compile libpypy-c.so and i have been getting numerous errors while doing so. I was trying to build pypy-1.8 release. I have couple of questions in this regard. 1)Does anybody know of an RPM containing libpypy-c.so for CentOS release 5.4? 2)Has anybody compiled libpypy-c.so,pypy-c for CentOS release 5.4 ? I would be delighted If someone can share the libpypy-c.so and pypy-c. -Srini

2 1

documentation of _immutable_?
by wlavrijsen＠lbl.gov 30 Oct '12

30 Oct '12

Hi, any place where I can find docs describing _immutable_ and _immutable_fields_ in some detail? As it happens, they don't quite do what I expected, which led to bugs. I'd like to know what their scope is and what the wildcards do. Right now I just removed them all to be safe (and allow some experimentation). But that is costing performance (20% extra overhead). Also, I see _attrs_ in quite a few places. What is it for? Thanks, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

2 3

unable to translate due to isinstance call
by wlavrijsen＠lbl.gov 28 Oct '12

28 Oct '12

Hi, my translation run fails with: -> assert s_value.contains(self.bindings[arg]) where s_value = SomeBool(knowntypedata={}) and self.bindings[arg] is SomeBool(const=True, knowntypedata={}). The cause is this: if isinstance(self.functions[0], CPPFunction): where the s_value above is op.result coming the isinstance() call. With moving things around, the problem comes and goes (hence I thought I had it fixed already, but it came back ...). Checking other calls to isinstance() in other modules, I can't see why it would not be allowed here? Also, what would be the difference between a const and a non-const bool? Any hints to get around this one are much appreciated! Thanks, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

3 6

Trouble translating llvm
by David Arthur 27 Oct '12

27 Oct '12

Running "python translate.py -Ojit --jit-backend=llvm" with pypy 1.9 (pypy-pypy-341e1e3821ff), I get [translation:info] JIT compiler generation... [Timer] Timings: [Timer] annotate --- 1354.8 s [Timer] rtype_lltype --- 1855.0 s [Timer] pyjitpl_lltype --- 0.2 s [Timer] =========================================== [Timer] Total: --- 3210.1 s [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 791, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 285, in _do [translation:ERROR] res = func() [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/translator/driver.py", line 397, in task_pyjitpl_lltype [translation:ERROR] backend_name=self.config.translation.jit_backend, inline=True) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/metainterp/warmspot.py", line 35, in apply_jit [translation:ERROR] kwds['CPUClass'] = getcpuclass(backend_name) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/detect_cpu.py", line 82, in getcpuclass [translation:ERROR] mod = __import__(modname, {}, {}, clsname) [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/runner.py", line 9, in <module> [translation:ERROR] from pypy.jit.backend.llvm import llvm_rffi [translation:ERROR] File "/Users/mumrah/Downloads/pypy-pypy-341e1e3821ff/pypy/jit/backend/llvm/llvm_rffi.py", line 4, in <module> [translation:ERROR] from pypy.translator.tool.cbuild import ExternalCompilationInfo, log [translation:ERROR] ImportError: cannot import name log Any insights? -- David Arthur

3 2

Re: [pypy-dev] [pypy-commit] pypy default: fix jit translation, don't lookinside this function until ptradd works with things besides ccharp
by Maciej Fijalkowski 25 Oct '12

25 Oct '12

On Thu, Oct 25, 2012 at 4:38 AM, alex_gaynor <noreply(a)buildbot.pypy.org> wrote: > Author: Alex Gaynor <alex.gaynor(a)gmail.com> > Branch: > Changeset: r58413:88158af0627f > Date: 2012-10-24 19:38 -0700 > http://bitbucket.org/pypy/pypy/changeset/88158af0627f/ > > Log: fix jit translation, don't lookinside this function until ptradd > works with things besides ccharp We can use raw_load and raw_store (from pypy.rlib.rawstorage) that can be understood by the JIT. making ptradd understood is also not a big deal. > > diff --git a/pypy/module/array/interp_array.py b/pypy/module/array/interp_array.py > --- a/pypy/module/array/interp_array.py > +++ b/pypy/module/array/interp_array.py > @@ -9,6 +9,7 @@ > from pypy.objspace.std.multimethod import FailedToImplement > from pypy.objspace.std.stdtypedef import SMM, StdTypeDef > from pypy.objspace.std.register_all import register_all > +from pypy.rlib import jit > from pypy.rlib.rarithmetic import ovfcheck, widen > from pypy.rlib.unroll import unrolling_iterable > from pypy.rlib.objectmodel import specialize, keepalive_until_here > @@ -466,6 +467,7 @@ > self.setlen(0) > self.fromsequence(w_lst) > > + @jit.dont_look_inside > def delslice__Array_ANY_ANY(space, self, w_i, w_j): > i = space.int_w(w_i) > if i < 0: > @@ -495,13 +497,13 @@ > rffi.c_memcpy( > rffi.cast(rffi.VOIDP, rffi.ptradd(self.buffer, i)), > rffi.cast(rffi.VOIDP, rffi.ptradd(oldbuffer, j)), > - (self.len - j) * mytype.bytes) > + (self.len - j) * mytype.bytes > + ) > self.len -= j - i > self.allocated = self.len > if oldbuffer: > lltype.free(oldbuffer, flavor='raw') > > - > # Add and mul methods > > def add__Array_Array(space, self, other): > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit

1 0

Splitting RPython and PyPy
by Maciej Fijalkowski 22 Oct '12

22 Oct '12

Hi Due to impressive work by Ronan Lamy, we're now able to split RPython and PyPy. Note that the fact of splitting this is not up to discussion, however, how we go about it is. During discussions with Armin we came up with the following plan: * We make a copy of pypy repo called rpython. it'll still live under pypy team on bitbucket. * We'll rename toplevel package of pypy to rpython, for the rpython part. Since we need to change ALL THE IMPORTS, we can do refactorings of imports now. Proposed changes: * to be moved from pypy to rpython: annotation, translator, rlib * pypy.config must be split somehow, same for tool, bin and for doc * move pypy.rpython namespace to rpython.rtyper * move pypy.rpython.lltypesystem to rpython.lltypesystem, same for ootype * pypy.rpython.memory becomes rpython.gc * pypy.objspace.flow becomes rpython.flowspace * testrunner and dotviewer can become independent packages * _pytest stays with pypy, however for rpython you can use whatever version of py.test you have installed. rpython test suite should also however be runnable under current contents of _pytest * RPython will come with setup.py and be a "normal" python package Cheers, fijal & armin

7 9

pypy-dev October 2012