Mailman 3 February 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

[Repost] PyPy 1.8 50% slower than PyPy 1.7 on my test case
by Laurent Vaucher 04 Jun '12

04 Jun '12

Sorry to repost, but does anyone have an idea about what I could do to track down the source of the slowdown? Should I run with some trace to try to compare how different parts behave? How could I do that? Thanks, Laurent. Hi and first of all, thanks for that great project. Now to my "problem". I'm doing some puzzle-solving, constraint processing with Python and on my particular program, PyPy 1.8 showed a 50% increase in running time over PyPy 1.7. I'm not doing anything fancy (no numpy, itertools, etc.). The program is here: https://github.com/slowfrog/hexiom To reproduce: - fetch hexiom2.py and level36.txt from github - run 'pypy hexiom2.py -sfirst level36.txt' On my machine (Windows 7) the timings are the following: Python 2.6.5/Windows 32 3m35s Python 2.7.1 (930f0bc4125a, Nov 27 2011, 11:58:57) [PyPy 1.7.0 with MSC v.1500 32 bit] 31s Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 18:31:47) [PyPy 1.8.0 with MSC v.1500 32 bit] 48s I'm using the default options. Do you have any idea what could be causing that? Thanks, Laurent.

3 5

Biting off more than I can chew
by Greg Bowyer 07 May '12

07 May '12

The other night I got this burning desire to recreate the Azul GPGC (the java pauseless collector) inside pypy, with a view that it could be used alongside the STM work to make pypy a low (no?) pause concurrent VM. When I started tackling the code I realised I might have bitten off a little more than I can chew. My question (probably one of many to irritate and annoy all the fine folks here) would be, is there a sensible way to compile into pypy a small amount of C code that can be used to bootstrap and bridge some esoteric c libraries into pypy, the code that I want to run, on startup of pypy would be the following https://bitbucket.org/GregBowyer/pypy-c4gc/changeset/0de575b3a8d1#chg-azm_m… I have noticed that there are a few bridges into direct c code in the pypy source, but I cannot fathom how these are linked into RPython -- Greg

3 5

Bringing Cython and PyPy closer together
by Stefan Behnel 01 Apr '12

01 Apr '12

Hi, I'm breaking out of the thread where this topic was started ("offtopic, ontopic, ...") because it is getting too long and unfocussed to follow. The current state of the discussion seems to be that PyPy provides ways to talk to C code, but nothing as complete as CPython's C-API in the sense that it allows efficient two-way communication between C code and Python objects. Thus, we need to either improve this or look for alternatives. In order to get us more focussed on what can be done and what the implications are, so that we may eventually be able to decide what should be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement Proposal). http://wiki.cython.org/enhancements/pypy Please add to it as you see fit. In general, it is ok for a CEP to present diverging arguments, but please try to give them a structure in order to keep the overall document focussed. Stefan

9 64

pypy2exe
by Paolo Basso 21 Mar '12

21 Mar '12

Perhaps this will sound like a newbie question but IMHO a lot of people would benefit of some clarification and more info.... Here's the point: what if I want to build an executable using pypy (possibly with a GUI)? I'm not a pro of course, but with cpython I'm able to do it quite easily thanks to py2exe and many GUI frameworks (Qt also comes with a designer). Is there or will be there something like a pypy2exe or already exists a step by step guide I missed? May be rpythonic (http://code.google.com/p/rpythonic/) useful? I apologize if this is the wrong place to discuss such a topic, I can post the question in a better context if you redirect me... Cheers, Paolo

2 3

Py_DecRef() in cpyext
by Stefan Behnel 08 Mar '12

08 Mar '12

Hi, having rewritten Cython's dict iteration to use normal iteration, I ran a little benchmark for comparison and it turned out to be (algorithmically) faster by more than an order of magnitude for a dict with 1000 items. That's pretty decent, although it's still about 100x slower than in CPython. I'm sure and certain that there's much more of this low hanging fruit to find all over the place. Next, I ran callgrind over my little benchmark and it showed that the dominating part of the running time had shifted from 99% in PyDict_Next() to 56% in Py_DecRef(). Sadly, there wasn't much more to be read out of it, because the graph was just full of hex numbers instead of names. To be attributed to the JIT compiled code, I guess. Looking at Py_DecRef(), however, left me somewhat baffled. I would have expected this to be the most intensively tuned function in all of cpyext, but it even started with this comment: """ # XXX Optimize these functions and put them into macro definitions """ Ok, sounds reasonable. Should have been done already, though. Especially when I took a look at object.h and saw that the Py_DECREF() macro *always* calls into it. Another surprise. I had understood in previous discussions that the refcount emulation in cpyext only counts C references, which I consider a suitable design. (I guess something as common as Py_None uses the obvious optimisation of always having a ref-count > 1, right? At least when not debugging...) So I changed the macros to use an appropriate C-level implementation: """ #define Py_INCREF(ob) ((((PyObject *)ob)->ob_refcnt > 0) ? \ ((PyObject *)ob)->ob_refcnt++ : (Py_IncRef((PyObject *)ob))) #define Py_DECREF(ob) ((((PyObject *)ob)->ob_refcnt > 1) ? \ ((PyObject *)ob)->ob_refcnt-- : (Py_DecRef((PyObject *)ob))) #define Py_XINCREF(op) do { if ((op) == NULL) ; else Py_INCREF(op); \ } while (0) #define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); \ } while (0) """ to tell the C compiler that it doesn't actually need to call into PyPy in most cases (note that I didn't use any branch prediction macros, but that shouldn't change all that much anyway). This shaved off a couple of cycles from my iteration benchmark, but much less than I would have liked. My intuition tells me that this is because almost all objects that appear in the benchmark are actually short-lived in C space so that pretty much every Py_DECREF() on them kills them straight away and thus calls into Py_DecRef() anyway. To be verified with a better test. I think this could be fixed in some way by figuring out if there is at least one live reference to it inside of PyPy (which is usually the case for container iteration and parameters, for example), and as long as that's the case, add one to the refcount to keep C code from needing to call back into PyPy unnecessarily. A finaliser could do that, I guess. Ok, next, I looked at the Py_DecRef() implementation itself and found it to be rather long. Since I wasn't sure what the JIT would make of it, please don't shout at me if any of the following comments ("^^") make no sense. @cpython_api([PyObject], lltype.Void) def Py_DecRef(space, obj): if not obj: return ^^ this would best be handled right in C space to avoid unnecessary call overhead and to allow the C compiler to drop this test when possible. assert lltype.typeOf(obj) == PyObject obj.c_ob_refcnt -= 1 if DEBUG_REFCOUNT: debug_refcount("DECREF", obj, obj.c_ob_refcnt, frame_stackdepth=3) ^^ ok, I assume debug code will automatically be removed completely if obj.c_ob_refcnt == 0: state = space.fromcache(RefcountState) ptr = rffi.cast(ADDR, obj) if ptr not in state.py_objects_r2w: # this is a half-allocated object, lets call the deallocator # without modifying the r2w/w2r dicts _Py_Dealloc(space, obj) ^^ this looks like an emergency special case to me - shouldn't be on the fast path... else: w_obj = state.py_objects_r2w[ptr] ^^ here, I wonder if the JIT is smart enough to detect the double weakref dict lookup in the (I guess) 99% code path and folds it into one? ^^ I actually wonder why this needs to use a weakref dict instead of a normal dict. It's ref-counted after all, so we *know* there's a live reference to it until further notice. Are weakref dicts as fast as normal dicts in PyPy? del state.py_objects_r2w[ptr] w_type = space.type(w_obj) if not w_type.is_cpytype(): _Py_Dealloc(space, obj) del state.py_objects_w2r[w_obj] # if the object was a container for borrowed references state.delete_borrower(w_obj) ^^ in some place here, I would have expected the use of a freelist for the C space object representations in order to avoid unnecessary allocations, especially for things like short tuples etc. Is this hidden somewhere else? Or does PyPy's runtime handle this kind of caching internally somewhere? else: if not we_are_translated() and obj.c_ob_refcnt < 0: message = "Negative refcount for obj %s with type %s" % ( obj, rffi.charp2str(obj.c_ob_type.c_tp_name)) print >>sys.stderr, message assert False, message ^^ This will also vanish automatically, right? So, from a quick and shallow look, it seems to me that this code could be improved. By taking work off the fast paths and avoiding create-delete cycles, something as ubiquitous as this function could help in making C-API code run a lot faster. Stefan

2 5

Seeking a bit of guidance
by Russel Winder 07 Mar '12

07 Mar '12

For some reason I had assumed that pypy could be a drop in replacement for python. I am on Debian Unstable and I am suspected that my assumption is being thwarted. In particular in trying to run SCons. |> python /home/Checkouts/Mercurial/SCons/bootstrap/src/script/scons.py scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... stopServerIfNeedBe(["allPDFs"], []) scons: done building targets. |> pypy /home/Checkouts/Mercurial/SCons/bootstrap/src/script/scons.py Traceback (most recent call last): File "app_main.py", line 51, in run_toplevel File "/home/Checkouts/Mercurial/SCons/bootstrap/src/script/scons.py", line 187, in <module> import SCons.Script ImportError: No module named SCons This may be a SCons problem or a Debian Python packaging issue but higher likelihood is a PyPy configuration issue? Basically I am not sure where to start. I use SCons from a Mercurial clone, Python from Debian packages, PyPy from then downloaded binary distribution. |> python --version Python 2.7.2+ |> pypy --version Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:11) [PyPy 1.7.0 with GCC 4.4.3] Help and guidance greatfully received. Thanks. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel(a)russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder

4 8

web_review
by Affleck, Melanie 02 Mar '12

02 Mar '12

Dear Pypy-dev, We will review your site and your web placement, at no-cost to you, now. Reply today for your complimentary analysis. Sincerely, Melanie Affleck Interplay_Marketing pypy-dev(a)codespeak.net 2/29/2012

4 4

Translation Error building with cx_Oracle
by Phillip Class 28 Feb '12

28 Feb '12

Hello, On Ubuntu 10.04 LTS 64-bit with Python 2.7, after cloning the latest repo I am trying to build pypy with cx_Oracle mod using the command: python translate.py -Ojit targetpypystandalone.py --withmod-oracle After quite awhile it fails with the following translation errors. Can somebody please take a look? Thanks! [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 309, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/translator/driver.py", line 814, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/translator/driver.py", line 287, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/user/Desktop/pypy/pypy/translator/driver.py", line 399, in task_pyjitpl_lltype [translation:ERROR] backend_name=self.config.translation.jit_backend, inline=True) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/jit/metainterp/warmspot.py", line 48, in apply_jit [translation:ERROR] warmrunnerdesc.finish() [translation:ERROR] File "/home/user/Desktop/pypy/pypy/jit/metainterp/warmspot.py", line 236, in finish [translation:ERROR] self.annhelper.finish() [translation:ERROR] File "/home/user/Desktop/pypy/pypy/rpython/annlowlevel.py", line 240, in finish [translation:ERROR] self.finish_annotate() [translation:ERROR] File "/home/user/Desktop/pypy/pypy/rpython/annlowlevel.py", line 259, in finish_annotate [translation:ERROR] ann.complete_helpers(self.policy) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 180, in complete_helpers [translation:ERROR] self.complete() [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 254, in complete [translation:ERROR] self.processblock(graph, block) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 452, in processblock [translation:ERROR] self.flowin(graph, block) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 512, in flowin [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 714, in consider_op [translation:ERROR] raise_nicer_exception(op, str(graph)) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 711, in consider_op [translation:ERROR] resultcell = consider_meth(*argcells) [translation:ERROR] File "<4506-codegen /home/user/Desktop/pypy/pypy/annotation/annrpython.py:749>", line 3, in consider_op_simple_call [translation:ERROR] return arg.simple_call(*args) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/unaryop.py", line 175, in simple_call [translation:ERROR] return obj.call(getbookkeeper().build_args("simple_call", args_s)) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/unaryop.py", line 706, in call [translation:ERROR] return bookkeeper.pbc_call(pbc, args) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/bookkeeper.py", line 668, in pbc_call [translation:ERROR] results.append(desc.pycall(schedule, args, s_previous_result, op)) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/description.py", line 976, in pycall [translation:ERROR] return self.funcdesc.pycall(schedule, args, s_previous_result, op) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/description.py", line 297, in pycall [translation:ERROR] result = schedule(graph, inputcells) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/bookkeeper.py", line 664, in schedule [translation:ERROR] return self.annotator.recursivecall(graph, whence, inputcells) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 395, in recursivecall [translation:ERROR] position_key) [translation:ERROR] File "/home/user/Desktop/pypy/pypy/annotation/annrpython.py", line 235, in addpendingblock [translation:ERROR] assert annmodel.unionof(s_oldarg, s_newarg) == s_oldarg [translation:ERROR] AssertionError': [translation:ERROR] .. v2309 = simple_call(v2301, v2302, v2303, v2304, v2305, v2306, v2307, v2308) [translation:ERROR] .. '(pypy.module.pypyjit.policy:49)PyPyJitIface._compile_hook' [translation:ERROR] Processing block: [translation:ERROR] block@226 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'> [translation:ERROR] in (pypy.module.pypyjit.policy:49)PyPyJitIface._compile_hook [translation:ERROR] containing the following operations: [translation:ERROR] v2309 = simple_call(v2301, v2302, v2303, v2304, v2305, v2306, v2307, v2308) [translation:ERROR] --end--

4 7

the program will make pypy say "Segment fault"
by Spacelee 27 Feb '12

27 Feb '12

on Ubuntu 64bit, 6G memory, 4Core, python2.7.2 Program: import datetime tmp_str1 = 'aaaaaaaaaa' * 10000 tmp_str2 = 'bbbbbbbbbb' * 10000 tmp_str3 = 'bbbbbbbbbb' * 10000 tmp_str4 = 'bbbbbbbbbb' * 10000 s = datetime.datetime.now() for i in xrange(100000): a = "%s%s%s%s" % (tmp_str1, tmp_str2, tmp_str3, tmp_str4) e = datetime.datetime.now() print str(e-s) s = datetime.datetime.now() for i in xrange(100000): a = tmp_str1 + tmp_str2 + tmp_str3 + tmp_str4 e = datetime.datetime.now() print str(e-s) Ouput: Segment Fault -- *Space Lee*

3 2