Mailman 3 March 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

update (+patch) on embedding pypy
by Roberto De Ioris 09 Jun '12

09 Jun '12

Hi everyone, i have finally managed to have a pypy plugin into uWSGI via libpypy-c. The first step was exporting a bunch of symbols in libpypy-c: "RPython_StartupCode", "pypy_g_ObjSpace_startup", "pypy_g_call_startup", "pypy_g_State_startup", "pypy_g_Module_startup", "pypy_g_pypy_module_cpyext_state_State" pypy_g_ObjSpace_startup and pypy_g_Module_startup are not strictly required, i have used them only for debugging some very specific areas. Attached there is a patch for latest tip. After that, i have defined a Py_Initialize() replacement: http://projects.unbit.it/uwsgi/browser/plugins/python/python_plugin.c#L8 and placed a bunch of #ifdef for unsupported functions (like PyFile_FromFile) Finally i have built libpypy with pypy translate.py -Ojit --shared and the uWSGI plugin with LDFLAGS="-L." pypy-c uwsgiconfig.py --plugin plugins/pypy core (LDFLAGS is used as i have libpypy-c in the current dir) Run uWSGI with the welcome app (and a bunch of processes): uwsgi --http-socket :8080 --plugin pypy --module welcome --master --processes 8 Incredibly the whole uwsgi api (that it is pretty complex, involving tons of c code) works flawlessly as well as the custom wsgi.input object (the whole upload test suite passes). Running python threads from the WSGI app works too, but mapping python threads to already available pthreads is a no go (all explodes, i will investigate on that) Importing applications as modules works normally, but importing via file requires implementing a bunch of c-api in pypy (i will address that soon) virtualenv, sys.argv and setting programname is not supported (no c api available) i can get rid of this too in the next few days. Important thing: i have tested it only on OSX -- Roberto De Ioris http://unbit.it

6 23

[Repost] PyPy 1.8 50% slower than PyPy 1.7 on my test case
by Laurent Vaucher 04 Jun '12

04 Jun '12

Sorry to repost, but does anyone have an idea about what I could do to track down the source of the slowdown? Should I run with some trace to try to compare how different parts behave? How could I do that? Thanks, Laurent. Hi and first of all, thanks for that great project. Now to my "problem". I'm doing some puzzle-solving, constraint processing with Python and on my particular program, PyPy 1.8 showed a 50% increase in running time over PyPy 1.7. I'm not doing anything fancy (no numpy, itertools, etc.). The program is here: https://github.com/slowfrog/hexiom To reproduce: - fetch hexiom2.py and level36.txt from github - run 'pypy hexiom2.py -sfirst level36.txt' On my machine (Windows 7) the timings are the following: Python 2.6.5/Windows 32 3m35s Python 2.7.1 (930f0bc4125a, Nov 27 2011, 11:58:57) [PyPy 1.7.0 with MSC v.1500 32 bit] 31s Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 18:31:47) [PyPy 1.8.0 with MSC v.1500 32 bit] 48s I'm using the default options. Do you have any idea what could be causing that? Thanks, Laurent.

3 5

Biting off more than I can chew
by Greg Bowyer 07 May '12

07 May '12

The other night I got this burning desire to recreate the Azul GPGC (the java pauseless collector) inside pypy, with a view that it could be used alongside the STM work to make pypy a low (no?) pause concurrent VM. When I started tackling the code I realised I might have bitten off a little more than I can chew. My question (probably one of many to irritate and annoy all the fine folks here) would be, is there a sensible way to compile into pypy a small amount of C code that can be used to bootstrap and bridge some esoteric c libraries into pypy, the code that I want to run, on startup of pypy would be the following https://bitbucket.org/GregBowyer/pypy-c4gc/changeset/0de575b3a8d1#chg-azm_m… I have noticed that there are a few bridges into direct c code in the pypy source, but I cannot fathom how these are linked into RPython -- Greg

3 5

STM proposal funding
by Armin Rigo 17 Apr '12

17 Apr '12

Hi all, For those of you who missed it: the proposal on Software Transactional Memory is launched ("remove the GIL but give a better way to write multi-threaded applications"). It is available at http://pypy.org/tmdonate.html . I plan to ask the PSF for some participation, following the push towards doing so at the PyCon PSF meeting. See http://mail.python.org/pipermail/python-dev/2012-March/117680.html , "Funding from the Python Software Foundation". A bientôt, Armin.

5 14

Bringing Cython and PyPy closer together
by Stefan Behnel 01 Apr '12

01 Apr '12

Hi, I'm breaking out of the thread where this topic was started ("offtopic, ontopic, ...") because it is getting too long and unfocussed to follow. The current state of the discussion seems to be that PyPy provides ways to talk to C code, but nothing as complete as CPython's C-API in the sense that it allows efficient two-way communication between C code and Python objects. Thus, we need to either improve this or look for alternatives. In order to get us more focussed on what can be done and what the implications are, so that we may eventually be able to decide what should be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement Proposal). http://wiki.cython.org/enhancements/pypy Please add to it as you see fit. In general, it is ok for a CEP to present diverging arguments, but please try to give them a structure in order to keep the overall document focussed. Stefan

9 64

Re: [pypy-dev] Speeding up zlib in standard library
by Peter Cock 01 Apr '12

01 Apr '12

(Replying on list - assuming Justin went off list my mistake) On Fri, Mar 16, 2012 at 4:54 AM, Justin Peel <peelpy(a)gmail.com> wrote: > Two things to mention. First, if you are going to use valgrind, you > will need to build your own pypy because, as far as I know, the > buildbot ones do not have the debug info so you won't have any helpful > function names in your profile. If I remember correctly, most of the > time used is in _operate in rzlib.py. > > Also, in my opinion, the next thing to try in speeding up this code is > to do a faster copy than a char by char copy for copying to the input > buffer that is sent to the external C function. I'm not sure if the > copying from the output buffer to the string builder is char by char > or not. Looking at the code for rzlib.py, _operate does seem to be the core function, and so likely the hot spot. https://bitbucket.org/pypy/pypy/src/default/pypy/rlib/rzlib.py It uses the StringBuilder class from rstring via the append_charpsize method, defined as follows: https://bitbucket.org/pypy/pypy/src/default/pypy/rlib/rstring.py def append_charpsize(self, s, size): l = [] for i in xrange(size): l.append(s[i]) self.l.append(self.tp("").join(l)) self._grow(size) So it is indeed doing a char by char copy of the string from Python to zlib (in my case to decompress a long chunk of data). I don't know enough about PyPy's internals to say if something naive like this would work faster (guessing looking at the append_slice method): def append_charpsize(self, s, size): assert 0 <= size self.l.append(s[0:size]) self._grow(size) Presumably to try this idea out I'm going to first need to get PyPy to build locally? Thanks, Peter

4 11

Profiling pypy code
by Timothy Baldridge 01 Apr '12

01 Apr '12

I've been trying to find ways to optimize my clojure-on-python project. And I've been running through my code in jitviewer. However it seems that jitviewer only displays the number of times a given loop was run. What should I look into for benchmarking actual times spent in each loop? It seems that my optimization efforts would be best spent on areas of code that have a high hit count AND have a long execution time. Any ideas? Thanks, Timothy -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

2 1

dotviewer bug on Windows system
by bookaa 31 Mar '12

31 Mar '12

On Windows system, simply run: pypy-pypy-2346207d9946\dotviewer>dotviewer.py ..\pypy\doc\image\JIT.dot will get a error: graphparse.PlainParseError: bad 'node' Finally I fix this problem by add 3 line in parse_plain function of dotviewer\graphparse.py: if lines[i].endswith('\\\n'): # line ending in '\' lines[i] = lines[i][:-2] + lines[i+1] del lines[i+1] elif lines[i].endswith('\\\r\n'): # line ending in '\' lines[i] = lines[i][:-3] + lines[i+1] del lines[i+1] bookaa

2 1

Re: [pypy-dev] [pypy-commit] pypy dynamic-specialized-tuple: kill an assert that doesn't hold now.
by Hakan Ardo 30 Mar '12

30 Mar '12

Hi, if the descr can change and assumptions are made about it you'll need to add it to virtualstate.NotVirtualStateInfo and check it in generalization_of and _generate_guards. Otherwise if a peeled loop becomes specialized to some specific descr and that descr is changed on a bridge, the bridge will jump to the peeled loop, but it has to jump to the preamble. On Fri, Mar 30, 2012 at 4:00 AM, alex_gaynor <noreply(a)buildbot.pypy.org> wrote: > Author: Alex Gaynor <alex.gaynor(a)gmail.com> > Branch: dynamic-specialized-tuple > Changeset: r54088:5b813268023d > Date: 2012-03-29 21:59 -0400 > http://bitbucket.org/pypy/pypy/changeset/5b813268023d/ > > Log: kill an assert that doesn't hold now. > > diff --git a/pypy/jit/metainterp/optimizeopt/optimizer.py b/pypy/jit/metainterp/optimizeopt/optimizer.py > --- a/pypy/jit/metainterp/optimizeopt/optimizer.py > +++ b/pypy/jit/metainterp/optimizeopt/optimizer.py > @@ -59,7 +59,6 @@ > def make_len_gt(self, mode, descr, val): > if self.lenbound: > assert self.lenbound.mode == mode > - assert self.lenbound.descr == descr > self.lenbound.bound.make_gt(IntBound(val, val)) > else: > self.lenbound = LenBound(mode, descr, IntLowerBound(val + 1)) > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit -- Håkan Ardö

2 4