Mailman 3 July 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor March 10, 2014

March 10, 2014

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Any publications regarding PyPy / trace-based JIT compiler?
by Michael Sioutis Nov. 11, 2012

Nov. 11, 2012

Dear PyPy team, I am in the process of writing a paper that will target some AI conference, and I would like to ask if there are any relevant publications of yours or in general that showcase the possible advantages of trace-based JIT compilation over method-based JIT compilation or static compilation. I just want to use 2 or 3 of them as references when I explain why my implementation is more scalable and robust when compared to some C/C++/Java implementations (appart from different data structures and algorithms). Thank you :) Mike

7 12

MinGW32 support PyPy with mscr90.dll
by bookaa Sept. 19, 2012

Sept. 19, 2012

Windows system, after set cc=mingw32-gcc PyPy can not pass many tests: pytest.py pypy/translator/test/test_unsimplify.py pytest.py pypy/translator/c/test/test_extfunc.py::test_dictlike_environ_getitem ... I find this is because MinGW32 gcc compile default use msvcrt.dll, but we need msvcr90.dll, as Python 2.7 use msvcr90.dll. >>> import ctypes >>> ctypes.util.find_msvcrt() 'msvcr90.dll'And even python.exe dependence on msvcr90.dllref to http://trac.seqan.de/wiki/HowTo/SelectASpecificMsvcrtVersionWithMinGw this is how I make MinGW32 gcc use msvcr90.dll instead: save these to you MinGW directory as file D:\MinGW\lib\gcc\mingw32\4.6.2\specs *cpp: %{posix:-D_POSIX_SOURCE} %{mthreads:-D_MT} -D__MSVCRT_VERSION__=0x0900 *libgcc: %{mthreads:-lmingwthrd} -lmingw32 %{shared-libgcc:-lgcc_s} %{!shared-libgcc:-lgcc_eh} -lgcc -lmoldname90 -lmingwex -lmsvcr90 I suggest this instruction should be add to PyPy doc: MinGW32 support

2 1

cpyext performance
by Stefan Behnel Aug. 31, 2012

Aug. 31, 2012

Hi, I've been working with one of the modules in the Python benchmark suite, namely nbody, and tried to make it run a little faster when compiled with Cython in PyPy. I managed to get a massive speed-up by avoiding some borrowed references during list iteration and using PySequence_GetItem() instead, but now some 50-60% of the runtime are spent in Py_DecRef(). I tried debugging into it, but so far, all my attempts to get anything useful out of PyPy or cpyext have failed. I applied this obvious change """ diff -r 20258fbf10d0 pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py Sun Jul 01 10:02:26 2012 +0200 +++ b/pypy/module/cpyext/pyobject.py Tue Jul 03 11:18:22 2012 +0200 @@ -340,13 +340,13 @@ if obj.c_ob_refcnt == 0: state = space.fromcache(RefcountState) ptr = rffi.cast(ADDR, obj) - if ptr not in state.py_objects_r2w: + try: + w_obj = state.py_objects_r2w.pop(ptr) + except KeyError: # this is a half-allocated object, lets call the deallocator # without modifying the r2w/w2r dicts _Py_Dealloc(space, obj) else: - w_obj = state.py_objects_r2w[ptr] - del state.py_objects_r2w[ptr] w_type = space.type(w_obj) if not w_type.is_cpytype(): _Py_Dealloc(space, obj) """ and it gave me a couple of percent, but beyond that, I'd have to know what paths are actually taken here and where most of the time is spent. Running it through callgrind yields nothing helpful, so I tried running it in plain Python. That failed with an OverflowError in ll2ctypes.py: """ Traceback (most recent call last): File "pypy-1.9/pypy/bin/py.py", line 187, in <module> sys.exit(main_(sys.argv)) File "pypy-1.9/pypy/bin/py.py", line 86, in main_ space = option.make_objspace(config) File "pypy-1.9/pypy/tool/option.py", line 45, in make_objspace None, None, ['Space']) File "pypy-1.9/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "pypy-1.9/pypy/objspace/std/objspace.py", line 18, in <module> from pypy.objspace.std.complexobject import W_ComplexObject File "pypy-1.9/pypy/objspace/std/complexobject.py", line 6, in <module> from pypy.objspace.std.floatobject import W_FloatObject, _hash_float File "pypy-1.9/pypy/objspace/std/floatobject.py", line 48, in <module> class W_FloatObject(W_AbstractFloatObject): File "pypy-1.9/pypy/objspace/std/floatobject.py", line 51, in W_FloatObject from pypy.objspace.std.floattype import float_typedef as typedef File "pypy-1.9/pypy/objspace/std/floattype.py", line 87, in <module> _double_format, _float_format = detect_floatformat() File "pypy-1.9/pypy/objspace/std/floattype.py", line 64, in detect_floatformat rffi.cast(rffi.DOUBLEP, buf)[0] = 9006104071832581.0 File "pypy-1.9/pypy/rpython/lltypesystem/lltype.py", line 1219, in __setitem__ self._obj.setitem(i, val) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 620, in setitem self._storage.contents._setitem(index, value, boundscheck=False) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 279, in _setitem items = self._indexable(index) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 259, in _indexable PtrType = self._get_ptrtype() File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 255, in _get_ptrtype raise e OverflowError: array too large """ No idea what this is supposed to tell me, except that I apparently hit yet another bug in PyPy. Now, having wasted enough time with this, I'd be happy if someone else could take over. I put the C code here that I used: http://cython.org/nbodybench.tar.bz2 You can build it without Cython, just run the included setupdu.py script over it and call pypy -c 'import nbody; print(nbody.test_nbody(1))' Then try to profile that. Stefan

7 51

Developer selection for Py3k
by Antonio Cuni Aug. 31, 2012

Aug. 31, 2012

Hi all, as you probably know, the Py3k [1] proposal is getting funded thanks to our generous donors. During the first round, three of use were selected: me, Alex Gaynor and Benjamin Peterson. However, due to unforeseen unavailability of Alex and Benjamin, we are now looking for one more developer to help with the py3k work. If you are interested in getting paid to work on the Py3k proposal, please apply by replying to this email. To be applicable you need to be an experienced PyPy developer, preferably with a previous experience on the Python Interpreter part of PyPy. ciao, Anto

6 7

Questions Regarding Scipy in Pypy (Not a Feature Request)
by Steven Jackson Aug. 9, 2012

Aug. 9, 2012

I know there is no current plan to implement scipy in pypy, but searching the PyPy website, I was not able to find the reason. If it is not to be included as a feature due to lack of interest or developer time, I am offering to begin rewriting scipy in pure python (I have quite a bit of free time to do this). If it has been discussed and actively excluded from pypy, I would like to know that before I waste too much time rewriting it. I have an active interest in being able to use some modules which depend on scipy while using the PyPy interpreter (especially pyBrain). I have seen the hack on the pypy blog: http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.ht… I could simply use the hack myself, but unless there's a reason not too, I think it'd be nice to have scipy available as a pure python module (perhaps called scipypy in the same manner as numpy). -- Steven Jackson

6 9

Help (again)
by Léonard de Haro Aug. 8, 2012

Aug. 8, 2012

Hi everyone! First of all, thaks for your answers last time. Second of all, I need your help again. I'm coding an imperative version of my interpret and to do so, I coded continuations. I used a similat technique to the one used in the Prolog interpret you advised me to take a look at last time. I define a serie of class all inheriting from a simulated-abstract Contk class. Every class has a method apply which contains instruction for what's next to do. All these methods take the same arguments and (except in one case) return the same kind of tuple. The interpret function uses 4 variables that are updated: expr, env, cont, val. Thus, the result of .apply(_,_,_,_) is the tuple for update. class Contk(object): def __init__(self,*arg): raise NotImplementedError("For abstract class") def apply(self,expr,env,val): raise NotImplementedError("For abstract class") class Endk(Contk): def __init__(self,val): self.val = val def apply(self,expr, env, val): return self.val class Idk(Contk): def __init__(self): pass def apply(self, expr, env, val): return expr, env, Endk(val), val There is a class for each continuation needed by the interpret. The trouble is, I get this error: [translation:ERROR] AnnotatorError': annotation of 'union' degenerated to SomeObject() [translation:ERROR] Simple call of incompatible family: [translation:ERROR] (KeyError getting at the binding!) [translation:ERROR] [translation:ERROR] In <FunctionGraph of (RPinterpretImperative:27)Idk.apply at 0x1b31be8>: [translation:ERROR] Happened at file RPinterpretImperative.py line 28 [translation:ERROR] [translation:ERROR] ==> return expr, env, Endk(val), val [translation:ERROR] [translation:ERROR] Previous annotation: [translation:ERROR] (none) [translation:ERROR] .. v0 = simple_call((classobj Endk), val_0) [translation:ERROR] .. '(RPinterpretImperative:27)Idk.apply' [translation:ERROR] Processing block: [translation:ERROR] block@12 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'> [translation:ERROR] in (RPinterpretImperative:27)Idk.apply [translation:ERROR] containing the following operations: [translation:ERROR] v0 = simple_call((classobj Endk), val_0) [translation:ERROR] v1 = newtuple(expr_0, env_0, v0, val_0) [translation:ERROR] --end-- So at first I thought it was due to tuple that would not accept newly created objects, but after a few tests, it appears that not. Then I tought the annotator couldn't realize by himself that returning an Endk object is the same as returning a Contk, so I annotated the whole file with assertion to force him to realize that my variable cont should be a Contk not any other kind of subclass, but it didn't work either. So I don't know what to do now. I've even tried to implement apply outside of the classes, or to make a function for each parameters returned, but nothing worked. You can find the source here : https://github.com/zebign1/RPython-internship/tree/master/Interpreters/ifF1… (treeClass.py and parser.py as tools, the bugged file is RPinterpretImperative.py) Thanks again. Leonard

3 3

__pycache__ folders
by Marcus Smith July 31, 2012

July 31, 2012

Hello: I'm working on trying to get the pip test suite passing for pypy. I need to figure out the mechanism that turns on/off the use of __pycache__ folders. locally in my pypy v1.9 install, I only see normal *.pyc files for installed distributions. In Travis service builds which also reports v1.9, there are __pycache__ folders with *.pypy-19.pyc files. How is this set or determined in pypy? any help appreciated, thanks Marcus

9 15

any tutorial for compiling pypy on windows?
by gelin yan July 31, 2012

July 31, 2012

Hi All I spent almost three hours for compiling pypy on my windows 7 and got nothing. During the period of translating, I saw many errors about no file "expat.h", "zlib.h" sth like that. I have tried to located those files on different paths but it still failed. I have searched on google a bit however there is only one post which give a brief introduction how to compile pypy 1.4 on windows. It is kinda outdated and obscure. At least I want to know where I can locate those third-party dependencies correctly so that translator can find them.Thanks. Regards gelin yan

2 2

Re: [pypy-dev] pypy-dev Digest, Vol 15, Issue 33
by Robert Grosse July 30, 2012

July 30, 2012

Are you trying to compile it on 64bit Windows? As I understand it, 64 bit Windows isn't supported. I had issues with this too. On Mon, Jul 30, 2012 at 6:00 AM, <pypy-dev-request(a)python.org> wrote: > Send pypy-dev mailing list submissions to > pypy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/pypy-dev > or, via email, send a message with subject or body 'help' to > pypy-dev-request(a)python.org > > You can reach the person managing the list at > pypy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pypy-dev digest..." > > > Today's Topics: > > 1. any tutorial for compiling pypy on windows? (gelin yan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 30 Jul 2012 17:09:46 +0800 > From: gelin yan <dynamicgl(a)gmail.com> > To: PyPy Developer Mailing List <pypy-dev(a)python.org> > Subject: [pypy-dev] any tutorial for compiling pypy on windows? > Message-ID: > < > CABkOF6QyqcFMjQEUmEksqY73xtpdHByumvYQCRkUMdtZ77QH9w(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi All > > I spent almost three hours for compiling pypy on my windows 7 and got > nothing. During the period of translating, I saw many errors about no file > "expat.h", "zlib.h" sth like that. I have tried to located those files on > different paths but it still failed. > > I have searched on google a bit however there is only one post which > give a brief introduction how to compile pypy 1.4 on windows. It is kinda > outdated and obscure. > > At least I want to know where I can locate those third-party > dependencies correctly so that translator can find them.Thanks. > > Regards > > gelin yan >

3 2