Mailman 3 August 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor 10 Mar '14

10 Mar '14

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

Any publications regarding PyPy / trace-based JIT compiler?
by Michael Sioutis 11 Nov '12

11 Nov '12

Dear PyPy team, I am in the process of writing a paper that will target some AI conference, and I would like to ask if there are any relevant publications of yours or in general that showcase the possible advantages of trace-based JIT compilation over method-based JIT compilation or static compilation. I just want to use 2 or 3 of them as references when I explain why my implementation is more scalable and robust when compared to some C/C++/Java implementations (appart from different data structures and algorithms). Thank you :) Mike

7 12

nightly builds require OpenSSL 1.0 ?
by Stefan Behnel 30 Sep '12

30 Sep '12

Hi, my nightly fetch-PyPy build job stopped working two days ago because the nightly builds now require libssl 1.0.0. It's not currently available on our integration test server. Is that dependency really necessary or was it just an accidental side effect of a system update? Stefan

5 9

MinGW32 support PyPy with mscr90.dll
by bookaa 19 Sep '12

19 Sep '12

Windows system, after set cc=mingw32-gcc PyPy can not pass many tests: pytest.py pypy/translator/test/test_unsimplify.py pytest.py pypy/translator/c/test/test_extfunc.py::test_dictlike_environ_getitem ... I find this is because MinGW32 gcc compile default use msvcrt.dll, but we need msvcr90.dll, as Python 2.7 use msvcr90.dll. >>> import ctypes >>> ctypes.util.find_msvcrt() 'msvcr90.dll'And even python.exe dependence on msvcr90.dllref to http://trac.seqan.de/wiki/HowTo/SelectASpecificMsvcrtVersionWithMinGw this is how I make MinGW32 gcc use msvcr90.dll instead: save these to you MinGW directory as file D:\MinGW\lib\gcc\mingw32\4.6.2\specs *cpp: %{posix:-D_POSIX_SOURCE} %{mthreads:-D_MT} -D__MSVCRT_VERSION__=0x0900 *libgcc: %{mthreads:-lmingwthrd} -lmingw32 %{shared-libgcc:-lgcc_s} %{!shared-libgcc:-lgcc_eh} -lgcc -lmoldname90 -lmingwex -lmsvcr90 I suggest this instruction should be add to PyPy doc: MinGW32 support

2 1

cpyext performance
by Stefan Behnel 31 Aug '12

31 Aug '12

Hi, I've been working with one of the modules in the Python benchmark suite, namely nbody, and tried to make it run a little faster when compiled with Cython in PyPy. I managed to get a massive speed-up by avoiding some borrowed references during list iteration and using PySequence_GetItem() instead, but now some 50-60% of the runtime are spent in Py_DecRef(). I tried debugging into it, but so far, all my attempts to get anything useful out of PyPy or cpyext have failed. I applied this obvious change """ diff -r 20258fbf10d0 pypy/module/cpyext/pyobject.py --- a/pypy/module/cpyext/pyobject.py Sun Jul 01 10:02:26 2012 +0200 +++ b/pypy/module/cpyext/pyobject.py Tue Jul 03 11:18:22 2012 +0200 @@ -340,13 +340,13 @@ if obj.c_ob_refcnt == 0: state = space.fromcache(RefcountState) ptr = rffi.cast(ADDR, obj) - if ptr not in state.py_objects_r2w: + try: + w_obj = state.py_objects_r2w.pop(ptr) + except KeyError: # this is a half-allocated object, lets call the deallocator # without modifying the r2w/w2r dicts _Py_Dealloc(space, obj) else: - w_obj = state.py_objects_r2w[ptr] - del state.py_objects_r2w[ptr] w_type = space.type(w_obj) if not w_type.is_cpytype(): _Py_Dealloc(space, obj) """ and it gave me a couple of percent, but beyond that, I'd have to know what paths are actually taken here and where most of the time is spent. Running it through callgrind yields nothing helpful, so I tried running it in plain Python. That failed with an OverflowError in ll2ctypes.py: """ Traceback (most recent call last): File "pypy-1.9/pypy/bin/py.py", line 187, in <module> sys.exit(main_(sys.argv)) File "pypy-1.9/pypy/bin/py.py", line 86, in main_ space = option.make_objspace(config) File "pypy-1.9/pypy/tool/option.py", line 45, in make_objspace None, None, ['Space']) File "pypy-1.9/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "pypy-1.9/pypy/objspace/std/objspace.py", line 18, in <module> from pypy.objspace.std.complexobject import W_ComplexObject File "pypy-1.9/pypy/objspace/std/complexobject.py", line 6, in <module> from pypy.objspace.std.floatobject import W_FloatObject, _hash_float File "pypy-1.9/pypy/objspace/std/floatobject.py", line 48, in <module> class W_FloatObject(W_AbstractFloatObject): File "pypy-1.9/pypy/objspace/std/floatobject.py", line 51, in W_FloatObject from pypy.objspace.std.floattype import float_typedef as typedef File "pypy-1.9/pypy/objspace/std/floattype.py", line 87, in <module> _double_format, _float_format = detect_floatformat() File "pypy-1.9/pypy/objspace/std/floattype.py", line 64, in detect_floatformat rffi.cast(rffi.DOUBLEP, buf)[0] = 9006104071832581.0 File "pypy-1.9/pypy/rpython/lltypesystem/lltype.py", line 1219, in __setitem__ self._obj.setitem(i, val) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 620, in setitem self._storage.contents._setitem(index, value, boundscheck=False) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 279, in _setitem items = self._indexable(index) File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 259, in _indexable PtrType = self._get_ptrtype() File "pypy-1.9/pypy/rpython/lltypesystem/ll2ctypes.py", line 255, in _get_ptrtype raise e OverflowError: array too large """ No idea what this is supposed to tell me, except that I apparently hit yet another bug in PyPy. Now, having wasted enough time with this, I'd be happy if someone else could take over. I put the C code here that I used: http://cython.org/nbodybench.tar.bz2 You can build it without Cython, just run the included setupdu.py script over it and call pypy -c 'import nbody; print(nbody.test_nbody(1))' Then try to profile that. Stefan

7 51

pypy android
by Wiktor Mizdal 31 Aug '12

31 Aug '12

Hi, In article "Almost There - PyPy's ARM Backend" on Pypy blog, in "The incomplete list of open topics:" is point "Build a version that runs on Android". Will somebody work on it? Wiktor Mizdal

2 1

pypy android
by Wiktor Mizdal 31 Aug '12

31 Aug '12

Hi, does Pypy for Android require root Android version? Wiktor Mizdal

5 4

Developer selection for Py3k
by Antonio Cuni 31 Aug '12

31 Aug '12

Hi all, as you probably know, the Py3k [1] proposal is getting funded thanks to our generous donors. During the first round, three of use were selected: me, Alex Gaynor and Benjamin Peterson. However, due to unforeseen unavailability of Alex and Benjamin, we are now looking for one more developer to help with the py3k work. If you are interested in getting paid to work on the Py3k proposal, please apply by replying to this email. To be applicable you need to be an experienced PyPy developer, preferably with a previous experience on the Python Interpreter part of PyPy. ciao, Anto

6 7

Re: [pypy-dev] pymongo compatibility broken with version 2.3
by Jan Riechers 30 Aug '12

30 Aug '12

On 30.08.2012 15:25, Amaury Forgeot d'Arc wrote: > 2012/8/30 Jan Riechers <janpeterr(a)freenet.de>: >> (env) D:\gg_webdev>pypy dbskin.py >> Fatal Python error: PyThreadState_Get: no current thread > > This error message does not appear anywhere in pypy code, > it certainly comes from CPython... > > This message is typical of an extension module that was not recompiled for pypy; > it still links with python27.dll, and the call to PyThreadState_Get() jumps into > an uninitialized CPython interpreter... > > Extension modules need to be recompiled for pypy. > Thank you both for this explanation, and sorry for any confusion. I installed by bad habit using easy_install instead of setup.py install procedure from the github source. All good by now, 2.3 works fine! And sorry Amaury for the direct email response. Jan

1 0

pymongo compatibility broken with version 2.3
by Jan Riechers 30 Aug '12

30 Aug '12

Hello, I know this is not particular subject for this mailing list, but this info might help someone else using pymongo for mongodb with pypy. --- Latest release of the mongodb driver, pymongo (version 2.3, off 29th, August) does not work anymore with Pypy 1.9 and fails to start up with something like this: " (env) D:\gg_webdev>pypy dbskin.py Fatal Python error: PyThreadState_Get: no current thread This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. " But from my code side, there is no greenlet or similar in usage. The problem just arrived when upgrading the driver. So I'm not sure whats the concrete problem with this new version - also after consulting the changelog of the driver ( https://jira.mongodb.org/browse/PYTHON/fixforversion/11146 ) there is nothing mentioned about threading or anything alike that - but it's definitely the culprid. Nevertheless, version 2.2.1 (pymongo-2.2.1_-py2.7.egg) does work like a charm. Jan

3 2