Mailman 3 August 2011 - pypy-dev

Saving and reloading JIT optimizations
by David Fraser Aug. 30, 2011

Aug. 30, 2011

Hi This is a fairly naive question from a newbie... The pypy JIT takes a while to work out which parts of python code need optimization etc, and only after that phase do the speedups become relevant. Have there been any efforts (indeed, is it a feasible idea at all) that look at saving these optimizations for future runs of the same codebase? The advantages would be for: * Small programs that get run frequently - you could pre-tune the JIT but running a longer batch and then save the results … [View More]

6 10

Duplicated guards
by Hakan Ardo Aug. 29, 2011

Aug. 29, 2011

Hi, I've been experimenting a bit with heuristics such as "not retracing loops with a lot of guards" or "not unrolling if the peel loop is not much better than the preamble". In some situations is seem possible to increase performace by such means (go: +37%, html5lib: +13%, rietveld: +8%). However I think the real problem is that we duplicate guards and that this leads to an explosion of bridges. I added test_excessive_bridgeing to test_ajit on the jit-limit_peeling branch. It is a simple … [View More]

3 3

Re: [pypy-dev] [COMMENT] Pull request #8 for pypy/pypy: added numpy.sort, tests, and docstring for numpy.array.sort
by Laura Creighton Aug. 29, 2011

Aug. 29, 2011

I'd like it if the comments on pull requests came with the name of the person making the request, in this case yasirs. Is this something we can do on our end, (I don't think so) or something we need to ask Bitbucket to change? Laura In a message of Sun, 28 Aug 2011 06:32:52 -0000, Bitbucket writes: >New comment on pull request: > >https://bitbucket.org/pypy/pypy/pull-request/8/added-numpysort-tests-and- >do= >cstring-for#comment-35 > >Alex Gaynor (alex_gaynor) said: … [View More]

2 1

STM on CPython
by Armin Rigo Aug. 27, 2011

Aug. 27, 2011

Hi, A follow-up to the blog post about Software Transactional Memory (STM) at http://morepypy.blogspot.com/ . First, there is a growing number of papers out there that seems to give solid evidence that STM is "better" than lock-based systems, for various definitions of "better". In fact Greg Wilson pointed to http://www.neverworkintheory.org/?p=122 ; it's a study of students, showing that it takes them less time to implement the exercise with STM than fine-grained locks --- but most … [View More]importantly, only 10% of the programs produced contain errors, as opposed to 70% with fine-grained locks. There are also more abstract definitions of "better", e.g. for my point of view I'd say that the limiting ingredient in locks is that they don't compose: http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_opera… . But more generally it seems to be emerging as a consensus in the last 5 years. Brett C. made in 2003 a post on python-dev that was rejected (see http://mail.python.org/pipermail/python-dev/2003-February/033259.html); it was about a hack to extend CPython's GIL, giving "atomic sections" --- which would be implemented very simply by not releasing the GIL for a while. This was rejected as a hack that only worked because we had a GIL to start with, and that would no longer work as soon as somebody came up with a great idea to remove it. Me, of course, at that time, I was sure that this was the correct decision as well. In retrospect, I would argue that this was possibly one of the most dreadful decision ever made in Python. It would be exactly STM (without actually running on more than on core, but that's an implementation detail). If we had it 8 years ago, we would have had all this time to get experience with STM in a nice language and how you don't need locks at all to do multithreading. And that would seem quite precisely in line with the original goal of Python, i.e. to make programming easier at the expense of raw performance (in this case it is even worse, because we would not have lost any performance, but only lost a hypothetical future path in which to gain more performance). So, all this to say: 8 years later, I implemented that on CPython: https://bitbucket.org/arigo/cpython-withatomic/overview (on the 2.7 branch). It is sadly a fork of CPython instead of a C extension module because it needs to access and change a few things in ceval.c. The total patch is 190 lines, and that's because I need a new type (very verbose). It implements "thread.atomic", which you use in a "with" statement. So you write "with atomic:" and you get precisely the simple case described here: http://en.wikipedia.org/wiki/Software_transactional_memory#Proposed_languag… . Fwiw it's probably also a minimal patch to PyPy. Sorry for being mostly off-topic for once... A bientôt, Armin. [View Less]

3 3

Re: [pypy-dev] [pypy-commit] pypy faster-nested-scopes: fix tests. no clue where the extra arraylen comes from?
by Hakan Ardo Aug. 27, 2011

Aug. 27, 2011

If a getarrayitem is moved into the short_preamble we also need guards on the array len as it might be out of bound at the end of some future bridge. After jit-short_from_state the jump of the loop is produced by inlining the short preamble there (that approach does not crash if the optimizers introduces new boxes). It is typically optimized out entierly. In case of an array len check the guard is optimized out while the arraylen remains. It should be killed by the backend though as it's result … [View More]is never used. On Fri, Aug 26, 2011 at 11:19 PM, cfbolz <noreply(a)buildbot.pypy.org> wrote: > Author: Carl Friedrich Bolz <cfbolz(a)gmx.de> > Branch: faster-nested-scopes > Changeset: r46806:77b0daebad60 > Date: 2011-08-26 23:25 +0200 > http://bitbucket.org/pypy/pypy/changeset/77b0daebad60/ > > Log: fix tests. no clue where the extra arraylen comes from? > > diff --git a/pypy/module/pypyjit/test_pypy_c/test_call.py b/pypy/module/pypyjit/test_pypy_c/test_call.py > --- a/pypy/module/pypyjit/test_pypy_c/test_call.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_call.py > @@ -67,24 +67,17 @@ > assert log.opnames(ops) == ["guard_value", > "getfield_gc", "guard_value", > "getfield_gc", "guard_value", > - "getfield_gc", "guard_nonnull_class"] > - # LOAD_GLOBAL of OFFSET but in different function partially folded > + "guard_not_invalidated"] > + # LOAD_GLOBAL of OFFSET but in different function folded > # away > - # XXX could be improved > ops = entry_bridge.ops_by_id('add', opcode='LOAD_GLOBAL') > - assert log.opnames(ops) == ["guard_value", "getfield_gc", "guard_value"] > + assert log.opnames(ops) == ["guard_not_invalidated"] > # > # two LOAD_GLOBAL of f, the second is folded away > ops = entry_bridge.ops_by_id('call', opcode='LOAD_GLOBAL') > - assert log.opnames(ops) == ["getfield_gc", "guard_nonnull_class"] > + assert log.opnames(ops) == [] > # > assert entry_bridge.match_by_id('call', """ > - p29 = getfield_gc(ConstPtr(ptr28), descr=<GcPtrFieldDescr pypy.objspace.std.celldict.ModuleCell.inst_w_value .*>) > - guard_nonnull_class(p29, ConstClass(Function), descr=...) > - p33 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_code .*>) > - guard_value(p33, ConstPtr(ptr34), descr=...) > - p35 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_w_func_globals .*>) > - p36 = getfield_gc(p29, descr=<GcPtrFieldDescr pypy.interpreter.function.Function.inst_closure .*>) > p38 = call(ConstClass(getexecutioncontext), descr=<GcPtrCallDescr>) > p39 = getfield_gc(p38, descr=<GcPtrFieldDescr pypy.interpreter.executioncontext.ExecutionContext.inst_topframeref .*>) > i40 = force_token() > @@ -100,19 +93,16 @@ > # ----------------------------- > loop, = log.loops_by_id('call') > assert loop.match(""" > - i12 = int_lt(i5, i6) > - guard_true(i12, descr=...) > + guard_not_invalidated(descr=...) > + i9 = int_lt(i5, i6) > + guard_true(i9, descr=...) > + i10 = force_token() > + i12 = int_add(i5, 1) > i13 = force_token() > - i15 = int_add(i5, 1) > - i16 = int_add_ovf(i15, i7) > - guard_no_overflow(descr=...) > - i18 = force_token() > - i20 = int_add_ovf(i16, 1) > - guard_no_overflow(descr=...) > - i21 = int_add_ovf(i20, i7) > + i15 = int_add_ovf(i12, 1) > guard_no_overflow(descr=...) > --TICK-- > - jump(p0, p1, p2, p3, p4, i21, i6, i7, p8, p9, p10, p11, descr=<Loop0>) > + jump(p0, p1, p2, p3, p4, i15, i6, i7, p8, descr=<Loop0>) > """) > > def test_method_call(self): > diff --git a/pypy/module/pypyjit/test_pypy_c/test_globals.py b/pypy/module/pypyjit/test_pypy_c/test_globals.py > --- a/pypy/module/pypyjit/test_pypy_c/test_globals.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_globals.py > @@ -20,11 +20,9 @@ > guard_value(p10, ConstPtr(ptr11), descr=...) > p12 = getfield_gc(p10, descr=<GcPtrFieldDescr .*W_DictMultiObject.inst_strategy .*>) > guard_value(p12, ConstPtr(ptr13), descr=...) > - p15 = getfield_gc(ConstPtr(ptr14), descr=<GcPtrFieldDescr .*ModuleCell.inst_w_value .*>) > - guard_isnull(p15, descr=...) > guard_not_invalidated(descr=...) > p19 = getfield_gc(ConstPtr(p17), descr=<GcPtrFieldDescr .*W_DictMultiObject.inst_strategy .*>) > guard_value(p19, ConstPtr(ptr20), descr=...) > p22 = getfield_gc(ConstPtr(ptr21), descr=<GcPtrFieldDescr .*ModuleCell.inst_w_value .*>) > guard_nonnull(p22, descr=...) > - """) > \ No newline at end of file > + """) > diff --git a/pypy/module/pypyjit/test_pypy_c/test_instance.py b/pypy/module/pypyjit/test_pypy_c/test_instance.py > --- a/pypy/module/pypyjit/test_pypy_c/test_instance.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_instance.py > @@ -52,7 +52,7 @@ > i10 = int_add_ovf(i5, i7) > guard_no_overflow(descr=...) > --TICK-- > - jump(p0, p1, p2, p3, p4, i10, i6, p7, i7, p8, descr=<Loop0>) > + jump(p0, p1, p2, p3, p4, i10, i6, i7, p8, descr=<Loop0>) > """) > > def test_getattr_with_dynamic_attribute(self): > @@ -151,6 +151,7 @@ > assert loop.match_by_id('loadattr', > ''' > guard_not_invalidated(descr=...) > + i16 = arraylen_gc(p10, descr=<GcPtrArrayDescr>) > i19 = call(ConstClass(ll_dict_lookup), _, _, _, descr=...) > guard_no_exception(descr=...) > i21 = int_and(i19, _) > diff --git a/pypy/module/pypyjit/test_pypy_c/test_math.py b/pypy/module/pypyjit/test_pypy_c/test_math.py > --- a/pypy/module/pypyjit/test_pypy_c/test_math.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_math.py > @@ -47,6 +47,7 @@ > assert loop.match(""" > i2 = int_lt(i0, i1) > guard_true(i2, descr=...) > + guard_not_invalidated(descr=...) > f1 = cast_int_to_float(i0) > i3 = float_eq(f1, inf) > i4 = float_eq(f1, -inf) > @@ -60,4 +61,4 @@ > i7 = int_add(i0, f1) > --TICK-- > jump(..., descr=) > - """) > \ No newline at end of file > + """) > diff --git a/pypy/module/pypyjit/test_pypy_c/test_string.py b/pypy/module/pypyjit/test_pypy_c/test_string.py > --- a/pypy/module/pypyjit/test_pypy_c/test_string.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_string.py > @@ -105,5 +105,5 @@ > i58 = int_add_ovf(i6, i57) > guard_no_overflow(descr=...) > --TICK-- > - jump(p0, p1, p2, p3, p4, p5, i58, i7, i8, p9, p10, descr=<Loop4>) > + jump(p0, p1, p2, p3, p4, p5, i58, i7, descr=<Loop4>) > """) > _______________________________________________ > pypy-commit mailing list > pypy-commit(a)python.org > http://mail.python.org/mailman/listinfo/pypy-commit > -- Håkan Ardö [View Less]

1 0

Re: [pypy-dev] status of scheme implementation
by Juergen Boemmels Aug. 26, 2011

Aug. 26, 2011

Hi Armin, >Hi Juergen, > >On Sun, Aug 21, 2011 at 3:15 PM, Juergen Boemmels <boemmels(a)web.de> wrote: >> I just found the scheme implementation >> https://bitbucket.org/pypy/lang-scheme/ >> but there is no history besides the initial import. >> >> Is this maintained? I.e. is it usful to send patches for improving this? > >As you may have noticed by now (sorry for the delay), no, this is not >really maintained any more... You are welcome to … [View More]

2 1

Re: [pypy-dev] [pypy-commit] pypy reflex-support: Remove elidable_promote that made the translation choke.
by Antonio Cuni Aug. 26, 2011

Aug. 26, 2011

Hi Wim, On 16/08/11 19:47, wlav wrote: > Author: Wim Lavrijsen<WLavrijsen(a)lbl.gov> > Branch: reflex-support > Changeset: r46550:17b6f87c70bc > Date: 2011-08-16 10:54 -0700 > http://bitbucket.org/pypy/pypy/changeset/17b6f87c70bc/ > > Log: Remove elidable_promote that made the translation choke. > > diff --git a/pypy/module/cppyy/interp_cppyy.py b/pypy/module/cppyy/interp_cppyy.py > --- a/pypy/module/cppyy/interp_cppyy.py > +++ b/pypy/module/cppyy/… [View More]

2 3

status of scheme implementation
by Juergen Boemmels Aug. 26, 2011

Aug. 26, 2011

Hi, I just found the scheme implementation https://bitbucket.org/pypy/lang-scheme/ but there is no history besides the initial import. Is this maintained? I.e. is it usful to send patches for improving this? And is this the right mailing list for asking questions about lang-scheme? Anyway, I played a little with it and found one annoyance: The translated parser does not report the position of the parse error. There was something prepared, but commented out (I think because it didn't … [View More]

2 1

List index out of range for many tests
by Mitchell Hashimoto Aug. 26, 2011

Aug. 26, 2011

Hello, I'm taking a look at some failing tests in the buildbot and trying to mirror them here on my local machine, but the tests are giving me the following error: File "/Users/mitchellh/code/3rdparty/python/pypy/pypy/interpreter/pyframe.py", line 193, in pushvalue self.locals_stack_w[depth] = w_object IndexError: list assignment index out of range The full stack trace: https://gist.github.com/e458477a6c60c3d15b9c And the command I'm running: pypy> CC=gcc python pytest.py -s lib-… [View More]

2 3

Incompatibility in binary distribution of libraries?
by Serhat Sevki Dincer Aug. 25, 2011

Aug. 25, 2011

Start with main.py and mylib.py with python2.7 that comes with ubuntu 11.04 i386, run python2.7 main.py the program runs correctly and a file mylib.pyc is produced. Then, if I remove or rename mylib.py file, pyc file is sufficient for main.py to work python2.7 main.py still works. With pypy 1.6, after I rename mylib.py, and try pypy main.py it complains with Traceback (most recent call last): File "app_main.py", line 53, in run_toplevel File "main.py", line 1, in <module> from … [View More]

2 1