Mailman 3 September 2011 - pypy-dev

Performance, json and standard library
by Maciej Fijalkowski 26 Sep '11

26 Sep '11

Hello. I would like to raise the topic of modifying standard library for performance reasons in *some* places. I know the policy so far is to avoid modifications as much as possible and in general I agree. For example the changes justinpeel made to bz2 (or tarfile? please remind me about details) were not good, since it seems equivalent changes can be achieved by tweaking the JIT. However, json comes to mind. The situation is as follows: * json in stdlib is some old version of simplejson * simplejson has been played with to improve the performance on top of pypy * there are reports that relatively simple changes will improve performance: https://bugs.pypy.org/issue868 * json in stdlib will *not* be updated for 2.7 series and even if it gets updated for 3.x it'll be incompatible So while I agree that ideally, JIT could handle whatever it has, but maybe json is an example good enough to warrant changes. There are people out there who would base migration to pypy on json performance for example. Any opinions? Cheers, fijal

6 9

Problems when trying to translate.
by Александр Седов 26 Sep '11

26 Sep '11

Translating PyPy on Linux with -O1 --continuation --withmod-_continuation I've got the error: ValueError: unknown value for translation.gc: 'ref' Googling gave me no results. Is there a way to fix this? PS: By the way, translation procedure reports about pypy.module.interp_mmap unreachable block in W_MMap.descr_getitem.

3 3

Re: [pypy-dev] Performance, json and standard library
by Laura Creighton 24 Sep '11

24 Sep '11

In a message of Sat, 24 Sep 2011 14:47:33 +0200, Maciej Fijalkowski writes: >Hello. > >I would like to raise the topic of modifying standard library for >performance reasons in *some* places. I know the policy so far is to >avoid modifications as much as possible and in general I agree. For >example the changes justinpeel made to bz2 (or tarfile? please remind >me about details) were not good, since it seems equivalent changes can >be achieved by tweaking the JIT. > >However, json comes to mind. The situation is as follows: > >* json in stdlib is some old version of simplejson >* simplejson has been played with to improve the performance on top of py >py >* there are reports that relatively simple changes will improve >performance: https://bugs.pypy.org/issue868 >* json in stdlib will *not* be updated for 2.7 series and even if it >gets updated for 3.x it'll be incompatible > >So while I agree that ideally, JIT could handle whatever it has, but >maybe json is an example good enough to warrant changes. There are >people out there who would base migration to pypy on json performance >for example. > >Any opinions? > >Cheers, >fijal I am at PyCON UK right now. I have already had 4 conversations with people who badly want better json performance. (And one person who says zip performance bad, is it? I told him to write a bug report), and its only lunch time of day 1 of the con. Faster json thus appears to be an itch that needs a lot of scratching around here. Laura

1 0

rlist and ll_delitem_nonneg index
by Zariko Taba 23 Sep '11

23 Sep '11

Hi pypy ! I hit an assert in pypy/annotation/annrpython.py in addpendingblock line 231 : assert annmodel.unionof(s_oldarg, s_newarg) == s_oldarg I think I found an explanation by digging in code : The treated block is in the "ll_delitem_nonneg" function, s_newarg is a "SomeInteger(nonneg=True)" and s_oldarg is a "SomeInteger(nonneg=False)" and in fact they are the "index" argument of this function. oldarg comes from a previous annotation using the "rtype_method_remove" of rlist.py. "rtype_method_remove" use "ll_listremove" which use : ll_delitem_nonneg(..., ll_listindex(...)) Looking at ll_listindex, it seems clear that the index can be proved as always positive. That's probably why s_oldarg is a "SomeInteger(nonneg=True)" The new annotation comes from the method "rtype_delitem" (class __extend__ in rlist.py). the index from hop is always positive, so the code : if hop.args_s[1].nonneg: llfn = ll_delitem_nonneg select "ll_delitem_nonneg" as the deletion function. But the index given to the annotator is taken from : v_lst, v_index = hop.inputargs(r_lst, Signed) v_index is a "Signed" The information concerning the "non negative" property of the index integer is lost here. index is now "SomeInteger(nonneg=False)" and trigs the assert. What do you think ? Zariko.

2 1

Question about byte-code hacking
by Steven D'Aprano 23 Sep '11

23 Sep '11

Hi guys, Over on the python-ideas mailing list, there is a long thread about the default argument hack in functions, used for micro-optimizations, early-binding, and monkey-patching. Various alternatives are being argued about. One proposal put forward involves bytecode manipulations to change global lookups to local so that one could have a decorator that "injects" a value into a copy of the function. What's the PyPy position on bytecode hacking? Good, bad, evil, don't mind either way? For those who care, the thread starts here: http://mail.python.org/pipermail/python-ideas/2011-September/011691.html (beware, it's long). Thanks in advance, -- Steven

4 6

Number of constants in a jitted rpython interpreter
by Zariko Taba 22 Sep '11

22 Sep '11

Hi pypy ! I'm still exploring rpython and I face a problem when adding a jit to an interpreter. In Assembler class (pypy.jit.codewriter.assembler), in emit_const method, it seems to be assumed that there is no more than 256 constants. (constant seems to be accessed in a array with a 1 byte index). If I try to translate an interpreter with more than 256 constant objects (like string ?), I get this error : [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/goal/translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 810, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 286, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 397, in task_pyjitpl_lltype [translation:ERROR] backend_name=self.config.translation.jit_backend, inline=True) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/metainterp/warmspot.py", line 42, in apply_jit [translation:ERROR] **kwds) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/metainterp/warmspot.py", line 199, in __init__ [translation:ERROR] self.codewriter.make_jitcodes(verbose=verbose) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/codewriter.py", line 72, in make_jitcodes [translation:ERROR] self.transform_graph_to_jitcode(graph, jitcode, verbose) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/codewriter.py", line 61, in transform_graph_to_jitcode [translation:ERROR] self.assembler.assemble(ssarepr, jitcode) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 35, in assemble [translation:ERROR] self.write_insn(insn) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 135, in write_insn [translation:ERROR] is_short = self.emit_const(x, kind, allow_short=True) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 108, in emit_const [translation:ERROR] self.code.append(chr(self.constants_dict[key])) [translation:ERROR] ValueError: character code not in range(256) With this snippet of code : self.constants_dict[key] = 256 - len(constants) If len(constants) is 257, then self.constants_dict[key] is -1 and chr(-1) raise the ValueError. I attached a (really) stupid example to reproduce. When I browse pypy sources in rpython, I can't believe there is less than 256 constants of type 'ref'. What do you think ? Did I miss something ? Thanks ! Zariko.

3 3

Sandbox examples
by Geoffrey Thomas 21 Sep '11

21 Sep '11

Hi, I'm looking at building a real application using PyPy's sandbox mode, and am having a harder time than I'd expect finding any examples of people using the sandbox in the "real world". Specifically, I'm not easily finding examples of interaction scripts other than pypy_interact.py in the PyPy source tree, and would be curious to take a look at how people virtualize things that pypy_interact.py currently doesn't handle. Would any of you have pointers to other interaction scripts, or at least would like to talk about how you've used the sandbox, even if you can't provide a pointer to the code? (I'm not subscribed to the list -- please keep me in the Cc on replies, or let me know if I'm supposed to subscribe.) Thanks, -- Geoffrey Thomas geofft(a)mit.edu

3 2

contributing to pypy
by Matthew Kaniaris 21 Sep '11

21 Sep '11

Hello, I'd like to contribute to pypy. I've been following the project for a while but don't know much about the internals apart from reading the docs. Does anyone have an idea for a good place to start? -kans

2 1

Spurious dict lookups in my JIT loops
by Boris 21 Sep '11

21 Sep '11

Hi, I've been trying out writing my own interpreter using the PyPy framework recently, as a bit of fun. I've been trying to get the JIT to optimize a trivial loop down to the minimal amount of operations. With judicious use of `_immutable_fields_` and `_virtualizable2_`, I've got pretty close. But I'm still seeing lots of calls to `ll_dict_lookup__dicttablePtr_Signed_Signed`, which don't correspond to any code in my interpreter. I don't think I even have any dicts that take integer keys. Could someone give me a hint where these are coming from and for what purpose? Or perhaps how to inspect the dicts or get further info? I append what I'm seeing in the logs (Note: I've augmented the logs to give the raw pointer). In this case, it is only looking up the value 21, but I've seen other values in addition when running other programs. The setarrayitem_gc calls are expected - it is nulling out the stack that was being used. Everything from i17 is unexpected. I tested on revisions 00711ff1e03d and 96a212b0688a. Thanks, Boris ############################# [3fd12ea6569db] {jit-log-opt-loop # Loop 0 : loop with 37 ops [p0, p1, p2, i3, p4, p5] debug_merge_point(0, '::Test$iinit:20') +113: i7 = int_add(i3, 1) debug_merge_point(0, '::Test$iinit:22') debug_merge_point(0, '::Test$iinit:23') debug_merge_point(0, '::Test$iinit:26') +116: setarrayitem_gc(p5, 0, ConstPtr(ptr9,0x0), descr=<GcPtrArrayDescr>) +126: setarrayitem_gc(p5, 1, ConstPtr(ptr11,0x0), descr=<GcPtrArrayDescr>) +133: i13 = uint_lt(i7, 10000) guard_true(i13, descr=<Guard2>) [p0, p1, p2, p4, i7] +145: i17 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr15,0x84c2958), 21, 21, descr=<SignedCallDescr>) +176: guard_no_exception(, descr=<Guard3>) [p0, i17, p1, p2, p4, i7] +189: i19 = int_and(i17, -2147483648) +195: i20 = int_is_true(i19) guard_true(i20, descr=<Guard4>) [p0, p1, p2, p4, i7] +204: i23 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr22,0x84c2978), 21, 21, descr=<SignedCallDescr>) +235: guard_no_exception(, descr=<Guard5>) [p0, i23, p1, p2, p4, i7] +248: i24 = int_and(i23, -2147483648) +254: i25 = int_is_true(i24) guard_true(i25, descr=<Guard6>) [p0, p1, p2, p4, i7] +263: i28 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr27,0x84c2988), 21, 21, descr=<SignedCallDescr>) +294: guard_no_exception(, descr=<Guard7>) [p0, i28, p1, p2, p4, i7] +307: i29 = int_and(i28, -2147483648) +313: i30 = int_is_true(i29) guard_true(i30, descr=<Guard8>) [p0, p1, p2, p4, i7] +322: i33 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr32,0x84c2998), 21, 21, descr=<SignedCallDescr>) +353: guard_no_exception(, descr=<Guard9>) [p0, i33, p1, p2, p4, i7] +366: i34 = int_and(i33, -2147483648) +372: i35 = int_is_true(i34) guard_false(i35, descr=<Guard10>) [p0, p1, p2, p4, i7] +381: i38 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr37,0x84c2968), 21, 21, descr=<SignedCallDescr>) +412: guard_no_exception(, descr=<Guard11>) [p0, i38, p1, p2, p4, i7] +425: i39 = int_and(i38, -2147483648) +431: i40 = int_is_true(i39) guard_true(i40, descr=<Guard12>) [p0, p1, p2, p4, i7] debug_merge_point(0, '::Test$iinit:20') +440: i41 = arraylen_gc(p5, descr=<GcPtrArrayDescr>) +440: jump(p0, p1, p2, i7, p4, p5, descr=<Loop0>) +448: --end of the loop-- [3fd12ea696ce9] jit-log-opt-loop}

4 6

Before I start hacking on numpy
by matti picus 20 Sep '11

20 Sep '11

I would really love to have 2 dimensional matrices in micronumpy, and am willing to donate some hours of coding. There seems to be a number of "heads" on the mercurial tree that use numpy in their keyword. Can anyone give me pointer as to what branch (maybe just tip?) would be the recommended one to continue with? Any coding guidelines? My goal is to get to an implementation of inverse on a smallish (5x5 matrix) so I can build some useful code. Matti

4 3