Mailman 3 September 2011 - pypy-dev

Performance, json and standard library
by Maciej Fijalkowski Sept. 27, 2011

Sept. 27, 2011

Hello. I would like to raise the topic of modifying standard library for performance reasons in *some* places. I know the policy so far is to avoid modifications as much as possible and in general I agree. For example the changes justinpeel made to bz2 (or tarfile? please remind me about details) were not good, since it seems equivalent changes can be achieved by tweaking the JIT. However, json comes to mind. The situation is as follows: * json in stdlib is some old version of simplejson * … [View More]

6 9

Problems when trying to translate.
by Александр Седов Sept. 26, 2011

Sept. 26, 2011

Translating PyPy on Linux with -O1 --continuation --withmod-_continuation I've got the error: ValueError: unknown value for translation.gc: 'ref' Googling gave me no results. Is there a way to fix this? PS: By the way, translation procedure reports about pypy.module.interp_mmap unreachable block in W_MMap.descr_getitem.

3 3

Re: [pypy-dev] Performance, json and standard library
by Laura Creighton Sept. 24, 2011

Sept. 24, 2011

In a message of Sat, 24 Sep 2011 14:47:33 +0200, Maciej Fijalkowski writes: >Hello. > >I would like to raise the topic of modifying standard library for >performance reasons in *some* places. I know the policy so far is to >avoid modifications as much as possible and in general I agree. For >example the changes justinpeel made to bz2 (or tarfile? please remind >me about details) were not good, since it seems equivalent changes can >be achieved by tweaking the JIT. > &… [View More]

1 0

rlist and ll_delitem_nonneg index
by Zariko Taba Sept. 23, 2011

Sept. 23, 2011

Hi pypy ! I hit an assert in pypy/annotation/annrpython.py in addpendingblock line 231 : assert annmodel.unionof(s_oldarg, s_newarg) == s_oldarg I think I found an explanation by digging in code : The treated block is in the "ll_delitem_nonneg" function, s_newarg is a "SomeInteger(nonneg=True)" and s_oldarg is a "SomeInteger(nonneg=False)" and in fact they are the "index" argument of this function. oldarg comes from a previous annotation using the "rtype_method_remove" of rlist.py. "… [View More]

2 1

Question about byte-code hacking
by Steven D'Aprano Sept. 23, 2011

Sept. 23, 2011

Hi guys, Over on the python-ideas mailing list, there is a long thread about the default argument hack in functions, used for micro-optimizations, early-binding, and monkey-patching. Various alternatives are being argued about. One proposal put forward involves bytecode manipulations to change global lookups to local so that one could have a decorator that "injects" a value into a copy of the function. What's the PyPy position on bytecode hacking? Good, bad, evil, don't mind either way?… [View More]

4 6

Number of constants in a jitted rpython interpreter
by Zariko Taba Sept. 22, 2011

Sept. 22, 2011

Hi pypy ! I'm still exploring rpython and I face a problem when adding a jit to an interpreter. In Assembler class (pypy.jit.codewriter.assembler), in emit_const method, it seems to be assumed that there is no more than 256 constants. (constant seems to be accessed in a array with a 1 byte index). If I try to translate an interpreter with more than 256 constant objects (like string ?), I get this error : [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [… [View More]translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/goal/translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 810, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 286, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/translator/driver.py", line 397, in task_pyjitpl_lltype [translation:ERROR] backend_name=self.config.translation.jit_backend, inline=True) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/metainterp/warmspot.py", line 42, in apply_jit [translation:ERROR] **kwds) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/metainterp/warmspot.py", line 199, in __init__ [translation:ERROR] self.codewriter.make_jitcodes(verbose=verbose) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/codewriter.py", line 72, in make_jitcodes [translation:ERROR] self.transform_graph_to_jitcode(graph, jitcode, verbose) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/codewriter.py", line 61, in transform_graph_to_jitcode [translation:ERROR] self.assembler.assemble(ssarepr, jitcode) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 35, in assemble [translation:ERROR] self.write_insn(insn) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 135, in write_insn [translation:ERROR] is_short = self.emit_const(x, kind, allow_short=True) [translation:ERROR] File "/home/olivier/workspace/talstai_ext/pypy-1.6-src/pypy/jit/codewriter/assembler.py", line 108, in emit_const [translation:ERROR] self.code.append(chr(self.constants_dict[key])) [translation:ERROR] ValueError: character code not in range(256) With this snippet of code : self.constants_dict[key] = 256 - len(constants) If len(constants) is 257, then self.constants_dict[key] is -1 and chr(-1) raise the ValueError. I attached a (really) stupid example to reproduce. When I browse pypy sources in rpython, I can't believe there is less than 256 constants of type 'ref'. What do you think ? Did I miss something ? Thanks ! Zariko. [View Less]

3 3

Sandbox examples
by Geoffrey Thomas Sept. 21, 2011

Sept. 21, 2011

Hi, I'm looking at building a real application using PyPy's sandbox mode, and am having a harder time than I'd expect finding any examples of people using the sandbox in the "real world". Specifically, I'm not easily finding examples of interaction scripts other than pypy_interact.py in the PyPy source tree, and would be curious to take a look at how people virtualize things that pypy_interact.py currently doesn't handle. Would any of you have pointers to other interaction scripts, or at … [View More]

3 2

contributing to pypy
by Matthew Kaniaris Sept. 21, 2011

Sept. 21, 2011

Hello, I'd like to contribute to pypy. I've been following the project for a while but don't know much about the internals apart from reading the docs. Does anyone have an idea for a good place to start? -kans

2 1

Spurious dict lookups in my JIT loops
by Boris Sept. 21, 2011

Sept. 21, 2011

Hi, I've been trying out writing my own interpreter using the PyPy framework recently, as a bit of fun. I've been trying to get the JIT to optimize a trivial loop down to the minimal amount of operations. With judicious use of `_immutable_fields_` and `_virtualizable2_`, I've got pretty close. But I'm still seeing lots of calls to `ll_dict_lookup__dicttablePtr_Signed_Signed`, which don't correspond to any code in my interpreter. I don't think I even have any dicts that take integer keys. … [View More]Could someone give me a hint where these are coming from and for what purpose? Or perhaps how to inspect the dicts or get further info? I append what I'm seeing in the logs (Note: I've augmented the logs to give the raw pointer). In this case, it is only looking up the value 21, but I've seen other values in addition when running other programs. The setarrayitem_gc calls are expected - it is nulling out the stack that was being used. Everything from i17 is unexpected. I tested on revisions 00711ff1e03d and 96a212b0688a. Thanks, Boris ############################# [3fd12ea6569db] {jit-log-opt-loop # Loop 0 : loop with 37 ops [p0, p1, p2, i3, p4, p5] debug_merge_point(0, '::Test$iinit:20') +113: i7 = int_add(i3, 1) debug_merge_point(0, '::Test$iinit:22') debug_merge_point(0, '::Test$iinit:23') debug_merge_point(0, '::Test$iinit:26') +116: setarrayitem_gc(p5, 0, ConstPtr(ptr9,0x0), descr=<GcPtrArrayDescr>) +126: setarrayitem_gc(p5, 1, ConstPtr(ptr11,0x0), descr=<GcPtrArrayDescr>) +133: i13 = uint_lt(i7, 10000) guard_true(i13, descr=<Guard2>) [p0, p1, p2, p4, i7] +145: i17 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr15,0x84c2958), 21, 21, descr=<SignedCallDescr>) +176: guard_no_exception(, descr=<Guard3>) [p0, i17, p1, p2, p4, i7] +189: i19 = int_and(i17, -2147483648) +195: i20 = int_is_true(i19) guard_true(i20, descr=<Guard4>) [p0, p1, p2, p4, i7] +204: i23 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr22,0x84c2978), 21, 21, descr=<SignedCallDescr>) +235: guard_no_exception(, descr=<Guard5>) [p0, i23, p1, p2, p4, i7] +248: i24 = int_and(i23, -2147483648) +254: i25 = int_is_true(i24) guard_true(i25, descr=<Guard6>) [p0, p1, p2, p4, i7] +263: i28 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr27,0x84c2988), 21, 21, descr=<SignedCallDescr>) +294: guard_no_exception(, descr=<Guard7>) [p0, i28, p1, p2, p4, i7] +307: i29 = int_and(i28, -2147483648) +313: i30 = int_is_true(i29) guard_true(i30, descr=<Guard8>) [p0, p1, p2, p4, i7] +322: i33 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr32,0x84c2998), 21, 21, descr=<SignedCallDescr>) +353: guard_no_exception(, descr=<Guard9>) [p0, i33, p1, p2, p4, i7] +366: i34 = int_and(i33, -2147483648) +372: i35 = int_is_true(i34) guard_false(i35, descr=<Guard10>) [p0, p1, p2, p4, i7] +381: i38 = call(ConstClass(ll_dict_lookup__dicttablePtr_Signed_Signed), ConstPtr(ptr37,0x84c2968), 21, 21, descr=<SignedCallDescr>) +412: guard_no_exception(, descr=<Guard11>) [p0, i38, p1, p2, p4, i7] +425: i39 = int_and(i38, -2147483648) +431: i40 = int_is_true(i39) guard_true(i40, descr=<Guard12>) [p0, p1, p2, p4, i7] debug_merge_point(0, '::Test$iinit:20') +440: i41 = arraylen_gc(p5, descr=<GcPtrArrayDescr>) +440: jump(p0, p1, p2, i7, p4, p5, descr=<Loop0>) +448: --end of the loop-- [3fd12ea696ce9] jit-log-opt-loop} [View Less]

4 6

Before I start hacking on numpy
by matti picus Sept. 20, 2011

Sept. 20, 2011

I would really love to have 2 dimensional matrices in micronumpy, and am willing to donate some hours of coding. There seems to be a number of "heads" on the mercurial tree that use numpy in their keyword. Can anyone give me pointer as to what branch (maybe just tip?) would be the recommended one to continue with? Any coding guidelines? My goal is to get to an implementation of inverse on a smallish (5x5 matrix) so I can build some useful code. Matti

4 3