Mailman 3 May 2013 - pypy-dev

How can I make more readable the C code obtained from the PyPy translate
by RCU 09 May '13

09 May '13

Hello. I am new to PyPy. I managed to write a few RPython programs and translate them with PyPy translate. As a few others have noticed, as well, (see for example http://mail.python.org/pipermail/pypy-dev/2010-December/006616.html, http://grokbase.com/t/python/pypy-dev/124mqreh2r/output-readable-c and https://bugs.pypy.org/issue1220), the generated C code is very cryptic (when compared to the input RPython script). As far as I understand, this is so because of the following facts: - the RPython code gets compiled to Python bytecode and then translated to more basic operations (an IR which I think it does not have a particular name in the PyPy toolchain - or does it? :) ) - heavy optimizations are being applied on this IR before generating code with the C backend. So, is there any simple way to generate more readable C code (more similar, if possible, to RPython script) - maybe some translate.py options I am missing? Thank you, Alex

3 2

PyPy bitbucket notifications
by Michael Foord 09 May '13

09 May '13

At some point I was added to the pypy user on bitbucket. Ever since I've received a stream of notifications about the pypy repo being forked. In an effort to reduce these emails I clicked on the "manage notifications" link from one of these emails and unwatched all the pypy repos. It seems I've managed to unwatch these repos on behalf of the pypy user! This is probably not a bad thing (and if it is my sincere apologies - it doesn't seem to be easily undoable). Any user who wants these notifications can simply watch these repos with their individual username rather than via the pypy user. All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

1 0

PyPy 2.0 alpha for ARM
by Maciej Fijalkowski 07 May '13

07 May '13

Hello. We're pleased to announce an alpha release of PyPy 2.0 for ARM. This is mostly a technology preview, as we know the JIT is not yet stable enough for the full release. However please try your stuff on ARM and report back. This is the first release that supports a range of ARM devices - anything with ARMv6 (like the Raspberry Pi) or ARMv7 (like Beagleboard, Chromebook, Cubieboard, etc.) that supports VFPv3 should work. We provide builds with support for both ARM EABI variants: hard-float and some older operating systems soft-float. This release comes with a list of limitations, consider it alpha quality, not suitable for production: * stackless support is missing. * assembler produced is not always correct, but we successfully managed to run large parts of our extensive benchmark suite, so most stuff should work. You can download the PyPy 2.0 alpha ARM release here: http://pypy.org/download.html Part of the work was sponsored by the `Raspberry Pi foundation`_. .. _`Raspberry Pi foundation`: http://www.raspberrypi.org/ What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3. It's fast due to its integrated tracing JIT compiler. This release supports ARM machines running Linux 32bit. Both hard-float ``armhf`` and soft-float ``armel`` builds are provided. ``armhf`` builds are created using the Raspberry Pi custom `cross-compilation toolchain`_ based on gcc-arm-linux-gnueabihf and should work on ARMv6 and ARMv7 devices running at least debian or ubuntu. ``armel`` builds are built using gcc-arm-linux-gnuebi toolchain provided by ubuntu and currently target ARMv7. If there is interest in other builds, such as gnueabi for ARMv6 or without requiring a VFP let us know in the comments or in IRC. .. _`cross-compilation toolchain`: https://github.com/raspberrypi Benchmarks ========== Everybody loves benchmarks. Here is a table of our benchmark suite (for ARM we don't provide it yet on http://speed.pypy.org, unfortunately). This is a comparison of Cortex A9 processor with 4M cache and Xeon W3580 with 8M of L3 cache. The set of benchmarks is a subset of what we run for http://speed.pypy.org that finishes in reasonable time. The ARM machine was provided by Calxeda. Columns are respectively: * benchmark name * PyPy speedup over CPython on ARM (Cortex A9) * PyPy speedup over CPython on x86 (Xeon) * speedup on Xeon vs Cortex A9, as measured on CPython * speedup on Xeon vs Cortex A9, as measured on PyPy * relative speedup (how much bigger the x86 speedup is over ARM speedup) (in case this table is not readable, please visit http://morepypy.blogspot.com/2013/05/pypy-20-alpha-for-arm.html) | Benchmark | PyPy vs CPython (arm) | PyPy vs CPython (x86) | x86 vs arm (pypy) | x86 vs arm (cpython) | relative speedup | | ai | 3.61 | 3.16 | 7.70 | 8.82 | 0.87 | | bm_mako | 3.41 | 2.11 | 8.56 | 13.82 | 0.62 | | chaos | 21.82 | 17.80 | 6.93 | 8.50 | 0.82 | | crypto_pyaes | 22.53 | 19.48 | 6.53 | 7.56 | 0.86 | | django | 13.43 | 11.16 | 7.90 | 9.51 | 0.83 | | eparse | 1.43 | 1.17 | 6.61 | 8.12 | 0.81 | | fannkuch | 6.22 | 5.36 | 6.18 | 7.16 | 0.86 | | float | 5.22 | 6.00 | 9.68 | 8.43 | 1.15 | | go | 4.72 | 3.34 | 5.91 | 8.37 | 0.71 | | hexiom2 | 8.70 | 7.00 | 7.69 | 9.56 | 0.80 | | html5lib | 2.35 | 2.13 | 6.59 | 7.26 | 0.91 | | json_bench | 1.12 | 0.93 | 7.19 | 8.68 | 0.83 | | meteor-contest | 2.13 | 1.68 | 5.95 | 7.54 | 0.79 | | nbody_modified | 8.19 | 7.78 | 6.08 | 6.40 | 0.95 | | pidigits | 1.27 | 0.95 | 14.67 | 19.66 | 0.75 | | pyflate-fast | 3.30 | 3.57 | 10.64 | 9.84 | 1.08 | | raytrace-simple | 46.41 | 29.00 | 5.14 | 8.23 | 0.62 | | richards | 31.48 | 28.51 | 6.95 | 7.68 | 0.91 | | slowspitfire | 1.28 | 1.14 | 5.91 | 6.61 | 0.89 | | spambayes | 1.93 | 1.27 | 4.15 | 6.30 | 0.66 | | sphinx | 1.01 | 1.05 | 7.76 | 7.45 | 1.04 | | spitfire | 1.55 | 1.58 | 5.62 | 5.49 | 1.02 | | spitfire_cstringio | 9.61 | 5.74 | 5.43 | 9.09 | 0.60 | | sympy_expand | 1.42 | 0.97 | 3.86 | 5.66 | 0.68 | | sympy_integrate | 1.60 | 0.95 | 4.24 | 7.12 | 0.60 | | sympy_str | 0.72 | 0.48 | 3.68 | 5.56 | 0.66 | | sympy_sum | 1.99 | 1.19 | 3.83 | 6.38 | 0.60 | | telco | 14.28 | 9.36 | 3.94 | 6.02 | 0.66 | | twisted_iteration | 11.60 | 7.33 | 6.04 | 9.55 | 0.63 | | twisted_names | 3.68 | 2.83 | 5.01 | 6.50 | 0.77 | | twisted_pb | 4.94 | 3.02 | 5.10 | 8.34 | 0.61 | It seems that Cortex A9, while significantly slower than Xeon, has higher slowdowns with a large interpreter (CPython) than a JIT compiler (PyPy). This comes as a surprise to me, especially that our ARM assembler is not nearly as polished as our x86 assembler. As for the causes, various people mentioned branch predictor, but I would not like to speculate without actually knowing. How to use PyPy? ================ We suggest using PyPy from a `virtualenv`_. Once you have a virtualenv installed, you can follow instructions from `pypy documentation`_ on how to proceed. This document also covers other `installation schemes`_. .. _`pypy documentation`: http://doc.pypy.org/en/latest/getting-started.html#installing-using-virtual… .. _`virtualenv`: http://www.virtualenv.org/en/latest/ .. _`installation schemes`: http://doc.pypy.org/en/latest/getting-started.html#installing-pypy .. _`PyPy and pip`: http://doc.pypy.org/en/latest/getting-started.html#installing-pypy We would not recommend using in production PyPy on ARM just quite yet, however the day of a stable PyPy ARM release is not far off. Cheers, fijal, bivab, arigo and the whole PyPy team

1 0

PyPy to asm.js
by Leonardo Santagada 06 May '13

06 May '13

asm.js[1] seems a somewhat reasonable target for pypy, maybe even the jit could be made to work with it (with an external javascript lib that calls eval for compilation of the asm.js traces). Is anyone looking it it? They are planning to add support for garbage collector in asm.js and also support for more interesting interaction between the asm.js binary and the javascript ecosystem. [1] http://asmjs.org/faq.html -- Leonardo Santagada

2 1

Leveraging PyPy to translate/compile into other scripting languages?
by James Masters 06 May '13

06 May '13

Hi, I have a need to translate Python code into source code in other scripting languages - some open and others proprietary. The scope of the problem space is very narrow... essentially the need is to define/lookup variables, perform arithmetic operations, define/call functions, etc. but nothing complex like regular expression handling or even opening a file. Some of the target languages are dynamically typed and others are statically typed, so some level of type awareness is needed (preferably through inference). I currently have a partial implementation which uses the Python AST module and some string templating to express the AST nodes in the other languages. I am about to move on to building a symbol table capability and tackle type inference. Before I head too far down this road, I keep thinking back to PyPy and RPython and wondering if I could use portions of this work to help me accomplish what I'm trying to do. Any comments? Thanks, James

1 0

Re: [pypy-dev] Newbie question: using PyPy to compile the source
by Haoyi Li 05 May '13

05 May '13

of a single function? MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=001a11c3715035453704dbec5495 --001a11c3715035453704dbec5495 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Doesn't PyPy have a "compile to binary" option? I thought it did, but I may be mistaken. Sent from my Windows Phone From: Phyo Arkar Sent: 5/4/2013 6:52 PM To: haoyi.sg(a)gmail.com Cc: pypy-dev(a)python.org Subject: Re: [pypy-dev] Newbie question: using PyPy to compile the source of a single function? Pypy is JIT. if you want to do such thing you better look for Nuitika and Cython. On Sun, May 5, 2013 at 5:02 AM, <haoyi.sg(a)gmail.com> wrote: > I=E2=80=99m looking for some way of programmatically using PyPy to compil= e a > snippet of python source code (probably a function def) into an optimized > binary, which I can call to pass data back and forth. The end goal is to > have something like this > > @PyPy > def expensive_function(arg): > ... expensive computation ... > return result > > using macros (https://github.com/lihaoyi/macropy) to perform this > conversion at import time. > > I have no idea if this is possible or not; could anyone here give me any > pointers or advice how to do this/why it is impossible? > > Thanks! > -Haoyi > > _______________________________________________ > pypy-dev mailing list > pypy-dev(a)python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > --001a11c3715035453704dbec5495 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><head><meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Cont= ent-Type"></head><body><div><div style=3D"font-family: Calibri,sans-serif; = font-size: 11pt;">Doesn't PyPy have a "compile to binary" option? I thought= it did, but I may be mistaken.<br><br>Sent from my Windows Phone<br></div>= </div><hr><span style=3D"font-family: Tahoma,sans-serif; font-size: 10pt; f= ont-weight: bold;">From: </span><span style=3D"font-family: Tahoma,sans-ser= if; font-size: 10pt;">Phyo Arkar</span><br><span style=3D"font-family: Taho= ma,sans-serif; font-size: 10pt; font-weight: bold;">Sent: </span><span styl= e=3D"font-family: Tahoma,sans-serif; font-size: 10pt;">5/4/2013 6:52 PM</sp= an><br><span style=3D"font-family: Tahoma,sans-serif; font-size: 10pt; font= -weight: bold;">To: </span><span style=3D"font-family: Tahoma,sans-serif; f= ont-size: 10pt;">haoyi.sg(a)gmail.com</span><br><span style=3D"font-family: T= ahoma,sans-serif; font-size: 10pt; font-weight: bold;">Cc: </span><span sty= le=3D"font-family: Tahoma,sans-serif; font-size: 10pt;">pypy-dev(a)python.org= </span><br><span style=3D"font-family: Tahoma,sans-serif; font-size: 10pt; = font-weight: bold;">Subject: </span><span style=3D"font-family: Tahoma,sans= -serif; font-size: 10pt;">Re: [pypy-dev] Newbie question: using PyPy to com= pile the source of a single function?</span><br><br></body></html><div dir= =3D"ltr">Pypy is JIT. if you want to do such thing you better look for Nuit= ika and Cython.</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail= _quote">On Sun, May 5, 2013 at 5:02 AM, <span dir=3D"ltr"><<a href=3D"m= ailto:haoyi.sg@gmail.com" target=3D"_blank">haoyi.sg(a)gmail.com</a>></spa= n> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div><div style=3D"font-family:Calibri,'= Segoe UI',Meiryo,'Microsoft YaHei UI','Microsoft JhengHei U= I','Malgun Gothic','Khmer UI','Nirmala UI',Tung= a,'Lao UI',Ebrima,sans-serif;font-size:16px"> <div>I=E2=80=99m looking for some way of programmatically using PyPy to com= pile a snippet of python source code (probably a function def) into an opti= mized binary, which I can call to pass data back and forth. The end goal is= to have something like this</div> <div>=C2=A0</div><div>@PyPy</div><div>def expensive_function(arg):</div><di= v>=C2=A0=C2=A0=C2=A0=C2=A0... expensive computation ...</div><div>=C2=A0=C2= =A0=C2=A0 return result</div><div>=C2=A0</div><div>using macros (<a href=3D= "https://github.com/lihaoyi/macropy" target=3D"_blank">https://github.com/l= ihaoyi/macropy</a>) to perform this conversion at import time.</div> <div>=C2=A0</div><div>I have no idea if this is possible or not; could anyo= ne here give me any pointers or advice how to do this/why it is impossible?= </div><div>=C2=A0</div><div>Thanks!</div><div>-Haoyi</div></div></div><br>_= ______________________________________________<br> pypy-dev mailing list<br> <a href=3D"mailto:pypy-dev@python.org">pypy-dev(a)python.org</a><br> <a href=3D"http://mail.python.org/mailman/listinfo/pypy-dev" target=3D"_bla= nk">http://mail.python.org/mailman/listinfo/pypy-dev</a><br> <br></blockquote></div><br></div> --001a11c3715035453704dbec5495--

3 2

Newbie question: using PyPy to compile the source of a single function?
by haoyi.sg＠gmail.com 04 May '13

04 May '13

I’m looking for some way of programmatically using PyPy to compile a snippet of python source code (probably a function def) into an optimized binary, which I can call to pass data back and forth. The end goal is to have something like this @PyPy def expensive_function(arg): ... expensive computation ... return result using macros (https://github.com/lihaoyi/macropy) to perform this conversion at import time. I have no idea if this is possible or not; could anyone here give me any pointers or advice how to do this/why it is impossible? Thanks! -Haoyi

2 1

Second compilation stage
by Hakan Ardo 02 May '13

02 May '13

Hi, there have been a bit of talk on improving loops with control flow, i.e. when there is no single dominant path through the loop. To have that discussion here let me make the following proposal (based on a proposal from Armin): How about aiming for something that starts out with running the current JIT with the unrolling disabled. That will produce a graph of traces which we could partition into one subgraph per loop. When we believe all guards are either traced or never failing, we use such subgraphs as preambles. That is, we copy the full graph to producing a peeled loop graph and optimized it. We focus the optimization on removing the remaining guards as they (hopefully in most cases) are loop invariant. Then we have a guard free graph of traces that we could pass to gcc/llvm and the resulting machine code are attached after the preamble. If it turns out to be necessary, we could later add support for handing guards to better support cases where we are unable optimize them out. -- Håkan Ardö

2 2

hexiom2 benchmark
by Michael Hudson-Doyle 02 May '13

02 May '13

Hi all, Long time no see! Apologies if this is not the right place to ask this question. I'm trying to run the benchmarks from speed.pypy.org on my system (just with cpython for now -- there is a chance that Linaro will be doing some work on the performance of Python on ARM at some point), but when I check out https://bitbucket.org/pypy/benchmarks/overview and run "./runner.py --fast" I get this: Running hexiom2... INFO:root:Running /usr/bin/python /home/mwhudson/src/benchmarks/own/hexiom2.py -n 5 INFO:root:Running /usr/bin/python /home/mwhudson/src/benchmarks/own/hexiom2.py -n 5 Traceback (most recent call last): File "./runner.py", line 302, in <module> main(sys.argv[1:]) File "./runner.py", line 283, in main full_store=full_store, branch=branch) File "./runner.py", line 41, in run_and_store results = perf.main(opts, funcs) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 1617, in main bench_result = func(base_cmd_prefix, changed_cmd_prefix, options) File "/home/mwhudson/src/benchmarks/benchmarks.py", line 17, in BM return SimpleBenchmark(Measure, *args, **kwds) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 462, in SimpleBenchmark return CompareBenchmarkData(base_data, changed_data, options) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 736, in CompareBenchmarkData return CompareMultipleRuns(base_times, changed_times, options) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 704, in CompareMultipleRuns significant, t_score = IsSignificant(base_times, changed_times) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 189, in IsSignificant t_score = TScore(sample1, sample2) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero is this just an artifact of running --fast? It seems the code could probably do with being a bit more robust? Cheers, mwh

3 4

Pypy is slower than Python
by cat street 02 May '13

02 May '13

You can test this code: import time def test(n): m = 10 vals = [] keys = [] for i in xrange(m): vals.append(i) keys.append('a%s'%i) d = None for i in xrange(n): d = dict(zip(keys, vals)) return d if __name__ == '__main__': st = time.time() print test(1000000) print 'use:', time.time() - st -- HomePage: cat.appspot.com

5 11