Mailman 3 November 2011 - pypy-dev

Translating pypy-1.7 with -O0
by David Naylor 24 Nov '11

24 Nov '11

Hi, I'm trying to translate pypy-1.7 with optimisations at level 0. I get the following error: # /usr/local/bin/pypy translate.py --source --gcrootfinder=shadowstack -- thread -O0 targetpypystandalone.py <snip/> [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 809, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 286, in _do [translation:ERROR] res = func() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 505, in task_database_c [translation:ERROR] database = cbuilder.build_database() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/c/genc.py", line 143, in build_database [translation:ERROR] sandbox=self.config.translation.sandbox) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/c/database.py", line 63, in __init__ [translation:ERROR] self.gctransformer = self.gcpolicy.transformerclass(translator) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/rpython/memory/gctransform/framework.py", line 151, in __init__ [translation:ERROR] GCClass, GC_PARAMS = choose_gc_from_config(translator.config) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/rpython/memory/gc/base.py", line 448, in choose_gc_from_config [translation:ERROR] config.translation.gc,)) [translation:ERROR] ValueError: unknown value for translation.gc: 'ref' Looking at base.py I see that ref and boehm GC are not in the list (and I thought -O0 used the boehm not the ref GC). I saw a previous message about this (re pypy-1.6) but no solution. Also: # file /usr/local/lib/libgc.so.1 /usr/local/lib/libgc.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, not stripped confirms I have the required library installed. Is there, perhaps, something I am missing? Regards

3 4

Bytecode-less pypy languages
by Timothy Baldridge 23 Nov '11

23 Nov '11

So I've started work on my Clojure(lisp) in pypy interpreter again. This time I've decided to take a very materialistic approach to the design. As such my interpreter does not run off of bytecode, instead it directly interprets the Clojure structures, truely treating data as code. The idea is that every object in the system must have a evauate() method, and optionally a invoke(args) method. These methods are implemented as follows: most objects return themselves when evalute() is run def evaulates to itself, and when invoked, creates a named var that is bound to the 2nd form. E.g. (def foo 1) creates a var named foo that points to IntObj(1) user defined functions create local bindings, and then invoke their contents symbols search local bindings and global defs and then return the result through evaluate() lists evaluate all their contents, and then invoke the first item passing it the rest of the list as arguments With this simple approach, I've been able to implement many of the clojure concepts with just a few lines of code. It really works well. Here is my question. When I get a full program written, I'll have hundreds of objects each with evaluate() and invoke() all will be immutable, but there really isn't any sort of loop. We can implement loops via recur: (def (fn (x) (if (< x 100) (recur (inc x)) x))) But that's about the only loop in the interpreter. So I know in "normal" jits you would "greenlight" the instruction pointer of the byte code interpreter. Will PyPy not be happy with a lack of an instruction pointer? If I flag every single class as immutable, will it be able to still a decent jit? I guess, what I'm asking is when do green and red variables need to be defined, is it ever okay to not define them? Will the jit even work? Thank you for your time, Timothy -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth)

3 2

PyPy 1.7 - widening the sweet spot
by Maciej Fijalkowski 23 Nov '11

23 Nov '11

================================== PyPy 1.7 - widening the sweet spot ================================== We're pleased to announce the 1.7 release of PyPy. As became a habit, this release brings a lot of bugfixes and performance improvements over the 1.6 release. However, unlike the previous releases, the focus has been on widening the "sweet spot" of PyPy. That is, classes of Python code that PyPy can greatly speed up should be vastly improved with this release. You can download the 1.7 release here: http://pypy.org/download.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work is ongoing, but not yet natively supported. The main topic of this release is widening the range of code which PyPy can greatly speed up. On average on our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up to **20 times** faster on some benchmarks. .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org Highlights ========== * Numerous performance improvements. There are too many examples which python constructs now should behave faster to list them. * Bugfixes and compatibility fixes with CPython. * Windows fixes. * PyPy now comes with stackless features enabled by default. However, any loop using stackless features will interrupt the JIT for now, so no real performance improvement for stackless-based programs. Contact pypy-dev for info how to help on removing this restriction. * NumPy effort in PyPy was renamed numpypy. In order to try using it, simply write:: import numpypy as numpy at the beginning of your program. There is a huge progress on numpy in PyPy since 1.6, the main feature being implementation of dtypes. * JSON encoder (but not decoder) has been replaced with a new one. This one is written in pure Python, but is known to outperform CPython's C extension up to **2 times** in some cases. It's about **20 times** faster than the one that we had in 1.6. * The memory footprint of some of our RPython modules has been drastically improved. This should impact any applications using for example cryptography, like tornado. * There was some progress in exposing even more CPython C API via cpyext. Things that didn't make it, expect in 1.8 soon ============================================== There is an ongoing work, which while didn't make it to the release, is probably worth mentioning here. This is what you should probably expect in 1.8 some time soon: * Specialized list implementation. There is a branch that implements lists of integers/floats/strings as compactly as array.array. This should drastically improve performance/memory impact of some applications * NumPy effort is progressing forward, with multi-dimensional arrays coming soon. * There are two brand new JIT assembler backends, notably for the PowerPC and ARM processors. Fundraising =========== It's maybe worth mentioning that we're running fundraising campaigns for NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any of those happen faster, we urge you to donate to `numpy proposal`_ or `py3k proposal`_. In case you want PyPy to progress, but you trust us with the general direction, you can always donate to the `general pot`_. .. _`numpy proposal`: http://pypy.org/numpydonate.html .. _`py3k proposal`: http://pypy.org/py3donate.html .. _`general pot`: http://pypy.org

13 24

Translating pypy-sandbox eats 13GB of RAM
by David Naylor 22 Nov '11

22 Nov '11

Hi While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 I saw the translating pypy consume 13.3GB of memory (column Res in top). This is under FreeBSD-9RC2/amd64. I've never noticed this memory consumtion when translating pypy-1.6 with the stackless option. Is this perhaps a bug with pypy-1.6 or the source code for pypy-1.7? I'll try again using pypy-1.7 to do the translating once it has been translated (I needed to restart since I was running the translations in parallel). Regards

4 4

Re: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM
by Laura Creighton 22 Nov '11

22 Nov '11

Are you using gcc 4.2? If so , you have this bug. https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391 and you need a newer gcc. (Or maybe an older one would work, too, but this one doesn't). Laura

2 1

Building pypy on freebsd 64bit
by Shadrin Eugene 19 Nov '11

19 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

New pypy website
by Maciej Fijalkowski 18 Nov '11

18 Nov '11

Hello everyone Ante Salinovic made us a great website for the upcoming 1.7 PyPy release. I would like to see some feedback on it. The preliminary version is now available at http://pypy-site.ep.io Among other things it's missing some more success stories as well as donate buttons for numpy and py3k. Cheers, fijal

6 6

Re: [pypy-dev] Slow sqlite user defined functions with pypy.
by William ML Leslie 17 Nov '11

17 Nov '11

Ack. On 17 November 2011 12:23, William ML Leslie <william.leslie.ttg(a)gmail.com> wrote: > On 17 November 2011 12:13, Elefterios Stamatogiannakis <estama(a)gmail.com> wrote: >> Pypy seems to not jit at all when a (pypy) Python function is called from C. > > Calls to native functions must be residualised, as there is no way to > tell what state gives rise to the call to the UDF. > > If there was a loop inside the UDF, that would still be compiled. But > the loop that occurs in the query must just call the C function in the > usual way, as the JIT has no idea what it might do. -- William Leslie

4 3

Pure Python Math Library
by Blaine 17 Nov '11

17 Nov '11

Does anyone know of a pure python math library? I've been playing around with berp <https://github.com/bjpop/berp/wiki>, which is a python3 to haskell translator and compiler, and it works great as long as you don't go crazy with C extensions. It's highly experimental but fun to play around with. The only thing that I really miss is being able to use the math module. I asked the maintainer if it is possible to map into haskell's math library, but in the mean time a pure python math library would fit nicely since it would be compiled along with the rest of the python. I'm looking for log, log10, ceil, and pow mostly for my personal needs. It's funny now that things like pypy and berp exist. I find myself trying to locate pure python routines (like DFT) that would have no reason to exist with cpython, but make lots of sense with pypy. Anyway, just wondering if anyone had heard of such a thing. How do you even implement log? Thanks, Blaine

6 13

Updated 'High Performance Python' tutorial (the one from EuroPython 2011)
by Ian Ozsvald 16 Nov '11

16 Nov '11

Dear all, I've published v0.2 of my High Performance Python tutorial write-up from the session I ran at EuroPython: http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from… Antonio - you asked earlier if the 'expanded math' version of the Mandelbrot solver (using doubles rather than complex numbers) would be faster - I've timed it and it is a bit faster with a nightly build of PyPy, but nowhere near as fast at ShedSkin's generated C output (details below). Maciej - thanks for pointing me at the numpy module. I've added a tiny section showing numpy in PyPy but I haven't converted the Mandelbrot solver to use it (even finishing v0.2 took longer than I'd thought). I'm hoping that some more exposure in the report might bring in more volunteers from outside. Here's a clip from the report in the PyPy section: "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. (Ian - the only difference with pure_python_2.py is that local dereferences in the tight loop are moved outside the loop, causing fewer dereference operations) As an additional test (not shown in the graphs) I ran pypy shedskin2.py 1000 1000 which runs the expanded math version of the shedskin variant below (this replaces complex numbers with floats and expands abs to avoid the square root). The shedskin2.py result takes 3.2 seconds (which is still much slower than the 0.4s version compiled using shedskin)." The pure_python src is here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… shedskin2.py is available here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/… I haven't tested whether the warm-up periods for PyPy are significant, possibly they account for much of the difference between ShedSkin and PyPy? I want to revisit this but for the next few weeks I have to go back to other projects. I hope the report brings in some new folk for PyPy, Ian. -- Ian Ozsvald (A.I. researcher, screencaster) ian(a)IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald

5 8