Mailman 3 January 2008 - pypy-dev

real-time gc ?
by Simon Burton 21 Feb '08

21 Feb '08

I've taken some statistics on how the boehm gc performs with our current application, it has terrible worst case performance, often stopping for hundreds of mS at a time. I tried calling GC_enable_incremental() but that seems to cause the code to explode. I don't have any idea how to obtain similar statistics for the refcounting gc, which is also non-realtime. Perhaps it is in fact worse than boehm in this regard. Over-all CPU usage is significantly less with boehm, but a lot can happen in 300mS of a garbage collection. I don't think it's feasable compiling the pypy interpreter with refcounting, it takes a truly vast amount of time (some hours...) It would be good using the pypy interpreter to script our rpython codebase, but this gc issue seems to be a blocker. Simon.

2 2

How to learn RPython as a general purpose language
by Josh Gilbert 04 Feb '08

04 Feb '08

I tried downloading the archives of this list and grepping, but couldn't find an answer to this question. If it's been discussed to death and I missed it I apologize. I realize that it is not recommended to use RPython as a general purpose language, a front end to C if you will. My question is how one might learn how to do so, regardless. I'm also interested in reasons why it shouldn't be done (preferably ones that can be resolved). ## References I've found a comment on LtU suggesting that it is extremely hard to learn, yet a valuable skill if you can cut it: http://lambda-the-ultimate.org/node/1260#comment-13972 I've also found a hello world example from 2006 (humorously labeled part 1): http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-par… Josh Gilbert.

6 5

reg. GCLS recursive test.
by Amund Tveit 29 Jan '08

29 Jan '08

Hi, I wrote a blog entry yesterday about using rpython for the great computer language shootout recursive program, and got a RunTime error when I input on the commandline 11 (1 to 10 goes fine). Just did an svn update of pypy to version 51115 and still get the error, here is the program: blog entry: http://amundblog.blogspot.com/2008/01/rpython-gclb-benchmark-recursive.html source file: recursive.py # The Computer Language Shootout # http://shootout.alioth.debian.org/ # based on bearophile's psyco program # slightly modified by Isaac Gouy def Ack(x, y): if x == 0: return y+1 if y == 0: return Ack(x-1, 1) return Ack(x-1, Ack(x, y-1)) def Fib(n): if n < 2: return 1 return Fib(n-2) + Fib(n-1) def FibFP(n): if n < 2.0: return 1.0 return FibFP(n-2.0) + FibFP(n-1.0) def Tak(x, y, z): if y < x: return Tak( Tak(x-1,y,z), Tak(y-1,z,x), Tak(z-1,x,y) ) return z def TakFP(x, y, z): if y < x: return TakFP( TakFP(x-1.0,y,z), TakFP(y-1.0,z,x), TakFP(z-1.0,x,y) ) return z from sys import argv, setrecursionlimit setrecursionlimit(20000) def main(argv): n = int(argv[1]) - 1 print "Ack(3,%d):" % (n+1), Ack(3, n+1) print "Fib(" + str(28.0+n) + "," + str(FibFP(28.0+n)) print "Tak(%d,%d,%d): %d" % (3*n, 2*n, n, Tak(3*n, 2*n, n)) print "Fib(3):", Fib(3) print "Tak(3.0,2.0,1.0):", TakFP(3.0, 2.0, 1.0) return 0 from pypy.translator.interactive import Translation t = Translation(main, standalone=True, gc='ref') t.source(backend='c') path = t.compile() print path Best regards, Amund

4 3

Converting from Python strings to C strings slow?
by Martin C. Martin 27 Jan '08

27 Jan '08

Hi, There seems to be a lot of overhead when passing a large string (23 Meg) to C compiled RPython code. For example, this code: def small(text): return 3 t = Translation(small) t.annotate() t.rtype() f3 = t.compile_c() st = time.time() z = f3(xml) print time.time() - st Whereas parsing the 16,000 XML elements using a regular expression only took 22 msec. Even reading the text file inside the compiled RPython is faster than passing it. Here's the code from rstr.py that (seems to be) doing the conversion. Any idea how I'd put timing code in there to see what's taking all the time? Any idea how to speed it up? class __extend__(pairtype(PyObjRepr, AbstractStringRepr)): def convert_from_to((r_from, r_to), v, llops): v_len = llops.gencapicall('PyString_Size', [v], resulttype=Signed) cstr = inputconst(Void, STR) v_result = llops.genop('malloc_varsize', [cstr, v_len], resulttype=Ptr(STR)) cchars = inputconst(Void, "chars") v_chars = llops.genop('getsubstruct', [v_result, cchars], resulttype=Ptr(STR.chars)) llops.gencapicall('PyString_ToLLCharArray', [v, v_chars]) string_repr = llops.rtyper.type_system.rstr.string_repr v_result = llops.convertvar(v_result, string_repr, r_to) return v_result Best, Martin

3 6

Re: [pypy-dev] Two problems
by Martin C. Martin 25 Jan '08

25 Jan '08

So it's not possible for RPython to call regular, non-RPython? Is there some wrapper I could write to let RPython call regular python code? Best, Martin Simon Burton wrote: > On Fri, 25 Jan 2008 10:30:17 -0500 > "Martin C. Martin" <martin(a)martincmartin.com> wrote: > >> Hi, >> >> I'm really impressed with the C code generated for some functions I'd >> like to use in my project. You guys rock! >> >> I've run into two problems this morning. In pypy-1.0.0, this: >> >> import re >> import os, sys >> from pypy.translator.interactive import Translation >> import py >> >> def mytest(): >> x = re.search('bar', 'foobar') >> return x.start() >> >> t = Translation(mytest) >> t.annotate() >> >> throws an exception: >> >> File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line >> 567, in simple_call >> return bltn.analyser(bltn.s_self, *args) >> File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line >> 19, in __call__ >> return self.im_func(firstarg, *args, **kwds) >> TypeError': method_get() takes exactly 3 arguments (2 given) >> .. v1 = simple_call((builtin_function_or_method get), v0) >> .. '(sre:213)_compile_star2' >> >> >> This is an InstanceMethod where im_self is a SomeDict and args is a >> tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, >> 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). >> It seems to be calling a method_get() with two args, SomeDict and >> SomeTuple, whereas it wants 3. Any idea how to fix this? > > I'm not sure you can use module "re" in rpython code. > > The error is because somewhere there is a .get(XX) method call on a dict. > In rpython the default arg for dict.get must be specified. > > Simon. > >

2 2

Two problems
by Martin C. Martin 25 Jan '08

25 Jan '08

Hi, I'm really impressed with the C code generated for some functions I'd like to use in my project. You guys rock! I've run into two problems this morning. In pypy-1.0.0, this: import re import os, sys from pypy.translator.interactive import Translation import py def mytest(): x = re.search('bar', 'foobar') return x.start() t = Translation(mytest) t.annotate() throws an exception: File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 567, in simple_call return bltn.analyser(bltn.s_self, *args) File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 19, in __call__ return self.im_func(firstarg, *args, **kwds) TypeError': method_get() takes exactly 3 arguments (2 given) .. v1 = simple_call((builtin_function_or_method get), v0) .. '(sre:213)_compile_star2' This is an InstanceMethod where im_self is a SomeDict and args is a tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). It seems to be calling a method_get() with two args, SomeDict and SomeTuple, whereas it wants 3. Any idea how to fix this? Thinking it might be fixed in svn, I got the latest code using: svn co http://codespeak.net/svn/pypy/dist pypy-dist Then tried to start pypy using: python pypy-dist/pypy/bin/py.py But got: File "/ita/downloaded/pypy-dist/pypy/module/posix/__init__.py", line 3, in ? from pypy.rpython.module.ll_os import RegisterOs File "/ita/downloaded/pypy-dist/pypy/rpython/module/ll_os.py", line 127 @registering_if(os, 'execv') ^ SyntaxError: invalid syntax This is all on 64 bit Linux, python 2.4.1. Any idea how I can get my simple function to compile? Best, Martin

2 1

Loops for gcc
by Armin Rigo 23 Jan '08

23 Jan '08

Hi all, Some results seem to show that GCC is more happy optimizing "for" and "while" loops than the equivalent loops written with a bunch of goto's like we produce in our C backend. Interestingly, it seems that what makes GCC unhappy is not the bunch of goto's, but only the absence of the explicit C-level loop. Example: For this piece of C code as someone would write by hand: while (n > 0) { ...body... } we generate the following instead, which defeats GCC's loop-specific optimizations: block1: cond = n > 0; if (cond) goto block2; goto block3; block2: ...body... goto block1; The smallest change I could find that make GCC produce good code again is to modify block1 and block2 like this: block1: cond = n > 0; while (cond) { /* trick */ goto block2; block1_back: cond = n > 0; /* the operations in block1 are repeated */ } goto block3; block2: ...body... goto block1_back; /* changed */ GCC doesn't seem to mind that the real body of the loop is not in the loop at all but somewhere else; I guess it just follows the chain of goto's and treat the result as equivalent to having the source code directly within the loop's { }. So I wrote a quick hack that detects (for now only the innermost) loops in graphs. More specifically it finds "loop head" blocks like block1, and "loop-closing links" like the link from block2 to block1. It then produces code like above. It's in the loops-for-gcc branch. It's quite obscure but seems to give a consistent 5% speed-up on pypy-c running richards... A bientot, Armin.

1 0

Workshop on Self-sustaining Systems (S3)
by Martin von Löwis 22 Jan '08

22 Jan '08

*** Workshop on Self-sustaining Systems (S3) 2008 *** May 15-16, 2008 Potsdam, Germany http://www.swa.hpi.uni-potsdam.de/s3/ -- Call for papers: The Workshop on Self-sustaining Systems (S3) is a forum for discussion of topics relating to computer systems and languages that are able to bootstrap, implement, modify, and maintain themselves. One property of these systems is that their implementation is based on small but powerful abstractions; examples include (amongst others) Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and Lisp. Such systems are the engines of their own replacement, giving researchers and developers great power to experiment with, and explore future directions from within, their own small language kernels. S3 will be take place May 15-16, 2008 at the Hasso-Plattner-Institute in Potsdam, Germany. It is an exciting opportunity for researchers and practitioners interested in self-sustaining systems to meet and share their knowledge, experience, and ideas for future research and development. -- Invited talk: Ian Piumarta: Late-bound Object Lambda Architectures (Viewpoints Research Institute, USA) -- Submissions and proceedings: S3 invites submissions of high-quality papers reporting original research, or describing innovative contributions to, or experience with, self-sustaining systems, their implementation, and their application. Papers that depart significantly from established ideas and practices are particularly welcome. Submissions must not have been published previously and must not be under review for any another refereed event or publication. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality. Revised papers will be published as post-proceedings in the Springer LNCS series. Papers should be submitted electronically via EasyChair at http://www.easychair.org/conferences/?conf=s3 in PDF format. Submissions must be written in English (the official language of the workshop) and must not exceed 20 pages. They should use the LNCS format, templates for which are available at http://www.springer.de/comp/lncs/authors.html. -- Venue: Hasso-Plattner-Institut (Potsdam, Germany) -- Important dates: Submission of papers: February 15, 2008 Author notification: April 11, 2008 Revised papers due: April 25, 2008 S3 workshop: May 15-16, 2008 Final papers for LNCS post-proceedings due: June 6, 2008 -- Chairs: * Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany) * Kim Rose (Viewpoints Research Institute, USA) -- Program committee: * Johan Brichau, Universite Catholique de Louvain, Belgium * Pascal Costanza, Vrije Universiteit Brussel, Belgium * Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium * Stephane Ducasse, INRIA Lille, France * Michael Haupt, Hasso-Plattner-Institut, Germany * Robert Hirschfeld, Hasso-Plattner-Institut, Germany * Dan Ingalls, Sun Microsystems Laboratories, USA * Martin von L?wis, Hasso-Plattner-Institut, Germany * Hidehiko Masuhara, University of Tokyo, Japan * Ian Piumarta, Viewpoints Research Institute, USA * David Ungar, IBM, USA -- Registration fees: Early (until April 18, 2008) * Regular participants: EUR 160 * Students: EUR 80 Late (after April 18, 2008) * Regular participants: EUR 170 * Students: EUR 90

1 0

Performance Regression with SVN version 50525
by Michael Schneider 16 Jan '08

16 Jan '08

Hello All, The pypy c translation time doubled for me with SVN revision 50525 (2:13:48 PM) *bigdog:* 50524: (2:13:48 PM) *bigdog:* Timer] Timings: (2:13:48 PM) *bigdog:* Timer] annotate - 318.9 s (2:13:48 PM) *bigdog:* Timer] rtype_lltype - 174.2 s (2:13:48 PM) *bigdog:* Timer] backendopt_lltype - 183.3 s (2:13:48 PM) *bigdog:* Timer] stackcheckinsertion_lltype - 62.6 s (2:13:48 PM) *bigdog:* Timer] database_c - 128.2 s (2:13:48 PM) *bigdog:* Timer] source_c - 69.3 s (2:13:48 PM) *bigdog:* Timer] compile_c - 0.1 s (2:13:48 PM) *bigdog:* Timer] ======================================== (2:13:48 PM) *bigdog:* Timer] Total: - 936.6 s (2:55:29 PM) *bigdog:* 50525 timings: (2:55:32 PM) *bigdog:* [Timer] Timings: (2:55:32 PM) *bigdog:* [Timer] annotate - 335.2 s (2:55:32 PM) *bigdog:* [Timer] rtype_lltype - 174.3 s (2:55:32 PM) *bigdog:* [Timer] backendopt_lltype - 183.5 s (2:55:32 PM) *bigdog:* [Timer] stackcheckinsertion_lltype - 1066.0 s (2:55:32 PM) *bigdog:* [Timer] database_c - 126.3 s (2:55:32 PM) *bigdog:* [Timer] source_c - 70.4 s (2:55:32 PM) *bigdog:* [Timer] compile_c - 0.1 s (2:55:32 PM) *bigdog:* [Timer] ========================================= (2:55:32 PM) *bigdog:* [Timer] Total: - 1955.8 s The translation spent inordinate amount of time in: (2:37:52 PM) *bigdog:* ..+*%+................................*%%..................................... (2:37:52 PM) *bigdog:* [backendopt:malloc] removed 3922 simple mallocs in total (2:37:52 PM) *bigdog:* [backendopt:mergeifblocks] starting to merge if blocks (2:37:52 PM) *bigdog:* ..+*%+................................*%%.............. (2:37:52 PM) *bigdog:* [translation:info] inserting stack checks... (2:37:52 PM) *bigdog:* .. (2:37:52 PM) *bigdog:* [rtyper] -=- specialized 6 more blocks -=- Translation memory use went from ~350 meg to ~1.3 gb (memory use are not exact, I just looked at the perfomrnace monitor when runnning) Hope this is helpful, Mike

3 7

pylib patch to enable mips and mipsel greenlets
by Alexandre Fayolle 14 Jan '08

14 Jan '08

Hi, I've received a patch by Thiemo Seufer <ths(a)networkno.de> to port pylib greenlets to mips and mipsel linux. It is available for examination at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=459520 -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science

1 0