Mailman 3 March 2011 - pypy-dev

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

cpyext reference counting and other gc's
by Dima Tisnek 02 May '11

02 May '11

Hey, I had a look at cpyext recently and saw that reference counting is emulated with, err, reference counting, seeminlgy in the referenced object itself. Does this mean that cpyext would not work with other gc's or is there some wrapping going on behind the scenes?

5 13

The JVM backend and Jython
by fwierzbicki＠gmail.com 14 Apr '11

14 Apr '11

Hi all, It was nice meeting up with many of you at PyCon! I've been thinking about the first steps towards collaboration between the Jython project and the PyPy project. It looks like it isn't going to be too long before we are all (CPython, PyPy, IronPython, Jython, etc) working on a single shared repository for all of our standard library .py code. In my ideal world there would come a day when there is also no standalone Java code in the Jython project: that is the shared standard library would contain all of Jython's .py files, and all of the Java would be generated from PyPy and Jython as a standalone project would disappear. It is possible that this is too ambitious, but big goals are more fun, right? In reality even if this where to get going, I imagine it would be a 10+ year plan :) So to my question - just how broken is the JVM backend? Are there workarounds that would allow the Java code to get generated? I ask because I would like to evaluate the generated Java parser as a potential replacement for our current ANTLR based parser. It seems like a nice baby step towards real collaboration since it seems like a relatively easy place to start. Clearly it would need adjustments to actually work for Jython - but I'd be able to look into that part. I don't think I have the time to try to unbreak the translation though... -Frank

6 12

Pypy custom interpreter JIT question
by Andrew Brown 06 Apr '11

06 Apr '11

Hello PyPy team, So I became interested in the translation toolchain plus JIT compiler generator, and in attempt to learn more about it, I set out to write a very simple interpreter for the language BF (brainf***). Simple enough, 8 opcodes, each with no arguments, and most have a one line implementation. Also plenty of examples out there to run, like a mandelbrot generator =) So I wrote up an interpreter for it in RPython. This worked great until I tried to enable the JIT option, at which point it would produce incorrect results. Strange, I thought I may have been using the hints incorrectly, I couldn't find too many details on exactly what the red and green vars should be. Here's the strange part though: I finally fixed it by changing an implementation detail that shouldn't have changed semantics at all. My implementation creates an instance of a Tape object which has two attributes: a list of integers representing the state of the machine, and a single integer identifying the current active cell on the tape. The implementation of each opcode was a method of this class, and the state of the program (what I passed as the "red" variable) was the instance of this class. After I manually factored the functionality of the class directly into the main dispatch loop and got rid of the class entirely, the JIT compiler started producing correct results. Can anyone help me figure out why my first attempt didn't work? Do red variables that are in class instances need to be handled different somehow? Here's the initial version that runs incorrectly when translated with JIT: https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py Here's the modified version that seems to work just fine: https://bitbucket.org/brownan/bf-interpreter/src/8095853278e9/targetbf.py In particular, note the elimination of the Tape object in the second version, and the differences in the mainloop function as well as the differences in the "red" variables. I've also included a few example BF programs if someone wants to try it out. The hanoi example crashes almost immediately with the first version translated with JIT. By the way, I've been translating it with the latest version of PyPy off of bitbucket. (latest as of a few weeks ago, that is) Thanks and great work on this project! -Andrew Brown

9 37

Re: [pypy-dev] Pypy custom interpreter JIT question
by Andrew Brown 31 Mar '11

31 Mar '11

Here's three emails that accidentally didn't get sent to the list. On Thu, Mar 31, 2011 at 10:41 AM, Dan Roberts <ademan555(a)gmail.com> wrote: > Hi Andrew, > Did you ever try your interpreter with the 99 bottles program? I got my > interpreter down faster than the beef interpreter ~4s vs ~15s on mandelbrot, > however even with that speed, both 'bf' and 'beef' trounced my interpreter > by an absurd amount. It seems like it probably was a problem with my code > base, when I first saw you were working on this I meant to ask you to try > 99bottles.bf and see if you had similar problems. I haven't had a chance > to examine where the problem is coming from though. > > Cheers, > Dan > On Thu, Mar 31, 2011 at 11:02 AM, Andrew Brown <brownan(a)gmail.com> wrote: > Hi Dan, > Did you mean to send this to the list as well? I only ask because it's easy > to hit "Reply" instead of "Reply All". > > Regardless, I have a 99 bottles program, in the comments it says it's > written by Andrew Paczkowski. I haven't mentioned it just because it runs > absurdly fast: 0.02 seconds or so (compared with 0.2s for running the py > code on cpython), so I didn't consider it a good test. I wanted something > that took a bit longer. > > I just searched for another and found one by Raphael Bois, but that runs in > 0.04 seconds. > > Perhaps you're using a different version of this program that's less > efficient and runs faster? (or maybe this really is just that fast?) > > Also, the mandelbrot program that I included in my repo takes 8.4 seconds > to run on my computer. Not quite the 4 second time you're getting (have you > published your interpreter anywhere? I'd like to look at it) I have a > feeling I've taken this interpreter as far as it will go without doing any > more intelligent inspection of the bf code directly. > > -Andrew > On Thu, Mar 31, 2011 at 11:20 AM, Dan Roberts <ademan555(a)gmail.com> wrote: > Hey, > Yeah, that's the second or third time I didn't reply to all lately :-/ > And my interpreter is on paste.pocoo.org somewhere, I can paste it again > when I go home today. I suspect there's something wrong with it though, > considering you're getting proper performance on 99bottles, it takes about 3 > minutes here! (On the same system where it wins on mandelbrot by >66% > against 'beef' or bf whichever one is faster) One immediately obvious > difference was your use of the bracket map, which I think is an awesome > idea. I may adopt it, currently I calculate how far backwards/forwards to > travel at "runtime" instead of preprocessing it. I made it pure so that it > would be constant folded by the JIT, but I suppose the 1000 iterations > before the JIT kicks in (per loop) could explain a large performance > difference. I could probably combine both techniques and cache the results > at runtime, by the second run, it'll be a dict lookup, so it'll be jitted > essentially the same, and I won't have to think about parsing to find > matching braces :-) > > If you can think of a good way to bring this discussion back on the mailing > list that'd be fine. > > Cheers, > Dan >

1 0

Re: [pypy-dev] Pypy custom interpreter JIT question
by Laura Creighton 31 Mar '11

31 Mar '11

In a message of Thu, 31 Mar 2011 14:33:40 +0200, Antonio Cuni writes: >On 31/03/11 14:28, Andrew Brown wrote: >> In any case, I'm satisfied with the speed. It's still beaten by a BF to > C >> translator combined with gcc -O2 though, that'd be a tough case to beat >. =) > >what happens if you combine the BF to C with gcc -O0 or -O1? > >Anyway, I think that if you feel like writing a post explaining your >experience with using pypy and its jit for writing an interpreter, we cou >ld >publish it on our blog. I suppose it would be useful/interesting for oth >er >people as well. > >What do the others think? I'd look forward to reading it. Laura

1 0

Hacking dropbox
by Antonio Cuni 29 Mar '11

29 Mar '11

Hi, I don't have any news from the dropbox side, but as an aside note, I found a project which is open source and implements the very same technique I used for dropbox, and additionally it also decompiles back to the source code: http://code.google.com/p/pyretic/ ciao, Anto

1 0

Google Summer of Code Idea Proposal
by Romain Guillebert 28 Mar '11

28 Mar '11

Hi I'm interested in doing the Summer of code on PyPy, from what I saw on the mailing list and on irc, I would like to work on 2 things which might interest you (there is no order of preference). * Python backend for Cython (to interface with C code, this could use use ctypes or an API exposed by PyPy), the backend would either produce python code or python bytecode and will allow PyPy to JIT the Cython extension (I talked about it with Alex Gaynor yesterday). * Improve the JVM backend, making it translate on the trunk and interface (R)Python code with Java. (As proposed by Antonio Cuni). I'm available from the end of May (around the 25th) to the beginning of September so this shouldn't be a problem. If this is OK, which one would you prefer ? Regards Romain

2 1

Re: [pypy-dev] Thinking about the GIL
by hyarion＠iinet.net.au 27 Mar '11

27 Mar '11

On Fri Mar 25 21:41 , Armin Rigo sent: >Hi, > >On Fri, Mar 18, 2011 at 1:44 AM, hyarion(a)iinet.net.au >hyarion(a)iinet.net.au> wrote: >> (...) Wrapping each bytecode in an STM >> transaction would give you an as-if-serial execution order, again with no guarantees >> about which order. You get transaction overheads instead of lock/unlock overheads, >> but some STM systems can be quite efficient for short transactions that rarely >> conflict. > >Yes, I also thought about this as one of the solutions that would "fit >the model" of PyPy by not needing changes all over the place. >However, I am unsure that the performance of STM is good enough for >that application so far. Maybe I'm wrong, but I fear (a priori, with >no precise experience) that it would be really too slow to wrap *all* >memory reads and writes with the STM machinery. Yeah, I suppose to get the performance I'm thinking of you probably need to know what you're sharing (and for it not to be everything). You wouldn't need all memory reads and writes to be transactional, just ones to interpreter-level values that are mutable and shared between threads, and ones to things that represent app-level shareable values. The STM system I worked on was for a declarative language with immutable data. In that context it turns out the system only had to be careful when retrieving references from STM; what the reference points to could still be used as normal non- shared data. In a language like Python I guess almost any bit of memory could potentially be (or later become) a shared value that another thread could be using. What's the "success-rate" of the JIT's malloc-removal like? You could omit the transactional overhead for those values. Would that get close enough for the important 20% of code? You could probably do some other jiggery-pokery under the hood to minimise the amount of data protected by STM overhead too. If I had more time I'd love to look at this sort of thing. -- Ben

2 1

Re: [pypy-dev] Google Summer of Code Idea Proposal
by Dan Roberts 27 Mar '11

27 Mar '11

Hi Romain, I just wanted to chime in that I think a Python backend for Cython would be awesome. You wouldn't want to generate bytecode for compatibility reasons: PyPy doesn't use exactly CPython's bytecode, and afaik IronPython and Jython both just use their host VM bytecode directly. Generating ctypes+pure python sources would be the most generally useful, and I think that'd be the best GSoC project. Some great follow-up work would be to skip ctypes on PyPy and generate different calls, as Antonio has found that even with his awesome optimizations, ctypes is significantly (I think it was 5x) slower than the alternative, rawffi I think, but don't quote me on that or the number. As for reviving the jvm backend, I think that's a very *interesting* project, I actually have looked at doing that myself, there's a tiny bit of work in the ootype-virtualrefs that I've done. You'll need to learn a ton about how RPython works, the translator, and so on. I'm not sure that one would get approval though, especially pitted against awesome projects like your ctypes project. But don't let me discourage you from applying for *both*, I just wouldn't put all my eggs in the jvm-backend basket. Cheers, Dan On Mar 27, 2011 1:07 PM, "Romain Guillebert" <romain.py(a)gmail.com> wrote:

1 0