Mailman 3 January 2008 - pypy-dev

Blog / Parser
by Toby Watson Jan. 11, 2008

Jan. 11, 2008

Hi, I've just read the blog post, "Visualizing a Python tokenizer" and it reminded me of this: "OMeta: an object oriented language for pattern matching" http://www.cs.ucla.edu/~awarth/papers/dls07.pdf OMeta is an extension and generalisation of the idea of PEGs*. It provides a nice way to describe a language both at the character level (tokens), the grammar itself and productions into the AST. Finally the grammars are extensible (possibly from within the language itself). The … [View More]

2 1

CLI on Windows
by Jean-Paul Calderone Jan. 10, 2008

Jan. 10, 2008

Hello all, I've played around a little bit with PyPy and Mono on Linux. Now I'm wondering what the requirements are to use/test the CLI features of PyPy on Windows. Can anyone give a list of the dependencies PyPy/CLI has on Windows? Jean-Paul

2 1

Re: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
by Carl Friedrich Bolz Jan. 4, 2008

Jan. 4, 2008

Hi Toby, (ccing pypy-dev) Toby Watson wrote: > I have a PyPy related idea - one I'd like to explore at the upcoming > sprint in Leysin. > > Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual > machine shipped in Flash and donated as Tamarin (to be the Mozilla/ > Firefox JS engine). This in turn would open a number of possibilities, > not least would be seeing a >>>> prompt sitting in a Flash app on a > web page. > > As I … [View More]understand it from some of the PyPy CLI/JVM docs I have been > reading it is possible certainly to generate an interpreter, but > possibly also a compiler for any particular platform, in this case > using the oobackend. > > Just quickly, because I know many people do not know about (or > care! ;-) what is happening in Flash, the recent version 9 release > introduced: > > * A new virtual machine (AVM2) > * With JITter compilation > * Support for prototype and class-based ECMAscript styles (should be > suitable as Python target) > * Dynamic bytecode generation _and loading_ is possible (should be > handy) > * ActionScript 3/ECMAScript has strong typing and other more mature > language features > > FYI: > * Bytecode is called Actionscript Byte Code (ABC) > * Another language (HaXe) has already targeted this VM successfully. > > With such a 'Flython' interpreter I would hope in time to be able to > bind to Adobe's native and library classes and hence by able to : > > * Compile swf files from Python (for Flash) source > * Build Flex apps (Adobe's GUI framework) > * Target AIR (Flash desktop apps) > > About myself, my interests: > > 30-something developer of many, many interactive systems using a lot > of different technology from web Python/PHP, through ActionScript to > OpenGL/C++, C#, etc. I have written many small parts of compilers and > language transformers, but never a whole compiler. I have experience > with Agile development environments and Test-driven development. > > I've started digging around using Adobe's flex compiler, the open- > source Tamarin engine, plus my own small swf decompiler and lots of > public information to understand how the AVM2 VM works and Adobe's > compiler infrastructure - separate compilation, linking, the bytecode > etc. > > I am actually interested in developing new language/programming styles > atop PyPy/Flython (if it did exist) and I like the Flash VM because it > is web-borne and so widely distributed - essentially people could try > out experiments in their browser without having to download/build > anything. > > I don't know how much is possible in a week and of course much depends > on me, but my goal would be to get a skeleton for the interpreter/ > translator and to sound out what all the issues are in targetting the > AVM, binding to native classes and so on. > > I was considering writing another compiler specifically for this task, > but after looking at Jython, IronPython, Boo and PyPy it became > apparent that PyPy is quite mature now, may save a lot of time, and > ultimately produce a more interesting result. > > I would hope that by adding another backend I could contribute to the > fleshing-out of PyPy, help participate in the general maturing of the > project and hopefully give the group a cool demo too. > > So...I hope to see some of you at the sprint and look forward to some > interesting discussions! I think this task is probably sprint-sized and relevant to PyPy. The question is whether some of the core-devs would like to work on it, so you can pair with that person. Then there are technical questions too, for example whether the tamarin VM is good enough to actually run all the full PyPy Python interpreter on top of it. Cheers, Carl Friedrich [View Less]

4 8

interrupts, preemption, threads
by Daniel Farina Jan. 4, 2008

Jan. 4, 2008

Hello list, I wasn't able to locate any of the keywords ['interrupt', 'preemption', 'thread'] in the web documentation, so I come to the list to ask what is roughly the status of these things in the generated PyPy virtual machines on whatever applicable platforms. What I am not asking about is PyPy's Python implemention of threads, but rather support for those writing their own interpreters. Example scenarios: - timeouts on arbitrary code fragments (conflated by the fact that non-… [View More]

2 3

JVM integration
by Paul deGrandis Jan. 3, 2008

Jan. 3, 2008

Anto, Niko, and All, I have begun working on PyPy again and am starting on jvm integration. After talking briefly with Carl on IRC, I looked at the code in CLI. I have a few questions: It appears as though the general approach is to identify entry points into CLI code (via dlls), use reflection, and make calls. This is also later used to make (r)python modules into dlls themselves (allowing for bidirectional integration). Is this correct? Also, how does the notion of PythonNet play into … [View More]

2 1

Re: [pypy-dev] AVM2 / Tamarin backend at the sprint
by Toby Watson Jan. 3, 2008

Jan. 3, 2008

Since there seems to be a little discussion here, may I ask: What would be a useful code / piece of understanding to bring along? Things I've been thinking about: * A Python lib for generating ABC byte code, ABC byte code files (to run on Tamarin) and SWFs (for Flash). * An understanding of how class linking / loading environment works in Tamarin / AVM2. * ??? What else to figure out on the target side ??? I'm thinking that I want to generate ABC and produce ABC files. They are … [View More]

1 0

Fwd: [pypy-sprint] AVM2 / Tamarin backend at the sprint
by Leonardo Santagada Jan. 3, 2008

Jan. 3, 2008

The right place to discuss this is pypy-dev Begin forwarded message: > From: Toby Watson <toby(a)thetobe.com> > Date: 3 de janeiro de 2008 19h18min11s GMT-02:00 > To: pypy-sprint(a)codespeak.net > Subject: [pypy-sprint] AVM2 / Tamarin backend at the sprint > > Hi Everyone, > > I have a PyPy related idea - one I'd like to explore at the upcoming > sprint in Leysin. > > Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual > machine shipped in … [View More]Flash and donated as Tamarin (to be the Mozilla/ > Firefox JS engine). This in turn would open a number of possibilities, > not least would be seeing a >>>> prompt sitting in a Flash app on a > web page. > > As I understand it from some of the PyPy CLI/JVM docs I have been > reading it is possible certainly to generate an interpreter, but > possibly also a compiler for any particular platform, in this case > using the oobackend. > > Just quickly, because I know many people do not know about (or > care! ;-) what is happening in Flash, the recent version 9 release > introduced: > > * A new virtual machine (AVM2) > * With JITter compilation > * Support for prototype and class-based ECMAscript styles (should be > suitable as Python target) > * Dynamic bytecode generation _and loading_ is possible (should be > handy) > * ActionScript 3/ECMAScript has strong typing and other more mature > language features > > FYI: > * Bytecode is called Actionscript Byte Code (ABC) > * Another language (HaXe) has already targeted this VM successfully. > > With such a 'Flython' interpreter I would hope in time to be able to > bind to Adobe's native and library classes and hence by able to : > > * Compile swf files from Python (for Flash) source > * Build Flex apps (Adobe's GUI framework) > * Target AIR (Flash desktop apps) > > About myself, my interests: > > 30-something developer of many, many interactive systems using a lot > of different technology from web Python/PHP, through ActionScript to > OpenGL/C++, C#, etc. I have written many small parts of compilers and > language transformers, but never a whole compiler. I have experience > with Agile development environments and Test-driven development. > > I've started digging around using Adobe's flex compiler, the open- > source Tamarin engine, plus my own small swf decompiler and lots of > public information to understand how the AVM2 VM works and Adobe's > compiler infrastructure - separate compilation, linking, the bytecode > etc. > > I am actually interested in developing new language/programming styles > atop PyPy/Flython (if it did exist) and I like the Flash VM because it > is web-borne and so widely distributed - essentially people could try > out experiments in their browser without having to download/build > anything. > > I don't know how much is possible in a week and of course much depends > on me, but my goal would be to get a skeleton for the interpreter/ > translator and to sound out what all the issues are in targetting the > AVM, binding to native classes and so on. > > I was considering writing another compiler specifically for this task, > but after looking at Jython, IronPython, Boo and PyPy it became > apparent that PyPy is quite mature now, may save a lot of time, and > ultimately produce a more interesting result. > > I would hope that by adding another backend I could contribute to the > fleshing-out of PyPy, help participate in the general maturing of the > project and hopefully give the group a cool demo too. > > So...I hope to see some of you at the sprint and look forward to some > interesting discussions! > > Kind regards, > Toby -- Leonardo Santagada [View Less]

1 0

llvm.gcroot
by Armin Rigo Jan. 2, 2008

Jan. 2, 2008

Hi Eric, hi all, A field report about the 'llvmgcroot' branch... I've been playing with ways to remove the "shadow stack" of GC roots. More precisely, with our own framework GCs, the problem is that in C there is no portable way to walk the current stack and locate the local variables containing pointers to GC-managed objects (let alone modify these pointers safely, in case of a moving GC). So what we do at the moment is to push/pop all such pointers in a custom shadow stack around each call,… [View More] everywhere. In LLVM there is a very recently implemented primitive called llvm.gcroot() which allows us to declare local variables which contain pointers to GC-managed objects. Then with the help of a few pages of custom C++ code to dynamically link with LLVM's native code backend 'llc', we can put extra data into the assembler that describes where these local variables are and how big each function's stack frame is. This is not platform-specific so far, in the sense that the same C++ code works on any platform. There is still a bit of platform-specific code, which is how exactly to interpret this data to walk the stack from our GC's collect() methods. For more info: C++ plugin for llc (see README.txt): http://codespeak.net/svn/user/arigo/hack/pypy-hack/stackrootwalker PyPy branch: http://codespeak.net/svn/pypy/branch/llvmgcroot with the platform-specific stack walking code: http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/rpython/memory/gctrans… This gives some modest speed-up over a plain llvm translation (which is slower than a C translation, which is itself slower than a --llvm-via-c, but the latter doesn't support llvm.gcroot(), obviously, as it goes through C as well). To activate this in the branch, use translate.py --llvmgcroot. Then I started playing with something similar for genc by parsing the .s files produced by gcc. Completely platform-specific, obviously. To active this in the branch, use translate.py --llvmgcroot --backend=c (sorry if it's a bit confusing). This uses the special 'asm' keyword of gcc to generate markers that are found by http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/translator/c/trackgcro… and we get data added to the .s file in the same format as the custom C++ GC plugin for LLVM. Definitely an obscure hack, though it's kind of safe in the sense that if gcc does too advanced optimizations it will not give obscure segfaults, but clean crashes in trackgcroot.py itself. It gives the same modest speed-ups for pypy-c. In both the C and the LLVM case, the presence of the gcroot markers currently prevent some low-level optimizations by, respecitvely, llc and gcc. The reason is that the local variables that are marked as GC roots cannot be simply "fast" variables that the compiler is free to put in any register and move around as needed. I'm not sure how to fix this for llc, but for gcc I have the project to try to extend trackgcroot.py. Finally, if we're talking about maximum performance, it might be possible (given enough time and obscure hacking) to use trackgcroot.py on the assembler files produced by gcc during a --llvm-via-c translation... by convincing LLVM to put in the C code things that look like calls, but that are really macros that expand to the asm() markers... A bientot, Armin. [View Less]

1 0