Mailman 3 October 2003 - pypy-dev

Re: LLVM and PyPy
by Armin Rigo 04 Nov '03

04 Nov '03

Hello Chris, On Fri, Oct 31, 2003 at 10:58:45AM -0600, Chris Lattner wrote: > These are definitely features that we plan to add, but just haven't gotten > to yet. In particular, Alkis is working on a Java front-end, which will > require similar features. In the beginning, we will probably just use a > conservative collector, eventually adding support for precise GC. Great! > We already have the capability of doing function-at-a-time code > generation: what is basic-block at a time generation used for? How do you > do global optimizations like register allocation? It is central to Psyco, the Python just-in-time specializer (http://psyco.sourceforge.net) whose techniques we plan to integrate with PyPy. Unlike other environments like Self, which collects execution profiles during interpretation and use them to recompile whole functions, Psyco has no interpretation stage: it directly emits a basic block and run it; the values found at run-time trigger the compilation of more basic blocks, which are run, and so on. So each function's machine code is a dynamic network of basic blocks which are various specialized versions of a bit of the original function. This network is not statically known, in particular because basic blocks often have a "switch" exit based on some value or type collected at run-time. Every new value encountered at this point triggers the compilation of a new switch case jumping to a new basic block. We will also certainly consider Self-style recompilations, as they allow more agressive optimizations. (Register allocation in Psyco is done using a simple round-robin scheme; code generation is very fast.) > That would be great! We've tossed around the idea of creating C bindings > for LLVM, which would make interfacing from other languages easier than > going directly to the C++ API, but we just haven't had a chance to yet. > Maybe you guys would be interested in helping with that project? Well, as the C++ API is nice and clean it is probably simpler to bind it directly to Python. We would probably go for Boost-Python, which makes C++ objects directly accessible to Python. But nothing is sure about this; maybe driving LLVM from LLVM code is closer to our needs. Is there a specific interface to do that? Is it possible to extract from LLVM the required code only, and link it with the final executable? In my experience, there are a few limitations of C that require explicit assembly code, like building calls dynamically (i.e. the caller's equivalent of varargs). A bientot, Armin.

2 3

Re: [LLVMdev] Re: LLVM and PyPy
by holger krekel 31 Oct '03

31 Oct '03

Hi Chris, [Chris Lattner Fri, Oct 31, 2003 at 10:58:45AM -0600] > On Fri, 31 Oct 2003, Armin Rigo & Holger Krekel wrote: > > > Hello Chris, > > > > We have been investigating your project and the good documentation > > and are very impressed. If we understood your goals correctly > > this seems like a good match for our ongoing and active PyPy project, > > a reimplementation of the Python language in Python. > > Cool. We are all big fans of Python here. :) That's good because we might want to recode some LLVM functionalities in Python :-) > > We'll definitely try using LLVM as our low-level backend. > > But actually we contact you now to ask you if you'd be interested > > in some bidirectional collaboration. > > I've read up a bit on PyPy, and it looks like LLVM could be a nice way to > get the JIT type interface that you would like. Also, making use of the > LLVM optimizer can make your staticly generated code nice and fast. :) Yes, but we also would want to dynamically emit and execute LLVM code. But a static translation is indeed our first goal :-) > We've tossed around the idea of creating C bindings > for LLVM, which would make interfacing from other languages easier than > going directly to the C++ API, but we just haven't had a chance to yet. > Maybe you guys would be interested in helping with that project? Thinking some more about it, we would probably try to translate our PyPy implementation into LLVM-code and also generate some glue-LLVM-code which allows us to programmatically drive LLVM from Python. Is LLVM able to "drive" itself? I mean can the LLVM-low-level object code generate LLVM-low-level object code and then execute it? This would fit nicely with PyPy because we are running ourselves (in 'abstract interpretation' mode) in order to generate a low-level representation of ourselves. This low-level representation is already close to LLVM's low-level view. So if the LLVM-code gets executed (beeing a python interpreter) it should be able to just-in-time-compile new LLVM code and execute it. With our architecture, for such a JIT we could reuse a good part of the code we already have for generating our low-level representation. It's a rather self-referential thing (also see our logo: http://codespeak.net/pypy/ :-). > > On another level, some of the PyPy core developers are actually also > > involved with the 'codespeak' site which aims at connecting interesting > > open source projects and provide new collaborative development services. > > The PyPy project is extensively using subversion which is a very interesting > > (and stable) alternative to cvs. So if you need any help with setting > > up some publically accessible infrastructure the codespeak guys will > > certainly welcome you. > > At this point, we're working like crazy to get important features > implemented in LLVM. We certainly acknowledge that CVS has severe > deficiencies, but in the near future we'll probably stay with it. > Perhaps after SVN 1.0 comes out... :) then we may want to mirror your cvs repo to subversion :-) The reason is that we want to provide consistent versions of all the libraries/modules/projects we use. And subversion makes this rather easy (if the other project is svn-controled, too), e.g. you can say 'i want to follow the HEAD version of LLVM in this branch' or 'i want to use this stable version of LLVM for my own stable-release'. Then you can just issue 'svn up' and you will have the desired versions on your working-copy. However, i can understand that you don't want to consider subversion right now and will stop advertising now :-) cheers, holger

2 1

LLVM - did we know this?
by Christian Tismer 29 Oct '03

29 Oct '03

Hi Friends, Richard Emslie just pushed my nose into something interesting. I don't remember, did we talk about this before? """ Anyways, I was wondering if anyone in pypy has investigated LLVM (http://llvm.cs.uiuc.edu/). Might be totally off the ball, but looks interesting nonetheless. Also look as if it has funding behind it which you guys could jump on. """ ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

5 9

Interesting project
by Stephan Diehl 28 Oct '03

28 Oct '03

Hi, I've just stumbled over this site: http://pliant.cx/ At a first glance, it looks like an operating system (on top of Linux) written in a python alike language (indentation is significant, so it MUST be similar to Python :-) Don't know, if this can be usefull, or not. Cheers Stephan

2 1

subversion repo will be upgraded **now**
by holger krekel 27 Oct '03

27 Oct '03

hello, we will be upgrading the subversion server libraries and need to dump the old database and restore it to the new format. This will take some time so expect some downtime regarding svn-accesses. The rest of apache will probably continue to run as usual, though. We'll reply to this posting if everything is stable again. We are pretty sure already, that you will need to run svn clients with at least version 0.30 (we'll also post some results for windows and linux clients which we will test the new server with). cheers, Holger & Jum

1 1

PyPy: sprint and news
by Armin Rigo 26 Oct '03

26 Oct '03

PyPy Sprint announcement & news from the project ================================================ We are coming close to a first experimental release of PyPy, a more flexible Python implementation written in Python. The sprint to make this happen will take place in Amsterdam, a city know to be reachable by cheap flights :-) This is 1) the announcement for the sprint; 2) news about the current state of PyPy; 3) some words about a proposal we recently submitted to the European Union. Amsterdam Sprint Details ------------------------ The Sprint will take place from the 14th of December to the 21st of December at the "Vrije Universiteit in Amsterdam", 14th-21th Dec 2003. thanks to Etienne Posthumus, who helps us to organize the event. The main goal will be a complete C translation of PyPy, probably still using a hacked Pyrex-variant as an intermediate layer and using CPython's runtime. We also plan to work on some fun frontends to PyPy like one based on pygame or a web browser to visualize interactions between interpreter and objectspace. If you want to participate on the sprint, please subscribe here http://codespeak.net/mailman/listinfo/pypy-sprint and list yourself on this wiki page http://codespeak.net/moin/pypy/moin.cgi/AmsterdamSprint where you will also find more information as the sprint date approaches. If you are just interested but don't know if you come then only subscribe to the mailing list. State of the PyPy project -------------------------- PyPy works pretty well but still on top of CPython. The double interpretation penalty makes it - as expected - incredibly slow :-) In the Berlin sprint we have thus started to work on the "translation" part, i.e. how this code should be translated into C. We can now translate simple functions to C-like code including some type annotations. For convenience, we are reusing a modified subset of Pyrex to generate the low-level C code. Thanks to Seo (who joined the project from south-korea) we also have a lisp-backend to fuel the endless c.l.py threads about python versus lisp :-) The goal of the next sprint is to complete this work so that we can translate the complete PyPy source into a huge Pyrex module, and then a big CPython extension module. True, the result is not independent from CPython, as it runs reusing its runtime environment. But it's probably an interesting enough state to make a public release from. The translation is done by generating a control flow of functions by means of abstract interpretation. IOW, we run the PyPy interpreter with a custom object space ("flowobjspace") which generates a control flow graph (including the elementary operations) which is then used to generate low-level code for backends. We also have preliminary type inference on the graphs, which can be used by the Pyrex generator to emit some C type declarations. Writing transformations and analysis of these graphs and displaying them with GraphViz's 'dot' is great fun! We certainly have a greater need than ever for graphical interactive tools to see, understand and debug all these graph manipulations and run tests of them. Currently it is a bit difficult to write a test that checks that a transformed graph "looks right"! What we expect from the Amsterdam sprint is thus: - a big not-too-slow "cpypy.so" extension module for CPython, where at least integer arithmetic is done efficiently - interactive tools to display and debug and test PyPy, visualizing control flow, call-graphs and state models. - improving and rewriting our testing tools to give us more control over the testing process, and to allow more interactive testing sessions. Other interesting News ---------------------- Before mid October, we also had a quite different Sprint. It was an approximately 10-day effort towards submitting a proposal to the EU. If it is accepted we will have resources to fund some developers working full- or parttime on the project. However, our "sprint driven development" will continue to play the central role for development of PyPy. There are especially two technical sections of the proposal which you might find interesting to read: "Scientific and technological objectives": http://codespeak.net/pypy/index.cgi?doc/funding/B1.0 "Detailed implementation plan" http://codespeak.net/pypy/index.cgi?doc/funding/B6.0 Maybe you want to read the whole proposal for other reasons, too, like making a EU project of your own or competing with us. Actually, with our sprints there is usually a lot of room for cooperation :-) Anyway, here is the PDF-url: http://codespeak.net/svn/pypy/trunk/doc/funding/proposal/part_b.pdf Everybody who thinks that he/she could help on the project is invited to join! Btw, the latest discussions about our sprint goals usually take place on the pypy-dev list: http://codespeak.net/mailman/listinfo/pypy-dev have fun, Armin & Holger

1 0

Refactored flow model
by Armin Rigo 26 Oct '03

26 Oct '03

Hello everybody, A bunch of commits from yesterday: Holger and I rewrote the classes that we use to represent control flow graphs. We had agreed during the whole Berlin sprint that the many XxxBranch classes weren't very practical to use, and we discussed an alternative at the end of the sprint. That's what we just implemented. There is also a much nicer dot-generator now. The graphs are becoming quite comprehensible now, much shorter, with nice colors, and all :-) Try running the tests with '-v' (either individual tests or the test_all.py script); if you installed dot, it will leave Postscript files in /tmp/usession-*/*.ps, with both the non-simplified and the simplified graph showing on the same page. A bientot, Armin.

1 0

Does RPython have slice?
by Seo Sanghyeon 24 Oct '03

24 Oct '03

I was playing with translator frontend, writing random snippets. And I discovered FlowObjSpace currently lacks newslice interface. Does RPython have slice? I checked StdObjSpace implementation. slicefind.sh is located at SVN repo user/sanxiyn/script. Slice is currently used only in stringojbect, namely split, strip, count, startswith, endswith, and repr. (Two matches in listobject doesn't need to be RPython.) It wouldn't be hard to rewrite them without slice. I remember a discussion on IRC whose log I posted for a while ago: in RPython, op should be able to be implemented efficiently, or be disallowed. And flowgraph transformation was suggested. Perhaps slice to sans-slice transformation should be written, and translators may or may not use the transformation? Seo Sanghyeon

2 1

Re: [pypy-dev] Improved Unit Test Framework
by Peter Hansen 19 Oct '03

19 Oct '03

(Sorry, Holger... I first sent this to you directly by accident, rather than just to the mailing list.) holger krekel wrote: > > [Peter Hansen Sun, Oct 19, 2003 at 09:14:43AM -0400] > > holger krekel wrote: > > > > - improvements to unittest itself > > > > Just what would those be? I still haven't heard a single useful > > improvement. There must be some (at least, that's obviously what > > a lot of people feel), but what are they? > > ok, here a few possible improvements for starters (i have a lot more in > mind :-) > > - beeing able to provide a function that executes a test-method > (today you have to replicate all the boilerplate in TestCase.__call__ > etc.pp.) Why do you need to do that? Or maybe I should say, can't you already do that? If you have a TestCase, I think you just call .run or something on it, same as when you specify a single class and (optionally) a test method name on the command line. I frequently run a single test using our framework wrapper, which just stole code from the existing runner in unittest. > - beeing able to raise e.g. TestSkip Error to indicate that a test > should not be run with the current setup. (You can't alway tell > that offhand in setUp). What does this do that raise TestSkipError (after defining it) would do not do? Continue running other tests? Seems like a property of the test runner, not the unittest side of things (assuming a new test runner, as we are by making the distinction into two categories, I think). > - allow providing extra options to the cmdline-interface > > - plan for/do other user-interfaces (web-driven, pygame, whatever :-) > > - provide means (a TestConsole) to easily get a list of failed tests > (interactively), keep the testing-process alive and repetitively redo > the failed tests. Or save off the info to some file and reload it > on rerunning the test. These are all properties of the test runner, aren't they? The existing unittest has its own version of this stuff, but I got the impression you were already accepting that a different runner would be needed. -Peter

2 1

Re: [pypy-dev] Improved Unit Test Framework
by Laura Creighton 19 Oct '03

19 Oct '03

In a message of Sat, 18 Oct 2003 01:55:36 +0200, holger krekel writes: <snip> >Apparently, almost every larger python project writes its own >unittest-workaround/extension with often similar features. It's sure >interesting to condense a list of useful features from all the different >approaches. <snip> >After all, i don't think that writing a more flexible (unit)test module >suiting our purposes is that hard. We already have quite a bit of it >and can rip from unittest.py. > >cheers, > > holger It might be nice to get a hold of other people and see what they would like to have in a unittest module. That way, even if we don't write it that way ourselves, we can make sure that it will be easy for them to add their heart's desire later. Laura

4 6