Mailman 3 March 2005 - pypy-dev

bug in http://codespeak.net/moin/pypy/moin.cgi/FrontPage?action=show
by Laura Creighton 10 Oct '05

10 Oct '05

If you click on the 'documentation' link you get a traceback! ooops. I will look at this later unless somebody beats me to it. Laura

5 4

Next step: gen???.py
by Armin Rigo 02 Apr '05

02 Apr '05

Hi there, This is a status report and a "what to do next" summary. The status is easily summarized as follows: the PyPy interpreter is quite complete and highly compliant with CPython, minus a number of dark corners that are only likely to bite user programs using the most introspective features of the standard library (most notably pickling). The flow graph and annotation subsystems work quite nicely too; we can successfully annotate more or less all of PyPy, or at least we are getting close to this goal. The "next" step is low-level code generation. Here, we have a rather large number of prototypes all around. The most complete one is genc.py, which is able to produce code for roughly the complete PyPy interpreter, but without using the annotations -- i.e. it is very slow code, and it is essentially dependent on CPython. Approaching the same goal but from the opposite direction, we have genllvm.py, which is only able to translate fully annotated graphs and doesn't have any mean of fall-back for things like faked objects which cannot be annotated. Finally there are other very incomplete or deprecated or planning-only back-ends: gencl, genpyrex, genjava. Not to mention geninterplevel whose goal is still different. The question is which line of work to focus on right now. All of these back-ends are interesting and worthwhile in the long run but we need to select a first one. There are basically 4 reasonable options: * Enhance genc.py. This is a step-by-step process, and any intermediate version can still be tested against the whole PyPy and against the snippet examples. Another advantage of genc is that it is the only option that doesn't depend on any external tool other than a C compiler. * Enhance genc.py using the C++ facility of function overloading for simplicity (basically, we would generate "z=add(x,y)" in the file and let the C++ compiler decide which version of add() to call based on the declared types of x and y). This might well be the easiest solution. A minor drawback is to require a C++ compiler. A possibly larger drawback is that the C++ compilation time might be quite larger, even for similar-looking code. (Having to know C++ in the first place shouldn't be that big a drawback if we don't use fancy C++ features.) * Go for genllvm.py. An obvious drawback is that we'd all have to install LLVM. The problem with genllvm right now is that it cannot make sense of unannotated code (or code containing the SomeObject annotation). We don't know yet for sure the quantity of such SomeObjects in the annotated PyPy source code, but a guess is that they occur mainly for "fake" stuff (file, long, unicode...). If so, there is one way around this problem. Carl pointed out that it *might* be easy to link the LLVM compiler output with CPython, possibly making a C extension module for CPython. If so, then we would add in genllvm support for "black box" PyObject pointers, and use a few functions from the CPython C-API to manipulate them. The goal here would be to modify the source code of the interpreter/module/objspace to reduce the number of operations that need to be performed on these "black boxes". For example, we could possibly reduce all these operations to method calls. In other words, we would say that using CPython objects like "long" at interp-level is temporarily OK but only if they are manipulated via method calls. This would make the genllvm support for them much easier. * genjava.py could be another option. It has a simpler type system, which matches ours quite well, but genjava doesn't exist yet at all (the one in the java/ subdirectory had a different goal in mind). We get memory management for free. If we add the requirement to compile with GCJ it could be easy to make a CPython extension module too, with the same problems and solutions about SomeObject as above. All in all, each 4 option is equally possible. If I have to pick one, I guess the first 2 options pass the additional criteria of "very good confidence that it will work within a couple of months". This is definitely biased by the fact that I'm not fluent in Java and know very little about LLVM, but also by my lack of knowledge about the ease of installation and integration of the corresponding tools. Then to pick one of the first two options, the second one (allowing some C++ facilities to sneak it) is my favourite. Comments and feed-back are welcome! Armin

8 12

Parsing prognosis, redux.
by Jonathan David Riehl 31 Mar '05

31 Mar '05

Hi all, I just wanted to chime in on the parser/compiler issue. I think we would all benefit if I spent some time documenting Basil, as Basil is a language development framework, and much more flexible (at least in design if not implementation) than Armin may suspect. The stuff I dropped into Pypy is divorced from Basil and loses some of its flexibility. This is the price one pays for not creating a cross project dependency. I've always had in mind the goal of being able to rapidly prototype Python variants in Python. If another project that coincides with Basil's scope is a part of the EU project, perhaps that is the better political solution for the Pypy project. The EU has paid (or will pay) for this work, and will pay for experts to work full time on integration of that work. My situation is much more in the air as I have dissertation work to undertake (which could still advance Basil significantly,) and will shortly lose most of my travel funding as my primary benefactor is leaving the university (although, I might still seek a travel grant from the PSF if I can show enough of a contribution.) I suspect that we will just have to look at both parsers and see which is the best technical fit, and not just in the short term (ie. which is easier to support.) The group will make a decision and we can move on from there. If the Logilab work does not yet include a compiler, perhaps we should start a discussion that will keep us from duplicating too much work on that front. Thanks! -Jon

1 0

Re: [pypy-svn] r10191 - in pypy/dist/pypy: interpreter objspace/std translator
by hpk＠trillke.net 31 Mar '05

31 Mar '05

Hi Christian, On Wed, Mar 30, 2005 at 23:35 +0200, tismer(a)codespeak.net wrote: > Author: tismer > Date: Wed Mar 30 23:35:23 2005 > New Revision: 10191 > ... > Log: > ... > applevelinterp-lified a lot more modules. > Besides those which are not R-Pythonic due to yield, > there are some things left: > > ... > > pyparser is not supported since it cannot be imported. Something was > not checked in by the one who added it. Do you mean Jonathan's python implementation of the parser module that Jonathan and i added at the Pycon sprint? This seems to work from a fresh checkout for me ... cheers, holger

2 1

Parser
by Armin Rigo 31 Mar '05

31 Mar '05

Hi all, Last sprint, one of the things we integrated into PyPy was Jon's parser. There was some potential collision here: the Logilab people, Ludovic in particular, has also been working on a parser for some time. Unfortunately, the sprint started on a Saturday and the integration was mostly done by Monday, so contacting Ludovic for feed-back was difficult. After a chat with Adrien, the situation seems to be as follows: the Logilab parser is roughly ready (needs polishing) and will be checked in somewhere soon. (Checking in regularly would be a good idea anyway, even for incomplete stuff...) It ressembles Jon's in that it parses Python's Grammar file and makes a parser from that, producing nested tuples in the format that the compiler package can turn into AST and then bytecode. We will have to compare the two parsers to know more precisely the current status. I suspect that the Logilab parser is more appropriate to Logilab's and DFKI's own goals in the WP09/WP10, which involve playing around with the syntax, using a visitor pattern to generate its output; as far as I understand it, Jon's parser is not meant to be nice and flexible but mainly just a rewrite in Python of CPython's own parser. This is a bit unfortunate, as it means that some work has been done twice. Not having a Logilab person at the PyCon sprint was certainly a problem, given that they should take the lead on the parsing and compiling stages (which, I admit freely, I'm not too interested in). I'll let Adrien and Ludovic comment on this... A bientôt, Armin

6 8

Any update on the latest sprint?
by Ben.Young＠risk.sungard.com 30 Mar '05

30 Mar '05

Hi there! I am just an interested observer in the PyPy project and was wondering if there was a summary anywhere of what went on at the latest sprint. It would be great to hear just how close you are getting to some of the translation stuff in particular (llvm seems very cool!) Cheer, Ben

1 0

Re: [pypy-svn] r10163 - in pypy/dist/pypy: module/builtin objspace/std
by Michael Hudson 29 Mar '05

29 Mar '05

arigo(a)codespeak.net writes: > Author: arigo > Date: Tue Mar 29 19:41:35 2005 > New Revision: 10163 > > Modified: > pypy/dist/pypy/module/builtin/app_descriptor.py > pypy/dist/pypy/objspace/std/typetype.py > Log: > Who would have though that the __doc__ attribute of types was delicate to get > right? It is specially so when we need a class like 'property' whose > instances also have a __doc__ slot... Crazyness. Oh, I knew this, if only from several years of reading python-checkins :) Cheers, mwh -- * vegai wears his reading bra. <vegai> umm, I mean glasses -- from Twisted.Quotes

1 0

releasing PyPy ...
by hpk＠trillke.net 29 Mar '05

29 Mar '05

Hi pypy-dev'ers, i think it is time to start releasing PyPy. I suggest the following release targets/characteristics: PyPy-0.6 in April: Python Implementation running on top of CPython with up-to-date documentation, nice entry points PyPy-0.7 in June: translate Python programs (written within a restricted subset of Python) to lower level languages, with actual documented and tested tools to do it. PyPy-0.8/0.9 in July/August: statically translated mostly self-contained compliant Python implemenetation PyPy-1.0 September: Milestone release: have a fully usable self-translating Python implementation with focus on flexibility and extensibility, _not_ on speed, let's name it the "Knuth release" :-) Technically we should put the releases into their own branches and tags so that we have the following layout in the repository: svn/pypy/ dist/... # ongoing development branch/ pypy-X.Y/ # always a snapshot of the pypy/dist directory tag/ pypy-X.Y.Z # always a snapshot of the pypy/dist directory moreover we very likely make available downloads as zip and tar files somewhere and also prep up the web site for representing release information etc.pp. IMO the first release in April should (or maybe even at Pycon?) focus on getting all the documentation, and entry points nice enough. I'd like to work a bit on this and discuss things during the pre-Pycon sprint, i guesss. comments, wishes & suggestions welcome! cheers, holger

2 2

Parsing prognosis.
by Jonathan David Riehl 21 Mar '05

21 Mar '05

I need some direction on how to proceed with the parser module. I don't recommend using it for the interpreter level compiler at this time as it takes several minutes to load. Perhaps it would be appropriate to add it to the library as a drop in replacement to the CPython parser module, and look at pulling it into the interpreter level after we have a C translator working well. Normally, I'd just put something into .../dist/pypy/lib and add some tests to test2, but there is a bit of additional infrastructure used by the parser. Is there a better place to put this? Possibly we should look into building yet another bytecode compiler and creating a new .../dist/pypy/parser package. Of course, I will add some documentation about how to update the parser (such as using PyPgen). Anyway, any thoughts are welcome, and (logical) instructions desirable. Thanks! -Jon

1 0

Re: summaries on builtin types/modules
by bernstorff_lehmann＠mail.tele.dk 20 Mar '05

20 Mar '05

just a heads up : For some reason the summaries doesn't work in either Internet Explorer or Opera (windows 2k). In these browsers you only get a blank window when you try : http://codespeak.net/~pedronis/deltareports/modules-index.html FireFox is ok though. I havent tried to understand why yet Anders Lehmann

5 6