From neal@metaslash.com Wed Aug 28 01:22:25 2002 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 27 Aug 2002 20:22:25 -0400 Subject: [Compiler-sig] Status of ast-branch Message-ID: <3D6C17C1.45377AE8@metaslash.com> Jeremy: I've been meaning to ask, but keep forgetting... What is the status of the ast branch? I know it's pretty out-dated now. How close is it to working? Do you have any specific tasks that need to be done? I've reviewed the checkins, but haven't had a chance to try out the branch. I'm still hoping the new AST/compiler can be implemented for 2.3. pychecker is pretty broken w/2.3 and it would be nice to have pychecker[23] working with the new compiler... and maybe even jython. Pretty big dream right now, especially given how little work I've done recently. Neal From jeremy@alum.mit.edu Wed Aug 28 16:21:56 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Wed, 28 Aug 2002 11:21:56 -0400 Subject: [Compiler-sig] Re: Status of ast-branch In-Reply-To: <3D6C17C1.45377AE8@metaslash.com> References: <3D6C17C1.45377AE8@metaslash.com> Message-ID: <15724.60052.484444.888002@slothrop.zope.com> >>>>> "NN" == Neal Norwitz writes: NN> Jeremy: I've been meaning to ask, but keep forgetting... NN> What is the status of the ast branch? I know it's pretty NN> out-dated now. How close is it to working? Do you have any NN> specific tasks that need to be done? I've reviewed the NN> checkins, but haven't had a chance to try out the branch. I think the ast-branch needs a solid week of effort to get it working. I have a number of uncommitted changes (on my machine at home) that wires everything together. The changes are primarily in pythonrun.c; the various functions that parse and compile code need to be updated to parse, convert to ast, and compile. I've been really overloaded with some work for Zope customers, so I haven't had as much time as I'd like to work on the ast-branch. I keep telling myself that things will easy up in September and I can find a week or two to concentrate on it. NN> I'm still hoping the new AST/compiler can be implemented for NN> 2.3. pychecker is pretty broken w/2.3 and it would be nice to NN> have pychecker[23] working with the new compiler... and maybe NN> even jython. Pretty big dream right now, especially given how NN> little work I've done recently. I'd also like to finish it all before the first 2.3 alpha release. If the work can get done in September, I think it will happen. I'd break down the work into the following tasks: - Finish ast.c - test it against are large sample of code - do sensible error handling and memory management - Write ast marshallers for C and Python - Define canonical binary representation for AST - Define Python representation for AST - Write marshallers to convert between C/Python/binary reps - Error checking - Detect syntax errors - Finish code generation pass - Handle all the rest of the grammar. Still to do: - function and classes - ~9 statement types - ~6 expression types - Write basic block to bytecode conversion - Much like pyassem in the compiler package The first two tasks are easily separated. The ast.c code could be repackaged as a C extension module that works with current Python. Then the ast can be tested against all sorts of real python code, and the generated AST can be marshalled into the binary representation to pass back to Python. I have a few notes on the binary representation, but I think it's a fairly straightforward problem. The AST definition (e.g. Python.asdl) can be used to generate a set of codes for each grammar production. Then you just need to define universal encodings for the basic types -- string, number, sequence, optional. The code generation and assembler could be worked on as separate components, but it would be hard to test them in isolation. It's more likely that I should finish up the assembler and get at least a minimal full compiler working. Once I get to that much done, it should be easier for someone to tackle a particular set of unhandle grammar productions. Jeremy From nas@python.ca Wed Aug 28 17:58:54 2002 From: nas@python.ca (Neil Schemenauer) Date: Wed, 28 Aug 2002 09:58:54 -0700 Subject: [Compiler-sig] Re: Status of ast-branch In-Reply-To: <15724.60052.484444.888002@slothrop.zope.com> References: <3D6C17C1.45377AE8@metaslash.com> <15724.60052.484444.888002@slothrop.zope.com> Message-ID: <20020828165854.GA430@glacier.arctrix.com> Jeremy Hylton wrote: > - Write ast marshallers for C and Python > - Define canonical binary representation for AST > - Define Python representation for AST > - Write marshallers to convert between C/Python/binary reps Is it going to be possible to create or modify ASTs from Python and pass them to the later stages of the compiler? Neil From jeremy@alum.mit.edu Wed Aug 28 14:07:54 2002 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Wed, 28 Aug 2002 09:07:54 -0400 Subject: [Compiler-sig] Re: Status of ast-branch In-Reply-To: <20020828165854.GA430@glacier.arctrix.com> References: <3D6C17C1.45377AE8@metaslash.com> <15724.60052.484444.888002@slothrop.zope.com> <20020828165854.GA430@glacier.arctrix.com> Message-ID: <15724.52010.78849.445538@slothrop.zope.com> >>>>> "NS" == Neil Schemenauer writes: NS> Jeremy Hylton wrote: >> - Write ast marshallers for C and Python >> - Define canonical binary representation for AST >> - Define Python representation for AST >> - Write marshallers to convert between C/Python/binary reps NS> Is it going to be possible to create or modify ASTs from Python NS> and pass them to the later stages of the compiler? Yes. I don't know exactly what the API should be. There should be one function that compiles source to an AST and another that compiles the AST to bytecode. I don't know if we add builtins, overload existing ones (like compile), extend the parser module, or come up with a new module. Jeremy From nas@python.ca Wed Aug 28 19:01:57 2002 From: nas@python.ca (Neil Schemenauer) Date: Wed, 28 Aug 2002 11:01:57 -0700 Subject: [Compiler-sig] Re: Status of ast-branch In-Reply-To: <15724.52010.78849.445538@slothrop.zope.com> References: <3D6C17C1.45377AE8@metaslash.com> <15724.60052.484444.888002@slothrop.zope.com> <20020828165854.GA430@glacier.arctrix.com> <15724.52010.78849.445538@slothrop.zope.com> Message-ID: <20020828180157.GA668@glacier.arctrix.com> Jeremy Hylton wrote: > >>>>> "NS" == Neil Schemenauer writes: > NS> Is it going to be possible to create or modify ASTs from Python > NS> and pass them to the later stages of the compiler? > > Yes. Excellent. > I don't know exactly what the API should be. There should be one > function that compiles source to an AST and another that compiles the > AST to bytecode. I think there should also be constructors or factory functions for each node in the AST. > I don't know if we add builtins, overload existing ones (like > compile), extend the parser module, or come up with a new module. I vote to put it in a module or package (reuse compiler?). I'm guessing the average user would not be messing with ASTs. Neil