From neal@metaslash.com Wed Aug 28 01:22:25 2002
From: neal@metaslash.com (Neal Norwitz)
Date: Tue, 27 Aug 2002 20:22:25 -0400
Subject: [Compiler-sig] Status of ast-branch
Message-ID: <3D6C17C1.45377AE8@metaslash.com>
Jeremy:
I've been meaning to ask, but keep forgetting...
What is the status of the ast branch? I know it's pretty
out-dated now. How close is it to working? Do you have
any specific tasks that need to be done? I've reviewed
the checkins, but haven't had a chance to try out the branch.
I'm still hoping the new AST/compiler can be implemented
for 2.3. pychecker is pretty broken w/2.3 and it would be nice
to have pychecker[23] working with the new compiler...
and maybe even jython. Pretty big dream right now,
especially given how little work I've done recently.
Neal
From jeremy@alum.mit.edu Wed Aug 28 16:21:56 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Wed, 28 Aug 2002 11:21:56 -0400
Subject: [Compiler-sig] Re: Status of ast-branch
In-Reply-To: <3D6C17C1.45377AE8@metaslash.com>
References: <3D6C17C1.45377AE8@metaslash.com>
Message-ID: <15724.60052.484444.888002@slothrop.zope.com>
>>>>> "NN" == Neal Norwitz <neal@metaslash.com> writes:
NN> Jeremy: I've been meaning to ask, but keep forgetting...
NN> What is the status of the ast branch? I know it's pretty
NN> out-dated now. How close is it to working? Do you have any
NN> specific tasks that need to be done? I've reviewed the
NN> checkins, but haven't had a chance to try out the branch.
I think the ast-branch needs a solid week of effort to get it
working. I have a number of uncommitted changes (on my machine at
home) that wires everything together. The changes are primarily in
pythonrun.c; the various functions that parse and compile code need to
be updated to parse, convert to ast, and compile.
I've been really overloaded with some work for Zope customers, so I
haven't had as much time as I'd like to work on the ast-branch. I
keep telling myself that things will easy up in September and I can
find a week or two to concentrate on it.
NN> I'm still hoping the new AST/compiler can be implemented for
NN> 2.3. pychecker is pretty broken w/2.3 and it would be nice to
NN> have pychecker[23] working with the new compiler... and maybe
NN> even jython. Pretty big dream right now, especially given how
NN> little work I've done recently.
I'd also like to finish it all before the first 2.3 alpha release. If
the work can get done in September, I think it will happen.
I'd break down the work into the following tasks:
- Finish ast.c
- test it against are large sample of code
- do sensible error handling and memory management
- Write ast marshallers for C and Python
- Define canonical binary representation for AST
- Define Python representation for AST
- Write marshallers to convert between C/Python/binary reps
- Error checking
- Detect syntax errors
- Finish code generation pass
- Handle all the rest of the grammar. Still to do:
- function and classes
- ~9 statement types
- ~6 expression types
- Write basic block to bytecode conversion
- Much like pyassem in the compiler package
The first two tasks are easily separated. The ast.c code could be
repackaged as a C extension module that works with current Python.
Then the ast can be tested against all sorts of real python code, and
the generated AST can be marshalled into the binary representation to
pass back to Python.
I have a few notes on the binary representation, but I think it's a
fairly straightforward problem. The AST definition (e.g. Python.asdl)
can be used to generate a set of codes for each grammar production.
Then you just need to define universal encodings for the basic types
-- string, number, sequence, optional.
The code generation and assembler could be worked on as separate
components, but it would be hard to test them in isolation. It's more
likely that I should finish up the assembler and get at least a
minimal full compiler working. Once I get to that much done, it
should be easier for someone to tackle a particular set of unhandle
grammar productions.
Jeremy
From nas@python.ca Wed Aug 28 17:58:54 2002
From: nas@python.ca (Neil Schemenauer)
Date: Wed, 28 Aug 2002 09:58:54 -0700
Subject: [Compiler-sig] Re: Status of ast-branch
In-Reply-To: <15724.60052.484444.888002@slothrop.zope.com>
References: <3D6C17C1.45377AE8@metaslash.com> <15724.60052.484444.888002@slothrop.zope.com>
Message-ID: <20020828165854.GA430@glacier.arctrix.com>
Jeremy Hylton wrote:
> - Write ast marshallers for C and Python
> - Define canonical binary representation for AST
> - Define Python representation for AST
> - Write marshallers to convert between C/Python/binary reps
Is it going to be possible to create or modify ASTs from Python and pass
them to the later stages of the compiler?
Neil
From jeremy@alum.mit.edu Wed Aug 28 14:07:54 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Wed, 28 Aug 2002 09:07:54 -0400
Subject: [Compiler-sig] Re: Status of ast-branch
In-Reply-To: <20020828165854.GA430@glacier.arctrix.com>
References: <3D6C17C1.45377AE8@metaslash.com>
<15724.60052.484444.888002@slothrop.zope.com>
<20020828165854.GA430@glacier.arctrix.com>
Message-ID: <15724.52010.78849.445538@slothrop.zope.com>
>>>>> "NS" == Neil Schemenauer <nas@python.ca> writes:
NS> Jeremy Hylton wrote:
>> - Write ast marshallers for C and Python
>> - Define canonical binary representation for AST
>> - Define Python representation for AST
>> - Write marshallers to convert between C/Python/binary reps
NS> Is it going to be possible to create or modify ASTs from Python
NS> and pass them to the later stages of the compiler?
Yes. I don't know exactly what the API should be. There should be
one function that compiles source to an AST and another that compiles
the AST to bytecode. I don't know if we add builtins, overload
existing ones (like compile), extend the parser module, or come up
with a new module.
Jeremy
From nas@python.ca Wed Aug 28 19:01:57 2002
From: nas@python.ca (Neil Schemenauer)
Date: Wed, 28 Aug 2002 11:01:57 -0700
Subject: [Compiler-sig] Re: Status of ast-branch
In-Reply-To: <15724.52010.78849.445538@slothrop.zope.com>
References: <3D6C17C1.45377AE8@metaslash.com> <15724.60052.484444.888002@slothrop.zope.com> <20020828165854.GA430@glacier.arctrix.com> <15724.52010.78849.445538@slothrop.zope.com>
Message-ID: <20020828180157.GA668@glacier.arctrix.com>
Jeremy Hylton wrote:
> >>>>> "NS" == Neil Schemenauer <nas@python.ca> writes:
> NS> Is it going to be possible to create or modify ASTs from Python
> NS> and pass them to the later stages of the compiler?
>
> Yes.
Excellent.
> I don't know exactly what the API should be. There should be one
> function that compiles source to an AST and another that compiles the
> AST to bytecode.
I think there should also be constructors or factory functions for each
node in the AST.
> I don't know if we add builtins, overload existing ones (like
> compile), extend the parser module, or come up with a new module.
I vote to put it in a module or package (reuse compiler?). I'm guessing
the average user would not be messing with ASTs.
Neil