[Python-Dev] AST branch is in?

[Jeremy Hylton]

On 10/20/05, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 10/20/05, Anthony Baxter <anthony at interlink.com.au> wrote:
> >
> > Could someone involved give a short email laying out what concrete (no
> > pun intended) advantages this new compiler gives us? Does it just
> > allow us to do new and interesting manipulations of the code during
> > compilation? Cleaner, easier to maintain, or the like?
>

I just wanted to clarify that Neal meant the abstract syntax, not the
grammar.  It should allow people to write tools to analyze Python
source code without having to worry about the often irrelevant details
of the exact tokens or the way they are parsed.  We should be able to
get to a state where tools using the AST work with Python and Jython
(and maybe IronPython, who knows).  The tokenize and parser modules
still exist for tools for which those details aren't irrelevant.

We should also think about how to migrate the compiler module from its
current AST to the new AST, although the backwards compatibility
issues there are a bit tricky.

> The Grammar is (was at one point at least) shared between Jython and
> would allow more tools to be able to share infrastructure.  The idea
> is to eventually be able to have [JP]ython output the same AST to
> tools.  There is quite a bit of generated code based on the Grammar.
> So some stuff should be easier.  Other stuff is just moved.  You still
> need to convert from the AST to the byte code.
>
> Hopefully it will be easier to do various sorts of optimization and
> general manipulation of an AST rather than what existed before.

I think it should be a lot easier to write tools for the C Python
compiler that do extra analysis or optimization.  The existing
peephole optimizer could be improved by integrating it with the
bytecode assembler (for example, eliminating all NOP bytecodes).

Jeremy