[Python-Dev] AST branch update

[Jeremy Hylton]

Neil and I have been working on the AST branch for the last week. 
We're nearly ready to merge the changes to the head.  I imagine we'll
do it this weekend, barring last minute glitches.

There are a few open issues that remain.  I'd like to merge the branch
before resolving them.  Please let me know if you disagree.

The current status of the AST branch is that only two tests fail:
test_trace and test_symtable.  The causes for these failures are
described below.  We did not merge the current head to the branch
again, but I diffed the test suite between head and branch and did not
see any substantive changes since the last merge.

Some of the finer points of generating the line number table (lnotab)
are wrong.  There is some very delicate code to support single
stepping with the debugger.  We'll get that fixed soon, but we'd like
to temporarily disable the failing tests in test_trace.

The symtable module exposed parts of the internal representation of
the old symbol table.  The representation changed, and the module is
going to need to change.  The old module was poorly documented and
tested, so I'd like to start over.  Again, I'd like to disable a
couple of failing tests until after the merge occurs.

I don't think the current test suite covers all of the possible syntax
errors that can be raised.  I'd like to add a new test suite that
covers all of the remaining cases, perhaps moving some existing tests
into this module as well.  I'd like to do that after the merge, which
means there may be some loose ends where syntax errors aren't handled
gracefully.

For those of you familiar with the ast work, I'll summarize the recent changes:

We added line numbers to expressions in the AST.  There are many cases
where a statement spans multiple lines.  We couldn't generate a
correct lnotab without knowing the lines that expressions occur on.

We merged the peephole optimizer into the new compiler and restored
PyNode_Compile() so that the parser module works again.  The parser
module will still expose the old parse trees (just what it's users
want).  We should probably provide a similar AST module, but I'm not
sure if we'll get to that.

We fixed some flawed logic in the symbol table for handling nested
scopes.  Luckily, the test cases for nested scopes are pretty
thorough.  They all pass now.

Jeremy