[Python-Dev] Questionable AST wibbles

[Neal Norwitz]

Jeremy,

There are a bunch of mods from the AST branch that got integrated into
head.  Hopefully, by doing this on python-dev more people will get
involved.  I'll describe  high level things first, but there will be a
ton of details later on.  If people don't want to see this crap on
python-dev, I can take this offline.

Highlevel overview of code size (rough decrease of 300 C source lines):
 * Python/compile.c -2733 (was 6822 now 4089)
 * Python/Python-ast.c +2281 (new file)
 * Python/asdl.c +92 (new file)
 * plus other minor mods

symtable.h has lots of changes to structs and APIs.  Not sure what
needs to be doc'ed.

I was very glad to see that ./python compileall.py Lib took virtually
the same time before and after AST.  Yeah!  Unfortunately, I can't say
the same for memory usage for running compileall:

Before AST: [10120 refs]
After AST:  [916096 refs]

I believe there aren't that many true memory leaks from running
valgrind.  Though there are likely some ref leaks.  Most of this is
probably stuff that we are just hanging on to that is not required.  I
will continue to run valgrind to find more problems.

A bunch of APIs changed and there is some additional name pollution. 
Since these are pretty internal APIs, I'm not sure that part is a big
deal.  I will try to find more name pollution and eliminate it by
prefixing with Py.

One API change which I think was a mistake was _Py_Mangle() losing 2
parameters (I think this was how it was a long time ago).  See
typeobject.c, Python.h, compile.c.

pythonrun.h has a bunch of changes.  I think a lot of the APIs
changed, but there might be backwards compatible macros.  I'm not
sure.  I need to review closely.

symtable.h has lots of changes to structs and APIs.  Not sure what
needs to be doc'ed.  Some #defines are history (I think they are in
the enum now):  TYPE_*.

code.h was added, but it mostly contains stuff from compile.h.  Should
we remove code.h and just put everything in compile.h?  This will
remove lots little changes.
code.h & compile.h are tightly coupled.  If we keep them separate, I
would like to see some other changes.

This probably is not a big deal, but I was surprised by this change:

+++ test_repr.py        20 Oct 2005 19:59:24 -0000      1.20
@@ -123,7 +123,7 @@

    def test_lambda(self):
        self.failUnless(repr(lambda x: x).startswith(
-            "<function <lambda"))
+            "<function lambda"))

This one may be only marginally worse (names w/parameter unpacking):

test_grammar.py

-    verify(f4.func_code.co_varnames == ('two', '.2', 'compound',
-                                        'argument',  'list'))
+    vereq(f4.func_code.co_varnames,
+          ('two', '.1', 'compound', 'argument',  'list'))

There are still more things I need to review.  These were the biggest
issues I found.  I don't think most are that big of a deal, just
wanted to point stuff out.

n