[Compiler-sig] More Jython progress

[Finn Bock]

>>>>>> "FB" == Finn Bock <bckfnn@worldonline.dk> writes:
>
>  FB> Hi, Based on the current AST tree and a modified CodeCompiler, I
>  FB> can now generate javabytecode. I'm sure there are still a few
>  FB> bugs in the generated code but it passes our initial (rather
>  FB> small) test suite.
>
>  FB> The patch to current jython CVS is available here:
>
>  FB> :
>  FB> http://sourceforge.net/tracker/?func=detail&atid=312867&aid=546737&group_id=12867
>
>  FB> The next phase for Jython is to port jythonc (the java
>  FB> sourcecode generator) to use the new AST tree.

[Jeremy]

>I had hoped to look over your code this weekend, but didn't get to
>it.  The subtleties of converting list comprehensions delayed me <0.6
>wink>.  Is it your intent to re-do the compiler(s) in Jython? 

Yes. One is done already, still one compiler to do.

If you want to look, the AST is created by org.python.p2.TreeBuilder
which is called by the actions specified in the JavaCC grammar. The
compiler are located in org.python.c2.CodeCompiler. In the c2 package
there are also a ScopesCompiler that handling the symbol types
(fast_locals, cells, etc) and an ArgListCompiler that deals with default
argument values and argtuple unpacking. All three compiler classes are
using the visitor pattern I outlined a while ago.

>In
>hindsight, it seems clear that you weren't doing this just to kill
>time, but I didn't realize that both Pythons were in for a compiler
>overhaul at the same time.

The size of the overhaul is significantly smaller for jython. Our old
syntax tree was almost node-by-node exactly the same as the new AST but
all the children was anonymous. Except for a few smaller differences
(like listcomp and function arguments) the transformation have been
straightforward.

The main reason I wanted to switch the new AST is because we have to
create yet another AST visitor, one that does on the fly interpretation
of the python code. I did not want to start on this visitor using the
old tree, I guessed it would be faster to switch the other compilers to
the new AST instead.

>  FB> Some observations about the AST:
>
>I'll have to think about these tomorrow.  I hope it's not too much
>trouble that I changed Dict.

The new way of representing Dict keys and values is rather unnatural for
jython because all the existing support functions and PyDictionary()
ctors assume that the elements are alternating keys and values.

It is not a big issue and I don't think the slowdown of rearranging the
elements will be noticable.

>  FB> The ListComp and the way I uses it bugs me a little. I'll admit
>  FB> it is a clever way of representing a listcomp but I have been
>  FB> reusing the visitFor() and visitIf() methods to generate the
>  FB> loop and branching code. Since I wanted to continue to do that I
>  FB> builds a series of For() and If() statements from the listcomp:
>
>I just looked at the compiler package and saw that it's visitFor() and
>visitListFor() are quite similar.  The visitIf() and visitListIf()
>aren't very similar, presumably because a lot of logic is in the
>visitListComp() method.
>
>    The compiler package uses a ListComp() object with two children --
>    a binding expression and a list of ListCompFor and ListCompIf
>    nodes.)
>
>I'd need to think harder about how the two kinds of fors and ifs could
>be merged here.  Perhaps you could accomplish this with helper methods
>instead of creating throwaway nodes?  _visit_generic_for() that could
>be called be either visitFor() or visitlistcomp()?

I'll think more about it, but the main problem is that visitFor() and
visitIf() are using recursion while a ListComp is a sequence.

regards,
finn