From bckfnn@worldonline.dk Thu May 23 13:07:44 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Thu, 23 May 2002 14:07:44 +0200
Subject: [Compiler-sig] More jython progress.
Message-ID: <3CECDB90.3080901@worldonline.dk>
Hi,
I now got a version of jythonc running with the new AST. As always
jythonc (the java sourcecode generator) is about twice as complicated as
the java bytecode generator. Sigh.
In jythonc, Jim Hugunin had added an additional abstraction layer on top
of the anonymous SimpleNode ast tree that jython have used so far. With
the new AST we no longer need the abstraction layer. Without that layer
the code in jythonc became (IMO) a lot simpler and easier to follow. The
new AST also means that the codegen in our two generators are far more
similar than before. A huge win because we always have to implement new
language features in both generators. IOW I'm very happy with the new
AST and with the use of asdl as a way of describing the AST. Good work
Jeremy.
Since jythonc is written in python I would like to work towards an
agreement on what the public python API to the AST should be like.
What package/module should contain the AST node classes?
In jython we have a scope analyzer that analyze a module and return
symboltables for module, class and function levels. I guess CPython must
also have some such thing. Should the compiler package have a public API
to such scope analyzer?
Similar we have future analyzer. Should we define a public API to this?
What should be the public names of the AST node classes? I have used the
name of the Constructors directly but I had to change the name of the
'except' production (to 'excepthandler').
Each field in a constructor is obviously a class instance attribute.
Each asdl constructor becomes a java class. That means new attributes
can't be added dynamicly. A strong limitation compared to a python class
instance. It also means that the __XXX__ attributes can't be changed or
depended on (such as __class__ and __name__).
The sequences in the asdl becomes java arrays. That means that the
elements in a sequence can be changed but the length of the sequence
can't. I think we should ban all attempts to modify the sequences.
Are these restrictions unacceptabe for the potential users of the AST
tree? I would like to reach some type of agreement soon.
regards,
finn
From bckfnn@worldonline.dk Thu May 23 13:28:16 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Thu, 23 May 2002 14:28:16 +0200
Subject: [Compiler-sig] Very experimental python AST tree and unpickling support.
Message-ID: <3CECE060.5030007@worldonline.dk>
Hi,
Jeremy suggested that the CPython view of the AST tree could be
implemented by pickling the C tree and unpickling a python tree.
I have made such a python tree, mostly in order to reach agreement on
how the python API of the tree should look like.
You can find my work here:
http://www.jython.org/ast/
Pythonast.py contains the tree classes including unpickle support and
asdl_py.py is used to create the Pythonast.py. I have not given much
though to the pickling format, it is just quick and dirty and I'm sure
it is full of bugs. The only pickling code I have is located in the java
tree so from jython I have create the 'os.py.pckl' which can be
unpickled as a 'Module' like this:
import Pythonast as ast
print ast.unpickle(open("os.py.pckl", "rb"))
Thoughs, comments and feedback welcome.
regards,
finn
From ecn@metaslash.com Thu May 23 14:25:51 2002
From: ecn@metaslash.com (Eric C. Newton)
Date: Thu, 23 May 2002 09:25:51 -0400
Subject: [Compiler-sig] More jython progress.
In-Reply-To: <3CECDB90.3080901@worldonline.dk>; from bckfnn@worldonline.dk on Thu, May 23, 2002 at 02:07:44PM +0200
References: <3CECDB90.3080901@worldonline.dk>
Message-ID: <20020523092551.A22805@ecn>
> What package/module should contain the AST node classes?
> What should be the public names of the AST node classes? I have used
> the name of the Constructors directly but I had to change the name
> of the 'except' production (to 'excepthandler').
Doesn't matter to me.
> In jython we have a scope analyzer that analyze a module and return
> symboltables for module, class and function levels. I guess CPython must
> also have some such thing.
The python version of the compiler has such a pass: compiler.symbols
> Should the compiler package have a public API to such scope
> analyzer?
That would be helpful.
> Similar we have future analyzer. Should we define a public API to this?
What does the future analyzer do?
> Each asdl constructor becomes a java class. That means new
> attributes can't be added dynamicly. A strong limitation compared to
> a python class instance. It also means that the __XXX__ attributes
> can't be changed or depended on (such as __class__ and __name__).
As long as isinstance() works, I can live with that. Right now the
__class__ name is used for the generic visit() implementation, but
that can be changed.
> The sequences in the asdl becomes java arrays. That means that the
> elements in a sequence can be changed but the length of the sequence
> can't. I think we should ban all attempts to modify the sequences.
I'm modifying the tree structure in only one case in PyChecker2, and I
think I can live without it.
I'm adding attributes to the tree _everywhere_. As PyChecker2 grinds
over the tree, it adds discovered information to the tree (things like
type information and cross-references). I can probably live without
it by tracking the information somewhere else.
-Eric
From bckfnn@worldonline.dk Thu May 23 18:13:32 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Thu, 23 May 2002 19:13:32 +0200
Subject: [Compiler-sig] More jython progress.
References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn>
Message-ID: <3CED233C.7040405@worldonline.dk>
[Eric C. Newton]
>>In jython we have a scope analyzer that analyze a module and return
>>symboltables for module, class and function levels. I guess CPython must
>>also have some such thing.
>
> The python version of the compiler has such a pass: compiler.symbols
I didn't know that. Thank you for the info.
>>Similar we have future analyzer. Should we define a public API to this?
>
> What does the future analyzer do?
It simply return an object with properties 'nested_scope' and 'division'
(in 2.1). It indicate which 'from __future__ import xxx' statement that
was discovered in the module.
>>The sequences in the asdl becomes java arrays. That means that the
>>elements in a sequence can be changed but the length of the sequence
>>can't. I think we should ban all attempts to modify the sequences.
>
> I'm modifying the tree structure in only one case in PyChecker2, and I
> think I can live without it.
The tree can still be modified, it is just the sequences that can't. It
is perfectly valid to assign a new sequence (list or tuple) to a
sequence attribute. I just want us to ban appending (and deleting) nodes
to a sequence.
> I'm adding attributes to the tree _everywhere_. As PyChecker2 grinds
> over the tree, it adds discovered information to the tree (things like
> type information and cross-references). I can probably live without
> it by tracking the information somewhere else.
Yeah, but you want to live without the ability to add new class instance
attributes?
In the old SimpleNode AST tree jython had over time added several
attributes to the nodes (such as scope info). So it obviously is usefull.
If we decide that the AST node must support attributes then I can also
add it to jython. I can simple do a kind of deepcopy of the tree that
returns real class instances in place of the java instances. It will
just make the AST tree building from python a little slower.
I don't think such dynamic attributes will survice across several
compiler passes. If f.ex a peephole optimizer is implemented in C the
tree must be pickled, optimized and unpickled. All dynamic attributes
will be lost.
Thank you for your comments.
regards,
finn
From ecn@metaslash.com Fri May 24 02:19:12 2002
From: ecn@metaslash.com (Eric C. Newton)
Date: Thu, 23 May 2002 21:19:12 -0400
Subject: [Compiler-sig] More jython progress.
In-Reply-To: <3CED233C.7040405@worldonline.dk>; from bckfnn@worldonline.dk on Thu, May 23, 2002 at 07:13:32PM +0200
References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn> <3CED233C.7040405@worldonline.dk>
Message-ID: <20020523211912.A27767@ecn>
> Yeah, but you want to live without the ability to add new class instance
> attributes?
Perhaps I'm just getting hung up on definitions.
I was translating "class instance attributes" as "object attributes"
since an "instance of a class" is called an "object". Now that I
think about it, it could mean "attributes which are stored in
classes".
If I can add attributes to the objects which make up the tree, then
great! I don't put any attributes in a class.
> If we decide that the AST node must support attributes then I can also
> add it to jython. I can simple do a kind of deepcopy of the tree that
> returns real class instances in place of the java instances. It will
> just make the AST tree building from python a little slower.
Slower is not better. I prefer to work in python, even if I have a
few more rules. If it gets slower, I'll be forced to go to C.
I tried to work with the C version of the AST for a few hours. It
wasn't pleasant.
-Eric
From bckfnn@worldonline.dk Fri May 24 12:22:14 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Fri, 24 May 2002 13:22:14 +0200
Subject: [Compiler-sig] More jython progress.
References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn> <3CED233C.7040405@worldonline.dk> <20020523211912.A27767@ecn>
Message-ID: <3CEE2266.9010405@worldonline.dk>
[Eric C. Newton]
>>Yeah, but you want to live without the ability to add new class instance
>>attributes?
>
> Perhaps I'm just getting hung up on definitions.
>
> I was translating "class instance attributes" as "object attributes"
> since an "instance of a class" is called an "object". Now that I
> think about it, it could mean "attributes which are stored in
> classes".
I meant it as "object attributes". We can't add new attributes to any of
the nodes in the AST tree.
I think that the correct python term isn't "object" (everything is an
object, even classes) but "class instance". I got that from:
http://www.python.org/doc/current/ref/types.html
but maybe I'm wrong.
>>If we decide that the AST node must support attributes then I can also
>>add it to jython. I can simple do a kind of deepcopy of the tree that
>>returns real class instances in place of the java instances. It will
>>just make the AST tree building from python a little slower.
>
> Slower is not better. I prefer to work in python, even if I have a
> few more rules. If it gets slower, I'll be forced to go to C.
The "little slower" I was talking about would only be for jython and I
don't think that the "little slower" speed would be much different from
the speed of the pickling phase suggested by Jeremy.
regards,
finn
From bckfnn@worldonline.dk Thu May 30 18:09:48 2002
From: bckfnn@worldonline.dk (Finn Bock)
Date: Thu, 30 May 2002 19:09:48 +0200
Subject: [Compiler-sig] Jython progress.
Message-ID: <3CF65CDC.9040104@worldonline.dk>
Hi,
Just to let you know, the AST patch for jython has now been integrated
into jython CVS. We can still make changes to the AST, at least until we
release jython-2.2, but I would like you to remember that for jython,
the AST is now out of the sandbox.
Obviously I hope that the new AST will be included in CPython-2.3. If it
isn't, we only get a huge improvement in the internal jython codes. If
the AST gets ready for 2.3 we will also get great compatibility with
CPython.
regards,
finn