So it looks like the AST branch has landed. Wooo! Well done to all who
were involved - it seems like it's been a huge amount of work.
Could someone involved give a short email laying out what concrete (no
pun intended) advantages this new compiler gives us? Does it just
allow us to do new and interesting manipulations of the code during
compilation? Cleaner, easier to maintain, or the like?
Anthony
--
Anthony Baxter
On 10/20/05, Anthony Baxter
Could someone involved give a short email laying out what concrete (no pun intended) advantages this new compiler gives us? Does it just allow us to do new and interesting manipulations of the code during compilation? Cleaner, easier to maintain, or the like?
The Grammar is (was at one point at least) shared between Jython and would allow more tools to be able to share infrastructure. The idea is to eventually be able to have [JP]ython output the same AST to tools. There is quite a bit of generated code based on the Grammar. So some stuff should be easier. Other stuff is just moved. You still need to convert from the AST to the byte code. Hopefully it will be easier to do various sorts of optimization and general manipulation of an AST rather than what existed before. Only time will tell if we can acheive many of the benefits, so it would be good if people could review the code and see if things look more complex/complicated and suggest improvements. I'm not all that familiar with the structure, I'm more of a hopeful consumer of it. HTH, n
On 10/20/05, Neal Norwitz
On 10/20/05, Anthony Baxter
wrote: Could someone involved give a short email laying out what concrete (no pun intended) advantages this new compiler gives us? Does it just allow us to do new and interesting manipulations of the code during compilation? Cleaner, easier to maintain, or the like?
I just wanted to clarify that Neal meant the abstract syntax, not the grammar. It should allow people to write tools to analyze Python source code without having to worry about the often irrelevant details of the exact tokens or the way they are parsed. We should be able to get to a state where tools using the AST work with Python and Jython (and maybe IronPython, who knows). The tokenize and parser modules still exist for tools for which those details aren't irrelevant. We should also think about how to migrate the compiler module from its current AST to the new AST, although the backwards compatibility issues there are a bit tricky.
The Grammar is (was at one point at least) shared between Jython and would allow more tools to be able to share infrastructure. The idea is to eventually be able to have [JP]ython output the same AST to tools. There is quite a bit of generated code based on the Grammar. So some stuff should be easier. Other stuff is just moved. You still need to convert from the AST to the byte code.
Hopefully it will be easier to do various sorts of optimization and general manipulation of an AST rather than what existed before.
I think it should be a lot easier to write tools for the C Python compiler that do extra analysis or optimization. The existing peephole optimizer could be improved by integrating it with the bytecode assembler (for example, eliminating all NOP bytecodes). Jeremy
On 10/20/05, Neal Norwitz
The Grammar is (was at one point at least) shared between Jython and would allow more tools to be able to share infrastructure. The idea is to eventually be able to have [JP]ython output the same AST to tools.
Hello Python-dev, My name is Frank Wierzbicki and I'm working on the Jython project. Does anyone on this list know more about the history of this Grammar sharing between the two projects? I've heard about some Grammar sharing between Jython and Python, and I've noticed that (most of) the jython code in /org/python/parser/ast is commented "Autogenerated AST node". I would definitely like to look at (eventually) coordinating with this effort. I've cross-posted to the Jython-dev list in case someone there has some insight. Thanks, Frank
On 10/25/05, Frank Wierzbicki
My name is Frank Wierzbicki and I'm working on the Jython project. Does anyone on this list know more about the history of this Grammar sharing between the two projects? I've heard about some Grammar sharing between Jython and Python, and I've noticed that (most of) the jython code in /org/python/parser/ast is commented "Autogenerated AST node". I would definitely like to look at (eventually) coordinating with this effort.
I've cross-posted to the Jython-dev list in case someone there has some insight.
Your best bet is to track down Jim Hugunin and see if he remembers. He's jimhug at microsoft.com or jim at hugunin.net. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On 10/25/05, Frank Wierzbicki
wrote: My name is Frank Wierzbicki and I'm working on the Jython project. Does anyone on this list know more about the history of this Grammar sharing between the two projects? I've heard about some Grammar sharing between Jython and Python, and I've noticed that (most of) the jython code in /org/python/parser/ast is commented "Autogenerated AST node". I would definitely like to look at (eventually) coordinating with this effort.
I've cross-posted to the Jython-dev list in case someone there has some insight.
Your best bet is to track down Jim Hugunin and see if he remembers. He's jimhug at microsoft.com or jim at hugunin.net.
no. this is all after Jim, its indeed a derived effort from the CPython own AST effort, just that we started using it quite a while ago. This is all after Jim was not involved with Jython anymore, Finn Bock started this.
On 10/25/05, Samuele Pedroni
Your best bet is to track down Jim Hugunin and see if he remembers. He's jimhug at microsoft.com or jim at hugunin.net.
no. this is all after Jim, its indeed a derived effort from the CPython own AST effort, just that we started using it quite a while ago. This is all after Jim was not involved with Jython anymore, Finn Bock started this.
Oops! Sorry for the misinformation. Shows how much I know. :( -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Frank Wierzbicki wrote:
On 10/20/05, *Neal Norwitz*
mailto:nnorwitz@gmail.com> wrote: The Grammar is (was at one point at least) shared between Jython and would allow more tools to be able to share infrastructure. The idea is to eventually be able to have [JP]ython output the same AST to tools.
Hello Python-dev,
My name is Frank Wierzbicki and I'm working on the Jython project. Does anyone on this list know more about the history of this Grammar sharing between the two projects? I've heard about some Grammar sharing between Jython and Python, and I've noticed that (most of) the jython code in /org/python/parser/ast is commented "Autogenerated AST node". I would definitely like to look at (eventually) coordinating with this effort.
I've cross-posted to the Jython-dev list in case someone there has some insight.
as far as I understand now Python trunk contains some generated AST representation C code created through the asdl_c.py script from an updated Python.asdl, these files live in http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Parser/ a parallel asdl_java.py existed in Python CVS sandbox (where the AST effort started) and was updated the last time the Jython own AST classes were generated with at the time version of Python.asdl (this was done by me if I remember correctly at some point in Jython 2.2 evolution, I think when the PyDev guys wanted a more up-to-date Jython parser to reuse): http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/python/nondist/sandbox/ast/asdl_java.py?content-type=text%2Fplain&rev=1.7 basically the new Python.asdl needs to be used, the asdl_java.py maybe updated and our compiler changed as necessary. regards.
On 10/20/05, Anthony Baxter
So it looks like the AST branch has landed. Wooo! Well done to all who were involved - it seems like it's been a huge amount of work.
Hear, hear. Great news! Thanks to Jeremy, Neil and all the others. I can't wait to check it out! -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 10/20/05, Guido van Rossum
On 10/20/05, Anthony Baxter
wrote: So it looks like the AST branch has landed. Wooo! Well done to all who were involved - it seems like it's been a huge amount of work.
Hear, hear. Great news! Thanks to Jeremy, Neil and all the others. I can't wait to check it out!
I want to thank all the people who made it possible by writing code and debugging. I hope this is a complete list: Armin Rigo Brett Cannon Grant Edwards John Ehresman Kurt Kaiser Neal Norwitz Neil Schemenauer Nick Coghlan Tim Peters And thanks to the PSF and PyCon organizers for hosting the formerly annual ast-branch sprints! Jeremy
Anthony Baxter wrote:
So it looks like the AST branch has landed. Wooo! Well done to all who were involved - it seems like it's been a huge amount of work.
Congratulations from this quarter, too. I really liked the structure of the new compiler in the limited time I spent working with it on the AST branch, and am glad it has made its way onto the HEAD for Python 2.5. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Anthony Baxter
Could someone involved give a short email laying out what concrete (no pun intended) advantages this new compiler gives us?
One advantage is that it decreases the coupling between the parser and the backend of the compiler. For example, it should be possible to replace the parser without modifying a lot of the compiler. Also, the concrete syntax tree (CST) generated by Python's parser is not a convenient data structure to deal with. Anyone who's used the 'parser' module probably experienced the pain: >>> parser.ast2list(parser.suite('a = 1')) [257, [266, [267, [268, [269, [320, [298, [299, [300, [301, [303, [304, [305, [306, [307, [308, [309, [310, [311, [1, 'a']]]]]]]]]]]]]]], [22, '='], [320, [298, [299, [300, [301, [303, [304, [305, [306, [307, [308, [309, [310, [311, [2, '1']]]]]]]]]]]]]]]]], [4, '']]], [0, '']]
Does it just allow us to do new and interesting manipulations of the code during compilation?
Well, that's a pretty big deal, IMHO. For example, adding pychecker-like functionality should be straight forward now. I also hope some of the namespace optimizations get explored (e.g. PEP 267).
Cleaner, easier to maintain, or the like?
At this point, the old and new compiler are pretty similar in terms of complexity. However, the new compiler is a much better base to build upon. Neil
On 10/21/05, Neil Schemenauer
Also, the concrete syntax tree (CST) generated by Python's parser is not a convenient data structure to deal with. Anyone who's used the 'parser' module probably experienced the pain:
>>> parser.ast2list(parser.suite('a = 1')) [257, [266, [267, [268, [269, [320, [298, [299, [300, [301, [303, [304, [305, [306, [307, [308, [309, [310, [311, [1, 'a']]]]]]]]]]]]]]], [22, '='], [320, [298, [299, [300, [301, [303, [304, [305, [306, [307, [308, [309, [310, [311, [2, '1']]]]]]]]]]]]]]]]], [4, '']]], [0, '']]
That's the fault of the 'parser' extension module though, and this affects tools using the parser module, not the bytecode compiler itself. The CST exposed to C programmers is slightly higher level. (But the new AST is higher level still, of course.) BTW, Elemental is letting me open-source a reimplementation of pgen in Python. This also includes a nifty way to generate ASTs. This should become available within a few weeks. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 21 Oct 2005 18:32:22 +0000 (UTC) nas at arctrix.com (Neil Schemenauer) wrote:
Does it just allow us to do new and interesting manipulations of the code during compilation?
Well, that's a pretty big deal, IMHO. For example, adding pychecker-like functionality should be straight forward now. I also hope some of the namespace optimizations get explored (e.g. PEP 267).
Is there a python interface ? Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com
On Tue, Oct 25, 2005 at 01:36:26PM +1000, Simon Burton wrote:
Is there a python interface ?
Not yet, as far as I know. FYI, all: please see the following weblog entry for a description of the AST branch: http://www.amk.ca/diary/2005/10/the_ast_branch_lands_1 If I got anything wrong, please offer corrections in the comments for that post. --amk
participants (10)
-
A.M. Kuchling
-
Anthony Baxter
-
Frank Wierzbicki
-
Guido van Rossum
-
Jeremy Hylton
-
Neal Norwitz
-
Neil Schemenauer
-
Nick Coghlan
-
Samuele Pedroni
-
Simon Burton