Module Suggestion: ast
Hi all, I would like to propose a new module for the stdlib for Python 2.6 and higher: "ast". The motivation for this module is the pending deprecation for compiler.ast which is widely used (debugging, template engines, code coverage etc.). _ast is a very solid module and is without a doubt easier to maintain then compiler.ast which was written in Python, it's lacking some features such as pretty printing the AST or traversing it. The idea of "ast" would be adding high level functionality for easier working with the AST. It currently provides these features: - pretty printing AST objects - a parse function as easier alias for compile() + flag - operator-node -> operator symbol mappings (eg: _ast.Add -> '+') - methods to modify lineno / col_offset (incrementing or copying the data over from existing nodes) - getting the fields of nodes as dicts - iterating over all child nodes - a function to get the docstring or an AST node - a node walker that yields all child-nodes recursively - a `NodeVistor` and `NodeTransformer` Additionally there is a `literate_eval` function in that module that can safely evaluate python literals in a string. Module and unittests are located in the Pocoo Sandbox HG repository: http://dev.pocoo.org/hg/sandbox/file/tip/ast/ast.py http://dev.pocoo.org/hg/sandbox/file/tip/ast/test_ast.py A slightly modified version of the ast.py module for Python 2.5 compatibility is currently in use for the Mako template engine to achieve support for Google's AppEngine. An example module for the NodeVisitor is in the repository which converts a Python AST back into Python source code: http://dev.pocoo.org/hg/sandbox/file/tip/ast/codegen.py Regards, Armin
Just a thought, but it would be great if this could be implemented over the top of a C layer that operates on real AST nodes (rather than the PyObject representation of those nodes). I'm working on stuff to perform code optimization at the AST level (see the tlee-ast-optimize branch), and the functionality you're describing may wind up being very useful to me. I've got more to say on the topic, but I'm at work right now. Just something to keep in mind. Cheers, T Armin Ronacher wrote:
Armin Ronacher schrieb:
If there are no further objections, I'll add this to PEP 361 so that the proposal doesn't get lost. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
2008/5/1 Georg Brandl <g.brandl@gmx.net>:
Excuse my confusion over process, but if this is to go into 2.6, does that mean it needs to be ready before the first beta? Or is there a more relaxed schedule for the stdlib (and if so, what is the deadline for the stdlib)? The same question probably applies to the stdlib reorg... Paul.
Paul Moore schrieb:
There is no feature freeze before the first beta. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
I'm in the process of writing C code for the purposes of traversing AST nodes in the AST optimization branch. This seems to be an ideal case for code generation based on the ASDL representation of the AST as we're currently doing for Python-ast.[ch]. I'm already considering this approach for some code I need to visit the C AST representation. We could likely write a similar generator for the PyObject representation of the AST, which might be useful for the proposed "ast" module. This would ensure that both implementations of the AST visitation code are always kept in step with the ASDL. The only real problem I can foresee with this is that the code in asdl_c.py is already getting pretty hairy. Adding to the mess is going to make it worse still. Maybe this will serve as a good opportunity to clean it up a little? Any objections? Cheers, T Paul Moore wrote:
Just a thought, but it would be great if this could be implemented over the top of a C layer that operates on real AST nodes (rather than the PyObject representation of those nodes). I'm working on stuff to perform code optimization at the AST level (see the tlee-ast-optimize branch), and the functionality you're describing may wind up being very useful to me. I've got more to say on the topic, but I'm at work right now. Just something to keep in mind. Cheers, T Armin Ronacher wrote:
Armin Ronacher schrieb:
If there are no further objections, I'll add this to PEP 361 so that the proposal doesn't get lost. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
2008/5/1 Georg Brandl <g.brandl@gmx.net>:
Excuse my confusion over process, but if this is to go into 2.6, does that mean it needs to be ready before the first beta? Or is there a more relaxed schedule for the stdlib (and if so, what is the deadline for the stdlib)? The same question probably applies to the stdlib reorg... Paul.
Paul Moore schrieb:
There is no feature freeze before the first beta. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
I'm in the process of writing C code for the purposes of traversing AST nodes in the AST optimization branch. This seems to be an ideal case for code generation based on the ASDL representation of the AST as we're currently doing for Python-ast.[ch]. I'm already considering this approach for some code I need to visit the C AST representation. We could likely write a similar generator for the PyObject representation of the AST, which might be useful for the proposed "ast" module. This would ensure that both implementations of the AST visitation code are always kept in step with the ASDL. The only real problem I can foresee with this is that the code in asdl_c.py is already getting pretty hairy. Adding to the mess is going to make it worse still. Maybe this will serve as a good opportunity to clean it up a little? Any objections? Cheers, T Paul Moore wrote:
participants (5)
-
"Martin v. Löwis" -
Armin Ronacher -
Georg Brandl -
Paul Moore -
Thomas Lee