AST manipulation and source code generation
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs? I was experimenting with possible translations for exception chaining and wanted to run some automated tests, so i started playing around with the compiler package to do source-to-source transformations. Then i started working on a way to do template-based substitution of ASTs and a way to spit source code back out, and i'm wondering if that might be good for experimenting with future Python features. (If there's already stuff out there for doing this, let me know -- i don't intend to duplicate existing work.) -- ?!ng
On 5/24/05, Ka-Ping Yee <python-dev@zesty.ca> wrote:
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs?
Sure. Eventually, we'll have to figure out how to unify the compiler package AST and the ast-branch AST, but don't let that delay you now.
I was experimenting with possible translations for exception chaining and wanted to run some automated tests, so i started playing around with the compiler package to do source-to-source transformations. Then i started working on a way to do template-based substitution of ASTs and a way to spit source code back out, and i'm wondering if that might be good for experimenting with future Python features.
(If there's already stuff out there for doing this, let me know -- i don't intend to duplicate existing work.)
I don't know of any existing work, but it certainly sounds useful. Jeremy
I wrote something like this (called pyunparse) a little while ago. It's not the cleanest code in the world, but it worked for my original use case (debugging Logix, which uses python ASTs as an IR): http://www.pycs.net/users/0000445/stories/7.html Cheers, /arg On May 24, 2005, at 9:56 AM, Jeremy Hylton wrote:
On 5/24/05, Ka-Ping Yee <python-dev@zesty.ca> wrote:
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs?
Sure. Eventually, we'll have to figure out how to unify the compiler package AST and the ast-branch AST, but don't let that delay you now.
I was experimenting with possible translations for exception chaining and wanted to run some automated tests, so i started playing around with the compiler package to do source-to-source transformations. Then i started working on a way to do template-based substitution of ASTs and a way to spit source code back out, and i'm wondering if that might be good for experimenting with future Python features.
(If there's already stuff out there for doing this, let me know -- i don't intend to duplicate existing work.)
I don't know of any existing work, but it certainly sounds useful.
Jeremy _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/andy% 40andygross.org
Ka-Ping, FWIW, I've also got an implementation, which is based on the parser module rather than the compiler module. Much simpler, imo, but whitespace isn't preserved (could be perhaps?). Anyway, take it or leave it. Links follow. chad ----- Subversion repository: http://svn.zetadev.com/repos/public/ASTutils/ The relevant method is 'ast2text' in ASTutils.py: http://svn.zetadev.com/repos/public/ASTutils/tags/0.2.0/ASTutils.py API documentation for this method: http://www.zetadev.com/software/ASTutils/latest/api/public/ASTutils.ASTutils... API documentation root: http://www.zetadev.com/software/ASTutils/latest/api/
On 5/24/05, Ka-Ping Yee <python-dev@zesty.ca> wrote:
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs?
the astng package from logilab's common library [1] extends compiler AST nodes with a bunch of methods, including a as_string method on each node allowing to regenerate a python source code from an ast (other methods are mainly to ease navigation in the tree or to extract higher level information from it). Currently it's implemented as a node's method instead of using a visitor pattern or the like, but that would be easily done. [1] http://www.logilab.org/projects/common/ -- Sylvain Thénault LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs?
Heh, so I guess the answer is "yes." BTW, how does the concept of AST transformations relate to the concept of (Lisp) macros? Am I right to think that they are similar? chad
On 5/26/05, Chad Whitacre <chad@zetaweb.com> wrote:
Would there be any interest in extending the compiler package with tools for AST transformations and for emitting Python source code from ASTs?
Heh, so I guess the answer is "yes."
BTW, how does the concept of AST transformations relate to the concept of (Lisp) macros? Am I right to think that they are similar?
I think they are similar, but two key differences are: - An AST transformation can transform existing syntax but doesn't allow you to create new syntax. - An AST transformation has to be explicitly invoked. A macro is part of the language proper and has a semantics for how and when macros are evaluated. Jeremy
Thanks Jeremy. Also wandered off-list w/ Ka-Ping; posting here for posterity. chad ----- chad: BTW, how does the concept of AST transformations relate to the concept of (Lisp) macros? Am I right to think that they are similar? ?!ng: Absolutely. In terms of mechanism, they're basically the same; the main difference is that in Lisp, the transformations are a part of the core language definition. ?!ng: Well, i should refine that a bit to say that the Lisp macro system is a little more specific. Whereas AST transformations in Python are open-ended (you could generate any result you want), the key interesting property of Lisp macros is that they are constrained to be "safe", in the sense that the bindings of variable names are always preserved. chad: Hmmm ... I don't follow python-dev closely but hasn't there been resistance to macros in Python? Are we saying macros may be a good idea after all? ?!ng: resistance -> Yes. ?!ng: good idea -> Not really. AST transformations are useful for experimenting with the language, but i don't think there is much enthusiasm for making these transformations a normal part of the way most programs are written.
?!ng: Well, i should refine that a bit to say that the Lisp macro system is a little more specific. Whereas AST transformations in Python are open-ended (you could generate any result you want), the key interesting property of Lisp macros is that they are constrained to be "safe", in the sense that the bindings of variable names are always preserved.
I'm not quite sure what ?!ng means by "the bindings of variable names are always preserved", but I conjecture that he is thinking of the "hygienic macros" in Scheme rather than the macros in Common Lisp which permit arbitrary code transformations. -- g
participants (6)
-
Andy Gross -
Chad Whitacre -
Gareth McCaughan -
Jeremy Hylton -
Ka-Ping Yee -
Sylvain Thénault