[Ironpython-users] Manipulating/Reversing the Ironpython AST

[Dino Viehland]

PTVS's parser (which is just a branch of the IronPython parser) supports this so that we could implement the refactoring features.  The nodes have a ToCodeString method on them which you can call to get back the original code.  You'll need to parse in verbatim mode like so:

            var parser = Parser.CreateParser(
                someTextReader,
                Project.LanguageVersion,
                new ParserOptions() { Verbatim = true, BindReferences = true }
            );
            var ast = parser.ParseFile();

And you'll need to pass the PythonAst that you get back into the ToCodeString as all of the verbatim round-trip info is stored on the top-level Ast node.

This will preserve comments, whitespace, etc...  You can modify the AST in any way you want and still call ToCodeString - the nodes which you insert will all get some default amount of formatting.  You'll want to pass in the original PythonAst that you used to parse the nodes with.  There's some random helper methods like CopyLeadingWhiteSpace and CopyTrailingNewLine on the Node classes for manipulating some of the round trip info - it's not exhaustive, just what we've needed for PTVS's refactoring support.  You can always download the source and add more or contribute back changes if you need some specific additions.

The parser is in Microsoft.PythonTools.Analysis.dll which is a stand-alone DLL (no dependencies on anything outside of the .NET framework).  If you want to get a copy of the DLL you can just download and install PTVS (http://pytools.codeplex.com/) and then the DLL will either be in:

VSINSTALLDIR\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\1.0
                or
%LOCALAPPDATA%\Microsoft\VisualStudio\10.0\Extensions\Microsoft\Python Tools for Visual Studio\1.0

Depending on if you did the default (all-users) install or the per-user install.

The only thing that's possibly missing is that we don't have a good AST re-writer, so you'll need to do that by hand or implement something like the DLR ExpressionVisitor class.

From: ironpython-users-bounces+dinov=exchange.microsoft.com at python.org [mailto:ironpython-users-bounces+dinov=exchange.microsoft.com at python.org] On Behalf Of Jay Riley
Sent: Friday, October 28, 2011 8:39 AM
To: ironpython-users at python.org
Subject: [Ironpython-users] Manipulating/Reversing the Ironpython AST

Hi all,

I was wondering if IronPython has the ability to reverse/unparse the AST back into a ource file? Or if someone has perhaps built something to do so?

Here's what I'm trying to do:

I have a game I'm working on, and I'm currently drafting up tools for it. The tools are being written in C# and are meant to make changing and editing game files easier. Several of the game files are written in python and are used to extend objects from the main game source. For instace, I have an Items.py file that contains the following (minimalized for example):

from ItemModule import *

import copy

class ScriptedItem(Item):
    def __init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower):
        Item.__init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower, Item.GetNextItemID())
    def Clone(self):
        return copy.deepcopy(self)

ItemLibrary.AddItem(ScriptedItem("Abounding Crystal", "A colourful crystal composed of many smaller crystals. It gives off a warm glow.", ItemType.SynthesisMaterial, ItemType.SynthesisMaterial, 0, ItemUsage.Unusable, 0, 50))

In this case, I'd like to provide a convenient front-end to allow editors to modify/add/delete items from the list. To do this, my editor need to be able to:


  1.  Find and list all the class types (in this example, it'd be only Scripted Item)
  2.  Find and list all created items (in this case there'd only be one, Abounding Crystal). I'd need to find the type (in this caseScriptedItem) and all the parameter values
  3.  Allow editing of parameters and the creation/removal of items.

I tried writing my own parser, but that became increasingly difficult as the complexity of the Item constructors went up. When I found IronPython and its ability to generate a walkable AST, I thought I'd found my solution, and indeed the AST makes it easy to accomplish 1 and 2 of my requirements. However, I'm currently stuck on how to write back changes made in my editor to the source file. My initial idea was to preserve the AST and modify values on the existing nodes for edited items and inject new nodes when new items were created. However even if I could get this to work correctly, I have no idea how to reconvert the AST back into a source file. When I asked on stackoverflow, I was told this is usually done using "prettyprinting" and had some suggestions to use python's "inspect" property.

I'm not sure how to use inspect to do what I want, and have some concerns over the amount of effort require to get "prettyprinting" correct, so i wanted to ask if anyone here has written a prettyprinter for ironpython or perhaps knows some other way to accomplish my three goals? Any help would be appreciated,

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20111028/804796f9/attachment.html>