[IronPython] Dear Lazy Web - IronPython 2 Parser

[Michael Foord]

Great! Thanks Dino.

With the following C# code:

using System;
using System.Collections;
using System.Collections.Generic;
using IronPython.Compiler.Ast;
using Microsoft.Scripting;

namespace ExpressionWalker
{
    public class ExpressionWalker : PythonWalker
    {
        public List<string> names = new List<string>();

        public override bool Walk(NameExpression node)
        {
            names.Add(SymbolTable.IdToString(node.Name));
            return true;    // means walk this node, not important for a
Name node.
        }
    }
}


The following Python code:

import clr
clr.AddReference('ExpressionWalker')
clr.AddReference('IronPython')
clr.AddReference('Microsoft.Scripting')

from IronPython import PythonEngineOptions
from IronPython.Hosting import PythonEngine
from IronPython.Compiler import Parser
from Microsoft.Scripting import CompilerContext, SourceCodeUnit

from ExpressionWalker import ExpressionWalker

expression = "A1 * 3 + B4"
pe = PythonEngine.CurrentEngine

s = SourceCodeUnit(pe, expression)
c = CompilerContext(s)
p = Parser.CreateParser(c, PythonEngineOptions())

e = p.ParseExpression()

w = ExpressionWalker()
e.Walk(w)

print list(w.names)

Produces:
['A1', 'B4']


This works in Silverlight (which is my intention). :-)

I'll add this to the cookbook, but I realise that parts of this
interface may be unstable...

Michael

Dino Viehland wrote:
> PythonWalker and MS.Ast.Walker are actually different - PythonWalker walks the IronPython AST before its transformed to a DLR AST, and MS.Ast.Walker walks the DLR AST.
>
> You can certainly party on the Expression object all you want.  But it'd be much easier (and less prone to break as we make changes to the AST) to create a subclass of PythonWalker.  This is really easy in C# because it's just:
>
> class MyWalker : PythonWalker {
>         public override bool Walk(NameExpression node) {
>                 Console.WriteLine(SymbolTable.IdToString(node.Name));
>                 return true;    // means walk this node, not important for a Name node.
>         }
> }
>
> Then:
> e = p.ParseExpression()
> e.Walk(MyWalker())
>
> >From Python it's a little more tricky because all the Walk nodes are overloaded - and by a little more tricky I mean you just need to test the type of node coming in :).
> (I haven't actually compiled any of this, but that's the theory).
>
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
> Sent: Monday, August 13, 2007 5:06 PM
> To: Discussion of IronPython
> Subject: Re: [IronPython] Dear Lazy Web - IronPython 2 Parser
>
> Thanks for the reply Dino.
>
> Leaving some kind of API exposed to the Python parser would be
> appreciated, rather than making it private.
>
> What I would like to do in this case, is discover what variable names a
> Python expression uses.
>
> I know how to do this for a Python AST (name nodes that are the first
> child of an atom node). I don't even need to modify the AST - I just
> want to pull the names out.
>
>  From what you said, I *think* that using Parser.CreateParser sounds the
> closest.
>
> With the help you provided, I got as far as the following code:
>
> import clr
> clr.AddReference('IronPython')
> clr.AddReference('Microsoft.Scripting')
>
> from IronPython import PythonEngineOptions
> from IronPython.Hosting import PythonEngine
> from IronPython.Compiler import Parser
> from Microsoft.Scripting import CompilerContext, SourceCodeUnit
>
> expression = "3 * 3"
> pe = PythonEngine.CurrentEngine
>
> s = SourceCodeUnit(pe, expression)
> c = CompilerContext(s)
> p = Parser.CreateParser(c, PythonEngineOptions())
>
> e = p.ParseExpression()
>
> This gets me a BinaryExpression object, which has interesting properties
> like 'Left', 'Right', 'Start' and 'End' and a Walk method that takes a
> PythonWalker. (Presumably BinaryExpression is a node type - the root
> node of this particular expression.)
>
> I can find Microsoft.Scripting.Ast.Walker (which has multiple overloads
> of Walk for different node types). I assume the PythonWalker is a
> subclass of this.
>
> Can I use the expression object to inspect its nodes without creating a
> walker?
>
> Thanks for the help. :-)
>
> Michael
>
> Dino Viehland wrote:
>   
>> I don't know if you figured this out yet but here goes...
>>
>> Our Parser class is still public (I'm not sure if this will change or not) but you can do (IronPython.Compiler.)Parser.CreateParser.  That takes a CompilerContext class which is going to point the parser at a SourceUnit that we are currently parsing.
>>
>> >From there you can call ParseFileInput and it'll parse the file and return you the IronPython AST.  The interesting part might be if you actually want to compile that code later on :).  In that case you might want to look at PythonScriptCompiler to see the rest of the process - it's really just one call to PythonScriptCompiler.BindAndTransform which also seems to be public (again, I'm not sure if that'll change or not).
>>
>> Alternately if you just want a way to paramterize some user code you might want to look at ScriptCompiler.ParseCodeDom.  This allows you to provide a limited CodeDom tree (we support CodeMemberMethod, and snippet statement / expressions and that's it) which you could create from some user input.  The nice thing about that is it won't be Python specific (although I'm guessing that's not a problem for you :) ).
>>
>>
>> -----Original Message-----
>> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
>> Sent: Wednesday, August 08, 2007 3:50 PM
>> To: Discussion of IronPython
>> Subject: [IronPython] Dear Lazy Web - IronPython 2 Parser
>>
>> Hello all,
>>
>> Sorry for being lazy, but...
>>
>> What is the easiest way of  building an AST from IronPython 2 code? I
>> would like an AST that I can modify and then flatten again in
>> Silverlight.... It needn't be a Python AST, an IronPython one is fine. I
>> only need access to 'name' nodes... I wish to change some of the names
>> and turn the AST back into code before exec'ing.
>>
>> :-)
>>
>> Thanks
>>
>>
>> Michael
>> http://www.ironpython.info/
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>>
>>     
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
>