[IronPython] Is it possible to get the DLR's AST when it fails to parse?

[Lee Culver]

Thanks Dino.  This (very) rough draft worked perfectly:

class Program
{
    static void Main(string[] args)
    {
        string cmd = "if one.two.three.four.";

        ScriptEngine engine = Python.CreateEngine();
        ScriptScope scope = engine.CreateScope();
        ScriptSource source = engine.CreateScriptSourceFromString(cmd, SourceCodeKind.InteractiveCode);

        SourceUnit unit = HostingHelpers.GetSourceUnit(source);
        Parser p = Parser.CreateParser(
            new CompilerContext(unit, new PythonCompilerOptions(), ErrorSink.Null),
            new PythonOptions()
            );

        PythonAst ast = p.ParseFile(false);
        Walker w = new Walker(cmd.Length);
        ast.Walk(w);

        Stack<string> members = w.GetResult();
        while (members.Count != 0)
            Console.WriteLine(members.Pop());
    }
}

public class Walker : PythonWalker
{
    int mIndex;
    Stack<string> mResult = null;
    public Walker(int index)
    {
        mIndex = index;
    }

    public Stack<string> GetResult()
    {
        return mResult;
    }

    public override bool Walk(MemberExpression node)
    {
        if (mIndex == node.Span.End.Index)
        {
            mResult = new Stack<string>();
            MemberExpression curr = node;

            while (curr.Target is MemberExpression)
            {
                curr = (MemberExpression)curr.Target;
                mResult.Push(curr.Name.ToString());
            }

            if (curr.Target is NameExpression)
                mResult.Push(((NameExpression)curr.Target).Name.ToString());

            return false;
        }

        return true;
    }
}

-Lee




-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland
Sent: Tuesday, December 01, 2009 4:15 PM
To: Discussion of IronPython
Subject: Re: [IronPython] Is it possible to get the DLR's AST when it fails to parse?



Tokenization would save some work but he could also just use the IronPython

parser.  It's unfortunately a small pain to create (it's overly tied to the

DLR APIs right now) but it's:



return Parser.CreateParser(

                new CompilerContext(sourceUnit, new PythonCompilerOptions(), ErrorSink.Null),

                new PythonOptions()

                );



And to get that SourceUnit you can do HostingHelpers.GetSourceUnit on the ScriptSource.



The resulting parsed expression will include ErrorExpression's where

we encounter invalid code.  You may also find that things like class and

function definitions can be missing their names if you are parsing

incomplete code like "def ".  Then you'll be working on the IronPython AST

instead of the DLR AST (but the two will be the same likely in

IronPython 2.6.1 - the ASTs are now reducible DLR ASTs).





> -----Original Message-----

> From: users-bounces at lists.ironpython.com [mailto:users-

> bounces at lists.ironpython.com] On Behalf Of Michael Foord

> Sent: Tuesday, December 01, 2009 4:07 PM

> To: Discussion of IronPython

> Subject: Re: [IronPython] Is it possible to get the DLR's AST when it fails to

> parse?

>

> No time for a longer reply, but you probably want to tokenize rather

> than parse.

>

> Michael

>

> Lee Culver wrote:

> >

> > I have been playing with an implementation of a simple GUI console for

> > IronPython (similar to IDLE, without it's text-editor IDE). I've

> > mostly brought it up to rough feature parity with IDLE, but I'd like

> > to add a rudimentary itellisense such that when you type something on

> > the order of "foo.", the interpreter can pop up a menu containing the

> > contents of foo (assuming it's something already defined in the local

> > scope). However, I'd really rather not get in the business of parsing

> > Python code. (That's what the DLR is for!)

> >

> > Is there any way I can use DLR to give me the AST up to the point at

> > which it failed to parse so that I can walk that and see if we are in

> > a valid place to look up the fields of an object? I looked at using

> > the ErrorListener class with ScriptSource.Compile, but it only gives

> > me "unexpected <eof> found" at the point where the "." is...which I

> > already knew.

> >

> > Basically, whenever the user presses the "." key in the console what I

> > would like to do is roughly this code:

> >

> > ScriptEngine engine = Python.CreateEngine();

> >

> > ScriptScope scope = engine.CreateScope();

> >

> > ScriptSource source = engine.CreateScriptSourceFromString("if foo.",

> > Microsoft.Scripting.SourceCodeKind.InteractiveCode);

> >

> > source.Compile(); // <-- This *should* fail. I want the AST when it does.

> >

> > Primarily, I'm trying to find chains of identifiers to walk so that I

> > can display the Nth identifier's fields:

> >

> > identifier1 . identifier2 . identifier3 . ...

> >

> > I need to distinguish this from things like this, because there's

> > nothing you can reasonably do to have intellisense in this case:

> >

> > Function(args) . identifier1 . identifier2

> >

> > (Yes, I realize that walking this chain can execute code and have side

> > effects, I'm willing to live with that for my purposes.) Is there any

> > mechanism through which I can have the DLR parse this for me and give

> > me the resulting AST that I can walk so I can see if we can pop up an

> > intellisense-like view?

> >

> > Can anyone recommend a better way to do this outside of simply writing

> > my own mini-parser?

> >

> > Thanks,

> >

> > -Lee Culver

> >

> > ------------------------------------------------------------------------

> >

> > _______________________________________________

> > Users mailing list

> > Users at lists.ironpython.com

> > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

> >

>

>

> --

> http://www.ironpythoninaction.com/

> http://www.voidspace.org.uk/blog

>

>

> _______________________________________________

> Users mailing list

> Users at lists.ironpython.com

> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________

Users mailing list

Users at lists.ironpython.com

http://lists.ironpython.com/listinfo.cgi/users-ironpython.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20091202/613fed7b/attachment.html>