[IronPython] Building via AST
Justin Chase
justin.m.chase at gmail.com
Thu Apr 15 21:56:07 CEST 2010
So this is awesome by the way. I added the import and
casting-to-an-interface code back in and it works like a charm. I'm
generating python code purely via AST!
On Thu, Apr 15, 2010 at 1:25 PM, Justin Chase <justin.m.chase at gmail.com>wrote:
> SUCCESS!
>
> Here's the final code:
> var runtime = Python.CreateRuntime();
>
> var functionDefinition = new FunctionDefinition(
> "Do",
> new[] { new Parameter("self") },
> new ReturnStatement(new ConstantExpression("hello python!")));
>
> var classDefinition = new IronPython.Compiler.Ast.ClassDefinition(
> "Example",
> new Expression[] { },
> functionDefinition);
>
> dynamic python = runtime.Compile(classDefinition);
> dynamic example = python.Example();
>
> Console.WriteLine(example.Do());
> Console.ReadKey(true);
>
> public static class DLRHelper
> {
> public static dynamic Compile(this ScriptRuntime runtime, params
> Statement[] statements)
> {
> var engine = runtime.GetEngineByFileExtension(".py");
> var context = (PythonContext)HostingHelpers.GetLanguageContext(engine);
>
> var globals = new PythonDictionary();
> globals.Add("__name__", "<dynamic>");
>
> var options = new PythonCompilerOptions(ModuleOptions.ExecOrEvalCode);
> var unit = new SourceUnit(context, NullTextContentProvider.Null, "",
> SourceCodeKind.AutoDetect);
>
> var moduleContext = new ModuleContext(globals, context);
> var codeContext = moduleContext.GlobalContext;
> var pythonAst = new PythonAst(
> new IronPython.Compiler.Ast.SuiteStatement(statements),
> false,
> ModuleOptions.ExecOrEvalCode,
> false,
> new CompilerContext(
> unit,
> options,
> ErrorSink.Default));
>
> pythonAst.Walk(new EnsureLocWalker());
>
> pythonAst.Bind();
> var lambda = (System.Linq.Expressions.LambdaExpression)pythonAst.Reduce();
> var func = (Func<CodeContext, FunctionCode, object>)lambda.Compile();
> var result = func(codeContext, null); // result is null
>
> return HostingHelpers.CreateScriptScope(engine, codeContext.GlobalScope);
> }
>
> // I created a walker called EnsureLocWalker that overrides all Walk
> methods and calls SetLoc to new SourceSpan(new SourceLocation(1, 1, 1), new
> SourceLocation(2, 2, 2));
> // and header when appropriate to new SourceLocation(1, 1, 1)
> }
>
>
> FYI your WithStatement, TryStatement and ForStatement are missing getters
> on their header property. That's frustrating. I want to be able to do this:
>
> if (node.Header == SourceLocation.Invalid)
> node.Header = new SourceLocation(1, 1, 1);
>
> Also it would be nice if the headers and locations were completely ignored
> if they're invalid instead of throwing exceptions. Also it would be *reaaaally
> *nice if you had a Walk(Node node) override on the PythonWalker. So you
> could do something to all nodes. Also, why have a million overrides for
> this? Can't you just make a generic PythonWalker? Then just have a single
> Walk(T node) and AfterWalk(T node). That would be a lot less gross.
>
> The reason why I want to do this is because I am working on a DSL tool (
> http://metasharp.codeplex.com) where I'm compiling down into an AST.
> Currently I have transforms to create CodeDom objects and linq Expression
> objects but I lack the ability to dynamically create Types or directly
> generate IL. Using this I think i can save a TON of work for myself.
>
> Thanks for all the help!
>
>
> On Wed, Apr 14, 2010 at 6:21 PM, Dino Viehland <dinov at microsoft.com>wrote:
>
>> Try calling SetLoc on the FunctionDefinition and also try setting the
>> Header property. I think if it’s some reasonable span it’ll work.
>>
>>
>>
>> *From:* users-bounces at lists.ironpython.com [mailto:
>> users-bounces at lists.ironpython.com] *On Behalf Of *Justin Chase
>> *Sent:* Wednesday, April 14, 2010 3:35 PM
>>
>> *To:* Discussion of IronPython
>> *Subject:* Re: [IronPython] Building via AST
>>
>>
>>
>> Ok awesome!
>>
>>
>>
>> So now I have:
>>
>> dynamic example = globals["Example"];
>>
>> dynamic instance = example();
>>
>> Console.WriteLine(instance.Do());
>>
>>
>>
>> But I'm getting an exception about a line number. Any idea how I can get
>> past that? I've tried a bunch of things but I'm not sure what's going on.
>>
>>
>>
>> ArgumentOutOfRangeException
>>
>> Specified argument was out of the range of valid values. Parameter name:
>> line must be greater than or equal to 1
>>
>> at Microsoft.Scripting.SourceLocation.ValidateLocation(Int32 index,
>> Int32 line, Int32 column)
>>
>> at Microsoft.Scripting.SourceLocation..ctor(Int32 index, Int32 line,
>> Int32 column)
>>
>> at IronPython.Compiler.Ast.FunctionDefinition.CreateFunctionLambda()
>>
>> at IronPython.Compiler.Ast.FunctionDefinition.EnsureFunctionLambda()
>>
>> at IronPython.Compiler.Ast.FunctionDefinition.GetLambda()
>>
>> at IronPython.Runtime.FunctionCode.get_Code()
>>
>> at IronPython.Runtime.FunctionCode.GetGeneratorOrNormalLambda()
>>
>> at IronPython.Runtime.FunctionCode.UpdateDelegate(PythonContext
>> context, Boolean forceCreation)
>>
>> at
>> IronPython.Runtime.FunctionCode.LazyCompileFirstTarget(PythonFunction
>> function)
>>
>> at
>> IronPython.Compiler.PythonCallTargets.OriginalCallTarget1(PythonFunction
>> function, Object arg0)
>>
>> at IronPython.Runtime.PythonFunction.FunctionCaller`1.Call1(CallSite
>> site, CodeContext context, Object func, T0 arg0)
>>
>> at
>> System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite
>> site, T0 arg0, T1 arg1, T2 arg2)
>>
>> at CallSite.Target(Closure , CallSite , Object )
>>
>> at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite
>> site, T0 arg0)
>>
>> at CallSite.Target(Closure , CallSite , Object )
>>
>> at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite
>> site, T0 arg0)
>>
>>
>>
>>
>>
>> On Wed, Apr 14, 2010 at 3:32 PM, Dino Viehland <dinov at microsoft.com>
>> wrote:
>>
>> The problem here is just that you actually have a nested code context when
>> for the top-level code you just need a single global context. Replace the
>> line:
>>
>>
>>
>> var codeContext = new CodeContext(
>>
>> new PythonDictionary(),
>>
>> moduleContext);
>>
>>
>>
>> with:
>>
>> var codeContext = moduleContext.GlobalContext;
>>
>>
>>
>> and it’ll work. The nested code context that you’re creating is only used
>> for when we’re inside of a function definition or a class definition.
>>
>>
>>
>> Also you can avoid creating the script scope and just pull Example out of
>> the globals dictionary as well.
>>
>>
>>
>> *From:* users-bounces at lists.ironpython.com [mailto:
>> users-bounces at lists.ironpython.com] *On Behalf Of *Justin Chase
>> *Sent:* Wednesday, April 14, 2010 11:24 AM
>>
>>
>> *To:* Discussion of IronPython
>> *Subject:* Re: [IronPython] Building via AST
>>
>>
>>
>> I feel like I'm sooooo close but something isn't quite working. Here is my
>> current code:
>>
>>
>>
>> var runtime = Python.CreateRuntime();
>>
>>
>> runtime.LoadAssembly(typeof(IExample).Assembly);
>>
>>
>>
>> var engine =
>> runtime.GetEngineByFileExtension(".py");
>>
>> var context =
>> (PythonContext)HostingHelpers.GetLanguageContext(engine);
>>
>>
>>
>> var globals = new PythonDictionary();
>>
>> globals.Add("__name__", "<test>"); // I
>> get an exception without this.
>>
>>
>>
>> var options = new
>> PythonCompilerOptions(ModuleOptions.ExecOrEvalCode |
>> ModuleOptions.Initialize);
>>
>> var unit = new SourceUnit(context,
>> NullTextContentProvider.Null, "", SourceCodeKind.Statements);
>>
>>
>>
>> var moduleContext = new
>> ModuleContext(globals, context);
>>
>> var codeContext = new CodeContext(
>>
>> new PythonDictionary(),
>>
>> moduleContext);
>>
>>
>>
>> var classDefinition = new
>> IronPython.Compiler.Ast.ClassDefinition(
>>
>> "Example",
>>
>> new Expression[] { },
>>
>> new FunctionDefinition(
>>
>> "Do",
>>
>> new[] { new
>> Parameter("self") },
>>
>> new
>> ReturnStatement(new ConstantExpression("hello python!"))));
>>
>>
>>
>> var pythonAst = new PythonAst(
>>
>> new
>> IronPython.Compiler.Ast.SuiteStatement(new Statement[] { /* import, */
>> classDefinition }),
>>
>> false,
>>
>>
>> ModuleOptions.ExecOrEvalCode,
>>
>> false,
>>
>> new CompilerContext(
>>
>> unit,
>>
>> options,
>>
>> new
>> ThrowsErrorSink()));
>>
>>
>>
>> pythonAst.Bind();
>>
>> var lambda =
>> (System.Linq.Expressions.LambdaExpression)pythonAst.Reduce();
>>
>> var func = (Func<CodeContext,
>> FunctionCode, object>)lambda.Compile();
>>
>> var result = func(codeContext, null); //
>> result is null
>>
>>
>>
>> dynamic python =
>> HostingHelpers.CreateScriptScope(engine, moduleContext.GlobalScope);
>>
>> *dynamic example = python.Example(); //
>> fails! Example is not in scope.*
>>
>> Console.WriteLine(example.Do());
>>
>> Console.ReadKey(true);
>>
>>
>>
>>
>>
>> Can anyone tell me what I'm missing? This has got to be pretty close.
>>
>>
>>
>>
>>
>> On Mon, Apr 12, 2010 at 7:51 PM, Justin Chase <justin.m.chase at gmail.com>
>> wrote:
>>
>> Awesome. I will thanks.
>>
>> On Apr 12, 2010 7:49 PM, "Dino Viehland" <dinov at microsoft.com> wrote:
>>
>> This might be possible. If you wrap this all up in a PythonAst object
>> (calling the constructor which takes a CompilerContext), call Bind on it
>> then you should get a LambdaExpression back out. You can Compile() on
>> that.
>>
>>
>>
>> But it’s not like this is well traveled territory and this only applies to
>> 2.6.1 (before that the trees weren’t DLR ASTs so they weren’t reducable).
>> When we do this ourselves we also call the produced delegate and flow in
>> some data. The delegate is going to want at least a FunctionCode object as
>> an argument but I think you could get away with passing null (at least as
>> long as no exceptions are thrown). The delegate might also want a
>> CodeContext object as well depending on the compilation mode we end up using
>> (which is based on the CompilerContext you give us). This you wouldn’t be
>> able to get away w/ passing null. But you can get one by doing new
>> ModuleContext(new PythonDictionary(), pythonContext).GlobalContext. The
>> HostingHelpers class can give you a LanguageContext from the ScriptEngine
>> for Python which you can cast to a PythonContext.
>>
>>
>>
>> Let me know if it works! J
>>
>>
>>
>> *From:* users-bounces at lists.ironpython.com [mailto:
>> users-bounces at lists.ironpython.com] *On Behalf Of *Justin Chase
>> *Sent:* Monday, April 12, 2010 4:09 PM
>>
>> To: Discussion of IronPython Subject: Re: [IronPython] Building via AST
>>
>> Ok, so at risk of being a nuissance I have one last question because I
>> feel like I'm half way t...
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>>
>>
>> --
>> Justin Chase
>> http://www.justnbusiness.com
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>>
>>
>> --
>> Justin Chase
>> http://www.justnbusiness.com
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>
>
> --
> Justin Chase
> http://www.justnbusiness.com
>
--
Justin Chase
http://www.justnbusiness.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20100415/dc508076/attachment.html>
More information about the Ironpython-users
mailing list