[Python-ideas] Enabling access to the AST for Python code

[Ben Hoyt]

Good to know -- thanks! -Ben

On Fri, May 22, 2015 at 10:40 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Thu, May 21, 2015 at 10:10 PM Ben Hoyt <benhoyt at gmail.com> wrote:
>>
>> Huh, interesting idea. I've never used import hooks. Looks like the
>> relevant macropy source code is here:
>>
>>
>> https://github.com/lihaoyi/macropy/blob/master/macropy/core/import_hooks.py
>>
>> So basically you would do the following:
>>
>> 1) intercept the import
>> 2) find the source code file yourself and read it
>> 3) call ast.parse() on the source string
>> 4) do anything you want to the AST, for example turn the "select(c for
>> c in Customer if sum(c.orders.price) > 1000" into whatever SQL or
>> other function calls
>> 5) pass the massaged AST to compile(), execute it and return the module
>>
>> Hmmm, yeah, I think you're basically suggesting macro-like processing
>> of the AST. Pretty cool, but not quite what I was thinking of ... I
>> was thinking select() would get an AST object at runtime and do stuff
>> with it.
>
>
> Depending on what version of Python you are targeting, it's actually simpler
> than that even to get it into the import system:
>
> Subclass importlib.machinery.SourceFileLoader and override source_to_code()
> to do your AST transformation and return your changed code object (basically
> your steps 3-5 above)
> Set a path hook that uses an instance of importlib.machinery.FileFinder
> which utilizes your custom loader
> There is no step 3
>
> I know this isn't what you're after, but I just wanted to let you know
> importlib has made this sort of thing fairly trivial to implement.
>
> -Brett
>
>>
>>
>> -Ben
>>
>> On Thu, May 21, 2015 at 9:51 PM, Andrew Barnert <abarnert at yahoo.com>
>> wrote:
>> > On May 21, 2015, at 18:18, Ben Hoyt <benhoyt at gmail.com> wrote:
>> >>
>> >> (I know that there's the "ast" module and ast.parse(), which can give
>> >> you an AST given a *source string*, but that's not very convenient
>> >> here.)
>> >
>> > Why not? Python modules are distributed as source. You can pretty easily
>> > write an import hook to intercept module loading at the AST level and
>> > transform it however you want. Or just use MacroPy, which wraps up all the
>> > hard stuff (especially 2.x compatibility) and provides a huge framework of
>> > useful tools. What do you want to do that can't be done that way?
>> >
>> > For many uses, you don't even have to go that far--code objects remember
>> > their source file and line number, which you can usually use to retrieve the
>> > text and regenerate the AST.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/