[Python-ideas] Enabling access to the AST for Python code

[Andrew Barnert]

On May 21, 2015, at 19:15, Ben Hoyt <benhoyt at gmail.com> wrote:
> 
> Oh wait, macropy already has this exact thing. They call it PINQ
> (kinda Python LINQ), and they're macro-compiling it to SQLAlchemy
> calls.

I didn't even realize he'd included this when suggesting MacroPy. :)

Anyway, most of his macros are pretty easy to read as sample code, so even if what he's done isn't exactly what you wanted, it should be a good foundation.

> https://github.com/lihaoyi/macropy#pinq-to-sqlalchemy
> 
> Wow.
> 
> -Ben
> 
>> On Thu, May 21, 2015 at 10:10 PM, Ben Hoyt <benhoyt at gmail.com> wrote:
>> Huh, interesting idea. I've never used import hooks. Looks like the
>> relevant macropy source code is here:
>> 
>> https://github.com/lihaoyi/macropy/blob/master/macropy/core/import_hooks.py

If you wanted to do this yourself, and only need to support 3.4+, it's a lot easier than the way MacroPy does it.

But of course it's even easier to just use MacroPy.

>> So basically you would do the following:
>> 
>> 1) intercept the import
>> 2) find the source code file yourself and read it
>> 3) call ast.parse() on the source string
>> 4) do anything you want to the AST, for example turn the "select(c for
>> c in Customer if sum(c.orders.price) > 1000" into whatever SQL or
>> other function calls
>> 5) pass the massaged AST to compile(), execute it and return the module
>> 
>> Hmmm, yeah, I think you're basically suggesting macro-like processing
>> of the AST. Pretty cool, but not quite what I was thinking of ... I
>> was thinking select() would get an AST object at runtime and do stuff
>> with it.

If you really want to, you can build a trivial import hook that just attaches the ASTs (to everything, or only to specific code) and then ignore the code and process the AST at runtime. If you actually need to use runtime information in the processing, that might be worth it, but otherwise it seems like you're just wasting time transforming and compiling the AST on every request. Of course you could build in a cache if the information isn't really dynamic, but in that case, using the code object and .pyc as a cache is a lot simpler and probably more efficient.

>> 
>> -Ben
>> 
>>> On Thu, May 21, 2015 at 9:51 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>>>> On May 21, 2015, at 18:18, Ben Hoyt <benhoyt at gmail.com> wrote:
>>>> 
>>>> (I know that there's the "ast" module and ast.parse(), which can give
>>>> you an AST given a *source string*, but that's not very convenient
>>>> here.)
>>> 
>>> Why not? Python modules are distributed as source. You can pretty easily write an import hook to intercept module loading at the AST level and transform it however you want. Or just use MacroPy, which wraps up all the hard stuff (especially 2.x compatibility) and provides a huge framework of useful tools. What do you want to do that can't be done that way?
>>> 
>>> For many uses, you don't even have to go that far--code objects remember their source file and line number, which you can usually use to retrieve the text and regenerate the AST.