[Python-ideas] Enabling access to the AST for Python code

[Ben Hoyt]

Hi Python Ideas folks,

(I previously posted a similar message on Python-Dev, but it's a
better fit for this list. See that thread here:
https://mail.python.org/pipermail/python-dev/2015-May/140063.html)

Enabling access to the AST for compiled code would make some cool
things possible (C# LINQ-style ORMs, for example), and not knowing too
much about this part of Python internals, I'm wondering how possible
and practical this would be.

Context: PonyORM (http://ponyorm.com/) allows you to write regular
Python generator expressions like this:

    select(c for c in Customer if sum(c.orders.price) > 1000)

which compile into and run SQL like this:

    SELECT "c"."id"
    FROM "Customer" "c"
    LEFT JOIN "Order" "order-1" ON "c"."id" = "order-1"."customer"
    GROUP BY "c"."id"
    HAVING coalesce(SUM("order-1"."total_price"), 0) > 1000

I think the Pythonic syntax here is beautiful. But the tricks PonyORM
has to go to get it are ... not quite so beautiful. Because the AST is
not available, PonyORM decompiles Python bytecode into an AST first,
and then converts that to SQL. (More details on all that from author's
EuroPython talk at http://pyvideo.org/video/2968)

PonyORM needs the AST just for generator expressions and
lambda functions, but obviously if this kind of AST access feature
were in Python it'd probably be more general.

I believe C#'s LINQ provides something similar, where if you're
developing a LINQ converter library (say LINQ to SQL), you essentially
get the AST of the code ("expression tree") and the library can do
what it wants with that.

(I know that there's the "ast" module and ast.parse(), which can give
you an AST given a *source string*, but that's not very convenient
here.)

What would it take to enable this kind of AST access in Python? Is it
possible? Is it a good idea?

-Ben