[Python-ideas] [Python-Ideas] AST Transformation Hooks for Domain Specific Languages

[Eric Snow]

On Fri, Apr 8, 2011 at 5:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> A few odds and ends from recent discussions finally clicked into
> something potentially interesting earlier this evening. Or possibly
> just something insane. I'm not quite decided on that point as yet (but
> leaning towards the latter).
>
> Anyway, without further ado, I present:
>
> AST Transformation Hooks for Domain Specific Languages
> ======================================================
>
> Consider:
>
> # In some other module
> ast.register_dsl("dsl.sql", dsl.sql.TransformAST)
>
> # In a module using that DSL
> import dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
>    select address
>    from people
>    where name = {name} and dob = {dob}
>
>
I like how you moved the from to after the parameter list.  It makes it less
complicated.  Annotations ( -> int: ) will still go after that, right?


>

Suppose that the standard AST for the latter looked something like:
>
>    DSL(syntax="dsl.sql",
>        name='lookup_address',
>        args=arguments(
>            args=[arg(arg='name',
>                      annotation=<Normal AST for "dsl.sql.char">),
>                  arg(arg='dob',
>                      annotation=<Normal AST for "dsl.sql.date">)],
>            vararg=None, varargannotation=None,
>            kwonlyargs=[], kwarg=None, kwargannotation=None,
>            defaults=[], kw_defaults=[]),
>        body=[Expr(value=Str(s='select address\nfrom people\nwhere
> name = {name} and dob = {dob}'))],
>        decorator_list=[],
>        returns=None)
>
> (For those not familiar with the AST, the above is actually just the
> existing Function node with a "syntax" attribute added)
>
> At *compile* time (note, *not* function definition time), the
> registered AST transformation hook would be invoked and would replace
> that DSL node with "standard" AST nodes.
>
>
So the AST transform would not necessarily return a Function node...


> For example, depending on the design of the DSL and its support code,
> the above example might be equivalent to:
>
>    @dsl.sql.escape_and_validate_args
>    def lookup_address(name: dsl.sql.char, dob: dsl.sql.date):
>       args = dict(name=name, dob=dob)
>       query = "select address\nfrom people\nwhere name = {name} and
> dob = {dob}"
>       return dsl.sql.cursor(query, args)
>
>
> As a simpler example, consider something like:
>
>    def f() from all_nonlocal:
>        x += 1
>        y -= 2
>
> That was translated at compile time into:
>
>    def f():
>        nonlocal x, y
>        x += 1
>        y -= 2
>
> My first pass at a rough protocol for the AST transformers suggests
> they would only need two methods:
>
>  get_cookie() - Magic cookie to add to PYC files containing instances
> of the DSL (allows recompilation to be forced if the DSL is updated)
>  transform_AST(node) - a DSL() node is passed in, expected to return
> an AST containing no DSL nodes (SyntaxError if one is found)
>
> Attempts to use an unregistered DSL would trigger SyntaxError
>
> So there you are, that's the crazy idea. The stoning of the heretic
> may now commence :)
>
> Where this idea came from was the various discussions about "make
> statement" style constructs and a conversation I had with Eric Snow at
> Pycon about function definition time really being *too late* to do
> anything particularly interesting that couldn't already be handled
> better in other ways. Some tricks Dave Malcolm had done to support
> Python level manipulation of the AST during compilation also played a
> big part, as did Eugene Toder's efforts to add an AST optimisation
> step to the compilation process.
>
>
I Like this idea better than the def-from with builders.  Since it is
compile-time, it certainly addresses some of the concerns that Guido brought
up, like run-time dynamic code generation and pyc files.  You simply have to
play within the confines of the AST, which seems like an effective
constraint.

Like Michael said, this would be great for trying out new language features.
  And like Eugene said, it may be worth doing in an import hook.  I've
actually been working on such a hook since the feedback I got for those
three ideas last week.  I doubt I will have anything right away, but I'll
let you know when I get there.  Regardless, I still like the idea of a
syntax that allows for the same freedom to define new stuff.

I wonder though if it would be subject to extensive abuse that would make
code harder to understand/use/debug.  I don't know that there are any other
language features that are extensively used that have his effect.  _maybe_
metaclasses, decorators, and the whole dot-lookup mechanism.  However, you
can wrap your head around those with a little explanation.  I guess this
would be similar.  It's good that the non-python is contained within the def
body.

I'm just thinking about what Raymond said last week about how all the
meta-programming stuff makes debugging harder.  I tend to agree.  I think of
the challenge of tracing through a problem I had in sqlalchemy a couple of
years ago that involved their use of metaclasses.

Anyway, love the idea.

-eric


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/12dfcedb/attachment.html>