[Python-ideas] AST Transformation Hooks for Domain Specific Languages

[Michael Foord]

On 8 April 2011 12:32, Nick Coghlan <ncoghlan at gmail.com> wrote:

> (Oops, let's try that again with the correct destination address this
> time...)
>
> A few odds and ends from recent discussions finally clicked into
> something potentially interesting earlier this evening. Or possibly
> just something insane. I'm not quite decided on that point as yet (but
> leaning towards the latter).
>
> Anyway, without further ado, I present:
>

Oops, sent my reply to the wrong list as well.

The essence of the proposal is to allow arbitrary syntax within "standard
python files". I don't think it stands much of a chance in core.

It would be an awesome tool for experimenting with new syntax and DSLs
though. :-)

Michael




>
> AST Transformation Hooks for Domain Specific Languages
> ======================================================
>
> Consider:
>
> # In some other module
> ast.register_dsl("dsl.sql", dsl.sql.TransformAST)
>
> # In a module using that DSL
> import dsl.sql
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
>   select address
>   from people
>   where name = {name} and dob = {dob}
>
>
> Suppose that the standard AST for the latter looked something like:
>
>   DSL(syntax="dsl.sql",
>       name='lookup_address',
>       args=arguments(
>           args=[arg(arg='name',
>                     annotation=<Normal AST for "dsl.sql.char">),
>                 arg(arg='dob',
>                     annotation=<Normal AST for "dsl.sql.date">)],
>           vararg=None, varargannotation=None,
>           kwonlyargs=[], kwarg=None, kwargannotation=None,
>           defaults=[], kw_defaults=[]),
>       body=[Expr(value=Str(s='select address\nfrom people\nwhere
> name = {name} and dob = {dob}'))],
>       decorator_list=[],
>       returns=None)
>
> (For those not familiar with the AST, the above is actually just the
> existing Function node with a "syntax" attribute added)
>
> At *compile* time (note, *not* function definition time), the
> registered AST transformation hook would be invoked and would replace
> that DSL node with "standard" AST nodes.
>
> For example, depending on the design of the DSL and its support code,
> the above example might be equivalent to:
>
>   @dsl.sql.escape_and_validate_args
>   def lookup_address(name: dsl.sql.char, dob: dsl.sql.date):
>      args = dict(name=name, dob=dob)
>      query = "select address\nfrom people\nwhere name = {name} and
> dob = {dob}"
>      return dsl.sql.cursor(query, args)
>
>
> As a simpler example, consider something like:
>
>   def f() from all_nonlocal:
>       x += 1
>       y -= 2
>
> That was translated at compile time into:
>
>   def f():
>       nonlocal x, y
>       x += 1
>       y -= 2
>
> My first pass at a rough protocol for the AST transformers suggests
> they would only need two methods:
>
>  get_cookie() - Magic cookie to add to PYC files containing instances
> of the DSL (allows recompilation to be forced if the DSL is updated)
>  transform_AST(node) - a DSL() node is passed in, expected to return
> an AST containing no DSL nodes (SyntaxError if one is found)
>
> Attempts to use an unregistered DSL would trigger SyntaxError
>
> So there you are, that's the crazy idea. The stoning of the heretic
> may now commence :)
>
> Where this idea came from was the various discussions about "make
> statement" style constructs and a conversation I had with Eric Snow at
> Pycon about function definition time really being *too late* to do
> anything particularly interesting that couldn't already be handled
> better in other ways. Some tricks Dave Malcolm had done to support
> Python level manipulation of the AST during compilation also played a
> big part, as did Eugene Toder's efforts to add an AST optimisation
> step to the compilation process.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110408/7d30cbcb/attachment.html>