[Python-ideas] AST Transformation Hooks for Domain Specific Languages

[Nick Coghlan]

On Sat, Apr 9, 2011 at 10:27 AM, Eugene Toder <eltoder at gmail.com> wrote:
>> Yep, but if you're at the point of using a DSL, that's likely to be
>> part of a larger framework which can take care of registering the DSL
>> for you before you import any modules that need it.
>
> Not necessarily. And registering makes debugging small pieces and
> playing in interactive interpreter less convenient.

Not really. All you would need is for it to be standard practice for
DSL definition modules to provide a __main__ clause that registers
themselves and uses runpy to execute the passed in arguments (once we
start converting some of the stdlib utilities like "pdb" to do that,
I'll even see if there is sufficiently commonality to make it worth
factoring out a "runpy.delegate_main()" helper function).

So a simple "python -mdsl.sql myfile.py" would run a file that uses
the DSL, while "python -i -mdsl.sql" would get you an interactive
interpreter that understood that DSL dialect.

Embedding imports inside functions has long been fragile, and is a
good recipe for deadlocking code (especially if process forks are
involved). I *really* don't want to advocate enshrining them as an
implicit part of a standard piece of syntax (even if that syntax has
no realistic chance of ever making it into the core language).

>> Yeah, the big downside is having to almost completely reimplement the
>> AST compiler in order to do it that way (since the built-in one would
>> choke on the syntax extension).
>
> I did not realize you're proposing to skip parsing of the body of dsl
> function and just tuck all the code into Str literal.
> I though the idea was to actually do an AST transformation, i.e. use
> Python parser first and rewrite resulting AST into something else.
> This is a classical approach and something people already do in
> Python, though it's not integrated with the language at the moment
> (e.g. there's no quote and no way to get AST for function).

That part is actually orthogonal to the core AST construction hook
concept. It just occurred to me as a possibility when I was writing
the SQL example.

Certainly, if I try implementing this, I won't do it that way - I'll
just add the "syntax" attribute to the Function node and do the
experiment as a true AST transformation process rather than implicitly
quoting the function body. That will make the AST generation tweaks
significantly simpler than the full-blown Mython-style quoting
concept.

> I don't know if I like the stringification idea. It allows much more
> freedom in dsl syntax, but to use that freedom one needs to implement
> a parser. Just rewriting AST is simpler. Also, the implementation is
> very different from an AST transform -- basically, the grammar needs
> to say that after you saw dsl function header the next suite needs to
> be saved as a string.

Reusing Python syntax would still be easy - you'd simply invoke
ast.parse() on the stringified body.

However, I'm not that fond of the idea either, so I doubt I'll
experiment with it (particularly since Mython already works that way).

> in module b will import module a at compile time and execute a.dec on
> foo's AST and resulting AST will be used.
> This can be done with a 'from' clause as well, if it gains importing
> capabilities -- that's just a syntactic difference. However, this
> seems very close to existing decorators, so similar syntax and
> semantics seem to make sense.

It also has the virtue of getting the DSL name in a more prominent
place, and allowing easier composition of DSLs. I'm sold :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia