looking for tips on how to implement "ruby-style" Domain Specific Language in Python

[J Kenneth King]

Jonathan Gardner <jgardner at jonathangardner.net> writes:

> On Jan 6, 12:24 pm, J Kenneth King <ja... at agentultra.com> wrote:
>> Jonathan Gardner <jgard... at jonathangardner.net> writes:
>> > On Jan 6, 8:18 am, sturlamolden <sturlamol... at yahoo.no> wrote:
>> >> On Jan 6, 4:32 pm, mark <mark.fi... at googlemail.com> wrote:
>>
>> >> > I want to implement a internal DSL in Python. I would like the syntax
>> >> > as human readable as possible.
>>
>> >> Also beware that Python is not Lisp. You cannot define new syntax (yes
>> >> I've seen the goto joke).
>>
>> > This isn't really true. You can, for instance, write a program (in
>> > Python) that takes your pseudo-Python and converts it into Python.
>> > This is what a number of templating libraries such as Mako do.
>>
>> Which is not even close to being the same.
>>
>> Lisp - the program source is also the data format
>>
>> Python - the program source is a string
>>
>> I could go on a really long rant about how the two are worlds apart, but
>> I'll let Google tell you if you're really interested.
>
> I get that Lisp is special because you can hack on the reader as it is
> reading the file in. This is strongly discouraged behavior, as far as
> I know, despite the number of cute hacks you can accomplish with it.

It is generally discouraged unless there's a reason for it.

> But consider that this really isn't different than having a program
> read in the lisp-with-modification source and spitting out pure lisp,
> to be read by an honest-to-gosh lisp program later.
>
> If that's the case, then Lisp and Python really aren't that different
> in this regard, except that you don't have the option of modifying the
> reader as it reads in the file.

I think you are missing the distinction.

Lisp expressions are also data structures. A Lisp expression can be
passed to functions and macros to be operated on before being
executed. When you're writing Lisp source, you're basically looking at
the AST on one level and when you start writing macros for your program,
you're creating a "DSL" or interface to that AST. Lisp source is
eventually expanded to a giant list that is consed by the evaluator (as
far as I understand it. I'm just getting into the compiler stuff
myself).

Consider:

(my-func 1 2 3)

This is just a list, the "primitive" data-type in Lisp! This piece of
"data" can be operated on by other bits of Lisp code because it is just
a list as far as Lisp is concerned.

In contrast, Python source is a string that needs to be parsed into
bytecode which is then run through the interpreter. The AST is
completely hidden from the source author. Python expressions are not
data types either and hence no macros -- I can't write a python function
that generates python code at compile time. I can only write a python
program that parses some other string and generates code that can be run
by another interpreter.

Consider:

for i in range(0, 100):
    do_something_interesting(i)

That's a pretty straight forward Python expression, but I can't do
anything with it -- it's not a unit of data, it's a string.

The distinction is not subtle by any means.