[Python-ideas] Python as meta-language

[Talin]

One of my long-standing interests is in mini-languages, particularly 
declarative languages. I'm always looking out for examples where a 
declarative language is used to represent some idea or concept that is 
not easily written in an imperative language. Examples are the behaviors 
of particle systems, kinematic constraints, formalized grammars, logical 
inferencing systems, query languages, and so on. In other words, you are 
using a language to describe a complex set of relationships, but you 
aren't giving specific commands to execute in a specific order.

Often such mini-languages are implemented by writing a custom parser. 
However, often this is not necessary if the underlying language (such as 
Python) is flexible enough. Python's ability to declare complex 
structures as literals, combined with its ability to overload operators, 
means that one can often embed the mini-language within the Python 
syntax itself, and use the Python compiler as the parser for your mini 
language. It also allows you a convenient means to "escape" back into 
the procedural world when needed.

Examples of the kind of things I am talking about include the SConstruct 
file format from SCONS and the SQLBuilder syntax from SQLObject. And 
although it's not directly related to Python, JSON has a lot of the same 
ideas - that is, using a scripting language source code as an efficient 
representation of complex data structures. And these are just a few of 
the many examples out there.

What I'd be interested in doing, in this python-ideas list, is 
brainstorming some ideas for how we can improve Python's ability to 
'host' other kinds of mini-languages within the Python syntax. We can 
start perhaps by examining some of the use cases I listed in the first 
paragraph - particle systems, etc - and see how one would want to 
represent those kinds of semantic structures within Python.

Of course, there are some languages (Lisp and Dylan come to mind), which 
are even more flexible in this regard - the languages can be 'morphed' 
out of all recognition to the original syntax. (For example, in Dylan, a 
'macro' was not a simple textual substitution as in C, but in fact added 
new production rules to the parser.) In I'm in no way advocating such a 
course. (Well, at least not at this moment :) ) I certainly recognize 
that there is a danger in making a language too 'plastic', in that it 
can easily be obfuscated with too much cleverness and lose it's 
identity. So I'm more interested in ideas that are subtle yet powerful.

I'm sure that there are lots of approaches to this general concept. I'm 
going to throw out a couple of ideas, but I am going to post them as 
separate replies to this email, and not right away - the reason is, I 
don't want this thread to be taken over by the discussion / criticism of 
those specific ideas, when I'm more interested in brainstorming the 
general concept.

-- Talin