Limits of Metaprogramming

[Tomasz Rola]

On Mon, 4 Aug 2008, Wilson wrote:

> Hi all,
> 

Howdy,

I am not sure if my remarks will be of any use for you, but here it goes.

> I have an interesting problem that I'm hoping can be solved with
> metaprogramming, but I don't know how far Python supports code
> generation (and I don't know if I'm taking the correct approach
> either... hence why I'm asking on this group):
> 
> I'd like to write a program that writes/manipulates a statemachine. My
> idea was that I would define states in a .py file along with their
> transitions and other metadata. I could then write a suite of programs
> that would manipulate these states in interesting ways, such as
> generate diagrams and documentation, test suites, do verifications and
> also edit them. These programs would import the .py file and use
> introspection upon the module's classes (states would be modelled as
> classes). For example, a documentation tool could use graphviz to draw
> the statediagram with the transition table and docstring...

Maybe you have actually two problems.

One problem is processing source code into something useful, but 
not executable. So it does not necessarily need metaprogramming to be 
solved, just some "code comprehension" (like get a python grammar, 
lex/yacc and read a lot of docs how to use them).

> My problem is that I don't know if it's possible to edit these states
> and then write them back to .py. Firstly, if my editing tool was to
> create a new state, I would want to create the class (using type) and
> attach it to the imported state module somehow. I can't find
> information on whether this is possible. Secondly, I'd like to
> manipulate the class and then use the inspect module to get the source
> and write it back to file. Now I'm pretty sure the inspect module will
> only write the source code as found in the original .py file. Is there
> a way to generate the code for a class or other python objects in a
> generic way? Or will I have to bite the bullet and write to custom
> file and generate code from that (like glade with XML)? Or is there
> another solution with templates that I'm over looking?

Well, "everything" is doable in a Turing-complete language :-). However, 
this one problem - code generation and using it on the fly - may be better 
solved with some other language, perhaps?

Not that I want to dissuade you (or anyone else) from using Python, but 
last time I checked the state of things (was a good couple of years ago, I 
wanted to solve some math-related problem with metaprogramming in Python) 
I decided I would do better if I learned Scheme. At least in Scheme, 
program structure is so simple, it is just a list (or a sequence of them), 
so manipulating code is IMHO much easier.

I mean, even if in theory every language in Turing-complete class is equal 
to each other (one can do the same kind of stuff in one as in the other), 
in practice they are not equal at all. So doing everything in one ultimate 
language of choice is not very wise from my point of view.

It does not mean you should immediately go to 

http://en.wikipedia.org/wiki/Scheme_(programming_language)

but you may do so, and then make an informed choice by yourself. To be 
better informed, you may also google for more material - introductions, 
opinions, comparisons and some first-hand experience.

On the other hand, writing "metafiles" and interpreters for them - ugh... 
:-). I have read somewhere, that some programmers resist using Scheme 
(or some other Lisp) for solving their problem and end up with writing 
their own Lisp-like (language|file format) and interpreter(s) for it. So 
you better watch yourself. If this effort is going to be something more 
permament than "done and go", then you will probably find yourself in the 
situation of reinventing the wheel, as your project goes more and more 
complex. If this is going to happen, you might use existing wheels and 
save some time for thinking on something better.

> Having the classes in memory, editing their attributes, adding methods
> and then serialising them to code has an elegance to it in my mind and
> would hopefully simplify things. The result would be directly
> executable by the interpreter itself and editable by python
> developers. But I fear that I may need an intermediary notation/format
> for storage. Any advice/hints would be very useful.

Yes, there is something elegant in this way of thinking, at least for me.

As I (indirectly) said above, I am not up to date with recent Python 
abilities (I still use Python for a lot of things, I just do not overuse 
it for things, that it seems to be not so good at). I hope someone knows 
better and steps in to prove me wrong - it is always good to learn 
something new. Also, if you have just one problem, then learning a new 
language because of it is a bit too much (or maybe not, if you enjoy it 
:-) ). But if you plan doing similar or more difficult things in the 
future, then knowing Scheme may be good for you. Even if you choose to not 
use it, you will be aware of what can be done with it so this knowledge 
can provide some ready to use templates.

And yes, there are also some other languages in Lisp family, but I think 
Scheme is best choice if you don't know any of them already. It is quite 
small, it is well defined (I mean, it has some real specification, 
instead of being "specified by its implementation") and there is a lot of 
info about it on the net that could be easily understood. And, last but 
not least, it has metaprogramming included with nice bunch of additional 
stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are 
other nice looking Scheme implementations but this one is probably best 
fitted for a beginner):

http://www.plt-scheme.org/

Regards,
Tomasz Rola

--
** A C programmer asked whether computer had Buddha's nature.      **
** As the answer, master did "rm -rif" on the programmer's home    **
** directory. And then the C programmer became enlightened...      **
**                                                                 **
** Tomasz Rola          mailto:tomasz_rola at bigfoot.com             **