macro FAQ

[Andrew Dalke]

Here's a proposed Q&A for the FAQ based on a couple recent
threads.  Appropriate comments appreciated

X.Y:  Why doesn't Python have macros like in Lisp or Scheme?

Before answering that, a clarification on what 'macro' means.
A Lisp macro is a way of modifying code when that code is first
defined.  It can rearrange the structure of the code, and add and
remove parts of it.  Unlike C's #define macro language, Lisp
macros understand the structure of the expression and the
context (or "closure") in which it is found.

Here's a simple example of what a macro might do for Python.
A complaint about Python is that constructors for simple data
types require typing the parameter names three times, as in

class Country:
    def __init__(self, name, capitol):
        self.name = name
        self.capitol = capitol

Suppose you could create a new way to create functions, which
looked at the argument list and automatically set the corresponding
member.  This would let you replace the above with something
more like

class Country:
    def __init__(self, name, capitol) [arg_assign]:
        pass

where the 'arg_assign' would use the function definition
and code to make a new function used in place of the
existing one and with the 'self.name = name' and
'self.capitol = capitol' code automatically added to
the front of the code.

This is a very powerful technique and especially nice for
high-performance code where the macro can optimize
code before it is actually evaluated.  Macros work well in
Lisp or Scheme, where the code is the data, all written
pretty much as a parse tree.

The first problem with implementing a macro system in Python
is coming up with a readable syntax; that is, how it's "spelled."
The example shown above, using the [] after the function
definition, was taken from a proposed syntax for adding
function modifiers like 'staticmethod' and 'classmethod' to
Python.

The second problem is that Python's code blocks do not
store the original parse tree, so there's nothing to manipulate.
If you use 'dir()' or the inspect module you'll see some fields
like the list of argument names and the default value to use
for the fields.  Nowhere will you find the original parse tree.
(The C implementation of Python does give access to the
Python byte code, but this is not part of the language
specification and Java implementation doesn't support it.)

These problems can be resolved, and Dylan shows that
an infix-based/Algol-like language can support macros
similar to those used in Lisp.

The deeper question is, should Python include macros?

People with a Lisp background might have a problem
understanding this viewpoint.  Macros add flexibility, and
more flexibilty lets programmers be more productive; or
so the argument goes.

Those against including macros in Python point out that
Python has a different focus.  It has a broad target audience.
Beginners can pick it up pretty easily yet advanced
developers also enjoy working in the language.  New features,
like new builtin functions, additional core types, expanding
the standard libraries, list comprensions, metaclasses and
deriving from object, and macros, all make it harder to
learn how to use Python well and shifts the balance more
towards advanced programmers.

More importantly, the detractors -- including those
with plenty of experience using macros in Lisp -- argue that
macros cause dialects to form.  Macros can modify other code to
make it fit the problem better, while functions only use other
code but make no modifications.  This makes them very
powerful but means that understanding a section of code
requires also knowing about any macros which might use the
code.  In an extreme case which wouldn't be used in real
projects, every * could be replaced with a +.

(As an aside, some proponents argue that macros and
functions are essentially the same.  Alex Martelli made
an interesting observation about one difference:  it's often
worthwhile to turn a set of code into a function even if it
is only called from one place, because it helps simplify
the code into smaller chucks which are easier to understand.
Macros, on the other hand, should almost never be used
unless they are used many times.)

With only one or a small group of people working together
on a project there is little problem.  Macros help in developing
idioms specific to the problem and group.  When groups
share code, they also share idioms, and anyone who has had
to deal with UK vs. US English knows the effect idioms have
in mututal understanding.

Those against macros say their benefits do not outweigh
the likelihood that the Python community will be broken up
into distinct subsets, where people from one group find it
hard to use code from another.

The proponents for macros say this has not been a problem
in Lisp.  Macros are misused, but so are functions and classes.
The same community feedback which encourages people to
use proper coding styles also encourages people to use macros
correctly.  Those against reassert that not only can it be a
problem but it's happened to them personally.

At this point the discussion usually breaks down, with the
Lispers pulling out the old canard about Python (and every
other language) being a half-complete, poor reimplementation
of Lisp, while Pythonistas trot out the expression "Lisp is like
a ball of mud. You can add any amount of mud to it and it
still looks like a ball of mud."

Those interested in actual language improvements then ask for
concrete examples of how macros might be used for Python.
So far, the examples given are ones which could equally well
be done in Python using metaclasses, properties, and other
existing mechanisms.  For example, the arg_assign example
given above could, I think, be done with a modified__new__.

The problem at this point seems to be that the two languages
have different enough object models and accepted best
practices that it's hard for an intuition based on Lisp to
know immediately how to apply that intuition to Python.

If you want to encourage Python to have macros, you must
therefore propose a syntax for it and give solid examples of
what the result would look like when applied to real Python
code.  Simply stating that macros are needed is not good
enough nor are examples showing how macros are useful
for Lisp code.  (Besides, believe it or not but some of us
don't know enough Lisp to follow your examples ;)

In the meanwhile, if you want to experiment with manipulating
code trees, you can use decompyle to turn byte code back
into Python source and the compiler module to turn source into
an AST and thence into byte code.

                     Andrew Dalke
                    dalke at dalkescientific.com