macro FAQ

[Doug Tolton]

Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyfhe47e705.fsf at lxplus085.cern.ch>...
> dtolton at yahoo.com (Doug Tolton) writes:
> 
> > Also note that the examples presented are purposefully trivial and
> > could in many instances be accomplished other ways,
> 
> I hoped that the examples that I presented satisfied the requirements
> that
> 
> a) They be simple enough to understand
> 
> b) not be (at least easily) implementable using existing Python
>    features
> 
> c) Not be completely useless
> 
> Did I fail ? :-)

No I don't think you failed, I just wanted to nail down that specific
point.  The reason I purposefully used trivial examples was to
illustrate the point the Macros don't somehow magically alter existing
source code, rather they operate on source code that is passed into
them.  I actually liked your examples, I was just hoping to come at
them from a slightly different example.
> 
> > Essentially Macros allow you to pass in some variables that are then
> > expanded into valid Python syntax which is then executed.
> 
> Well, not just variables, data in general, some of which might be
> variables, some of which might be variable names (identifiers) and
> some of which might be (peudo-) code.
> 
> > Question: Can't I accomplish this same functionality via string
> > parsing and eval?
> > 
> > Answer: Eval will accept any valid Python expression and execute it. 
> > This is however regarded by many as a security hole.  Macros allow
> > this type of expressiveness while limiting many of the security risks.
> > With a Macro you can more specifically control what types of actions
> > are permissible.
> 
> I don't think that this is relevant (or true, for that matter). Lisp
> macros do not expand arbitrary data that a user passes in, they appear
> as literals in your source code. The security issues are the same as
> those surrounding "os.system('rm -rf /')" appearing literally in your
> source code.
> 
> What is relevant, is the pain involved in doing it via strings. The
> (pseudo) code that you pass to a Lisp macro, not only looks exactly
> like Lisp code, but it is automatically tokenized and parsed for you,
> and very easily manipulatable.
> 

Again, I wanted to focus the discussion for a moment to help alleviate
people's fear of Macro's, as they seem to be based on
mis-understanding what they are about.  Please forgive my over
simplification.

> > # Note, Jacek's method of using indentation is an interesting
> > alternative to calling them as a
> > # function.
> 
> It's probably worth pointing out, again, how important Lisp's
> uniformity of syntax is, for its macro system. Generally speaking, a
> macro accepts as arguments some mixture of data, identifiers and
> expressions (let's forget about the existence of Python statements for
> now, they complicate the situation even more). How do you pass in a
> mixture of data and code?
> 
> Data is usually presented in a tuple:
> 
>    foo(datum1, datum2, datum3) 
> 
> Expressions are usually presented on separate lines:
> 
>    def foo(...):
>        expression1
>        expression2
>        expression3
> 
> Let's try to think of what a Python with-open-file macro call would
> look like.
> 
> In CL, with-open-file takes
> 
> 1) a list containing an identifier (a symbol), a pathname (let's call
>    it a string) with some optional keywords.
> 
> 2) arbitrary code
> 
> It then binds the identifer to a stream associated with the file, and
> expands the source code inside an exception handling environment.
> 
> So, the expansion of a (simplistic) Python with_open_file might look
> like this:
> 
>     outfile = file("foo/bar", "w")
>     try:
>         print >> outfile, foo(...)
>         for i in range(10):
>             print >> outfile, bar(...)
>     except:
>         perform the necessary closing and cleaning up
> 
> But what should the corresponding call to with_open_file look like?
> 
> We have to pass in the identifier "outfile" (this is where it is very
> handy to have symbols as a first-class data type, as in Lisp), and the
> file name "foo/bar" ... but then we also have to pass in the chunk of
> code 
> 
>     print >> outfile, foo(...)
>     for i in range(10):
>         print >> outfile, bar(i)
>     
> In Lisp it would look something like this:
> 
>     (with-open-file (outfile "foo/bar" :direction :output)
>       (format outfile "~a" (foo ...))
>       (dotimes (i 10)
>         (format outfile "~&~a" (bar i))))
> 
> Now, I refuse to contemplate passing the source code in a string, in
> the Python version.
> 
> Maybe we have to accept the distinction between data-like data, and
> code-like data, having the macro accept both a tuple of arguments, and
> blocks of code, so the call might look thus:
> 
>     with_open_file("outfile", "foo/bar", "w"):
>         code:
>             print >> outfile, foo(...)
>             for i in range(10):
>                 print >> outfile, bar(...)
> 
> (If you need to pass in more than one code block, each one would go in
> a seperate "code:" block.)
> 
> What could the definition look like?
> 
>     defmacro with_open_file(stream, filename, mode)(block1)
>         expand(stream) = file(filename, mode)
>         try:
>             expand(block1)
>         except:
>             perform the necessary closing and clearing up
> 
> (expand would be a function which gets replaced by its argument, at
> macro expansion time.)
> 
> Hmm, just about bearable, I guess, but I doubt that it generalizes
> very well.
> 
> Note that we didn't transform the source code argument, just pasted it
> in directly. If we wanted to manipulate it first, we'd be in for
> serious suffering.
> 
> > You could clearly accomplish this functionality using other means,
> 
> I think it is best to steer clear of such examples, as experience
> shows that some people will not be able to resist the temptation to
> conclude (and suggest) that there is no point to macros, and the
> discussion gets derailed. We've seen enough of "you could do this
> quite easily without macros, but ...", and not enough of "This would
> be soooo painful without macros ..." (I won't say "impossible", as
> there is always Turing completeness.)
> 
> What was wrong with my original examples? I'd like to know, so that I
> can replace them with better ones.

I agree some people have been derailed by that concept, but I'd hoped
that making it painfully clear that this wasn't the only way to do it,
nor was it necessarily a representative sample of Macros, that maybe
some people would see conceptually what Macros are in an extremely
simple way.  The Hello World of Macros.

Again, I thought your examples were fine, I just wanted to simplify
them to help the discussion along.  If you hate them that much, we can
forget I ever posted them. :-p

Doug Tolton