Macros for Python

I thought this may be of interest to some people on this list, even if not strictly an "idea". I'm working on MacroPy <https://github.com/lihaoyi/macropy>, a little pure-python library that allows user-defined AST rewrites as part of the import process (using PEP 302). In short, it makes mucking around with Python's semantics so easy as to be almost trivial: you write a function that takes an AST and returns an AST, register it as a macro, and you're off to the races. To give a sense of it, I just finished implementing Scala/Groovy style anonymous lambdas: map(f%(_ + 1), [1, 2, 3])#[2, 3, 4] reduce(f%(_ + _), [1, 2, 3])#6 ...which took about half an hour and 30 lines of code, start to finish. We're currently working on implementing destructuring-pattern-matching on objects (i.e. like in Haskell/Scala) and a clone of .NET's LINQ to SQL. It's still very much a work in progress, but we have a list of pretty cool macros already done, which shows off what you can do with it. If anyone else was thinking about messing around with the semantics of the Python language but was too scared to jump into the CPython internals, this offers a somewhat easier path. Hope this was interesting to somebody! -Haoyi

On Wed, Apr 24, 2013 at 6:49 AM, Haoyi Li <haoyi.sg@gmail.com> wrote:
you write a function that takes an AST and returns an AST, register it as a macro, and you're off to the races.
Insane and insanely brilliant. I'm not going to touch this until a lot of smoke blows over.
a clone of .NET's LINQ to SQL.
Sounds awesome. This is completely uncharted territory for me. I'd love to hear how this pans out in a year or so. Seems like it's very powerful and can help a lot in shaping Python based Domain Specific Languages, but perhaps too powerful to the point where it'll end up a horrific and complicated bug magnet. Yuval Greenfield

On 4/23/2013 11:49 PM, Haoyi Li wrote:
From the readme ''' String Interpolation a, b = 1, 2 c = s%"%{a} apple and %{b} bananas" print c #1 apple and 2 bananas ''' I am a little surprised that you would base a cutting edge extension on Py 2. Do you have it working with 3.3 also? '''Unlike the normal string interpolation in Python, MacroPy's string interpolation allows the programmer to specify the variables to be interpolated inline inside the string.''' Not true as I read that. a, b = 1, 2 print("{a} apple and {b} bananas".format(**locals())) print("%(a)s apple and %(b)s bananas" % locals()) #1 apple and 2 bananas #1 apple and 2 bananas I rather like the anon funcs with anon params. That only works when each param is only used once in the expression, but that restriction is the normal case. I am interested to see what you do with pattern matching. tjr

>I am a little surprised that you would base a cutting edge extension on Py 2. Do you have it working with 3.3 also? It's not really a cutting edge extension yet, it's more a completely-crazy "you did WHAT?" proof of concept to explore the space of possibilities. 2.7 was what we had installed, so we just ran with it. Haven't done any testing at all on 3.4, but if the project turns out well (i.e. the functionality is actually usable, and people are interested) we could look at porting it. I don't think the core of the system will change much, but the individual macros may have to be re-written since the ASTs are slightly different. > a, b = 1, 2 print("{a} apple and {b} bananas".format(**locals())) print("%(a)s apple and %(b)s bananas" % locals()) Yes, you can do it like that. You can't do more complex stuff though, like "%{a ** b} is %{a} to the power of %{b}" Perhaps I should put it in the readme, since I already have a unit test for it. You actually can get a syntax like that without macros, using stack-introspection, locals-trickery and lots of `eval`. The question is whether you consider macros more "extreme" than stack-introspection, locals-trickery and `eval`! A JIT compiler will probably be much happier with macros. On Wed, Apr 24, 2013 at 10:35 AM, Terry Jan Reedy <tjreedy@udel.edu> wrote: > On 4/23/2013 11:49 PM, Haoyi Li wrote: > >> I thought this may be of interest to some people on this list, even if >> not strictly an "idea". >> >> I'm working on MacroPy <https://github.com/lihaoyi/**macropy<https://github.com/lihaoyi/macropy>>, >> a little >> >> pure-python library that allows user-defined AST rewrites as part of the >> import process (using PEP 302). >> > > From the readme > ''' > String Interpolation > > a, b = 1, 2 > c = s%"%{a} apple and %{b} bananas" > print c > #1 apple and 2 bananas > ''' > I am a little surprised that you would base a cutting edge extension on Py > 2. Do you have it working with 3.3 also? > > '''Unlike the normal string interpolation in Python, MacroPy's string > interpolation allows the programmer to specify the variables to be > interpolated inline inside the string.''' > > Not true as I read that. > > a, b = 1, 2 > print("{a} apple and {b} bananas".format(**locals())) > print("%(a)s apple and %(b)s bananas" % locals()) > #1 apple and 2 bananas > #1 apple and 2 bananas > > I rather like the anon funcs with anon params. That only works when each > param is only used once in the expression, but that restriction is the > normal case. > > I am interested to see what you do with pattern matching. > > tjr > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas@python.org > http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas> >

On Apr 24, 2013, at 8:05, Haoyi Li <haoyi.sg@gmail.com> wrote:
You actually can get a syntax like that without macros, using stack-introspection, locals-trickery and lots of `eval`. The question is whether you consider macros more "extreme" than stack-introspection, locals-trickery and `eval`! A JIT compiler will probably be much happier with macros.
That last point makes this approach seem particularly interesting to me, which makes me wonder: Is your code CPython specific, or does it also work with PyPy (or Jython or Iron)? While PyPy is obviously a whole lot easier to mess with in the first place than CPython, having macros at the same language level as your code is just as interesting in both implementations.

I haven't tested in on various platforms, so hard to say for sure. MacroPy basically relies on a few things: - exec/eval - PEP 302 - the ast module All of these are pretty old pieces of python (almost 10 years old!) so it's not some new-and-fancy functionality. Jython seems to have all of them, I couldn't find any information about PyPy. When the project is more mature and I have some time, I'll see if I can get it to work cross platform. If anyone wants to fork the repo and try it out, that'd be great too! -Haoyi On Wed, Apr 24, 2013 at 11:55 AM, Andrew Barnert <abarnert@yahoo.com> wrote:

One use case I have is for Twisted's inlineCallbacks. I forked the pypy project to implement the await-keyword. Basically it transforms: def async_function(deferred_param): a = await deferred_param b = await some_call(a) return b into: @defer.inlineCallbacks def async_function(deferred_param): a = yield deferred_param b = yield some_call(a) yield defer.returnValue(b) Are such things possible? And if so, what lines of code would pdb show during introspection of the code? It's interesting, but when macros become more complicated, the debugging of these things can turn out to be really hard, I think. 2013/4/24 Haoyi Li <haoyi.sg@gmail.com>:

@Jonathan: That would be possible, although I can't say I know how to do it. A naive macro that wraps everything and has a "substitute awaits for yields, wrap them in inlineCallbacks(), and substitute returns for returnValue()s" may work, but I'm guessing it would run into a forest of edge cases where the code isn't so simple (what if you *want* a return? etc.). pdb *should* show the code after macro expansion. Without source maps, I'm not sure there's any way around that, so debugging may be hard. Of course, if the alternative is macros of forking the interpreter, maybe macros is the easier way to do it =) Debugging a buggy custom-forked interpreter probably isn't easy either! On Wed, Apr 24, 2013 at 5:48 PM, Jonathan Slenders <jonathan@slenders.be>wrote:

I pushed a simple implementation of case classes<https://github.com/lihaoyi/macropy#case-classes> using Macros, as well as a really nice to use parser combinator library<https://github.com/lihaoyi/macropy#parser-combinators>. The case classes are interesting because they overlap a lot with enumerations: auto-generated __str__, __repr__, inheritence via nesting, they can have members and methods, etc. They also show off pretty well how far Python's syntax (and semantic!) can be stretched using macros, so if anyone still has some crazy ideas for enumerations and wants to prototype them without hacking the CPython interpreter, this is your chance! Thanks! -Haoyi On Wed, Apr 24, 2013 at 3:15 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

Just an update for people who are interested, The project (https://github.com/lihaoyi/macropy) is more or less done for now, in its current state as a proof of concept/demo. Almost all of it runs perfectly on both CPython and PyPy, except for the pattern matcher which has some bugs on PyPy we haven't ironed out yet. Jython doesn't work at all: it seems to handle a number of things about the ast module pretty differently from either PyPy or CPython. We've got a pretty impressive list of feature demos: - Quasiquotes <https://github.com/lihaoyi/macropy#quasiquotes>, a quick way to manipulate fragments of a program - String Interpolation<https://github.com/lihaoyi/macropy#string-interpolation>, a common feature in many languages - Pyxl <https://github.com/lihaoyi/macropy#pyxl-integration>, integrating XML markup into a Python program - Tracing <https://github.com/lihaoyi/macropy#tracing> and Smart Asserts<https://github.com/lihaoyi/macropy#smart-asserts> - Case Classes <https://github.com/lihaoyi/macropy#case-classes>, easy Algebraic Data Types <https://en.wikipedia.org/wiki/Algebraic_data_type> from Scala - Pattern Matching <https://github.com/lihaoyi/macropy#pattern-matching> from the Functional Programming world - LINQ to SQL <https://github.com/lihaoyi/macropy#linq-to-sql> from C# - Quick Lambdas <https://github.com/lihaoyi/macropy#quick-lambdas> from Scala and Groovy, - Parser Combinators<https://github.com/lihaoyi/macropy#parser-combinators>, inspired by Scala's<http://www.suryasuravarapu.com/2011/04/scala-parser-combinators-win.html> . And have pushed a release to PyPI (https://pypi.python.org/pypi/MacroPy), to make it easier for people to download it and mess around. Hopefully somebody will find this useful in messing around with the Python language! Thanks! -Haoyi On Sat, Apr 27, 2013 at 11:05 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

MacroPy is awesome both conceptually and feature-demo-wise - thanks for working on and sharing this! Simple, elegant things are hardest to come by - to utilize module loading hooks for AST transformation seems really natural from hindsight, except no-one did that before. On Wed, May 8, 2013 at 11:04 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

Am 13.05.2013 19:02, schrieb Mart Sõmermaa:
*cough* https://bitbucket.org/birkenfeld/karnickel Georg

We were aware of Karnickel before we started, along with MetaPython ( https://code.google.com/p/metapython/) and Pyxl ( https://github.com/dropbox/pyxl) Apart from being abandoned, neither of the first two really demonstrates any usability (although Pyxl is used quite heavily), which is why we went ahead with MacroPy. On Mon, May 13, 2013 at 1:06 PM, Georg Brandl <g.brandl@gmx.net> wrote:

On Wed, Apr 24, 2013 at 6:49 AM, Haoyi Li <haoyi.sg@gmail.com> wrote:
you write a function that takes an AST and returns an AST, register it as a macro, and you're off to the races.
Insane and insanely brilliant. I'm not going to touch this until a lot of smoke blows over.
a clone of .NET's LINQ to SQL.
Sounds awesome. This is completely uncharted territory for me. I'd love to hear how this pans out in a year or so. Seems like it's very powerful and can help a lot in shaping Python based Domain Specific Languages, but perhaps too powerful to the point where it'll end up a horrific and complicated bug magnet. Yuval Greenfield

On 4/23/2013 11:49 PM, Haoyi Li wrote:
From the readme ''' String Interpolation a, b = 1, 2 c = s%"%{a} apple and %{b} bananas" print c #1 apple and 2 bananas ''' I am a little surprised that you would base a cutting edge extension on Py 2. Do you have it working with 3.3 also? '''Unlike the normal string interpolation in Python, MacroPy's string interpolation allows the programmer to specify the variables to be interpolated inline inside the string.''' Not true as I read that. a, b = 1, 2 print("{a} apple and {b} bananas".format(**locals())) print("%(a)s apple and %(b)s bananas" % locals()) #1 apple and 2 bananas #1 apple and 2 bananas I rather like the anon funcs with anon params. That only works when each param is only used once in the expression, but that restriction is the normal case. I am interested to see what you do with pattern matching. tjr

>I am a little surprised that you would base a cutting edge extension on Py 2. Do you have it working with 3.3 also? It's not really a cutting edge extension yet, it's more a completely-crazy "you did WHAT?" proof of concept to explore the space of possibilities. 2.7 was what we had installed, so we just ran with it. Haven't done any testing at all on 3.4, but if the project turns out well (i.e. the functionality is actually usable, and people are interested) we could look at porting it. I don't think the core of the system will change much, but the individual macros may have to be re-written since the ASTs are slightly different. > a, b = 1, 2 print("{a} apple and {b} bananas".format(**locals())) print("%(a)s apple and %(b)s bananas" % locals()) Yes, you can do it like that. You can't do more complex stuff though, like "%{a ** b} is %{a} to the power of %{b}" Perhaps I should put it in the readme, since I already have a unit test for it. You actually can get a syntax like that without macros, using stack-introspection, locals-trickery and lots of `eval`. The question is whether you consider macros more "extreme" than stack-introspection, locals-trickery and `eval`! A JIT compiler will probably be much happier with macros. On Wed, Apr 24, 2013 at 10:35 AM, Terry Jan Reedy <tjreedy@udel.edu> wrote: > On 4/23/2013 11:49 PM, Haoyi Li wrote: > >> I thought this may be of interest to some people on this list, even if >> not strictly an "idea". >> >> I'm working on MacroPy <https://github.com/lihaoyi/**macropy<https://github.com/lihaoyi/macropy>>, >> a little >> >> pure-python library that allows user-defined AST rewrites as part of the >> import process (using PEP 302). >> > > From the readme > ''' > String Interpolation > > a, b = 1, 2 > c = s%"%{a} apple and %{b} bananas" > print c > #1 apple and 2 bananas > ''' > I am a little surprised that you would base a cutting edge extension on Py > 2. Do you have it working with 3.3 also? > > '''Unlike the normal string interpolation in Python, MacroPy's string > interpolation allows the programmer to specify the variables to be > interpolated inline inside the string.''' > > Not true as I read that. > > a, b = 1, 2 > print("{a} apple and {b} bananas".format(**locals())) > print("%(a)s apple and %(b)s bananas" % locals()) > #1 apple and 2 bananas > #1 apple and 2 bananas > > I rather like the anon funcs with anon params. That only works when each > param is only used once in the expression, but that restriction is the > normal case. > > I am interested to see what you do with pattern matching. > > tjr > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas@python.org > http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas> >

On Apr 24, 2013, at 8:05, Haoyi Li <haoyi.sg@gmail.com> wrote:
You actually can get a syntax like that without macros, using stack-introspection, locals-trickery and lots of `eval`. The question is whether you consider macros more "extreme" than stack-introspection, locals-trickery and `eval`! A JIT compiler will probably be much happier with macros.
That last point makes this approach seem particularly interesting to me, which makes me wonder: Is your code CPython specific, or does it also work with PyPy (or Jython or Iron)? While PyPy is obviously a whole lot easier to mess with in the first place than CPython, having macros at the same language level as your code is just as interesting in both implementations.

I haven't tested in on various platforms, so hard to say for sure. MacroPy basically relies on a few things: - exec/eval - PEP 302 - the ast module All of these are pretty old pieces of python (almost 10 years old!) so it's not some new-and-fancy functionality. Jython seems to have all of them, I couldn't find any information about PyPy. When the project is more mature and I have some time, I'll see if I can get it to work cross platform. If anyone wants to fork the repo and try it out, that'd be great too! -Haoyi On Wed, Apr 24, 2013 at 11:55 AM, Andrew Barnert <abarnert@yahoo.com> wrote:

One use case I have is for Twisted's inlineCallbacks. I forked the pypy project to implement the await-keyword. Basically it transforms: def async_function(deferred_param): a = await deferred_param b = await some_call(a) return b into: @defer.inlineCallbacks def async_function(deferred_param): a = yield deferred_param b = yield some_call(a) yield defer.returnValue(b) Are such things possible? And if so, what lines of code would pdb show during introspection of the code? It's interesting, but when macros become more complicated, the debugging of these things can turn out to be really hard, I think. 2013/4/24 Haoyi Li <haoyi.sg@gmail.com>:

@Jonathan: That would be possible, although I can't say I know how to do it. A naive macro that wraps everything and has a "substitute awaits for yields, wrap them in inlineCallbacks(), and substitute returns for returnValue()s" may work, but I'm guessing it would run into a forest of edge cases where the code isn't so simple (what if you *want* a return? etc.). pdb *should* show the code after macro expansion. Without source maps, I'm not sure there's any way around that, so debugging may be hard. Of course, if the alternative is macros of forking the interpreter, maybe macros is the easier way to do it =) Debugging a buggy custom-forked interpreter probably isn't easy either! On Wed, Apr 24, 2013 at 5:48 PM, Jonathan Slenders <jonathan@slenders.be>wrote:

I pushed a simple implementation of case classes<https://github.com/lihaoyi/macropy#case-classes> using Macros, as well as a really nice to use parser combinator library<https://github.com/lihaoyi/macropy#parser-combinators>. The case classes are interesting because they overlap a lot with enumerations: auto-generated __str__, __repr__, inheritence via nesting, they can have members and methods, etc. They also show off pretty well how far Python's syntax (and semantic!) can be stretched using macros, so if anyone still has some crazy ideas for enumerations and wants to prototype them without hacking the CPython interpreter, this is your chance! Thanks! -Haoyi On Wed, Apr 24, 2013 at 3:15 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

Just an update for people who are interested, The project (https://github.com/lihaoyi/macropy) is more or less done for now, in its current state as a proof of concept/demo. Almost all of it runs perfectly on both CPython and PyPy, except for the pattern matcher which has some bugs on PyPy we haven't ironed out yet. Jython doesn't work at all: it seems to handle a number of things about the ast module pretty differently from either PyPy or CPython. We've got a pretty impressive list of feature demos: - Quasiquotes <https://github.com/lihaoyi/macropy#quasiquotes>, a quick way to manipulate fragments of a program - String Interpolation<https://github.com/lihaoyi/macropy#string-interpolation>, a common feature in many languages - Pyxl <https://github.com/lihaoyi/macropy#pyxl-integration>, integrating XML markup into a Python program - Tracing <https://github.com/lihaoyi/macropy#tracing> and Smart Asserts<https://github.com/lihaoyi/macropy#smart-asserts> - Case Classes <https://github.com/lihaoyi/macropy#case-classes>, easy Algebraic Data Types <https://en.wikipedia.org/wiki/Algebraic_data_type> from Scala - Pattern Matching <https://github.com/lihaoyi/macropy#pattern-matching> from the Functional Programming world - LINQ to SQL <https://github.com/lihaoyi/macropy#linq-to-sql> from C# - Quick Lambdas <https://github.com/lihaoyi/macropy#quick-lambdas> from Scala and Groovy, - Parser Combinators<https://github.com/lihaoyi/macropy#parser-combinators>, inspired by Scala's<http://www.suryasuravarapu.com/2011/04/scala-parser-combinators-win.html> . And have pushed a release to PyPI (https://pypi.python.org/pypi/MacroPy), to make it easier for people to download it and mess around. Hopefully somebody will find this useful in messing around with the Python language! Thanks! -Haoyi On Sat, Apr 27, 2013 at 11:05 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

MacroPy is awesome both conceptually and feature-demo-wise - thanks for working on and sharing this! Simple, elegant things are hardest to come by - to utilize module loading hooks for AST transformation seems really natural from hindsight, except no-one did that before. On Wed, May 8, 2013 at 11:04 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:

Am 13.05.2013 19:02, schrieb Mart Sõmermaa:
*cough* https://bitbucket.org/birkenfeld/karnickel Georg

We were aware of Karnickel before we started, along with MetaPython ( https://code.google.com/p/metapython/) and Pyxl ( https://github.com/dropbox/pyxl) Apart from being abandoned, neither of the first two really demonstrates any usability (although Pyxl is used quite heavily), which is why we went ahead with MacroPy. On Mon, May 13, 2013 at 1:06 PM, Georg Brandl <g.brandl@gmx.net> wrote:
participants (7)
-
Andrew Barnert
-
Georg Brandl
-
Haoyi Li
-
Jonathan Slenders
-
Mart Sõmermaa
-
Terry Jan Reedy
-
Yuval Greenfield