On Sun, Feb 23, 2020 at 5:17 PM Andrew Barnert
On Feb 23, 2020, at 05:44, André Roberge
wrote: In an attempt to make it easier to create import hooks to "play" with non-standard syntax, I have created a library that is simply called "ideas" [2]. It includes quite a few toy examples meant to illustrate what can be done. It also includes a slightly less minimal and hopefully less clumsy way of filtering the token stream. A reasonably complete documentation is available [3].
This is very cool. It’s something I keep wanting to do but never get around to it. I even wrote [a blog post about how to hack on Python without hacking on CPython][1]
Yes, I was familiar with it; see the bottom of https://aroberge.github.io/ideas/docs/html/motivation.html
I’ve only skimmed your intro so far, but a couple of early comments.
First, one of the parts that always blocked me was coming up with a good way to write token-level transformers. It looks like you solved that; maybe it’s worth pulling the tokenizer tools out into a separate library to post on PyPI separately?
I've thought of doing that ... but decided to wait to get more feedback on this entire project and have both the import hook and the tokenizer tools "mature and grow together" a bit.
Second, I always get sidetracked with the idea that it would be great if we could feed a user-generated token stream into the compiler (the same way we can feed a user-generated AST into the next stage) instead of needing to monkey around with untokenizing. This isn’t too hard to do, but it is a big enough change that it was always hard to convince anyone it was worth it. Your module might change that.
Finally, the other tricky bit is making hacks that span levels. For example, a text or token hook transformer that tags things in a way that can be used by an AST or bytecode transformer—or even by an actual CPython patch to the codegen or ceval stages (so you can write the new semantics of your proposed idea, but not have to write the new syntax part). I had a hard time explaining this in my blog post, but maybe you can do better. :)
I doubt it very much as understanding the non-Python code in CPython is well beyond my comfort level.
Also, if you haven’t looked at MacroPy, you definitely should.
I did look at it in the past based on some recommendations (perhaps yours) on Python-ideas. Time to revisit it... And maybe reference it in the docs. Sometimes it’s so much easier to write
something as a MacroPy macro that, even if you have to make the syntax a little more clunky than you wanted, it’s worth doing that way. [1]: http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking... http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking...