Meta: ideas for Python [-ideas]
Unfortunately, the boilerplate to write an import hook is more complicated than you’d
(Apologies if this appears twice: I attempted to create a new thread using the Web interface but do not see any sign of it having been created.) When people suggest changes to Python's syntax on this list, it is often suggested that they might want to try doing so using an import hook. In fact, this was suggested as recently as two days ago when Andrew Barnert wrote [1] like (and pretty hard to figure out the first time), and the support for filtering on the token stream (the most obvious way to do this one) rather than the text stream, AST, or bytecode is pretty minimal and clumsy. In an attempt to make it easier to create import hooks to "play" with non-standard syntax, I have created a library that is simply called "ideas" [2]. It includes quite a few toy examples meant to illustrate what can be done. It also includes a slightly less minimal and hopefully less clumsy way of filtering the token stream. A reasonably complete documentation is available [3]. I was planning to work some more on it but the recent comment by Andrew Barnert gave me the impetus to finish a first draft and mention it publicly. While it "works for me", consider it to be a pre-alpha release, which is ready for your suggestions, criticisms, and other comments. My hope is that eventually enough examples will be included so that, instead of simply suggesting "why don't you write an import hook for testing your proposed syntax", one could say "you can use transformations similar to that of examples X and Y of ideas to create an import hook for testing your proposed syntax". André Roberge [1] https://mail.python.org/archives/list/python-ideas@python.org/message/UNL62E... [2] Code on github: https://github.com/aroberge/ideas; see the documentation for the choice of name. [3] Documentation: https://aroberge.github.io/ideas/docs/html/
On Sun, Feb 23, 2020 at 1:01 PM Eric Fahlgren <ericfahlgren@gmail.com> wrote:
If you come up with something that you find useful to you, do not hesitate to submit it as an additional example. You don't have to do anything fancy: you can simply email me the code with either a link to a discussion on Python-ideas or a very brief description of what the purpose is. I can take care of the rest. Of course, you can always do more if you want when submitting an example. André
On Feb 23, 2020, at 05:44, André Roberge <andre.roberge@gmail.com> wrote:
In an attempt to make it easier to create import hooks to "play" with non-standard syntax, I have created a library that is simply called "ideas" [2]. It includes quite a few toy examples meant to illustrate what can be done. It also includes a slightly less minimal and hopefully less clumsy way of filtering the token stream. A reasonably complete documentation is available [3].
This is very cool. It’s something I keep wanting to do but never get around to it. I even wrote [a blog post about how to hack on Python without hacking on CPython][1] that was supposed to come with a library like this and some examples, but never wrote the library for it, so instead I just updated it to reference my old super-hacky 3.4 examples, making it a lot less useful. I’ve only skimmed your intro so far, but a couple of early comments. First, one of the parts that always blocked me was coming up with a good way to write token-level transformers. It looks like you solved that; maybe it’s worth pulling the tokenizer tools out into a separate library to post on PyPI separately? Second, I always get sidetracked with the idea that it would be great if we could feed a user-generated token stream into the compiler (the same way we can feed a user-generated AST into the next stage) instead of needing to monkey around with untokenizing. This isn’t too hard to do, but it is a big enough change that it was always hard to convince anyone it was worth it. Your module might change that. Finally, the other tricky bit is making hacks that span levels. For example, a text or token hook transformer that tags things in a way that can be used by an AST or bytecode transformer—or even by an actual CPython patch to the codegen or ceval stages (so you can write the new semantics of your proposed idea, but not have to write the new syntax part). I had a hard time explaining this in my blog post, but maybe you can do better. :) Also, if you haven’t looked at MacroPy, you definitely should. And maybe reference it in the docs. Sometimes it’s so much easier to write something as a MacroPy macro that, even if you have to make the syntax a little more clunky than you wanted, it’s worth doing that way. [1]: http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking...
On Sun, Feb 23, 2020 at 5:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Yes, I was familiar with it; see the bottom of https://aroberge.github.io/ideas/docs/html/motivation.html
I've thought of doing that ... but decided to wait to get more feedback on this entire project and have both the import hook and the tokenizer tools "mature and grow together" a bit.
I doubt it very much as understanding the non-Python code in CPython is well beyond my comfort level.
Also, if you haven’t looked at MacroPy, you definitely should.
I did look at it in the past based on some recommendations (perhaps yours) on Python-ideas. Time to revisit it... And maybe reference it in the docs. Sometimes it’s so much easier to write
On Sun, Feb 23, 2020 at 1:01 PM Eric Fahlgren <ericfahlgren@gmail.com> wrote:
If you come up with something that you find useful to you, do not hesitate to submit it as an additional example. You don't have to do anything fancy: you can simply email me the code with either a link to a discussion on Python-ideas or a very brief description of what the purpose is. I can take care of the rest. Of course, you can always do more if you want when submitting an example. André
On Feb 23, 2020, at 05:44, André Roberge <andre.roberge@gmail.com> wrote:
In an attempt to make it easier to create import hooks to "play" with non-standard syntax, I have created a library that is simply called "ideas" [2]. It includes quite a few toy examples meant to illustrate what can be done. It also includes a slightly less minimal and hopefully less clumsy way of filtering the token stream. A reasonably complete documentation is available [3].
This is very cool. It’s something I keep wanting to do but never get around to it. I even wrote [a blog post about how to hack on Python without hacking on CPython][1] that was supposed to come with a library like this and some examples, but never wrote the library for it, so instead I just updated it to reference my old super-hacky 3.4 examples, making it a lot less useful. I’ve only skimmed your intro so far, but a couple of early comments. First, one of the parts that always blocked me was coming up with a good way to write token-level transformers. It looks like you solved that; maybe it’s worth pulling the tokenizer tools out into a separate library to post on PyPI separately? Second, I always get sidetracked with the idea that it would be great if we could feed a user-generated token stream into the compiler (the same way we can feed a user-generated AST into the next stage) instead of needing to monkey around with untokenizing. This isn’t too hard to do, but it is a big enough change that it was always hard to convince anyone it was worth it. Your module might change that. Finally, the other tricky bit is making hacks that span levels. For example, a text or token hook transformer that tags things in a way that can be used by an AST or bytecode transformer—or even by an actual CPython patch to the codegen or ceval stages (so you can write the new semantics of your proposed idea, but not have to write the new syntax part). I had a hard time explaining this in my blog post, but maybe you can do better. :) Also, if you haven’t looked at MacroPy, you definitely should. And maybe reference it in the docs. Sometimes it’s so much easier to write something as a MacroPy macro that, even if you have to make the syntax a little more clunky than you wanted, it’s worth doing that way. [1]: http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking...
On Sun, Feb 23, 2020 at 5:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Yes, I was familiar with it; see the bottom of https://aroberge.github.io/ideas/docs/html/motivation.html
I've thought of doing that ... but decided to wait to get more feedback on this entire project and have both the import hook and the tokenizer tools "mature and grow together" a bit.
I doubt it very much as understanding the non-Python code in CPython is well beyond my comfort level.
Also, if you haven’t looked at MacroPy, you definitely should.
I did look at it in the past based on some recommendations (perhaps yours) on Python-ideas. Time to revisit it... And maybe reference it in the docs. Sometimes it’s so much easier to write
participants (3)
-
Andrew Barnert -
André Roberge -
Eric Fahlgren