On 4/19/2021 12:44 PM, Guido van Rossum wrote:

We had a similar thing at Dropbox, where `# coding: pyxl` would enable a preprocessor that allowed HTML embedded in the Python code. It translated this to function calls and string literals.

There were however several drawbacks:

- Installing the codec is a bit tricky, and if you don't have it the code can't run - Other tooling won't know about the new syntax (someone went through heroic efforts to write a PyCharm extension for pyxl but it got stale very quickly) - The codec slows down the import process - Keeping line numbers accurate in the codec is painful - Occasionally, when debugging mysteries, you end up needing to see the source code after preprocessing, which requires yet another custom tool

In the end the project was withdrawn and we switched to more mainstream templating solutions.

I suspect that many of the issues with pyxl would also plague using this approach as a way to customize typing syntax, and I don't think it would be an improvement over the status quo.

How does this "similar thing" compare to the recently announced imphook module?

Hello, On Mon, 19 Apr 2021 20:23:08 -0700 Guido van Rossum <guido@python.org> wrote: []

...

How does this "similar thing" compare to the recently announced imphook module?

For one, pyxl is a better name. :-)

I would humbly disagree ;-).

Seriously, as long as the purpose is to allow using a different grammar (for part of the file) that is then transpiled to proper Python, all but the first issue apply exactly the same.

That's one of the biggest differences - imphook (https://pypi.org/project/imphook/) written with the idea that macro capabilities have always been first-class citizen of the Python ecosystem (well, maybe 25 years of its 30 years). But it's also written with the full understanding and acknowledgment that there's a disconnect between advanced Python users, who have been using Python macro capabilities for decades, and mainstream community. For the latter, the use of these facilities was always complicated and esoteric. And given the attitude of some core CPython developers (examples: self-policing like in PEP511 or telling pyxl anecdotes), there's suspicion that the state of complication and confusion is held on the purpose (of course, with the good intentions of saving users from even more confusion).

Getting the hook to run (my first bullet) actually sounds *more* complicated, because you have to include code in your `__main__` module that installs the hook(s).

Right, importing a module is a complicated task for Python users, or should be presented as such. Well, then there's also a runner which allows to not patch your code to use macro facilities: python3 -m imphook -i mod_conf example_conf.py But that's command line, so of course it's still complicated. Well, maybe not that much, and people who were able to deal with "python2", "python3", "python3.5", "python3.6", "py", etc. can deal with that too.

But I have zero experience with imphook, I just skimmed its PyPI home page, which has an example that calls ``` imphook.add_import_hook(hook, (".conf",)) ```

That's the code which needs to be run by "macro" implementations, not by users of a particular "macro". -- Best regards, Paul mailto:pmiscml@gmail.com

Hello, On Tue, 20 Apr 2021 11:08:56 +0200 Victor Stinner <vstinner@python.org> wrote:

I proposed PEP 511 "API for code transformers" for Python 3.6 (in 2016) and it was rejected: https://www.python.org/dev/peps/pep-0511/#rejection-notice

Well, it wasn't rejected, it was self-rejected on the thought-crime grounds. It's however a big philosophical question what's worse for Python - adding questionable re: implementation aspects features (PEP649), NIH way to handle exceptions (PEP654) (a couple of random examples, really), - or embrace macro-level extensibility, which is usually a part of any advanced programming language out there. (And which is de-facto available for Python either). []

IMO the most important part of the PEP 511 was to change the pyc filename depending on the code transformer used to modify the code.

Right, for "production" macro usage, the question of tracking macro dependencies for precompiled bytecode (and invalidating it if dependencies changed) is important. PEP 638 (https://www.python.org/dev/peps/pep-0638/) also touches on that, but I also don't see it to be well thought out/articulated. Instead of adhoc version/flavor identifiers (which in large-scale will likely lead to version hell as we know), it literally should be that a .pyc file can depend on multiple (not one, like now) .py files (or maybe module names). [] -- Best regards, Paul mailto:pmiscml@gmail.com