[Python-ideas] PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not

[Andrew Barnert]

On Jan 27, 2016, at 07:39, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> Hi,
> 
> Thank you for all feedback on my PEP 511. It looks like the current
> blocker point is the unclear status of "language extensions": code
> tranformers which deliberately changes the Python semantics. I would
> like to discuss how we should register them. I think that the PEP 511
> must discuss "language extensions" even if it doesn't have to propose
> a solution to make their usage easier. It's an obvious usage of code
> transformers. If possible, I would like to find a compromise to
> support them, but make it explicit that they change the Python
> semantics.

Is this really necessary?

If someone is testing a language change locally, and just wants to use your (original) API for his tests instead of the more complicated alternative of building an import hook, it works fine. If he can't deploy that way, that's fine.

If someone builds a transformer that adds a feature in a way that makes it a pure superset of Python, he should be fine with running it on all files, so your API works fine. And if some files that didn't use any of the new features get .pyc files that imply they did, so what?

If someone builds a transformer that only runs on files with a different extension, he already needs an import hook, so he might as well just call his transformer from the input hook, same as he does today.

So... What case is served by this new, more complicated API that wasn't already served by your original, simple one (remembering that import hooks are already there as a fallback)?

> By the way, I discussed with Joseph Jevnik who wrote codetransformer
> (bytecode transformer) and lazy_python (AST transformer). He wrote me:
> 
> "One concern that I have though is that transformers are registered
> globally. I think that the decorators in codetransformer do a good job
> of signalling to reader the scope of some new code generation."
> 
> Currently, the PEP 511 doesn't provide a way to register a code
> transformer but only use it under some conditions. For example, if
> fatoptimizer is registered, all .pyc files will be called
> file.cpython-36.fat-0.pyc even if fatoptimizer was disabled.

That doesn't really answer his question, unless you're trying to add some syntax that's like a decorator but for an entire module, to be used in addition to the existing more local class and function decorators?

> Petr Viktorin wrote that language extensions "target specific modules,
> with which they're closely coupled: The modules won't run without the
> transformer. And with other modules, the transformer either does
> nothing (as with MacroPy, hopefully), or would fail altogether (as
> with Hy). So, they would benefit from specific packages opting in. The
> effects of enabling them globally range from inefficiency (MacroPy) to
> failures or needing workarounds (Hy)."

It seems like you're trying to find a declarative alternative to every possible use for an imperative import hook. If you can pull that off, it would be cool--but is it really necessary for your proposal?

Does your solution have to make it possible for MacroPy and Hy to drop their complicated import hooks and just register transformers, for it to be a useful solution?

If the problem you're trying to solve is just making it easier for MacroPy and Hy to coexist with the new transformers, maybe just solve that. For example, if it's too hard for them to decorate .pyc names in a way that fits in with your system, maybe adding a function to get the pre-hook pyc name and to set the post-hook one (e.g., to insert "-pymacro-" in the middle of it) would be sufficient.

If there's something that can't be solved in a similar way--e.g., if you think your proposal has to make macropy.console (or whatever he calls the "macros in the REPL" feature) either automatic or at least a lot easier--then maybe that's a different story, but it would be nice to see the rationale for why we need to solve that today. (Couldn't it be added in 3.7, after people have gotten experience with using 3.6 transformers?)