On 01/17/2016 12:48 PM, Victor Stinner wrote:
2016-01-16 12:06 GMT+01:00 Petr Viktorin <encukou@gmail.com <javascript:;>>:
This PEP addresses two things that would benefit from different approaches: let's call them optimizers and extensions.
Optimizers, such as your FAT, don't change Python semantics. They're designed to run on *all* code, including the standard library. It makes sense to register them as early in interpreter startup as possible, but if they're not registered, nothing breaks (things will just be slower). Experiments with future syntax (like when async/await was being developed) have the same needs.
Syntax extensions, such as MacroPy or Hy, tend to target specific modules, with which they're closely coupled: The modules won't run without the transformer. And with other modules, the transformer either does nothing (as with MacroPy, hopefully), or would fail altogether (as with Hy). So, they would benefit from specific packages opting in. The effects of enabling them globally range from inefficiency (MacroPy) to failures or needing workarounds (Hy).
To be clear, Hylang will not benefit from my PEP. That's why it is not mentioned in the PEP.
"Syntax extensions" only look like a special case of optimizers. I'm not sure that it's worth to make them really different.
There is an important difference: optimizers should be installed globally. But modules that don't opt in to a specific syntax extension should not get compiled with it.
The PEP is designed optimizers. It would be good to stick to that use case, at least as far as the registration is concerned. I suggest noting in the documentation that Python semantics *must* be preserved, and renaming the API, e.g.::
My API examples seem to have led the conversation astray. The point I wanted to make is that "syntax extensions" need a registration API that only enables them for specific modules. I admit the particular examples weren't very well thought out. I'm not proposing adding *any* of them to the PEP: I'd be happy if the PEP stuck to the "optimizers" use case and do that well. The "extensions" case is worth another PEP, which can reuse the transformers API (probably integrating it with importlib), but not the registration API.
I would prefer to do that differently: always register transformers very early, but configure each transformer to only apply it on some files. The transformer can use the filename (file extension? importlib is currently restricted to .py files by default no?), it can use a special variable in the file (ex: fatoptimizer searchs for a __fatoptimizer__ variable which is used to configure the optimizer), a configuration loaded when the transformer is created, etc.
Why very early? If a syntax extension is used in some package, it should only be activated right before that package is imported. And ideally it shouldn't get a chance to be activated on other packages. importlib is not restricted to .py (it can do .zip, .pyc, .so, etc. out of the box). Actually, with import hooks, the *source* file that uses the DSL can use a different extension (as opposed to the *.pyc getting a different tag, as for optimizers). For example, a library using a SQL DSL could look like:: __init__.py (imports a package to set up the transformer) queries.sqlpy __pycache__/ __init__.cpython-36.opt-0.pyc queries.cpython-36.opt-0.pyc That is probably what you want for syntax extensions. You can't really look at special variables in the file, because the transformer needs to be enabled before the code is compiled -- especially if text/tokenstream transforms are added, so the file might not be valid "vanilla Python". What's left is making it easy to register an import hook with a specific PEP 511 transformer -- but again, that can be a different PEP.