2016-01-16 12:06 GMT+01:00 Petr Viktorin <<a href="javascript:;" onclick="_e(event, 'cvml', 'encukou@gmail.com')">encukou@gmail.com</a>>:<br>
> This PEP addresses two things that would benefit from different<br>
> approaches: let's call them optimizers and extensions.<br>
><br>
> Optimizers, such as your FAT, don't change Python semantics. They're<br>
> designed to run on *all* code, including the standard library. It makes<br>
> sense to register them as early in interpreter startup as possible, but<br>
> if they're not registered, nothing breaks (things will just be slower).<br>
> Experiments with future syntax (like when async/await was being<br>
> developed) have the same needs.<br>
><br>
> Syntax extensions, such as MacroPy or Hy, tend to target specific<br>
> modules, with which they're closely coupled: The modules won't run<br>
> without the transformer. And with other modules, the transformer either<br>
> does nothing (as with MacroPy, hopefully), or would fail altogether (as<br>
> with Hy). So, they would benefit from specific packages opting in. The<br>
> effects of enabling them globally range from inefficiency (MacroPy) to<br>
> failures or needing workarounds (Hy).<br>
<br>
To be clear, Hylang will not benefit from my PEP. That's why it is not mentioned in the PEP.<br>
<br>
"Syntax extensions" only look like a special case of optimizers. I'm not sure that it's worth to make them really different.<br>
<br>
> The PEP is designed optimizers. It would be good to stick to that use<br>
> case, at least as far as the registration is concerned. I suggest noting<br>
> in the documentation that Python semantics *must* be preserved, and<br>
> renaming the API, e.g.::<br>
><br>
> sys.set_global_optimizers([])<br>
<br>
I would prefer to not restrict the PEP to a specific usage.<br>
<br>
> The "transformer" API can be used for syntax extensions as well, but the<br>
> registration needs to be different so the effects are localized. For<br>
> example it could be something like::<br>
><br>
> importlib.util.import_with_transformer(<br>
> 'mypackage.specialmodule', MyTransformer())<br>
<br>
Brett may help on this part. I don't think that it's the best way to use importlib. importlib is already pluggable. As I wrote in the PEP, MacroPy uses an import hook. (Maybe it should continue to use an import hook?)<br>
<br>
> or a special flag in packages::<br>
><br>
> __transformers_for_submodules__ = [MyTransformer()]<br>
<br>
Does it mean that you have to parse a .py file to then decide how to transform it? It will slow down compilation of code not using transformers.<br>
<br>
I would prefer to do that differently: always register transformers very early, but configure each transformer to only apply it on some files. The transformer can use the filename (file extension? importlib is currently restricted to .py files by default no?), it can use a special variable in the file (ex: fatoptimizer searchs for a __fatoptimizer__ variable which is used to configure the optimizer), a configuration loaded when the transformer is created, etc.<br>
<br>
> or extendeding exec (which you actually might want to add to the PEP, to<br>
> make giving examples easier)::<br>
><br>
> exec("print('Hello World!')", transformers=[MyTransformer()])<br>
<br>
There are a lot of ways to load, compile and execute code. Starting to add optional parameters will end as my old PEP 410 ( <a href="https://www.python.org/dev/peps/pep-0410/" target="_blank">https://www.python.org/dev/peps/pep-0410/</a> ) which was rejected because it added an optional parameter a lot of functions (at least 16 functions!). (It was not the only reason to reject the PEP.)<br>
<br>
Brett Canon proposed to add hooks to importlib, but it would restrict the feature to imports. See use cases in the PEP, I would like to use the same code transformers everywhere.<br>
<br>
> Another thing: this snippet from the PEP sounds too verbose::<br>
><br>
> transformers = sys.get_code_transformers()<br>
> transformers.insert(0, new_cool_transformer)<br>
> sys.set_code_transformers(transformers)<br>
><br>
> Can this just be a list, as with sys.path? Using the "optimizers" term::<br>
><br>
> sys.global_optimizers.insert(0, new_cool_transformer)<br>
<br>
set_code_transformers() checks the transformer name and ensures that the transformer has at least a AST transformer or a bytecode transformer. That's why it's a function and not a simple list.<br>
<br>
set_code_transformers() also gets the AST and bytecode transformers methods only once, to provide a simple C structure for PyAST_CompileObject (bytecode transformers) and PyParser_ASTFromStringObject (AST transformers).<br>
<br>
Note: sys.implementation.cache_tag is modifiable without any check. If you mess it, importlib will probably fail badly. And the newly added sys.implementation.optim_tag can also be modified without any check.<br>
<br>
> This::<br>
><br>
> def code_transformer(code, consts, names, lnotab, context):<br>
><br>
> It's a function, so it would be better to name it::<br>
><br>
> def transform_code(code):<br>
<br>
Fair enough :-) But I want the context parameter to pass additional information.<div><br></div><div>Note: if we pass a code object, the filename is already in the code object, but there are other informations (see below).<br><div>
<br>
> And this::<br>
><br>
> def ast_transformer(tree, context):<br>
><br>
> might work better with keyword arguments::<br>
><br>
> def transform_ast(tree, *, filename, **kwargs):<br>
><br>
> otherwise people might use context objects with other attributes than<br>
> "filename", breaking when a future PEP assigns a specific meaning to them.<br>
<br>
The idea of a context object is to be "future-proof". Future versions of Python can add new attributes without having to modify all code transformers (or even worse, having to use kind of "#ifdef" in the code depending on the Python version).<br>
<br>
> It actually might be good to make the code transformer API extensible as<br>
> well, and synchronize with the AST transformer::<br>
><br>
> def transform_code(code, *, filename, **kwargs):<br>
<br>
**kwargs and context is basically the same, but I prefer a single parameter rather than an ugly **kwargs. IMHO "**kwargs" cannot be called an API.<br>
<br>
By the way, I added lately the bytecode transformers to the PEP. In fact, we already can more informations to its context:<br>
<br>
* compiler_flags: flags like<br>
* optimization_level (int): 0, 1 or 2 depending on the -O and -OO command line options<br>
* interactive (boolean): True if interactive mode<br>
* etc.<br>
<br>
=> see the compiler structure in Python/compile.c.<br>
<br>
We will have to check that these attributes make sense to other Python implementations, or make it clear in the PEP that as sys.implementation, each Python implementation can add specific attributes, and only a few of them are always available.<br>
<br>
Victor<br>
</div></div>