[Python-ideas] PEP 511: API for code transformers

[Petr Viktorin]

On 01/15/2016 05:10 PM, Victor Stinner wrote:
> Hi,
> 
> This PEP 511 is part of a serie of 3 PEP (509, 510, 511) adding an API
> to implement a static Python optimizer specializing functions with
> guards.
> 
> If the PEP is accepted, it will solve a long list of issues, some
> issues are old, like #1346238 which is 11 years old ;-) I found 12
> issues:
> 
> * http://bugs.python.org/issue1346238
> * http://bugs.python.org/issue2181
> * http://bugs.python.org/issue2499
> * http://bugs.python.org/issue2506
> * http://bugs.python.org/issue4264
> * http://bugs.python.org/issue7682
> * http://bugs.python.org/issue10399
> * http://bugs.python.org/issue11549
> * http://bugs.python.org/issue17068
> * http://bugs.python.org/issue17430
> * http://bugs.python.org/issue17515
> * http://bugs.python.org/issue26107
> 
> I worked to make the PEP more generic that "this hook is written for
> FAT Python". Please read the full PEP to see a long list of existing
> usages in Python of code transformers.
> 
> You may read again the discussion which occurred 4 years ago about the
> same topic:
> https://mail.python.org/pipermail/python-dev/2012-August/121286.html
> (the thread starts with an idea of AST optimizer, but is moves quickly
> to a generic API to transform the code)
> 
> Thanks to Red Hat for giving me time to experiment on this.
> Victor
> 
> HTML version:
> https://www.python.org/dev/peps/pep-0510/#changes

Victor,
Thanks for your efforts on making Python faster!

This PEP addresses two things that would benefit from different
approaches: let's call them optimizers and extensions.

Optimizers, such as your FAT, don't change Python semantics. They're
designed to run on *all* code, including the standard library. It makes
sense to register them as early in interpreter startup as possible, but
if they're not registered, nothing breaks (things will just be slower).
Experiments with future syntax (like when async/await was being
developed) have the same needs.

Syntax extensions, such as MacroPy or Hy, tend to target specific
modules, with which they're closely coupled: The modules won't run
without the transformer. And with other modules, the transformer either
does nothing (as with MacroPy, hopefully), or would fail altogether (as
with Hy). So, they would benefit from specific packages opting in. The
effects of enabling them globally range from inefficiency (MacroPy) to
failures or needing workarounds (Hy).

The PEP is designed optimizers. It would be good to stick to that use
case, at least as far as the registration is concerned. I suggest noting
in the documentation that Python semantics *must* be preserved, and
renaming the API, e.g.::

    sys.set_global_optimizers([])

The "transformer" API can be used for syntax extensions as well, but the
registration needs to be different so the effects are localized. For
example it could be something like::

    importlib.util.import_with_transformer(
        'mypackage.specialmodule', MyTransformer())

or a special flag in packages::

    __transformers_for_submodules__ = [MyTransformer()]

or extendeding exec (which you actually might want to add to the PEP, to
make giving examples easier)::

    exec("print('Hello World!')", transformers=[MyTransformer()])

or making it easier to write an import hook with them, etc...

but all that would probably be out of scope for your PEP.


Another thing: this snippet from the PEP sounds too verbose::

    transformers = sys.get_code_transformers()
    transformers.insert(0, new_cool_transformer)
    sys.set_code_transformers(transformers)

Can this just be a list, as with sys.path? Using the "optimizers" term::

    sys.global_optimizers.insert(0, new_cool_transformer)



This::

    def code_transformer(code, consts, names, lnotab, context):

It's a function, so it would be better to name it::

    def transform_code(code):

And this::

    def ast_transformer(tree, context):

might work better with keyword arguments::

    def transform_ast(tree, *, filename, **kwargs):

otherwise people might use context objects with other attributes than
"filename", breaking when a future PEP assigns a specific meaning to them.

It actually might be good to make the code transformer API extensible as
well, and synchronize with the AST transformer::

    def transform_code(code, *, filename, **kwargs):