[Python-checkins] peps: PEP 511: add bytecode transformers

[victor.stinner]

https://hg.python.org/peps/rev/95de80b56938
changeset:   6189:95de80b56938
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri Jan 15 12:54:09 2016 +0100
summary:
  PEP 511: add bytecode transformers

files:
  pep-0511.txt |  107 +++++++++++++++++++++-----------------
  1 files changed, 60 insertions(+), 47 deletions(-)


diff --git a/pep-0511.txt b/pep-0511.txt
--- a/pep-0511.txt
+++ b/pep-0511.txt
@@ -1,5 +1,5 @@
 PEP: 511
-Title: API for AST transformers
+Title: API for code transformers
 Version: $Revision$
 Last-Modified: $Date$
 Author: Victor Stinner <victor.stinner at gmail.com>
@@ -12,12 +12,12 @@
 Abstract
 ========
 
-Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
-command line option to change ``.pyc`` filenames. Raise an
+Propose an API to register AST and bytecode transformers. Add also ``-o
+OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
 ``ImportError`` exception on import if the ``.pyc`` file is missing and
-the AST transformers required to transform the code are missing.
-AST transformers are not needed code transformed ahead of time (loaded
-from ``.pyc`` files).
+the code transformers required to transform the code are missing.  code
+transformers are not needed code transformed ahead of time (loaded from
+``.pyc`` files).
 
 
 Rationale
@@ -132,18 +132,18 @@
 Use Cases
 =========
 
-This section give examples of use cases explaining when and how AST
+This section give examples of use cases explaining when and how code
 transformers will be used.
 
 Interactive interpreter
 -----------------------
 
-It will be possible to use AST transformers with the interactive
+It will be possible to use code transformers with the interactive
 interpreter which is popular in Python and commonly used to demonstrate
 Python.
 
 The code is transformed at runtime and so the interpreter can be slower
-when expensive AST transformers are used.
+when expensive code transformers are used.
 
 Build a transformed package
 ---------------------------
@@ -153,7 +153,7 @@
 A transformer can have a configuration. The configuration is not stored
 in the package.
 
-All ``.pyc`` files of the package must be transformed with the same AST
+All ``.pyc`` files of the package must be transformed with the same code
 transformers and the same transformers configuration.
 
 It is possible to build different ``.pyc`` files using different
@@ -181,7 +181,7 @@
 optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are
 created during the installation.
 
-AST transformers of the optimizer tag are required. Otherwise, the
+Code transformers of the optimizer tag are required. Otherwise, the
 installation fails with an error.
 
 
@@ -191,10 +191,10 @@
 It will be possible to execute transformed code.
 
 Raise an ``ImportError`` exception on import if the ``.pyc`` file of the
-current optimizer tag is missing and the AST transformers required to
+current optimizer tag is missing and the code transformers required to
 transform the code are missing.
 
-The interesting point here is that AST transformers are not needed to
+The interesting point here is that code transformers are not needed to
 execute the transformed code if all required ``.pyc`` files are already
 available.
 
@@ -202,46 +202,57 @@
 Changes
 =======
 
-This PEP proposes to add an API to register AST transformers.
+This PEP proposes to add an API to register code transformers.
 
 The transformation can done ahead of time. It allows to implement
 powerful but expensive transformations.
 
 
-API for AST transformers
-------------------------
+API for code transformers
+-------------------------
 
-Add new functions to register AST transformers:
+Add new functions to register code transformers:
 
-* ``sys.set_ast_transformers(transformers)``: set the list of AST
-  transformers
-* ``sys.get_ast_transformers()``: get the list of AST
+* ``sys.set_code_transformers(transformers)``: set the list of code
+  transformers and update ``sys.implementation.optim_tag``
+* ``sys.get_code_transformers()``: get the list of code
   transformers.
 
-The order of AST transformers matter. Running transformer A and then
+The order of code transformers matter. Running transformer A and then
 transformer B can give a different output than running transformer B an
 then transformer A.
 
-API of an AST transformer:
+API of an code transformer:
 
-* An AST transformer is a callable object with the prototype::
+* An code transformer is a class with ``ast_transformer()`` and/or
+  ``bytecode_transformer()`` methods and a ``name`` attribute. Example::
 
-    def ast_transformer(tree, context):
-        ...
-        return tree
+    class MyTransformer:
+        name = 'my_transformer'
 
-  where *tree* is an AST tree and *context* is an object with a
-  ``filename`` attribute (``str``). New attributes may be added to
-  *context* in the future.
+        def bytecode_transformer(self, bytecode, ...):
+            ...
+            return tree
 
-* It must return an AST tree.
-* It must have a ``name`` attribute (``str``): short string used to identify an
-  optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters:
-  ``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used
-  to join AST transformer names to build the optimizer tag.
-* The transformer is called after the creation of the AST by the parser
-  and before the compilation to bytecode
-* It can modify the AST tree in place, or create a new AST tree.
+        def ast_transformer(self, tree, context):
+            ...
+            return tree
+
+  ``bytecode_transformer()``: *bytecode* is bytes objects.
+
+  ``ast_transformer()``: *tree* is an AST tree and *context* is an
+  object with a ``filename`` attribute (``str``). New attributes may be
+  added to *context* in the future. It must return an AST tree. It can
+  modify the AST tree in place, or create a new AST tree.
+
+* The ``name`` attribute (``str``) must be a short string used to
+  identify an optimizer. The name must not contain ``.`` (dot), ``-``
+  (dash) characters or directory separators: ``.`` is used to separated
+  fields in a ``.pyc`` filename and ``-`` is used to join AST
+  transformer names to build the optimizer tag.
+* The AST transformer is called after the creation of the AST by the
+  parser and before the compilation to bytecode
+* The bytecode transformer is after the compilation to bytecode
 
 .. note::
    It would be nice to pass the fully qualified name of a module in the
@@ -267,23 +278,23 @@
   ``opt``. Remove also the special case for the optimizer level ``0``
   with the default optimizer tag ``'opt'`` to simplify the code.
 * When loading a module, if the ``.pyc`` file is missing but the ``.py``
-  is available, the ``.py`` is only used if AST optimizers have the same
-  optimizer tag than the current tag, otherwise an ``ImportError``
+  is available, the ``.py`` is only used if code optimizers have the
+  same optimizer tag than the current tag, otherwise an ``ImportError``
   exception is raised.
 
 Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
 be compiled to import a module::
 
-    def get_ast_optim_tag():
-        transformers = sys.get_ast_transformers()
+    def transformers_tag():
+        transformers = sys.get_code_transformers()
         if not transformers:
             return 'opt'
         return '-'.join(transformer.name for transformer in transformers)
 
     def use_py():
-        return (get_ast_transformers() == sys.implementation.optim_tag)
+        return (transformers_tag() == sys.implementation.optim_tag)
 
-The order of ``sys.get_ast_transformers()`` matter. For example, the
+The order of ``sys.get_code_transformers()`` matter. For example, the
 ``fat`` transformer followed by the ``pythran`` transformer gives the
 optimizer tag ``fat-pythran``.
 
@@ -345,7 +356,7 @@
 
 
 AST transformer
-----------------
+---------------
 
 Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
 
@@ -365,13 +376,15 @@
         def __init__(self):
             self.transformer = KnightsWhoSayNi()
 
-        def __call__(self, tree, context):
+        def ast_transformer(self, tree, context):
             self.transformer.visit(tree)
             return tree
 
 
-    # register the AST transformer
-    sys.set_ast_transformers([ASTTransformer()])
+    # append our AST transformer after existing transformers
+    transformers = sys.get_code_transformers()
+    transformers.append(ASTTransformer())
+    sys.set_code_transformers(transformers)
 
     # execute code which will be transformed by ast_transformer()
     exec("print('Hello World!')")

-- 
Repository URL: https://hg.python.org/peps