On Fri, 15 Jan 2016 at 08:11 Victor Stinner <victor.stinner@gmail.com> wrote:
[SNIP]
 
Optimizer tag
-------------

Changes:

* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
  The default optimization tag is ``'opt'``.
* Add a new ``-o OPTIM_TAG`` command line option to set
  ``sys.implementation.optim_tag``.

Changes on ``importlib``:

* ``importlib`` uses ``sys.implementation.optim_tag`` to build the
  ``.pyc`` filename to importing modules, instead of always using
  ``opt``. Remove also the special case for the optimizer level ``0``
  with the default optimizer tag ``'opt'`` to simplify the code.
* When loading a module, if the ``.pyc`` file is missing but the ``.py``
  is available, the ``.py`` is only used if code optimizers have the
  same optimizer tag than the current tag, otherwise an ``ImportError``
  exception is raised.

Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
be compiled to import a module::

    def transformers_tag():
        transformers = sys.get_code_transformers()
        if not transformers:
            return 'noopt'
        return '-'.join(transformer.name
                        for transformer in transformers)

    def use_py():
        return (transformers_tag() == sys.implementation.optim_tag)

The order of ``sys.get_code_transformers()`` matter. For example, the
``fat`` transformer followed by the ``pythran`` transformer gives the
optimizer tag ``fat-pythran``.

The behaviour of the ``importlib`` module is unchanged with the default
optimizer tag (``'opt'``).

I just wanted to point out to people that the key part of this PEP is the change in semantics of `-O` accepting an argument. Without this change there is no way to cause import to pick up on optimized .pyc files that you want it to use without abusing pre-existing .pyc filenames.

This also means that everything else is optional. That doesn't mean it shouldn't be considered, mind you, as it makes using AST and bytecode transformers more practical. But some `-O` change that allows user-defined optimization tags is needed for any of this to work reasonably. From there it's theoretically possible for someone to write their own compileall that pre-compiles all Python code to .pyc files with a specific optimization tag which they specify with `-O` using their own AST and bytecode transformers and hence not need the transformation features built into sys/import.

I should also point out that this does get tricky in terms of how to handle the stdlib if you have not pre-compiled it, e.g., if the first module imported by Python is the encodings module then how to make sure the AST optimizers are ready to go by the time that import happens?

And lastly, Victor proposes that all .pyc files get an optimization tag. While there is nothing technically wrong with that, PEP 488 purposefully didn't do that in the default case for backwards-compatibility, so that will need to be at least mentioned in the PEP.