[Import-SIG] PEP for the removal of PYO files

[Brett Cannon]

Here is the latest draft. I think the biggest bit is the expanded section
of the Open Issues with a few more formatting proposals and Nick's
suggestion to let the common case of no optimizations lead to no level
being specified in the file name (I also changed the potential PEP # as 487
got snagged). Otherwise a sentence about getting to generate all
optimization levels upfront and the empty string suppressing the inclusion
of the optimization level are the other substantive changes.

PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:

Abstract
========

This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in bytecode repository
directory (i.e., the ``__pycache__`` directory).


Rationale
=========

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` is specified, including ``-OO``). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. Put in terms of optimization levels and
the file extension:

  - 0: ``.pyc``
  - 1 (``-O``): ``.pyo``
  - 2 (``-OO``): ``.pyo``

The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.

In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.

As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments.


Proposal
========

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate a PYC file will be incorporated into the bytecode
file name. Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression to::

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion with "0" while being
so close to the interpreter version number.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of
the period in the file name to delineate semantically different
concepts.

For example, the bytecode file name of ``importlib.cpython-35.pyc``
would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been
passed to the interpreter then instead of
``importlib.cpython-35.pyo`` the file name would be
``importlib.cpython-35.opt-2.pyc``.


Implementation
==============

importlib
---------

As ``importlib.util.cache_from_source()`` is the API that exposes
bytecode file paths as while as being directly used by importlib, it
requires the most critical change. As of Python 3.4, the function's
signature is::

  importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to::

  importlib.util.cache_from_source(path, debug_override=None, *,
optimization=None)

The introduced ``optimization`` keyword-only parameter will control
what optimization level is specified in the file name. If the
argument is ``None`` then the current optimization level of the
interpreter will be assumed. Any argument given for ``optimization``
will be passed to ``str()`` and must have ``str.isalnum()`` be true,
else ``ValueError`` will be raised (this prevents invalid characters
being used in the file name). If the empty string is passed in for
``optimization`` then the addition of the optimization will be
suppressed, reverting to the file name format which predates this
PEP.

It is expected that beyond Python's own
0-2 optimization levels, third-party code will use a hash of
optimization names to specify the optimization level, e.g.
``hashlib.sha256(','.join(['dead code elimination', 'constant
folding'])).hexdigest()``.
While this might lead to long file names, it is assumed that most
users never look at the contents of the __pycache__ directory and so
this won't be an issue.

The ``debug_override`` parameter will be deprecated. As the parameter
expects a boolean, the integer value of the boolean will be used as
if it had been provided as the argument to ``optimization`` (a
``None`` argument will mean the same as for ``optimization``). A
deprecation warning will be raised when ``debug_override`` is given a
value other than ``None``, but there are no plans for the complete
removal of the parameter as this time (but removal will be no later
than Python 4).

The various module attributes for importlib.machinery which relate to
bytecode file suffixes will be updated [7]_. The
``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will
both be documented as deprecated and set to the same value as
``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and
``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be
not later than Python 4).

All various finders and loaders will also be updated as necessary,
but updating the previous mentioned parts of importlib should be all
that is required.


Rest of the standard library
----------------------------

The various functions exposed by the ``py_compile`` and
``compileall`` functions will be updated as necessary to make sure
they follow the new bytecode file name semantics [6]_, [1]_. The CLI
for the ``compileall`` module will not be directly affected (the
``-b`` flag will be implicitly as it will no longer generate ``.pyo``
files when ``-O`` is specified).


Compatibility Considerations
============================

Any code directly manipulating bytecode files from Python 3.2 on
will need to consider the impact of this change on their code (prior
to Python 3.2 -- including all of Python 2 -- there was no
__pycache__ which already necessitates bifurcating bytecode file
handling support). If code was setting the ``debug_override``
argument to ``importlib.util.cache_from_source()`` then care will be
needed if they want the path to a bytecode file with an optimization
level of 2. Otherwise only code **not** using
``importlib.util.cache_from_source()`` will need updating.

As for people who distribute bytecode-only modules (i.e., use a
bytecode file instead of a source file), they will have to choose
which optimization level they want their bytecode files to be since
distributing a ``.pyo`` file with a ``.pyc`` file will no longer be
of any use. Since people typically only distribute bytecode files for
code obfuscation purposes or smaller distribution size then only
having to distribute a single ``.pyc`` should actually be beneficial
to these use-cases. And since the magic number for bytecode files
changed in Python 3.5 to support PEP 465 there is no need to support
pre-existing ``.pyo`` files [8]_.


Rejected Ideas
==============

N/A


Open Issues
===========

Formatting of the optimization level in the file name
-----------------------------------------------------

Using the "opt-" prefix and placing the optimization level between
the cache tag and file extension is not critical. All options which
have been considered are:

* ``importlib.cpython-35.opt-0.pyc``
* ``importlib.cpython-35.opt0.pyc``
* ``importlib.cpython-35.o0.pyc``
* ``importlib.cpython-35.O0.pyc``
* ``importlib.cpython-35.0.pyc``
* ``importlib.cpython-35-O0.pyc``
* ``importlib.O0.cpython-35.pyc``
* ``importlib.o0.cpython-35.pyc``
* ``importlib.0.cpython-35.pyc``

These were initially rejected either because they would change the
sort order of bytecode files, possible ambiguity with the cache tag,
or were not self-documenting enough.


Not specifying the optimization level when it is at 0
-----------------------------------------------------

It has been suggested that for the common case of when the
optimizations are at level 0 that the entire part of the file name
relating to the optimization level be left out. This would allow for
file names of ``.pyc`` files to go unchanged, potentially leading to
less backwards-compatibility issues.

It would also allow a potentially redundant bit of information to be
left out of the file name if an implementation of Python did not
allow for optimizing bytecode. This would only occur, though, if the
interpreter didn't support ``-O`` **and** didn't implement the ast
module, else user's could implement their own optimizations.

Arguments against allow for this is "explicit is better than
implicit" and "special cases aren't special enough to break the
rules". There are also currently no Python 3 interpreters that don't
support ``-O``, so a potential Python 3 implementation which doesn't
allow bytecode optimization is entirely theoretical at the moment.


References
==========

.. [1] The compileall module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [2] The astoptimizer project
   (https://pypi.python.org/pypi/astoptimizer)

.. [3] ``importlib.util.cache_from_source()``
   (
https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source
)

.. [4] Implementation of ``importlib.util.cache_from_source()`` from
CPython 3.4.3rc1
   (
https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437
)

.. [5] PEP 3147, PYC Repository Directories, Warsaw
   (http://www.python.org/dev/peps/pep-3147)

.. [6] The py_compile module
   (https://docs.python.org/3/library/compileall.html#module-compileall)

.. [7] The importlib.machinery module
   (
https://docs.python.org/3/library/importlib.html#module-importlib.machinery)

.. [8] ``importlib.util.MAGIC_NUMBER``
   (
https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER
)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20150301/08751533/attachment-0001.html>