<div dir="ltr">I'm in a good mood today and I think this is a great idea! That's not to say that I'm accepting it as-is (I haven't read it fully) but I expect that there are very few downsides and it won't break much. (There's of course always going to be someone who always uses -O and somehow depends on the existence of .pyo files, but they should have seen it coming with __pycache__ and the new version-specific extensions. :-)<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 27, 2015 at 9:06 AM, Brett Cannon <span dir="ltr"><<a href="mailto:bcannon@gmail.com" target="_blank">bcannon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Here is my proposed PEP to drop .pyo files from Python. Thanks to Barry's work in PEP 3147 this really shouldn't have much impact on user's code (then again, bytecode files are basically an implementation detail so it shouldn't impact hardly anyone directly).</div><div><br></div><div>One thing I would appreciate is if people have more motivation for this. While the maintainer of importlib in me wants to see this happen, the core developer in me thinks the arguments are a little weak. So if people can provide more reasons why this is a good thing that would be appreciated.</div><div><br></div><div><br></div><div>PEP: 487</div><div>Title: Elimination of PYO files</div><div>Version: $Revision$</div><div>Last-Modified: $Date$</div><div>Author: Brett Cannon <<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>></div><div>Status: Draft</div><div>Type: Standards Track</div><div>Content-Type: text/x-rst</div><div>Created: 20-Feb-2015</div><div>Post-History:</div><div><br></div><div>Abstract</div><div>========</div><div><br></div><div>This PEP proposes eliminating the concept of PYO files from Python.</div><div>To continue the support of the separation of bytecode files based on</div><div>their optimization level, this PEP proposes extending the PYC file</div><div>name to include the optimization level in bytecode repository</div><div>directory (i.e., the ``__pycache__`` directory).</div><div><br></div><div><br></div><div>Rationale</div><div>=========</div><div><br></div><div>As of today, bytecode files come in two flavours: PYC and PYO. A PYC</div><div>file is the bytecode file generated and read from when no</div><div>optimization level is specified at interpreter startup (i.e., ``-O``</div><div>is not specified). A PYO file represents the bytecode file that is</div><div>read/written when **any** optimization level is specified (i.e., when</div><div>``-O`` is specified, including ``-OO``). This means that while PYC</div><div>files clearly delineate the optimization level used when they were</div><div>generated -- namely no optimizations beyond the peepholer -- the same</div><div>is not true for PYO files. Put in terms of optimization levels and</div><div>the file extension:</div><div><br></div><div> - 0: ``.pyc``</div><div> - 1 (``-O``): ``.pyo``</div><div> - 2 (``-OO``): ``.pyo``</div><div><br></div><div>The reuse of the ``.pyo`` file extension for both level 1 and 2</div><div>optimizations means that there is no clear way to tell what</div><div>optimization level was used to generate the bytecode file. In terms</div><div>of reading PYO files, this can lead to an interpreter using a mixture</div><div>of optimization levels with its code if the user was not careful to</div><div>make sure all PYO files were generated using the same optimization</div><div>level (typically done by blindly deleting all PYO files and then</div><div>using the `compileall` module to compile all-new PYO files [1]_).</div><div>This issue is only compounded when people optimize Python code beyond</div><div>what the interpreter natively supports, e.g., using the astoptimizer</div><div>project [2]_.</div><div><br></div><div>In terms of writing PYO files, the need to delete all PYO files</div><div>every time one either changes the optimization level they want to use</div><div>or are unsure of what optimization was used the last time PYO files</div><div>were generated leads to unnecessary file churn.</div><div><br></div><div>As for distributing bytecode-only modules, having to distribute both</div><div>``.pyc`` and ``.pyo`` files is unnecessary for the common use-case</div><div>of code obfuscation and smaller file deployments.</div><div><br></div><div><br></div><div>Proposal</div><div>========</div><div><br></div><div>To eliminate the ambiguity that PYO files present, this PEP proposes</div><div>eliminating the concept of PYO files and their accompanying ``.pyo``</div><div>file extension. To allow for the optimization level to be unambiguous</div><div>as well as to avoid having to regenerate optimized bytecode files</div><div>needlessly in the `__pycache__` directory, the optimization level</div><div>used to generate a PYC file will be incorporated into the bytecode</div><div>file name. Currently bytecode file names are created by</div><div>``importlib.util.cache_from_source()``, approximately using the</div><div>following expression defined by PEP 3147 [3]_, [4]_, [5]_::</div><div><br></div><div> '{name}.{cache_tag}.pyc'.format(name=module_name,</div><div> cache_tag=sys.implementation.cache_tag)</div><div><br></div><div>This PEP proposes to change the expression to::</div><div><br></div><div> '{name}.{cache_tag}.opt-{optimization}.pyc'.format(</div><div> name=module_name,</div><div> cache_tag=sys.implementation.cache_tag,</div><div> optimization=str(sys.flags.optimize))</div><div><br></div><div>The "opt-" prefix was chosen so as to provide a visual separator</div><div>from the cache tag. The placement of the optimization level after</div><div>the cache tag was chosen to preserve lexicographic sort order of</div><div>bytecode file names based on module name and cache tag which will</div><div>not vary for a single interpreter. The "opt-" prefix was chosen over</div><div>"o" so as to be somewhat self-documenting. The "opt-" prefix was</div><div>chosen over "O" so as to not have any confusion with "0" while being</div><div>so close to the interpreter version number.</div><div><br></div><div>A period was chosen over a hyphen as a separator so as to distinguish</div><div>clearly that the optimization level is not part of the interpreter</div><div>version as specified by the cache tag. It also lends to the use of</div><div>the period in the file name to delineate semantically different</div><div>concepts.</div><div><br></div><div>For example, the bytecode file name of ``importlib.cpython-35.pyc``</div><div>would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been</div><div>passed to the interpreter then instead of</div><div>``importlib.cpython-35.pyo`` the file name would be</div><div>``importlib.cpython-35.opt-2.pyc``.</div><div><br></div><div><br></div><div>Implementation</div><div>==============</div><div><br></div><div>importlib</div><div>---------</div><div><br></div><div>As ``importlib.util.cache_from_source()`` is the API that exposes</div><div>bytecode file paths as while as being directly used by importlib, it</div><div>requires the most critical change. As of Python 3.4, the function's</div><div>signature is::</div><div><br></div><div> importlib.util.cache_from_source(path, debug_override=None)</div><div><br></div><div>This PEP proposes changing the signature in Python 3.5 to::</div><div><br></div><div> importlib.util.cache_from_source(path, debug_override=None, *, optimization=None)</div><div><br></div><div>The introduced ``optimization`` keyword-only parameter will control</div><div>what optimization level is specified in the file name. If the</div><div>argument is ``None`` then the current optimization level of the</div><div>interpreter will be assumed. Any argument given for ``optimization``</div><div>will be passed to ``str()`` and must have ``str.isalnum()`` be true,</div><div>else ``ValueError`` will be raised (this prevents invalid characters</div><div>being used in the file name). It is expected that beyond Python's own</div><div>0-2 optimization levels, third-party code will use a hash of</div><div>optimization names to specify the optimization level, e.g.</div><div>``hashlib.sha256(','.join(['dead code elimination', 'constant folding'])).hexdigest()``.</div><div><br></div><div>The ``debug_override`` parameter will be deprecated. As the parameter</div><div>expects a boolean, the integer value of the boolean will be used as</div><div>if it had been provided as the argument to ``optimization`` (a</div><div>``None`` argument will mean the same as for ``optimization``). A</div><div>deprecation warning will be raised when ``debug_override`` is given a</div><div>value other than ``None``, but there are no plans for the complete</div><div>removal of the parameter as this time (but removal will be no later</div><div>than Python 4).</div><div><br></div><div>The various module attributes for importlib.machinery which relate to</div><div>bytecode file suffixes will be updated [7]_. The</div><div>``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will</div><div>both be documented as deprecated and set to the same value as</div><div>``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and</div><div>``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be</div><div>not later than Python 4).</div><div><br></div><div>All various finders and loaders will also be updated as necessary,</div><div>but updating the previous mentioned parts of importlib should be all</div><div>that is required.</div><div><br></div><div><br></div><div>Rest of the standard library</div><div>----------------------------</div><div><br></div><div>The various functions exposed by the ``py_compile`` and</div><div>``compileall`` functions will be updated as necessary to make sure</div><div>they follow the new bytecode file name semantics [6]_, [1]_.</div><div><br></div><div><br></div><div>Compatibility Considerations</div><div>============================</div><div><br></div><div>Any code directly manipulating bytecode files from Python 3.2 on</div><div>will need to consider the impact of this change on their code (prior</div><div>to Python 3.2 -- including all of Python 2 -- there was no</div><div>__pycache__ which already necessitates bifurcating bytecode file</div><div>handling support). If code was setting the ``debug_override``</div><div>argument to ``importlib.util.cache_from_source()`` then care will be</div><div>needed if they want the path to a bytecode file with an optimization</div><div>level of 2. Otherwise only code **not** using</div><div>``importlib.util.cache_from_source()`` will need updating.</div><div><br></div><div>As for people who distribute bytecode-only modules, they will have</div><div>to choose which optimization level they want their bytecode files to</div><div>be since distributing a ``.pyo`` file with a ``.pyc`` file will no</div><div>longer be of any use. Since people typically only distribute bytecode</div><div>files for code obfuscation purposes or smaller distribution size</div><div>then only having to distribute a single ``.pyc`` should actually be</div><div>beneficial to these use-cases.</div><div><br></div><div><br></div><div>Rejected Ideas</div><div>==============</div><div><br></div><div>N/A</div><div><br></div><div><br></div><div>Open Issues</div><div>===========</div><div><br></div><div>Formatting of the optimization level in the file name</div><div>-----------------------------------------------------</div><div><br></div><div>Using the "opt-" prefix and placing the optimization level between</div><div>the cache tag and file extension is not critical. Other options which</div><div>were considered are:</div><div><br></div><div>* ``importlib.cpython-35.o0.pyc``</div><div>* ``importlib.cpython-35.O0.pyc``</div><div>* ``importlib.cpython-35.0.pyc``</div><div>* ``importlib.cpython-35-O0.pyc``</div><div>* ``importlib.O0.cpython-35.pyc``</div><div>* ``importlib.o0.cpython-35.pyc``</div><div>* ``importlib.0.cpython-35.pyc``</div><div><br></div><div>These were initially rejected either because they would change the</div><div>sort order of bytecode files, possible ambiguity with the cache tag,</div><div>or were not self-documenting enough.</div><div><br></div><div><br></div><div>References</div><div>==========</div><div><br></div><div>.. [1] The compileall module</div><div> (<a href="https://docs.python.org/3/library/compileall.html#module-compileall" target="_blank">https://docs.python.org/3/library/compileall.html#module-compileall</a>)</div><div><br></div><div>.. [2] The astoptimizer project</div><div> (<a href="https://pypi.python.org/pypi/astoptimizer" target="_blank">https://pypi.python.org/pypi/astoptimizer</a>)</div><div><br></div><div>.. [3] ``importlib.util.cache_from_source()``</div><div> (<a href="https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source" target="_blank">https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source</a>)</div><div><br></div><div>.. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1</div><div> (<a href="https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437" target="_blank">https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437</a>)</div><div><br></div><div>.. [5] PEP 3147, PYC Repository Directories, Warsaw</div><div> (<a href="http://www.python.org/dev/peps/pep-3147" target="_blank">http://www.python.org/dev/peps/pep-3147</a>)</div><div><br></div><div>.. [6] The py_compile module</div><div> (<a href="https://docs.python.org/3/library/compileall.html#module-compileall" target="_blank">https://docs.python.org/3/library/compileall.html#module-compileall</a>)</div><div><br></div><div>.. [7] The importlib.machinery module</div><div> (<a href="https://docs.python.org/3/library/importlib.html#module-importlib.machinery" target="_blank">https://docs.python.org/3/library/importlib.html#module-importlib.machinery</a>)</div><div><br></div><div><br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div><div><br></div><div><br></div><div> </div><div>..</div><div> Local Variables:</div><div> mode: indented-text</div><div> indent-tabs-mode: nil</div><div> sentence-end-double-space: t</div><div> fill-column: 70</div><div> coding: utf-8</div><div> End:</div><div><br></div></div>
<br>_______________________________________________<br>
Import-SIG mailing list<br>
<a href="mailto:Import-SIG@python.org">Import-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/import-sig" target="_blank">https://mail.python.org/mailman/listinfo/import-sig</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido">python.org/~guido</a>)</div>
</div>