<div dir="ltr">Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488 implemented before the PyCon sprints are over (work will be tracked in <a href="http://bugs.python.org/issue23731">http://bugs.python.org/issue23731</a>).<br></div><br><div class="gmail_quote">On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum <<a href="mailto:guido@python.org">guido@python.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Awesome, that's what I was hoping. Accepted! Congrats and thank you very much for writing the PEP and guiding the discussion.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 20, 2015 at 4:00 PM, Brett Cannon <span dir="ltr"><<a href="mailto:bcannon@gmail.com" target="_blank">bcannon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span>On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum <<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>I am willing to be the BDFL for this PEP. I have tried to skim the recent discussion (only python-dev) and I don't see much remaining controversy. HOWEVER... The PEP is not clear (or at least too subtle) about the actual name for optimization level 0. If I have foo.py, and I compile it three times with three different optimization levels (no optimization; -O; -OO), and then I look in __pycache__, would I see this:<br></div><div><br># (1)<br></div>foo.cpython-35.pyc<br></div>foo.cpython-35.opt-1.pyc<br>foo.cpython-35.opt-2.pyc<br><br></div>Or would I see this?<br><br># (2)<br>foo.cpython-35.opt-0.pyc<br>foo.cpython-35.opt-1.pyc<br>foo.cpython-35.opt-2.pyc<br></div></div></blockquote><div><br></div></span><div>#1</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div>Your lead-in ("I have decided to have the default case of no optimization levels mean that the .pyc file name will have <b>no</b> optimization level specified in the name and thus be just as it is today.") makes me think I should expect (1), but I can't actually pinpoint where the language of the PEP says this.<br></div></blockquote><div><br></div></span><div>It was meant to be explained by "<span style="font-size:13.1999998092651px;line-height:19.7999992370605px">When no optimization level is specified, the </span><span style="font-size:13.1999998092651px;line-height:19.7999992370605px">pre-PEP ``.pyc`` file name will be used (i.e., no change in file name</span></div><div style="font-size:13.1999998092651px;line-height:19.7999992370605px">semantics)", but obviously it's a bit too subtle. I just updated the PEP with an explicit list of bytecode file name examples based on no -O, -O, and -OO.</div><span><font color="#888888"><div style="font-size:13.1999998092651px;line-height:19.7999992370605px"><br></div><div style="font-size:13.1999998092651px;line-height:19.7999992370605px">-Brett</div></font></span><div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon <span dir="ltr"><<a href="mailto:bcannon@gmail.com" target="_blank">bcannon@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I have decided to have the default case of no optimization levels mean that the .pyc file name will have <b>no</b> optimization level specified in the name and thus be just as it is today. I made this decision due to potential backwards-compatibility issues -- although I expect them to be minutes -- and to not force other implementations like PyPy to have some bogus value set since they don't have .pyo files to begin with (PyPy actually uses bytecode for -O and don't bother with -OO since PyPy already uses a bunch of memory when running).<div><br></div><div>Since this closes out the last open issue, I need either a BDFL decision or a BDFAP to be assigned to make a decision. Guido?</div><div><br></div><div>======================================</div><div><br></div><div><div>PEP: 488</div><div>Title: Elimination of PYO files</div><div>Version: $Revision$</div><div>Last-Modified: $Date$</div><div>Author: Brett Cannon <<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>></div><div>Status: Draft</div><div>Type: Standards Track</div><div>Content-Type: text/x-rst</div><div>Created: 20-Feb-2015</div><div>Post-History:</div><div> 2015-03-06</div><div> 2015-03-13</div><div> 2015-03-20</div><div><br></div><div>Abstract</div><div>========</div><div><br></div><div>This PEP proposes eliminating the concept of PYO files from Python.</div><div>To continue the support of the separation of bytecode files based on</div><div>their optimization level, this PEP proposes extending the PYC file</div><div>name to include the optimization level in the bytecode repository</div><div>directory when it's called for (i.e., the ``__pycache__`` directory).</div><div><br></div><div><br></div><div>Rationale</div><div>=========</div><div><br></div><div>As of today, bytecode files come in two flavours: PYC and PYO. A PYC</div><div>file is the bytecode file generated and read from when no</div><div>optimization level is specified at interpreter startup (i.e., ``-O``</div><div>is not specified). A PYO file represents the bytecode file that is</div><div>read/written when **any** optimization level is specified (i.e., when</div><div>``-O`` **or** ``-OO`` is specified). This means that while PYC</div><div>files clearly delineate the optimization level used when they were</div><div>generated -- namely no optimizations beyond the peepholer -- the same</div><div>is not true for PYO files. To put this in terms of optimization</div><div>levels and the file extension:</div><div><br></div><div> - 0: ``.pyc``</div><div> - 1 (``-O``): ``.pyo``</div><div> - 2 (``-OO``): ``.pyo``</div><div><br></div><div>The reuse of the ``.pyo`` file extension for both level 1 and 2</div><div>optimizations means that there is no clear way to tell what</div><div>optimization level was used to generate the bytecode file. In terms</div><div>of reading PYO files, this can lead to an interpreter using a mixture</div><div>of optimization levels with its code if the user was not careful to</div><div>make sure all PYO files were generated using the same optimization</div><div>level (typically done by blindly deleting all PYO files and then</div><div>using the `compileall` module to compile all-new PYO files [1]_).</div><div>This issue is only compounded when people optimize Python code beyond</div><div>what the interpreter natively supports, e.g., using the astoptimizer</div><div>project [2]_.</div><div><br></div><div>In terms of writing PYO files, the need to delete all PYO files</div><div>every time one either changes the optimization level they want to use</div><div>or are unsure of what optimization was used the last time PYO files</div><div>were generated leads to unnecessary file churn. The change proposed</div><div>by this PEP also allows for **all** optimization levels to be</div><div>pre-compiled for bytecode files ahead of time, something that is</div><div>currently impossible thanks to the reuse of the ``.pyo`` file</div><div>extension for multiple optimization levels.</div><div><br></div><div>As for distributing bytecode-only modules, having to distribute both</div><div>``.pyc`` and ``.pyo`` files is unnecessary for the common use-case</div><div>of code obfuscation and smaller file deployments. This means that</div><div>bytecode-only modules will only load from their non-optimized</div><div>``.pyc`` file name.</div><div><br></div><div><br></div><div>Proposal</div><div>========</div><div><br></div><div>To eliminate the ambiguity that PYO files present, this PEP proposes</div><div>eliminating the concept of PYO files and their accompanying ``.pyo``</div><div>file extension. To allow for the optimization level to be unambiguous</div><div>as well as to avoid having to regenerate optimized bytecode files</div><div>needlessly in the `__pycache__` directory, the optimization level</div><div>used to generate the bytecode file will be incorporated into the</div><div>bytecode file name. When no optimization level is specified, the</div><div>pre-PEP ``.pyc`` file name will be used (i.e., no change in file name</div><div>semantics). This increases backwards-compatibility while also being</div><div>more understanding of Python implementations which have no use for</div><div>optimization levels (e.g., PyPy[10]_).</div><div><br></div><div>Currently bytecode file names are created by</div><div>``importlib.util.cache_from_source()``, approximately using the</div><div>following expression defined by PEP 3147 [3]_, [4]_, [5]_::</div><div><br></div><div> '{name}.{cache_tag}.pyc'.format(name=module_name,</div><div> cache_tag=sys.implementation.cache_tag)</div><div><br></div><div>This PEP proposes to change the expression when an optimization</div><div>level is specified to::</div><div><br></div><div> '{name}.{cache_tag}.opt-{optimization}.pyc'.format(</div><div> name=module_name,</div><div> cache_tag=sys.implementation.cache_tag,</div><div> optimization=str(sys.flags.optimize))</div><div><br></div><div>The "opt-" prefix was chosen so as to provide a visual separator</div><div>from the cache tag. The placement of the optimization level after</div><div>the cache tag was chosen to preserve lexicographic sort order of</div><div>bytecode file names based on module name and cache tag which will</div><div>not vary for a single interpreter. The "opt-" prefix was chosen over</div><div>"o" so as to be somewhat self-documenting. The "opt-" prefix was</div><div>chosen over "O" so as to not have any confusion in case "0" was the</div><div>leading prefix of the optimization level.</div><div><br></div><div>A period was chosen over a hyphen as a separator so as to distinguish</div><div>clearly that the optimization level is not part of the interpreter</div><div>version as specified by the cache tag. It also lends to the use of</div><div>the period in the file name to delineate semantically different</div><div>concepts.</div><div><br></div><div>For example, if ``-OO`` had been passed to the interpreter then instead</div><div>of ``importlib.cpython-35.pyo`` the file name would be</div><div>``importlib.cpython-35.opt-2.pyc``.</div><div><br></div><div>It should be noted that this change in no way affects the performance</div><div>of import. Since the import system looks for a single bytecode file</div><div>based on the optimization level of the interpreter already and</div><div>generates a new bytecode file if it doesn't exist, the introduction</div><div>of potentially more bytecode files in the ``__pycache__`` directory</div><div>has no effect in terms of stat calls. The interpreter will continue</div><div>to look for only a single bytecode file based on the optimization</div><div>level and thus no increase in stat calls will occur.</div><div><br></div><div>The only potentially negative result of this PEP is the probable</div><div>increase in the number of ``.pyc`` files and thus increase in storage</div><div>use. But for platforms where this is an issue,</div><div>``sys.dont_write_bytecode`` exists to turn off bytecode generation so</div><div>that it can be controlled offline.</div><div><br></div><div><br></div><div>Implementation</div><div>==============</div><div><br></div><div>importlib</div><div>---------</div><div><br></div><div>As ``importlib.util.cache_from_source()`` is the API that exposes</div><div>bytecode file paths as well as being directly used by importlib, it</div><div>requires the most critical change. As of Python 3.4, the function's</div><div>signature is::</div><div><br></div><div> importlib.util.cache_from_source(path, debug_override=None)</div><div><br></div><div>This PEP proposes changing the signature in Python 3.5 to::</div><div><br></div><div> importlib.util.cache_from_source(path, debug_override=None, *, optimization=None)</div><div><br></div><div>The introduced ``optimization`` keyword-only parameter will control</div><div>what optimization level is specified in the file name. If the</div><div>argument is ``None`` then the current optimization level of the</div><div>interpreter will be assumed (including no optimization). Any argument</div><div>given for ``optimization`` will be passed to ``str()`` and must have</div><div>``str.isalnum()`` be true, else ``ValueError`` will be raised (this</div><div>prevents invalid characters being used in the file name). If the</div><div>empty string is passed in for ``optimization`` then the addition of</div><div>the optimization will be suppressed, reverting to the file name</div><div>format which predates this PEP.</div><div><br></div><div>It is expected that beyond Python's own two optimization levels,</div><div>third-party code will use a hash of optimization names to specify the</div><div>optimization level, e.g.</div><div>``hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest()``.</div><div>While this might lead to long file names, it is assumed that most</div><div>users never look at the contents of the __pycache__ directory and so</div><div>this won't be an issue.</div><div><br></div><div>The ``debug_override`` parameter will be deprecated. As the parameter</div><div>expects a boolean, the integer value of the boolean will be used as</div><div>if it had been provided as the argument to ``optimization`` (a</div><div>``None`` argument will mean the same as for ``optimization``). A</div><div>deprecation warning will be raised when ``debug_override`` is given a</div><div>value other than ``None``, but there are no plans for the complete</div><div>removal of the parameter at this time (but removal will be no later</div><div>than Python 4).</div><div><br></div><div>The various module attributes for importlib.machinery which relate to</div><div>bytecode file suffixes will be updated [7]_. The</div><div>``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will</div><div>both be documented as deprecated and set to the same value as</div><div>``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and</div><div>``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be</div><div>not later than Python 4).</div><div><br></div><div>All various finders and loaders will also be updated as necessary,</div><div>but updating the previous mentioned parts of importlib should be all</div><div>that is required.</div><div><br></div><div><br></div><div>Rest of the standard library</div><div>----------------------------</div><div><br></div><div>The various functions exposed by the ``py_compile`` and</div><div>``compileall`` functions will be updated as necessary to make sure</div><div>they follow the new bytecode file name semantics [6]_, [1]_. The CLI</div><div>for the ``compileall`` module will not be directly affected (the</div><div>``-b`` flag will be implicit as it will no longer generate ``.pyo``</div><div>files when ``-O`` is specified).</div><div><br></div><div><br></div><div>Compatibility Considerations</div><div>============================</div><div><br></div><div>Any code directly manipulating bytecode files from Python 3.2 on</div><div>will need to consider the impact of this change on their code (prior</div><div>to Python 3.2 -- including all of Python 2 -- there was no</div><div>__pycache__ which already necessitates bifurcating bytecode file</div><div>handling support). If code was setting the ``debug_override``</div><div>argument to ``importlib.util.cache_from_source()`` then care will be</div><div>needed if they want the path to a bytecode file with an optimization</div><div>level of 2. Otherwise only code **not** using</div><div>``importlib.util.cache_from_source()`` will need updating.</div><div><br></div><div>As for people who distribute bytecode-only modules (i.e., use a</div><div>bytecode file instead of a source file), they will have to choose</div><div>which optimization level they want their bytecode files to be since</div><div>distributing a ``.pyo`` file with a ``.pyc`` file will no longer be</div><div>of any use. Since people typically only distribute bytecode files for</div><div>code obfuscation purposes or smaller distribution size then only</div><div>having to distribute a single ``.pyc`` should actually be beneficial</div><div>to these use-cases. And since the magic number for bytecode files</div><div>changed in Python 3.5 to support PEP 465 there is no need to support</div><div>pre-existing ``.pyo`` files [8]_.</div><div><br></div><div><br></div><div>Rejected Ideas</div><div>==============</div><div><br></div><div>Completely dropping optimization levels from CPython</div><div>----------------------------------------------------</div><div><br></div><div>Some have suggested that instead of accommodating the various</div><div>optimization levels in CPython, we should instead drop them</div><div>entirely. The argument is that significant performance gains would</div><div>occur from runtime optimizations through something like a JIT and not</div><div>through pre-execution bytecode optimizations.</div><div><br></div><div>This idea is rejected for this PEP as that ignores the fact that</div><div>there are people who do find the pre-existing optimization levels for</div><div>CPython useful. It also assumes that no other Python interpreter</div><div>would find what this PEP proposes useful.</div><div><br></div><div><br></div><div>Alternative formatting of the optimization level in the file name</div><div>-----------------------------------------------------------------</div><div><br></div><div>Using the "opt-" prefix and placing the optimization level between</div><div>the cache tag and file extension is not critical. All options which</div><div>have been considered are:</div><div><br></div><div>* ``importlib.cpython-35.opt-1.pyc``</div><div>* ``importlib.cpython-35.opt1.pyc``</div><div>* ``importlib.cpython-35.o1.pyc``</div><div>* ``importlib.cpython-35.O1.pyc``</div><div>* ``importlib.cpython-35.1.pyc``</div><div>* ``importlib.cpython-35-O1.pyc``</div><div>* ``importlib.O1.cpython-35.pyc``</div><div>* ``importlib.o1.cpython-35.pyc``</div><div>* ``importlib.1.cpython-35.pyc``</div><div><br></div><div>These were initially rejected either because they would change the</div><div>sort order of bytecode files, possible ambiguity with the cache tag,</div><div>or were not self-documenting enough. An informal poll was taken and</div><div>people clearly preferred the formatting proposed by the PEP [9]_.</div><div>Since this topic is non-technical and of personal choice, the issue</div><div>is considered solved.</div><div><br></div><div><br></div><div>Embedding the optimization level in the bytecode metadata</div><div>---------------------------------------------------------</div><div><br></div><div>Some have suggested that rather than embedding the optimization level</div><div>of bytecode in the file name that it be included in the file's</div><div>metadata instead. This would mean every interpreter had a single copy</div><div>of bytecode at any time. Changing the optimization level would thus</div><div>require rewriting the bytecode, but there would also only be a single</div><div>file to care about.</div><div><br></div><div>This has been rejected due to the fact that Python is often installed</div><div>as a root-level application and thus modifying the bytecode file for</div><div>modules in the standard library are always possible. In this</div><div>situation integrators would need to guess at what a reasonable</div><div>optimization level was for users for any/all situations. By</div><div>allowing multiple optimization levels to co-exist simultaneously it</div><div>frees integrators from having to guess what users want and allows</div><div>users to utilize the optimization level they want.</div><div><br></div><div><br></div><div>References</div><div>==========</div><div><br></div><div>.. [1] The compileall module</div><div> (<a href="https://docs.python.org/3/library/compileall.html#module-compileall" target="_blank">https://docs.python.org/3/library/compileall.html#module-compileall</a>)</div><div><br></div><div>.. [2] The astoptimizer project</div><div> (<a href="https://pypi.python.org/pypi/astoptimizer" target="_blank">https://pypi.python.org/pypi/astoptimizer</a>)</div><div><br></div><div>.. [3] ``importlib.util.cache_from_source()``</div><div> (<a href="https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source" target="_blank">https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source</a>)</div><div><br></div><div>.. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1</div><div> (<a href="https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437" target="_blank">https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437</a>)</div><div><br></div><div>.. [5] PEP 3147, PYC Repository Directories, Warsaw</div><div> (<a href="http://www.python.org/dev/peps/pep-3147" target="_blank">http://www.python.org/dev/peps/pep-3147</a>)</div><div><br></div><div>.. [6] The py_compile module</div><div> (<a href="https://docs.python.org/3/library/compileall.html#module-compileall" target="_blank">https://docs.python.org/3/library/compileall.html#module-compileall</a>)</div><div><br></div><div>.. [7] The importlib.machinery module</div><div> (<a href="https://docs.python.org/3/library/importlib.html#module-importlib.machinery" target="_blank">https://docs.python.org/3/library/importlib.html#module-importlib.machinery</a>)</div><div><br></div><div>.. [8] ``importlib.util.MAGIC_NUMBER``</div><div> (<a href="https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER" target="_blank">https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER</a>)</div><div><br></div><div>.. [9] Informal poll of file name format options on Google+</div><div> (<a href="https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm" target="_blank">https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm</a>)</div><div><br></div><div>.. [10] The PyPy Project</div><div> (<a href="http://pypy.org/" target="_blank">http://pypy.org/</a>)</div><div><br></div><div><br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div><div><br></div><div><br></div><div> </div><div>..</div><div> Local Variables:</div><div> mode: indented-text</div><div> indent-tabs-mode: nil</div><div> sentence-end-double-space: t</div><div> fill-column: 70</div><div> coding: utf-8</div><div> End:</div></div><div><br></div></div>
<br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/guido%40python.org" target="_blank">https://mail.python.org/mailman/options/python-dev/guido%40python.org</a><br>
<br></blockquote></div></div><div class="gmail_extra"><br><br clear="all"><br>-- <br><div>--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div></blockquote></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div>--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div>
</blockquote></div>