Mailman 3 March 2015 - Python-Dev

Google Summer of Code
by Jim Baker 21 Mar '15

21 Mar '15

Jython plans to participate in the Google Summer of Code for 2015. If you are interested, I have outlined a number of projects on our ideas page that students could work on: - Work on JyNI, which adds C extension API support to Jython - Performance optimizations, including startup time - Python bytecode compiler for Python source (must be written in Java!) See https://github.com/jythontools/gsoc2015 for the ideas page. Pull requests and comments gladly accepted to expand this list of ideas. The ideal student must be a very strong Java developer, with an existing track record of projects on such sites as GitHub and BitBucket. Knowledge of C and especially of the CPython source code invariably helps. Ideally you will also become a Jython committer as a result of your intense participation this summer. Note that the deadline for student proposals is very soon, Friday March 27. I apologize we are just getting our participation for GSOC together, but we have also been busy finalizing the Jython 2.7.0 release candidate. -- - Jim jim.baker(a){colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker

1 0

Final call for PEP 488: eliminating PYO files
by Brett Cannon 21 Mar '15

21 Mar '15

I have decided to have the default case of no optimization levels mean that the .pyc file name will have *no* optimization level specified in the name and thus be just as it is today. I made this decision due to potential backwards-compatibility issues -- although I expect them to be minutes -- and to not force other implementations like PyPy to have some bogus value set since they don't have .pyo files to begin with (PyPy actually uses bytecode for -O and don't bother with -OO since PyPy already uses a bunch of memory when running). Since this closes out the last open issue, I need either a BDFL decision or a BDFAP to be assigned to make a decision. Guido? ====================================== PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 2015-03-13 2015-03-20 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in the bytecode repository directory when it's called for (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` **or** ``-OO`` is specified). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. To put this in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. This means that bytecode-only modules will only load from their non-optimized ``.pyc`` file name. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate the bytecode file will be incorporated into the bytecode file name. When no optimization level is specified, the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name semantics). This increases backwards-compatibility while also being more understanding of Python implementations which have no use for optimization levels (e.g., PyPy[10]_). Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression when an optimization level is specified to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion in case "0" was the leading prefix of the optimization level. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, if ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. It should be noted that this change in no way affects the performance of import. Since the import system looks for a single bytecode file based on the optimization level of the interpreter already and generates a new bytecode file if it doesn't exist, the introduction of potentially more bytecode files in the ``__pycache__`` directory has no effect in terms of stat calls. The interpreter will continue to look for only a single bytecode file based on the optimization level and thus no increase in stat calls will occur. The only potentially negative result of this PEP is the probable increase in the number of ``.pyc`` files and thus increase in storage use. But for platforms where this is an issue, ``sys.dont_write_bytecode`` exists to turn off bytecode generation so that it can be controlled offline. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as well as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed (including no optimization). Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own two optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter at this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicit as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== Completely dropping optimization levels from CPython ---------------------------------------------------- Some have suggested that instead of accommodating the various optimization levels in CPython, we should instead drop them entirely. The argument is that significant performance gains would occur from runtime optimizations through something like a JIT and not through pre-execution bytecode optimizations. This idea is rejected for this PEP as that ignores the fact that there are people who do find the pre-existing optimization levels for CPython useful. It also assumes that no other Python interpreter would find what this PEP proposes useful. Alternative formatting of the optimization level in the file name ----------------------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-1.pyc`` * ``importlib.cpython-35.opt1.pyc`` * ``importlib.cpython-35.o1.pyc`` * ``importlib.cpython-35.O1.pyc`` * ``importlib.cpython-35.1.pyc`` * ``importlib.cpython-35-O1.pyc`` * ``importlib.O1.cpython-35.pyc`` * ``importlib.o1.cpython-35.pyc`` * ``importlib.1.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. An informal poll was taken and people clearly preferred the formatting proposed by the PEP [9]_. Since this topic is non-technical and of personal choice, the issue is considered solved. Embedding the optimization level in the bytecode metadata --------------------------------------------------------- Some have suggested that rather than embedding the optimization level of bytecode in the file name that it be included in the file's metadata instead. This would mean every interpreter had a single copy of bytecode at any time. Changing the optimization level would thus require rewriting the bytecode, but there would also only be a single file to care about. This has been rejected due to the fact that Python is often installed as a root-level application and thus modifying the bytecode file for modules in the standard library are always possible. In this situation integrators would need to guess at what a reasonable optimization level was for users for any/all situations. By allowing multiple optimization levels to co-exist simultaneously it frees integrators from having to guess what users want and allows users to utilize the optimization level they want. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_fro… ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py… ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) .. [9] Informal poll of file name format options on Google+ (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) .. [10] The PyPy Project (http://pypy.org/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

3 5

Fwd: class os.DirEntry is confusing,
by 罗勇刚(Yonggang Luo) 21 Mar '15

21 Mar '15

---------- Forwarded message ---------- From: 罗勇刚(Yonggang Luo) <luoyonggang(a)gmail.com> Date: 2015-03-21 2:24 GMT+08:00 Subject: class os.DirEntry is confusing, To: Python <python-ideas(a)python.org> Should be replaced with os.FsEntry or os.ScanEntry -- 此致礼罗勇刚 Yours sincerely, Yonggang Luo -- 此致礼罗勇刚 Yours sincerely, Yonggang Luo

2 1

pep 7
by Neil Girdhar 21 Mar '15

21 Mar '15

If ever someone wants to clean up the repository to conform to PEP 7, I wrote a program that catches a couple hundred PEP 7 violations in ./Python alone (1400 in the whole codebase): import os import re def grep(path, regex): reg_obj = re.compile(regex, re.M) res = [] for root, dirs, fnames in os.walk(path): for fname in fnames: if fname.endswith('.c'): path = os.path.join(root, fname) with open(path) as f: data = f.read() for m in reg_obj.finditer(data): line_number = sum(c == '\n' for c in data[:m.start()]) + 1 res.append("{}: {}".format(path, line_number)) return res for pattern in [ r'^\s*\|\|', r'^\s*\&\&', r'} else {', r'\<return\s*\(', ]: print("Searching for", pattern) print("\n".join(grep('.', pattern))) In my experience, it was hard to write PEP 7 conforming code when the surrounding code is inconsistent. Best, Neil

3 5

2.7.9 regression in argparse
by anatoly techtonik 20 Mar '15

20 Mar '15

Just a pointer for possible regression http://bugs.python.org/issue23058 -- anatoly t.

2 1

Summary of Python tracker Issues
by Python tracker 20 Mar '15

20 Mar '15

ACTIVITY SUMMARY (2015-03-13 - 2015-03-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4798 (-19) closed 30685 (+81) total 35483 (+62) Open issues with patches: 2258 Issues opened (49) ================== #2211: Cookie.Morsel interface needs update http://bugs.python.org/issue2211 reopened by haypo #22181: os.urandom() should use Linux 3.17 getrandom() syscall http://bugs.python.org/issue22181 reopened by haypo #22359: Remove incorrect uses of recursive make http://bugs.python.org/issue22359 reopened by doko #22516: Windows Installer won't - even when using "just for me"option http://bugs.python.org/issue22516 reopened by NaCl #23187: Segmentation fault, possibly asyncio related http://bugs.python.org/issue23187 reopened by haypo #23637: Warnings error with non-ascii chars. http://bugs.python.org/issue23637 reopened by serhiy.storchaka #23661: Setting a exception side_effect on a mock from create_autospec http://bugs.python.org/issue23661 opened by Ignacio Rossi #23662: Cookie.domain is undocumented http://bugs.python.org/issue23662 opened by Malcolm Smith #23663: Crash on failure in ctypes on Cygwin http://bugs.python.org/issue23663 opened by elieux #23664: Modernize HTML output of difflib.HtmlDiff.make_file() http://bugs.python.org/issue23664 opened by berker.peksag #23665: Provide IDLE menu option to set command-line arguments http://bugs.python.org/issue23665 opened by rhettinger #23666: Add shell session logging option to IDLE http://bugs.python.org/issue23666 opened by rhettinger #23667: IDLE to provide option for making trailing whitespace visible http://bugs.python.org/issue23667 opened by rhettinger #23668: Support os.[f]truncate on Windows http://bugs.python.org/issue23668 opened by steve.dower #23669: test_socket.NonBlockingTCPTests failing due to race condition http://bugs.python.org/issue23669 opened by steve.dower #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 opened by freakboy3742 #23671: string.Template doesn't work with the self keyword argument http://bugs.python.org/issue23671 opened by serhiy.storchaka #23672: IDLE can crash if file name contains non-BMP Unicode character http://bugs.python.org/issue23672 opened by kamisky #23674: super() documentation isn't very clear http://bugs.python.org/issue23674 opened by Tapani Kiiskinen #23675: glossary entry for 'method resolution order' links to somethin http://bugs.python.org/issue23675 opened by r.david.murray #23676: Add support of UnicodeTranslateError in standard error handler http://bugs.python.org/issue23676 opened by serhiy.storchaka #23677: Mention dict and set comps in library reference http://bugs.python.org/issue23677 opened by FrankMillman #23680: Sporadic freeze in test_interrupted_write_retry_text http://bugs.python.org/issue23680 opened by pitrou #23684: urlparse() documentation does not account for default scheme http://bugs.python.org/issue23684 opened by vadmium #23686: Update Windows and OS X installer OpenSSL to 1.0.2a http://bugs.python.org/issue23686 opened by alex #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 opened by wolma #23689: Memory leak in Modules/sre_lib.h http://bugs.python.org/issue23689 opened by abacabadabacaba #23690: re functions never release GIL http://bugs.python.org/issue23690 opened by abacabadabacaba #23691: re.finditer iterator is not reentrant, but doesn't protect aga http://bugs.python.org/issue23691 opened by abacabadabacaba #23692: Undocumented feature prevents re module from finding certain m http://bugs.python.org/issue23692 opened by abacabadabacaba #23693: timeit accuracy could be better http://bugs.python.org/issue23693 opened by rbcollins #23695: idiom for clustering a data series into n-length groups http://bugs.python.org/issue23695 opened by Paddy McCarthy #23697: Module level map & submit for concurrent.futures http://bugs.python.org/issue23697 opened by ncoghlan #23698: Fix inconsistencies in behaviour and document it correctly for http://bugs.python.org/issue23698 opened by pythonhacker #23699: Add a macro to ease writing rich comparisons http://bugs.python.org/issue23699 opened by encukou #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 opened by Claudiu.Popa #23702: docs.python.org/3/howto/descriptor.html still refers to "unbou http://bugs.python.org/issue23702 opened by pfalcon #23703: urljoin() with no directory segments duplicates filename http://bugs.python.org/issue23703 opened by vadmium #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 opened by rhettinger #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 opened by rhettinger #23706: pathlib.Path.write_text should include a newline argument http://bugs.python.org/issue23706 opened by daniel.ugra #23710: C API doc for PyObject_HEAD is outdated http://bugs.python.org/issue23710 opened by tpievila #23712: Experiment: Assume that exact unicode hashes are perfect disc http://bugs.python.org/issue23712 opened by rhettinger #23713: intermittent failure of multiprocessing unit test test_imap_un http://bugs.python.org/issue23713 opened by davin #23717: strptime() with year-weekday pair can produce invalid data http://bugs.python.org/issue23717 opened by serhiy.storchaka #23718: strptime() can produce invalid date with negative year day http://bugs.python.org/issue23718 opened by serhiy.storchaka #23719: PEP 475: port test_eintr to Windows http://bugs.python.org/issue23719 opened by haypo #23721: Set up a daily test coverage run http://bugs.python.org/issue23721 opened by brett.cannon #23722: During metaclass.__init__, super() of the constructed class do http://bugs.python.org/issue23722 opened by Martin.Teichmann Most recent 15 issues with no replies (15) ========================================== #23722: During metaclass.__init__, super() of the constructed class do http://bugs.python.org/issue23722 #23721: Set up a daily test coverage run http://bugs.python.org/issue23721 #23719: PEP 475: port test_eintr to Windows http://bugs.python.org/issue23719 #23718: strptime() can produce invalid date with negative year day http://bugs.python.org/issue23718 #23710: C API doc for PyObject_HEAD is outdated http://bugs.python.org/issue23710 #23706: pathlib.Path.write_text should include a newline argument http://bugs.python.org/issue23706 #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 #23702: docs.python.org/3/howto/descriptor.html still refers to "unbou http://bugs.python.org/issue23702 #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 #23697: Module level map & submit for concurrent.futures http://bugs.python.org/issue23697 #23692: Undocumented feature prevents re module from finding certain m http://bugs.python.org/issue23692 #23691: re.finditer iterator is not reentrant, but doesn't protect aga http://bugs.python.org/issue23691 #23675: glossary entry for 'method resolution order' links to somethin http://bugs.python.org/issue23675 #23669: test_socket.NonBlockingTCPTests failing due to race condition http://bugs.python.org/issue23669 Most recent 15 issues waiting for review (15) ============================================= #23713: intermittent failure of multiprocessing unit test test_imap_un http://bugs.python.org/issue23713 #23712: Experiment: Assume that exact unicode hashes are perfect disc http://bugs.python.org/issue23712 #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 #23703: urljoin() with no directory segments duplicates filename http://bugs.python.org/issue23703 #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 #23699: Add a macro to ease writing rich comparisons http://bugs.python.org/issue23699 #23689: Memory leak in Modules/sre_lib.h http://bugs.python.org/issue23689 #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 #23677: Mention dict and set comps in library reference http://bugs.python.org/issue23677 #23676: Add support of UnicodeTranslateError in standard error handler http://bugs.python.org/issue23676 #23671: string.Template doesn't work with the self keyword argument http://bugs.python.org/issue23671 #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 #23668: Support os.[f]truncate on Windows http://bugs.python.org/issue23668 #23663: Crash on failure in ctypes on Cygwin http://bugs.python.org/issue23663 Top 10 most discussed issues (10) ================================= #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 14 msgs #23187: Segmentation fault, possibly asyncio related http://bugs.python.org/issue23187 13 msgs #2211: Cookie.Morsel interface needs update http://bugs.python.org/issue2211 12 msgs #23648: PEP 475 meta issue http://bugs.python.org/issue23648 11 msgs #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 11 msgs #23441: rlcompleter: tab on empty prefix => insert spaces http://bugs.python.org/issue23441 9 msgs #23546: Windows, 'Edit withIDLE', and multplie installed versions http://bugs.python.org/issue23546 9 msgs #23680: Sporadic freeze in test_interrupted_write_retry_text http://bugs.python.org/issue23680 9 msgs #8027: distutils fail to determine c++ linker with unixcompiler if us http://bugs.python.org/issue8027 8 msgs #23042: ctypes module doesn't build on FreeBSD x86 http://bugs.python.org/issue23042 8 msgs Issues closed (78) ================== #2052: Allow changing difflib._file_template character encoding. http://bugs.python.org/issue2052 closed by berker.peksag #12155: queue example doesn't stop worker threads http://bugs.python.org/issue12155 closed by haypo #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 closed by haypo #12500: Skip test_ssl.test_connect_ex() on connection error http://bugs.python.org/issue12500 closed by haypo #14555: clock_gettime/settime/getres: Add more clock identifiers http://bugs.python.org/issue14555 closed by haypo #15612: Rewrite StringIO to use the _PyUnicodeWriter API http://bugs.python.org/issue15612 closed by haypo #17515: Add sys.setasthook() to allow to use a custom AST optimizer http://bugs.python.org/issue17515 closed by haypo #17628: str==str: compare the first character before calling memcmp() http://bugs.python.org/issue17628 closed by haypo #17911: traceback: add a new thin class storing a traceback without st http://bugs.python.org/issue17911 closed by rbcollins #18507: import_init() should not use Py_FatalError() but return an err http://bugs.python.org/issue18507 closed by haypo #18983: Specify time unit for timeit CLI http://bugs.python.org/issue18983 closed by rbcollins #19233: test_io.test_interrupted_write_retry_text() hangs on Solaris 1 http://bugs.python.org/issue19233 closed by haypo #19428: marshal: error cases are not documented http://bugs.python.org/issue19428 closed by haypo #19564: test_context() of test_multiprocessing_spawn hangs on "x86 Gen http://bugs.python.org/issue19564 closed by haypo #19654: test_tkinter sporadic skipped on "x86 Tiger 3.x" buildbot http://bugs.python.org/issue19654 closed by haypo #19748: test_time failures on AIX http://bugs.python.org/issue19748 closed by haypo #19816: test.regrtest: use tracemalloc to detect memory leaks? http://bugs.python.org/issue19816 closed by haypo #20114: Sporadic failure of test_semaphore_tracker() of test_multiproc http://bugs.python.org/issue20114 closed by haypo #20204: pydocs fails for some C implemented classes http://bugs.python.org/issue20204 closed by serhiy.storchaka #20600: test_create_server_ssl_verify_failed() failure on "PPC64 AIX 3 http://bugs.python.org/issue20600 closed by haypo #20614: test.script_helper should copy SYSTEMROOT environment variable http://bugs.python.org/issue20614 closed by haypo #20680: Pickle socket enums by names http://bugs.python.org/issue20680 closed by ethan.furman #20717: test_4_daemon_threads() fails randomly on "x86 Windows Server http://bugs.python.org/issue20717 closed by haypo #20719: testPythonOrg() of test_robotparser fails with the new www.pyt http://bugs.python.org/issue20719 closed by haypo #21644: Optimize bytearray(int) constructor to use calloc() http://bugs.python.org/issue21644 closed by haypo #21734: compilation of the _ctypes module fails on OpenIndiana: ffi_pr http://bugs.python.org/issue21734 closed by haypo #21735: test_threading.test_main_thread_after_fork_from_nonmain_thread http://bugs.python.org/issue21735 closed by haypo #21788: Rework Python finalization http://bugs.python.org/issue21788 closed by haypo #22333: test_threaded_import.test_parallel_meta_path() failed on x86 W http://bugs.python.org/issue22333 closed by haypo #22521: ctypes compilation fails on FreeBSD: Undefined symbol "ffi_cal http://bugs.python.org/issue22521 closed by haypo #22634: importing _ctypes failed: undefined symbol: ffi_call_win32 http://bugs.python.org/issue22634 closed by berker.peksag #22826: Support context management protocol in bkfile http://bugs.python.org/issue22826 closed by serhiy.storchaka #22861: [2.7] ssl._dnsname_match() and unicode http://bugs.python.org/issue22861 closed by haypo #22903: unittest creates non-picklable errors http://bugs.python.org/issue22903 closed by pitrou #22937: unittest errors can't show locals http://bugs.python.org/issue22937 closed by rbcollins #22971: test_pickle: "Fatal Python error: Cannot recover from stack ov http://bugs.python.org/issue22971 closed by haypo #22984: test_json.test_endless_recursion(): "Fatal Python error: Canno http://bugs.python.org/issue22984 closed by haypo #23001: Accept mutable bytes-like objects http://bugs.python.org/issue23001 closed by serhiy.storchaka #23136: BUG in how _strptime() handles week 0 http://bugs.python.org/issue23136 closed by serhiy.storchaka #23285: PEP 475 - EINTR handling http://bugs.python.org/issue23285 closed by haypo #23303: test_license_exists_at_url() of test_site fails on "x86 XP-4 3 http://bugs.python.org/issue23303 closed by haypo #23333: asyncio: add a new Protocol.connection_failed() method http://bugs.python.org/issue23333 closed by haypo #23353: generator bug with exception: tstate->exc_value is not cleared http://bugs.python.org/issue23353 closed by pitrou #23419: Faster default __reduce__ for classes without __init__ http://bugs.python.org/issue23419 closed by serhiy.storchaka #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 closed by haypo #23549: heapq docs should be more precise about how to access the smal http://bugs.python.org/issue23549 closed by eli.bendersky #23568: unittest.mock.MagicMock doesn't support __rdivmod__ http://bugs.python.org/issue23568 closed by berker.peksag #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 closed by rbcollins #23593: Update Windows and OS X installer OpenSSL to 1.0.2 http://bugs.python.org/issue23593 closed by ned.deily #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 closed by steve.dower #23624: str.center inconsistent with format "^" http://bugs.python.org/issue23624 closed by rhettinger #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 closed by brett.cannon #23631: 3.5 (a2) traceback regression snarls Idle http://bugs.python.org/issue23631 closed by rbcollins #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 closed by pitrou #23644: g++ module compile fails with ‘_Atomic’ does not name a ty http://bugs.python.org/issue23644 closed by haypo #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 closed by python-dev #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 closed by haypo #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 closed by h-miyajima #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 closed by Dan Nawrocki #23673: IntEnum is unpicklable by previous Python versions http://bugs.python.org/issue23673 closed by ethan.furman #23678: imaplib status command cannot handle folder name containing wh http://bugs.python.org/issue23678 closed by r.david.murray #23679: SSL Ciphers RC4 http://bugs.python.org/issue23679 closed by benjamin.peterson #23681: Have -b warn when directly comparing ints and bytes http://bugs.python.org/issue23681 closed by serhiy.storchaka #23682: distutils docs still talk about compatibility with Python 2.2 http://bugs.python.org/issue23682 closed by berker.peksag #23683: allow timeit to run expensive reset code per repeats http://bugs.python.org/issue23683 closed by rbcollins #23685: Fix usage of PyMODINIT_FUNC http://bugs.python.org/issue23685 closed by haypo #23687: Stacktrace identifies wrong line in multiline list comprehensi http://bugs.python.org/issue23687 closed by r.david.murray #23694: PEP 475: handle EINTR in fileutils.c http://bugs.python.org/issue23694 closed by haypo #23696: zipimport: chain ImportError to OSError http://bugs.python.org/issue23696 closed by haypo #23700: tempfile.NamedTemporaryFile can close too early if used as ite http://bugs.python.org/issue23700 closed by serhiy.storchaka #23707: PEP 475: os.urandom() doesn't handle EINTR correctly http://bugs.python.org/issue23707 closed by haypo #23708: PEP 475: Add _Py_read() and _Py_write() functions http://bugs.python.org/issue23708 closed by haypo #23709: Refactor ossaudiodev: use _Py_read and _Py_write with the Py_b http://bugs.python.org/issue23709 closed by haypo #23711: ConfigParser module blames on section less ini file http://bugs.python.org/issue23711 closed by vadmium #23714: Set comprehension + eval doesn't work inside function definiti http://bugs.python.org/issue23714 closed by ned.deily #23715: PEP 475: handle EINTR in the signal module http://bugs.python.org/issue23715 closed by haypo #23716: test_multiprocessing_spawn hangs on x86 Ubuntu Shared 3.x http://bugs.python.org/issue23716 closed by haypo #23720: __del__() order is broken since 3.4.0 http://bugs.python.org/issue23720 closed by pitrou

1 0

PEP 489: Redesigning extension module loading
by Petr Viktorin 18 Mar '15

18 Mar '15

Hello, On import-sig, I've agreed to continue Nick Coghlan's work on making extension modules act more like Python ones, work well with PEP 451 (ModuleSpec), and encourage proper subinterpreter and reloading support. Here is the resulting PEP. I don't have a patch yet, but I'm working on it. There's a remaining open issue: providing a tool that can be run in test suites to check if a module behaves well with subinterpreters/reloading. I believe it's out of scope for this PEP but speak out if you disagree. Please discuss on import-sig. ======================= PEP: 489 Title: Redesigning extension module loading Version: $Revision$ Last-Modified: $Date$ Author: Petr Viktorin <encukou(a)gmail.com>, Stefan Behnel <stefan_ml(a)behnel.de>, Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: import-sig(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2013 Python-Version: 3.5 Post-History: 23-Aug-2013, 20-Feb-2015 Resolution: Abstract ======== This PEP proposes a redesign of the way in which extension modules interact with the import machinery. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve them by bringing extension modules closer to the way Python modules behave; specifically to hook into the ModuleSpec-based loading mechanism introduced in PEP 451. Extensions that do not require custom memory layout for their module objects may be executed in arbitrary pre-defined namespaces, paving the way for extension modules being runnable with Python's ``-m`` switch. Other extensions can use custom types for their module implementation. Module types are no longer restricted to types.ModuleType. This proposal makes it easy to support properties at the module level and to safely store arbitrary global state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Extension authors are encouraged to take these issues into account when using the new API. Motivation ========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed (PEP 302). A ModuleSpec object (PEP 451) is used to hold information about the module, and passed to the relevant hooks. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. The initialisation function is not passed ModuleSpec information about the loaded module, such as the __file__ or fully-qualified name. This hinders relative imports and resource loading. This is specifically a problem for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of __file__ and __name__ information hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The other disadvantage of the discrepancy is that existing Python programmers learning C cannot effectively map concepts between the two domains. As long as extension modules are fundamentally different from pure Python ones in the way they're initialised, they are harder for people to pick up without relying on something like cffi, SWIG or Cython to handle the actual extension module creation. Currently, extension modules are also not added to sys.modules until they are fully initialized, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. Without the fully qualified module name, it is not trivial to correctly add the module to sys.modules either. Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or reloading, and, while it is possible with the current infrastructure to support these features, it is neither easy nor efficient. Addressing these issues was the goal of PEP 3121, but many extensions, including some in the standard library, took the least-effort approach to porting to Python 3, leaving these issues unresolved. This PEP keeps the backwards-compatible behavior, which should reduce pressure and give extension authors adequate time to consider these issues when porting. The current process =================== Currently, extension modules export an initialisation function named "PyInit_modulename", named after the file name of the shared library. This function is executed by the import machinery and must return either NULL in the case of an exception, or a fully initialised module object. The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef struct. It then continues to initialise it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the fully qualified name of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The proposal ============ The current extension module initialisation will be deprecated in favour of a new initialisation scheme. Since the current scheme will continue to be available, existing code will continue to work unchanged, including binary compatibility. Extension modules that support the new initialisation scheme must export the public symbol "PyModuleExec_modulename", and optionally "PyModuleCreate_modulename", where "modulename" is the name of the module. This mimics the previous naming convention for the "PyInit_modulename" function. If defined, these symbols must resolve to C functions with the following signatures, respectively:: int (*PyModuleExecFunction)(PyObject* module) PyObject* (*PyModuleCreateFunction)(PyObject* module_spec) The PyModuleExec function ------------------------- The PyModuleExec function is used to implement "loader.exec_module" defined in PEP 451. It function will be called to initialize a module. (Usually, this amounts to setting the module's initial attributes.) This happens in two situations: when the module is first initialized for a given (sub-)interpreter, and possibly later when the module is reloaded. When PyModuleExec is called, the module has already been added to sys.modules, and import-related attributes specified in PEP 451 [#pep-0451-attributes]_) have been set on the module. The "module" argument receives the module object to initialize. If PyModuleCreate is defined, "module" will generally be the the object returned by it. It is possible for a custom loader to pass any object to PyModuleExec, so this function should check and fail with TypeError if the module's type is unsupported. Any other assumptions should also be checked. If PyModuleCreate is not defined, PyModuleExec is expected to operate on any Python object for which attributes can be added by PyObject_GetAttr* and retrieved by PyObject_SetAttr*. This allows loading an extension into a pre-created module, making it possible to run it as __main__ in the future, participate in certain lazy-loading schemes [#lazy_import_concerns]_, or enable other creative uses. If PyModuleExec replaces the module's entry in sys.modules, the new object will be used and returned by importlib machinery. (This mirrors the behavior of Python modules. Note that for extensions, implementing PyModuleCreate is usually a better solution for the use cases this serves.) The function must return ``0`` on success, or, on error, set an exception and return ``-1``. The PyModuleCreate function --------------------------- The optional PyModuleCreate function is used to implement "loader.create_module" defined in PEP 451. By exporting it, an extension module indicates that it uses a custom module object. This prevents loading the extension in a pre-created module, but gives greater flexibility in allowing a custom C-level layout of the module object. Most extensions will not need to implement this function. The "module_spec" argument receives a "ModuleSpec" instance, as defined in PEP 451. When called, this function must create and return a module object, or set an exception and return NULL. There is no requirement for the returned object to be an instance of types.ModuleType. Any type can be used, as long as it supports setting and getting attributes, including at least the import-related attributes specified in PEP 451 [#pep-0451-attributes]_. This follows the current support for allowing arbitrary objects in sys.modules and makes it easier for extension modules to define a type that exactly matches their needs for holding module state. Note that when this function is called, the module's entry in sys.modules is not populated yet. Attempting to import the same module again (possibly transitively), may lead to an infinite loop. Extension authors are advised to keep PyModuleCreate minimal, an in particular to not call user code from it. If PyModuleCreate is not defined, the default loader will construct a module object as if with PyModule_New. Initialization helper functions ------------------------------- For two initialization tasks previously done by PyModule_Create, two functions are introduced:: int PyModule_SetDocString(PyObject *m, const char *doc) int PyModule_AddFunctions(PyObject *m, PyMethodDef *functions) These set the module docstring, and add the module functions, respectively. Both will work on any Python object that supports setting attributes. They return ``0`` on success, and on failure, they set the exception and return ``-1``. PyCapsule convenience functions ------------------------------- Instead of custom module objects, PyCapsule will become the preferred mechanism for storing per-module C data. Two new convenience functions will be added to help with this. * :: PyObject *PyModule_AddCapsule( PyObject *module, const char *module_name, const char *attribute_name, void *pointer, PyCapsule_Destructor destructor) Add a new PyCapsule to *module* as *attribute_name*. The capsule name is formed by joining *module_name* and *attribute_name* by a dot. This convenience function can be used from a module initialization function instead of separate calls to PyCapsule_New and PyModule_AddObject. Returns a borrowed reference to the new capsule, or NULL (with exception set) on failure. * :: void *PyModule_GetCapsulePointer( PyObject *module, const char *module_name, const char *attribute_name) Returns the pointer stored in *module* as *attribute_name*, or NULL (with an exception set) on failure. The capsule name is formed by joining *module_name* and *attribute_name* by a dot. This convenience function can be used instead of separate calls to PyObject_GetAttr and PyCapsule_GetPointer. Extension authors are encouraged to define a macro to call PyModule_GetCapsulePointer and cast the result to an appropriate type. Generalizing PyModule_* functions --------------------------------- The following functions and macros will be modified to work on any object that supports attribute access: * PyModule_GetNameObject * PyModule_GetName * PyModule_GetFilenameObject * PyModule_GetFilename * PyModule_AddIntConstant * PyModule_AddStringConstant * PyModule_AddIntMacro * PyModule_AddStringMacro * PyModule_AddObject The PyModule_GetDict function will continue to only work on true module objects. This means that it should not be used on extension modules that only define PyModuleExec. Legacy Init ----------- If PyModuleExec is not defined, the import machinery will try to initialize the module using the PyModuleInit hook, as described in PEP 3121. If PyModuleExec is defined, PyModuleInit will be ignored. Modules requiring compatibility with previous versions of CPython may implement PyModuleInit in addition to the new hook. Subinterpreters and Interpreter Reloading ----------------------------------------- Extensions using the new initialization scheme are expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly. The mechanism is designed to make this easy, but care is still required on the part of the extension author. No user-defined functions, methods, or instances may leak to different interpreters. To achieve this, all module-level state should be kept in either the module dict, or in the module object. A simple rule of thumb is: Do not define any static data, except built-in types with no mutable or user-settable class attributes. Module Reloading ---------------- Reloading an extension module will re-execute its PyModuleInit function. Similar caveats apply to reloading an extension module as to reloading a Python module. Notably, attributes or any other state of the module are not reset before reloading. Additionally, due to limitations in shared library loading (both dlopen on POSIX and LoadModuleEx on Windows), it is not generally possible to load a modified library after it has changed on disk. Therefore, reloading extension modules is of limited use. Multiple modules in one library ------------------------------- To support multiple Python modules in one shared library, the library must export appropriate PyModuleExec_<name> or PyModuleCreate_<name> hooks for each exported module. The modules are loaded using a ModuleSpec with origin set to the name of the library file, and name set to the module name. Note that this mechanism can currently only be used to *load* such modules, not to *find* them. XXX: This is an existing issue; either fix it/wait for a fix or provide an example of how to load such modules. Implementation ============== XXX - not started Open issues =========== We should expose some kind of API in importlib.util (or a better place?) that can be used to check that a module works with reloading and subinterpreters. Related issues ============== The runpy module will need to be modified to take advantage of PEP 451 and this PEP. This is out of scope for this PEP. Previous Approaches =================== Stefan Behnel's initial proto-PEP [#stefans_protopep]_ had a "PyInit_modulename" hook that would create a module class, whose ``__init__`` would be then called to create the module. This proposal did not correspond to the (then nonexistent) PEP 451, where module creation and initialization is broken into distinct steps. It also did not support loading an extension into pre-existing module objects. Nick Coghlan proposed the Create annd Exec hooks, and wrote a prototype implementation [#nicks-prototype]_. At this time PEP 451 was still not implemented, so the prototype does not use ModuleSpec. References ========== .. [#lazy_import_concerns] https://mail.python.org/pipermail/python-dev/2013-August/128129.html .. [#pep-0451-attributes] https://www.python.org/dev/peps/pep-0451/#attributes .. [#stefans_protopep] https://mail.python.org/pipermail/python-dev/2013-August/128087.html .. [#nicks-prototype] https://mail.python.org/pipermail/python-dev/2013-August/128101.html Copyright ========= This document has been placed in the public domain.

4 4

Possible wrong behavior of the dict?
by Zaur Shibzukhov 17 Mar '15

17 Mar '15

Hello! In order to explain, let define subclass of dict: class Pair: def __init__(self, key, val): self.key = key self.val = val class MyDict(dict): # def __init__(self, *args, **kwds): if len(args) > 1: raise TypeError('Expected at most 1 arguments, but got %d' % len(args)) for key, val in args[0]: self[key] = val for key, val in kwds.items(): self[key] = val def __getitem__(self, key): pair = dict.__getitem__(key) return pair.value def __setitem__(self, key, val): if key in self: pair = dict.__getitem__(key) pair.value = value else: pair = Pair(key, val) dict.__setitem__(self, key, pair) def values(self): for key in self: p = dict.__getitem__(self, key) yield p.value def items(self): for key, p in dict.__iter__(self): yield p.key, p.value The simple test give me strange result: >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>> dict(d) {'a': <__main__.Pair at 0x104ca9e48>, 'b': <__main__.Pair at 0x104ca9e80>, 'c': <__main__.Pair at 0x104ca9eb8>} instead of {'a':1, 'b':2, 'c':3}. Is this right behavior of the dict? --- Zaur Shibzukhov

3 7

Re: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support
by Paul Moore 15 Mar '15

15 Mar '15

On 24 February 2015 at 18:24, Guido van Rossum <guido(a)python.org> wrote: > Here's my review. I really like where this is going but I have a few > questions and suggestions (I can't help myself :-). OK, I've updated both the PEP and the patch based on follow-up discussions. I think (again!) it is ready to go. I've included the updated PEP inline below, it'll be available at peps.python.org as soon as someone has a chance to upload it. Thanks to everyone for the various comments. If I've missed anything that someone thinks I'd said I would change, please let me know. Paul PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth(a)gmail.com>, Paul Moore <p.f.moore(a)gmail.com> Discussions-To: https://mail.python.org/pipermail/python-dev/2015-February/138277.html Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support ======================================== Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the ``__main__`` module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than ``.py``) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications" and "Windowed Python ZIP Applications", and providing some simple tooling to manage the format. A New Python ZIP Application Extension ====================================== The terminology "Python Zip Application" will be the formal term used for a zip-format archive that contains Python code in a form that can be directly executed by Python (specifically, it must have a ``__main__.py`` file in the root directory of the archive). The extension ``.pyz`` will be formally associated with such files. The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python Zip Applications" with the platform launcher so they can be executed. A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a windowed application, indicating whether the console should appear when running the app. On Unix, it would be ideal if the ``.pyz`` extension and the name "Python Zip Application" were registered (in the mime types database?). However, such an association is out of scope for this PEP. Python Zip applications can be prefixed with a ``#!`` line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) On Unix, this allows the OS to run the file with the correct interpreter, via the standard "shebang" support. On Windows, the Python launcher implements shebang support. However, it is always possible to execute a ``.pyz`` application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module ================================== This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. More complete tools for managing Python Zip Applications are encouraged as 3rd party applications on PyPI. Currently, pyzzer [5]_ and pex [6]_ are two such tools. Module Interface ---------------- The zipapp module will provide the following functions: ``create_archive(source, target=None, interpreter=None, main=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create an application archive from *source*. The source can be any of the following: * The name of a directory, in which case a new application archive will be created from the content of that directory. * The name of an existing application archive file, in which case the file is copied to the target. The file name should include the ``.pyz`` extension, if required. * A file object open for reading in bytes mode. The content of the file should be an application archive, and the file object is assumed to be positioned at the start of the archive. The *target* argument determines where the resulting archive will be written: * If it is the name of a file, the archive will be written to that file. * If it is an open file object, the archive will be written to that file object, which must be open for writing in bytes mode. * If the target is omitted (or None), the source must be a directory and the target will be a file with the same name as the source, with a ``.pyz`` extension added. The *interpreter* argument specifies the name of the Python interpreter with which the archive will be executed. It is written as a "shebang" line at the start of the archive. On Unix, this will be interpreted by the OS, and on Windows it will be handled by the Python launcher. Omitting the *interpreter* results in no shebang line being written. If an interpreter is specified, and the target is a filename, the executable bit of the target file will be set. The *main* argument specifies the name of a callable which will be used as the main program for the archive. It can only be specified if the source is a directory, and the source does not already contain a ``__main__.py`` file. The *main* argument should take the form "pkg.module:callable" and the archive will be run by importing "pkg.module" and executing the given callable with no arguments. It is an error to omit *main* if the source is a directory and does not contain a ``__main__.py`` file, as otherwise the resulting archive would not be executable. If a file object is specified for *source* or *target*, it is the caller's responsibility to close it after calling create_archive. When copying an existing archive, file objects supplied only need ``read`` and ``readline``, or ``write`` methods. When creating an archive from a directory, if the target is a file object it will be passed to the ``zipfile.ZipFile`` class, and must supply the methods needed by that class. ``get_interpreter(archive)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Returns the interpreter specified in the shebang line of the *archive*. If there is no shebang, the function returns ``None``. The *archive* argument can be a filename or a file-like object open for reading in bytes mode. Command Line Usage ------------------ The zipapp module can be run with the python ``-m`` flag. The command line interface is as follows:: python -m zipapp directory [options] Create an archive from the given directory. An archive will be created from the contents of that directory. The archive will have the same name as the source directory with a .pyz extension. The following options can be specified: -o archive / --output archive The destination archive will have the specified name. The given name will be used as written, so should include the ".pyz" extension. -p interpreter / --python interpreter The given interpreter will be written to the shebang line of the archive. If this option is not given, the archive will have no shebang line. -m pkg.mod:fn / --main pkg.mod:fn The source directory must not have a __main__.py file. The archiver will write a __main__.py file into the target which calls fn from the module pkg.mod. The behaviour of the command line interface matches that of ``zipapp.create_archive()``. In addition, it is possible to use the command line interface to work with an existing archive:: python -m zipapp app.pyz --show Displays the shebang line of an archive. Output is of the form Interpreter: /usr/bin/env or Interpreter: <none> and is intended for diagnostic use, not for scripts. python -m zipapp app.pyz -o newapp.pyz [-p interpreter] Copy app.pyz to newapp.pyz, modifying the shebang line based on the -p option (as for creating an archive, no -p option means remove the shebang line). Specifying a destination is mandatory. In-place modification of an archive is *not* supported, as the risk of damaging archives is too great for a simple tool. As noted, the archives are standard zip files, and so can be unpacked using any standard ZIP utility or Python's zipfile module. For this reason, no interfaces to list the contents of an archive, or unpack them, are provided or needed. FAQ --- Are you sure a standard ZIP utility can handle ``#!`` at the beginning? Absolutely. The zipfile specification allows for arbitrary data to be prepended to a zipfile. This feature is commonly used by "self-extracting zip" programs. If your archive program can't handle this, it is a bug in your archive program. Isn't zipapp just a very thin wrapper over the zipfile module? Yes. If you prefer to build your own Python zip application archives using other tools, they will work just as well. The zipapp module is a convenience, nothing more. Why not use just use a ``.zip`` or ``.py`` extension? Users expect a ``.zip`` file to be opened with an archive tool, and expect a ``.py`` file to contain readable text. Both would be confusing for this use case. How does this compete with existing package formats? The sdist, bdist and wheel formats are designed for packaging of modules to be installed into an existing Python installation. They are not intended to be used without installing. The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script. Rejected Proposals ================== Convenience Values for Shebang Lines ------------------------------------ Is it worth having "convenience" forms for any of the common interpreter values? For example, ``-p 3`` meaning the same as ``-p "/usr/bin/env python3"``. It would save a lot of typing for the common cases, as well as giving cross-platform options for people who don't want or need to understand the intricacies of shebang handling on "other" platforms. Downsides are that it's not obvious how to translate the abbreviations. For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3", "python3", or something else? Also, there is no obvious short form for the key case of "/usr/bin/env python" (any available version of Python), which could easily result in scripts being written with overly-restrictive shebang lines. Overall, this seems like there are more problems than benefits, and as a result has been dropped from consideration. Registering ``.pyz`` as a Media Type ------------------------------------ It was suggested [3]_ that the ``.pyz`` extension should be registered in the Unix database of extensions. While it makes sense to do this as an equivalent of the Windows installer registering the extension, the ``.py`` extension is not listed in the media types database [4]_. It doesn't seem reasonable to register ``.pyz`` without ``.py``, so this idea has been omitted from this PEP. An interested party could arrange for *both* ``.py`` and ``.pyz`` to be registered at a future date. Default Interpreter ------------------- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. Unix users have problems with this behaviour, as the default for the python command on many distributions is Python 2, and it is felt that this PEP should prefer Python 3 by default. However, using a command of ``python3`` can result in unexpected behaviour for Windows users, where the default behaviour of the launcher for the command ``python`` is commonly customised by users, but the behaviour of ``python3`` may not be modified to match. As a result, the principle "in the face of ambiguity, refuse to guess" has been invoked, and archives have no shebang line unless explicitly requested. On Windows, the archives will still be run (with the default Python) by the launcher, and on Unix, the archives can be run by explicitly invoking the desired Python interpreter. Command Line Tool to Manage Shebang Lines ----------------------------------------- It is conceivable that users would want to modify the shebang line for an existing archive, or even just display the current shebang line. This is tricky to do so with existing tools (zip programs typically ignore prepended data totally, and text editors can have trouble editing files containing binary data). The zipapp module provides functions to handle the shebang line, but does not include a command line interface to that functionality. This is because it is not clear how to provide one without the resulting interface being over-complex and potentially confusing. Changing the shebang line is expected to be an uncommon requirement. Reference Implementation ======================== A reference implementation is at http://bugs.python.org/issue23491. References ========== .. [1] Allow interpreter to execute a zip file (http://bugs.python.org/issue1739468) .. [2] Feature is not documented (http://bugs.python.org/issue17359) .. [3] Discussion of adding a .pyz mime type on python-dev (https://mail.python.org/pipermail/python-dev/2015-February/138338.html) .. [4] Register of media types (http://www.iana.org/assignments/media-types/media-types.xhtml) .. [5] pyzzer - A tool for creating Python-executable archives (https://pypi.python.org/pypi/pyzzer) .. [6] pex - The PEX packaging toolchain (https://pypi.python.org/pypi/pex) The discussion of this PEP took place on the python-dev mailing list, in the thread starting at https://mail.python.org/pipermail/python-dev/2015-February/138277.html Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

6 10

backwards and forwards compatibility, the exact contents of pickle files, and IntEnum
by Ethan Furman 15 Mar '15

15 Mar '15

I'm not sure exactly how to phrase this inquiry, so please bear with me. What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the contents of pickle files created at those protocols, and we have different versions of Python. Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4. I'm thinking the answer is yes, it should be. So how do we fix it? There are a couple different options: - modify IntEnum pickle methods to return the name only - modify IntEnum pickle methods to pickle just like the int they represent The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version. The second option has the advantage that the actual pickle contents are the same [1] as in previous versions. So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object? -- ~Ethan~

4 5