Mailman 3 March 2015 - Python-Dev

Re: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum
by Ethan Furman March 15, 2015

March 15, 2015

Bringing back on list. :) On 03/15/2015 12:18 AM, Glenn Linderman wrote: > On 3/14/2015 11:58 PM, Ethan Furman wrote: >> On 03/14/2015 11:46 PM, Glenn Linderman wrote: >> >>> In general, it would seem to me that each version of Python could >>> add object types that are foreign to prior versions, and so as long >>> as version N can unpickle anything produced by versions <=N, all >>> would be well. >> >> The question to me is … [View More]

1 0

Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask
by Loevborg, Soeren Jakob March 15, 2015

March 15, 2015

Hi, I'm looking for feedback on issue 22941, "IPv4Interface arithmetic changes subnet mask". As stated in the bug, I'd be happy to write a patch, if anyone would comment on my proposal. http://bugs.python.org/issue22941 Thank you, Søren Løvborg --- Addition and subtraction of integers are documented for ipaddress.IPv4Address and ipaddress.IPv6Address, but also work for IPv4Interface and IPv6Interface (though the only documentation of this is a brief mention that the Interface classes … [View More]

6 9

Minor update to Python 3.5 release schedule
by Larry Hastings March 15, 2015

March 15, 2015

I always intended all my releases to be on Sundays--that all the release engineering work is done on weekends, which is generally easier for everybody. But I goofed up the 3.5 release schedule and had proposed 3.5.0a3 to be released Saturday March 28th. With the assent of the team I bumped it forward a day to Sunday March 29th. I apologize for my goof. Cheers, //arry/

1 0

Why does python use relative instead of absolute path when calling LoadLibrary*
by David Cournapeau March 14, 2015

March 14, 2015

Hi, While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. Is there any defined behaviour that depends on this path to be relative ? Context ---… [View More]

4 4

Summary of Python tracker Issues
by Python tracker March 14, 2015

March 14, 2015

ACTIVITY SUMMARY (2015-03-06 - 2015-03-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4817 (+13) closed 30604 (+49) total 35421 (+62) Open issues with patches: 2266 Issues opened (49) ================== #11726: clarify that linecache only works on files that can be decoded http://bugs.python.org/issue11726 reopened by r.david.murray #23566: … [View More]

1 0

PEP 488: elimination of PYO files
by Brett Cannon March 13, 2015

March 13, 2015

Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that *some* optimization took place, not *what* optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. There is no patch yet, but this is not a complicated change and I could get it done at the sprints at PyCon if necessary (I suspect updating the test suite will take the … [View More]most work). There are currently two open issues, although one is purely a bikeshed topic on formatting of file names so I don't really consider it open for change from what is proposed in the PEP without Guido saying he hates my preference or someone having a really good argument for some alternative. The second open issue on the common case file name is something to reasonably debate and come to consensus on. ======================= PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in bytecode repository directory (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` is specified, including ``-OO``). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. Put in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate a PYC file will be incorporated into the bytecode file name. Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion with "0" while being so close to the interpreter version number. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, the bytecode file name of ``importlib.cpython-35.pyc`` would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as while as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed. Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own 0-2 optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['dead code elimination', 'constant folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter as this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== N/A Open Issues =========== Formatting of the optimization level in the file name ----------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-0.pyc`` * ``importlib.cpython-35.opt0.pyc`` * ``importlib.cpython-35.o0.pyc`` * ``importlib.cpython-35.O0.pyc`` * ``importlib.cpython-35.0.pyc`` * ``importlib.cpython-35-O0.pyc`` * ``importlib.O0.cpython-35.pyc`` * ``importlib.o0.cpython-35.pyc`` * ``importlib.0.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. Not specifying the optimization level when it is at 0 ----------------------------------------------------- It has been suggested that for the common case of when the optimizations are at level 0 that the entire part of the file name relating to the optimization level be left out. This would allow for file names of ``.pyc`` files to go unchanged, potentially leading to less backwards-compatibility issues (although Python 3.5 introduces a new magic number for bytecode so all bytecode files will have to be regenerated regardless of the outcome of this PEP). It would also allow a potentially redundant bit of information to be left out of the file name if an implementation of Python did not allow for optimizing bytecode. This would only occur, though, if the interpreter didn't support ``-O`` **and** didn't implement the ast module, else users could implement their own optimizations. Arguments against allow this special case is "explicit is better than implicit" and "special cases aren't special enough to break the rules". There are also currently no Python 3 interpreters that don't support ``-O``, so a potential Python 3 implementation which doesn't allow bytecode optimization is entirely theoretical at the moment. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_fro… ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py… ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

15 31

Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
by Isaac Schwabacher March 12, 2015

March 12, 2015

On 15-03-08, nha pham wrote: > > We can optimize the TimSort algorithm by optimizing its binary insertion sort. > > The current version of binary insertion sort use this idea: > > Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. > > I suggest another idea: > > Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next … [View More]

4 7

Not documented special methods
by Serhiy Storchaka March 11, 2015

March 11, 2015

There are many special names used in Python core and the stdlib, but absent in the documentation index [1]. If you see names that are defined or used in the module or area maintained by you, please add references to these names to the index and document them if it is needed. Repeat the lists here. Module level names used in pydoc: __author__ __credits__ __date__ __version__ Module level name used in doctest: __test__ Other module level names: __about__ … [View More] (heapq only) __copyright__ (many modules) __cvsid__ (tarfile only) __docformat__ (doctest only) __email__ (test_with and test_keywordonlyarg only) __libmpdec_version__ (decimal only) __status__ (logging only) type attributes (mostly used in tests): __abstractmethods__ (used in abc, functools) __base__ __basicsize__ __dictoffset__ __flags__ (used in inspect, copyreg) __itemsize__ __weakrefoffset__ super() attributes: __self_class__ __thisclass__ Used in sqlite: __adapt__ __conform__ Used in ctypes: __ctype_be__ __ctype_le__ __ctypes_from_outparam__ Used in unittest: __unittest_expecting_failure__ __unittest_skip__ __unittest_skip_why__ float methods, for testing: __getformat__ __setformat__ Used in IDLE RPC: __attributes__ __methods__ Others: __alloc__ (bytearray method) __args__ (used in bdb) __build_class__ (builtins function, used in eval loop) __builtins__ (module attribute) __decimal_context__ (used in decimal) __exception__ (used in pdb) __getinitargs__ (used in pickle, datetime) __initializing__ (used in importlib) __isabstractmethod__ (function/method/descriptor attribute, used in abc, functools, types) __ltrace__ (used in eval loop, never set) __members__ (Enum attribute, used in many modules) __mp_main__ (used in multiprocessing) __new_member__ (Enum attribute, used in enum internally) __newobj__ (copyreg function, used in pickle, object.__reduce_ex__) __newobj_ex__ (copyreg function, used in pickle, object.__reduce_ex__) __objclass__ (descriptor/enum attribute, used in inspect, pydoc, doctest, multiprocessing) __prepare__ (metaclass method, used in builtins.__build_class__, types) __pycache__ (cache directory name) __return__ (used in pdb) __signature__ (used in inspect, never set) __sizeof__ (standard method, used in sys.getsizeof) __slotnames__ (used in object.__getstate__ for caching) __text_signature__ (function/method/descriptor attribute, used in inspect) __trunc__ (used in math.trunc, int, etc) __warningregistry__ (used in warnings) __weakref__ (used in weakref) __wrapped__ (used in inspect, functools, contextlib, asyncio) [1] http://bugs.python.org/issue23639 [View Less]

1 0

Re: [Python-Dev] str.lstrip bug?
by Isaac Schwabacher March 11, 2015

March 11, 2015

On 15-03-10, Ethan Furman wrote: > On 03/10/2015 10:57 AM, Isaac Schwabacher wrote: > > On 15-03-10, Facundo Batista wrote: > >> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: > >> > >>> tiny➜ ~ python > >>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > >>> [GCC 4.8.1] on linux2 > >>> Type "help", "copyright", "credits" or "license" for more information. > >>> >>> a='device_info' > >>&… [View More]

1 0

Re: [Python-Dev] str.lstrip bug?
by Isaac Schwabacher March 11, 2015

March 11, 2015

On 15-03-10, Facundo Batista wrote: > On Tue, Mar 10, 2015 at 2:27 PM, lou xiao <lox.xiao(a)gmail.com> wrote: > > > tiny➜ ~ python > > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > > [GCC 4.8.1] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> a='device_info' > >>>> a.lstrip('device') > > '_info' > >>>> a.lstrip('device_') > > 'nfo' > > On one hand, … [View More]

2 1