Mailman 3 March 2015 - Python-Dev

Re: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum
by Ethan Furman 15 Mar '15

15 Mar '15

Bringing back on list. :) On 03/15/2015 12:18 AM, Glenn Linderman wrote: > On 3/14/2015 11:58 PM, Ethan Furman wrote: >> On 03/14/2015 11:46 PM, Glenn Linderman wrote: >> >>> In general, it would seem to me that each version of Python could >>> add object types that are foreign to prior versions, and so as long >>> as version N can unpickle anything produced by versions <=N, all >>> would be well. >> >> The question to me is because we are not so much adding something >> brand-new (socket.AF_INET has been there for ages), as changing the >> type of something already existing -- I'm just not sure if we need >> "end-result" compatibility (achieved by the first option) or if we >> need "bug for bug" compatibility (achieved by the second option). > > Yes, the fact that it is not brand-new does add a mind-twist to the compatibility expectations. But while it is not > brand-new, it is changed. Exactly. IntEnum was advertised as being a "drop-in replacement" with the only change being better reprs and strs; so being unable to un-pickle in prior versions invalidates that claim. Part 2 of my question (Part 1 of which is "which method do we use to fix the IntEnum being used in the stdlib") is: do we just change the IntEnum subclasses (such as socket.AddressFamily) to have the fixed behavior, or do we change IntEnum itself? Having thought about it for a few minutes, we cannot make that change in IntEnum itself as it only works if the names are exported to the global namespace -- so this fix will work for the stdlib (since we export the names), but won't work otherwise. Unless someone steps forward and says that the actual pickle contents must be the same, I'm going to go with pickling the name: this is both unpicklable in prior versions, and doesn't lose the type in present versions. Just to be clear, there are no issue unpickling in the same version, only in prior versions. -- ~Ethan~

1 0

Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask
by Loevborg, Soeren Jakob 15 Mar '15

15 Mar '15

Hi, I'm looking for feedback on issue 22941, "IPv4Interface arithmetic changes subnet mask". As stated in the bug, I'd be happy to write a patch, if anyone would comment on my proposal. http://bugs.python.org/issue22941 Thank you, Søren Løvborg --- Addition and subtraction of integers are documented for ipaddress.IPv4Address and ipaddress.IPv6Address, but also work for IPv4Interface and IPv6Interface (though the only documentation of this is a brief mention that the Interface classes inherit from the respective Address classes). That's good. The problem is that adding/subtracting an integer to an Interface object changes the subnet mask (to max_prefixlen), something which is 1) not documented and 2) not the least surprising result. >>> import ipaddress >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 IPv4Interface('10.0.0.2/32') >>> ipaddress.IPv6Interface('fd00::1/64') + 1 IPv6Interface('fd00::2/128') Proposed behavior: >>> import ipaddress >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 IPv4Interface('10.0.0.2/8') >>> ipaddress.IPv6Interface('fd00::1/64') + 1 IPv6Interface('fd00::2/64') It all comes down to a judgment call: is this a bug, or expected (but undocumented) behavior? In PEP 387 lingo: Is this a "reasonable bug fix"? Or is it a "design mistake", thus calling for a FutureWarning first, and the actual change later? >>> ipaddress.IPv4Interface('1.2.3.4/16') + 1 __main__:1: FutureWarning: Arithmetic on IPv4Interface objects will change in Python 3.6. If you rely on the current behavior, change 'interface + n' to 'IPv4Interface(interface.ip + n, 32)'

6 9

Minor update to Python 3.5 release schedule
by Larry Hastings 14 Mar '15

14 Mar '15

I always intended all my releases to be on Sundays--that all the release engineering work is done on weekends, which is generally easier for everybody. But I goofed up the 3.5 release schedule and had proposed 3.5.0a3 to be released Saturday March 28th. With the assent of the team I bumped it forward a day to Sunday March 29th. I apologize for my goof. Cheers, //arry/

1 0

Why does python use relative instead of absolute path when calling LoadLibrary*
by David Cournapeau 13 Mar '15

13 Mar '15

Hi, While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. Is there any defined behaviour that depends on this path to be relative ? Context ----------- The reason why I am interested in this is the potential use of SetDllDirectory to share dlls between multiple python extensions. Currently, the only solutions I am aware of are: 1. putting the dlls in the PATH 2. bundling the dlls side by side the .pyd 3. patching packages to use preloading (using e.g. ctypes) I am investigating a solution 4, where the dlls would be put in a separate "private" directory only known of python itself, without the need to modify PATH. Patching python to use SetDllDirectory("some private paths specific to a python interpreter") works perfectly, except that it slightly changes the semantics of LoadLibraryEx not to look for dlls in the current directory. This breaks importing extensions built in place, unless I modify the call in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195 from: hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH) to hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH) That seems to work, but I am quite worried about changing any import semantics by accident. David

4 4

Summary of Python tracker Issues
by Python tracker 13 Mar '15

13 Mar '15

ACTIVITY SUMMARY (2015-03-06 - 2015-03-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4817 (+13) closed 30604 (+49) total 35421 (+62) Open issues with patches: 2266 Issues opened (49) ================== #11726: clarify that linecache only works on files that can be decoded http://bugs.python.org/issue11726 reopened by r.david.murray #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 reopened by haypo #23599: single and double quotes stripped upon paste with MacPorts lib http://bugs.python.org/issue23599 opened by Jeff Doak #23600: tizinfo.fromutc changed for tzinfo wih StdOffset=0, DstOffset= http://bugs.python.org/issue23600 opened by peterjclaw #23601: use small object allocator for dict key storage http://bugs.python.org/issue23601 opened by jtaylor #23602: Implement __format__ for Fraction http://bugs.python.org/issue23602 opened by tuomas.suutari #23603: Embedding Python3.4 - PyUnicode_Check fails (MinGW-W64) http://bugs.python.org/issue23603 opened by Ashish Sadanandan #23605: Use the new os.scandir() function in os.walk() http://bugs.python.org/issue23605 opened by haypo #23606: ctypes.util.find_library("c") no longer makes sense http://bugs.python.org/issue23606 opened by steve.dower #23607: Inconsistency in datetime.utcfromtimestamp(Decimal) http://bugs.python.org/issue23607 opened by serhiy.storchaka #23609: Export PyModuleObject in moduleobject.h http://bugs.python.org/issue23609 opened by h.venev #23611: Pickle nested names with protocols < 4 http://bugs.python.org/issue23611 opened by serhiy.storchaka #23612: 3.5.0a2 Windows installer does not remove 3.5.0a1 http://bugs.python.org/issue23612 opened by steve.dower #23614: Opaque error message on UTF-8 decoding to surrogates http://bugs.python.org/issue23614 opened by Rosuav #23616: Idle: conflict between loop execution and undo shortcut. http://bugs.python.org/issue23616 opened by Davide Okami #23618: PEP 475: handle EINTR in the socket module http://bugs.python.org/issue23618 opened by haypo #23619: Python 3.5.0a2 installer fails on Windows http://bugs.python.org/issue23619 opened by pmoore #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 opened by pmoore #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 opened by serhiy.storchaka #23623: Python 3.5 docs need to clarify how to set PATH, etc http://bugs.python.org/issue23623 opened by pmoore #23624: str.center inconsistent with format "^" http://bugs.python.org/issue23624 opened by Vedran.Čačić #23626: Windows per-user install of 3.5a2 doesn't associate .py files http://bugs.python.org/issue23626 opened by pmoore #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 opened by brett.cannon #23630: support multiple hosts in create_server/start_server http://bugs.python.org/issue23630 opened by sebastien.bourdeauducq #23631: 3.5 (a2) traceback regression snarls Idle http://bugs.python.org/issue23631 opened by terry.reedy #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 opened by pitrou #23633: Improve py launcher help, index, and doc http://bugs.python.org/issue23633 opened by terry.reedy #23634: os.fdopen reopening a read-only fd on windows http://bugs.python.org/issue23634 opened by mattip #23636: Add scgi to urllib.parse.uses_netloc http://bugs.python.org/issue23636 opened by anthonyryan1 #23637: Warnings error with non-ascii chars. http://bugs.python.org/issue23637 opened by Lukáš.Němec #23639: Not documented special names http://bugs.python.org/issue23639 opened by serhiy.storchaka #23640: Enum.from_bytes() is broken http://bugs.python.org/issue23640 opened by serhiy.storchaka #23641: Got rid of bad dunder names http://bugs.python.org/issue23641 opened by serhiy.storchaka #23642: Interaction of ModuleSpec and C Extension Modules http://bugs.python.org/issue23642 opened by dabeaz #23644: g++ module compile fails with ‘_Atomic’ does not name a ty http://bugs.python.org/issue23644 opened by Joshua.J.Cogliati #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 opened by gdementen #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 opened by haypo #23647: imaplib.py MAXLINE value is too low for gmail http://bugs.python.org/issue23647 opened by arnt #23648: PEP 475 meta issue http://bugs.python.org/issue23648 opened by haypo #23649: tarfile not re-entrant for multi-threading http://bugs.python.org/issue23649 opened by sgnn7 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 opened by WanderingLogic #23653: Make inspect._empty test to False http://bugs.python.org/issue23653 opened by Lucretiel #23654: infinite loop in faulthandler._stack_overflow http://bugs.python.org/issue23654 opened by WanderingLogic #23655: Memory corruption using pickle over pipe to subprocess http://bugs.python.org/issue23655 opened by nagle #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 opened by h-miyajima #23657: Don't do isinstance checks in zipapp http://bugs.python.org/issue23657 opened by brett.cannon #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 opened by Dan Nawrocki #23659: csv.register_dialect doc string http://bugs.python.org/issue23659 opened by Vladimir Ulupov #23660: Turtle left/right inverted when using different coordinates or http://bugs.python.org/issue23660 opened by aroberge Most recent 15 issues with no replies (15) ========================================== #23660: Turtle left/right inverted when using different coordinates or http://bugs.python.org/issue23660 #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 #23653: Make inspect._empty test to False http://bugs.python.org/issue23653 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 #23648: PEP 475 meta issue http://bugs.python.org/issue23648 #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 #23626: Windows per-user install of 3.5a2 doesn't associate .py files http://bugs.python.org/issue23626 #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 #23609: Export PyModuleObject in moduleobject.h http://bugs.python.org/issue23609 #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 #23578: struct.pack error messages do not indicate which argument was http://bugs.python.org/issue23578 #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 Most recent 15 issues waiting for review (15) ============================================= #23657: Don't do isinstance checks in zipapp http://bugs.python.org/issue23657 #23654: infinite loop in faulthandler._stack_overflow http://bugs.python.org/issue23654 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 #23649: tarfile not re-entrant for multi-threading http://bugs.python.org/issue23649 #23647: imaplib.py MAXLINE value is too low for gmail http://bugs.python.org/issue23647 #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 #23641: Got rid of bad dunder names http://bugs.python.org/issue23641 #23640: Enum.from_bytes() is broken http://bugs.python.org/issue23640 #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 #23630: support multiple hosts in create_server/start_server http://bugs.python.org/issue23630 #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 #23618: PEP 475: handle EINTR in the socket module http://bugs.python.org/issue23618 #23611: Pickle nested names with protocols < 4 http://bugs.python.org/issue23611 #23607: Inconsistency in datetime.utcfromtimestamp(Decimal) http://bugs.python.org/issue23607 Top 10 most discussed issues (10) ================================= #23605: Use the new os.scandir() function in os.walk() http://bugs.python.org/issue23605 24 msgs #23619: Python 3.5.0a2 installer fails on Windows http://bugs.python.org/issue23619 23 msgs #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 14 msgs #23246: distutils fails to locate vcvarsall with Visual C++ Compiler f http://bugs.python.org/issue23246 11 msgs #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 11 msgs #23602: Implement __format__ for Fraction http://bugs.python.org/issue23602 10 msgs #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 10 msgs #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 9 msgs #23633: Improve py launcher help, index, and doc http://bugs.python.org/issue23633 9 msgs #12916: Add inspect.splitdoc http://bugs.python.org/issue12916 8 msgs Issues closed (46) ================== #5527: multiprocessing won't work with Tkinter (under Linux) http://bugs.python.org/issue5527 closed by davin #7503: multiprocessing AuthenticationError "digest sent was rejected" http://bugs.python.org/issue7503 closed by davin #9191: winreg.c:Reg2Py() may leak memory (in unusual circumstances) http://bugs.python.org/issue9191 closed by Claudiu.Popa #10805: traceback.print_exception throws AttributeError when exception http://bugs.python.org/issue10805 closed by Claudiu.Popa #11705: sys.excepthook doesn't work in imported modules http://bugs.python.org/issue11705 closed by Claudiu.Popa #17403: Robotparser fails to parse some robots.txt http://bugs.python.org/issue17403 closed by berker.peksag #19909: Best practice docs for new SSL features http://bugs.python.org/issue19909 closed by christian.heimes #20617: test_ssl does not use mock http://bugs.python.org/issue20617 closed by berker.peksag #20659: Want to make a class method a property by combining decorators http://bugs.python.org/issue20659 closed by ncoghlan #20876: python -m test test_pathlib fails http://bugs.python.org/issue20876 closed by pitrou #21610: load_module not closing opened files http://bugs.python.org/issue21610 closed by python-dev #22522: sys.excepthook doesn't receive the traceback when called from http://bugs.python.org/issue22522 closed by Claudiu.Popa #22524: PEP 471 implementation: os.scandir() directory scanning functi http://bugs.python.org/issue22524 closed by haypo #22853: Multiprocessing.Queue._feed deadlocks on import http://bugs.python.org/issue22853 closed by serhiy.storchaka #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 closed by serhiy.storchaka #23003: traceback.{print_exc,print_exception,format_exc,format_excepti http://bugs.python.org/issue23003 closed by berker.peksag #23051: multiprocessing.pool methods imap()[_unordered()] deadlock http://bugs.python.org/issue23051 closed by serhiy.storchaka #23081: Document PySequence_List(o) as equivalent to list(o) http://bugs.python.org/issue23081 closed by berker.peksag #23138: cookiejar parses cookie value as int with empty name-value pai http://bugs.python.org/issue23138 closed by serhiy.storchaka #23192: Generator return value ignored in lambda function http://bugs.python.org/issue23192 closed by serhiy.storchaka #23279: test_site/test_startup_imports fails when mpl_toolkit or logil http://bugs.python.org/issue23279 closed by brett.cannon #23432: Duplicate content in SystemExit documentation http://bugs.python.org/issue23432 closed by berker.peksag #23467: Improve byte formatting compatibility between Py2 and Py3 http://bugs.python.org/issue23467 closed by python-dev #23486: Enum member lookup is 20x slower than normal class attribute l http://bugs.python.org/issue23486 closed by python-dev #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 closed by brett.cannon #23526: Silence resource warnings in test_httplib http://bugs.python.org/issue23526 closed by vadmium #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 closed by haypo #23579: Amazon.com links http://bugs.python.org/issue23579 closed by python-dev #23581: unittest.mock.MagicMock doesn't support matmul (@) operator http://bugs.python.org/issue23581 closed by berker.peksag #23595: Split the math module into _math (C) + math (py) http://bugs.python.org/issue23595 closed by haypo #23598: No backspace on curses on iTerm (mac) http://bugs.python.org/issue23598 closed by ned.deily #23604: Python 3.4 and 2.7 installation no Script folder and no pip in http://bugs.python.org/issue23604 closed by Daiyue Weng #23608: ssl.VERIFY_X509_TRUSTED_FIRST: Incorrect value for versionadde http://bugs.python.org/issue23608 closed by python-dev #23610: -m arg behavior http://bugs.python.org/issue23610 closed by zach.ware #23613: searchindex.js is annoying http://bugs.python.org/issue23613 closed by georg.brandl #23615: Reloading tokenize breaks tokenize.open() http://bugs.python.org/issue23615 closed by serhiy.storchaka #23617: Incorrect plural "includes" on Python Standard Library front p http://bugs.python.org/issue23617 closed by ned.deily #23620: cross-type comparison is different in python minor versions http://bugs.python.org/issue23620 closed by r.david.murray #23625: load_module() docs: zipped eggs are not loaded. http://bugs.python.org/issue23625 closed by brett.cannon #23627: Incorrect string formatting of float values in Python 2.7 http://bugs.python.org/issue23627 closed by eric.smith #23629: Default __sizeof__ is wrong for var-sized objects http://bugs.python.org/issue23629 closed by pitrou #23635: Python won't install correctly. http://bugs.python.org/issue23635 closed by Player72 #23638: shutil.copytree makedirs exception http://bugs.python.org/issue23638 closed by r.david.murray #23643: Segmentation Fault with Large Simple Function http://bugs.python.org/issue23643 closed by Michael Klein #23650: xml.etree.ElementTreee.write can't parse its own output http://bugs.python.org/issue23650 closed by ned.deily #23651: Typo in argparse allow_abrev http://bugs.python.org/issue23651 closed by berker.peksag

1 0

PEP 488: elimination of PYO files
by Brett Cannon 13 Mar '15

13 Mar '15

Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that *some* optimization took place, not *what* optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. There is no patch yet, but this is not a complicated change and I could get it done at the sprints at PyCon if necessary (I suspect updating the test suite will take the most work). There are currently two open issues, although one is purely a bikeshed topic on formatting of file names so I don't really consider it open for change from what is proposed in the PEP without Guido saying he hates my preference or someone having a really good argument for some alternative. The second open issue on the common case file name is something to reasonably debate and come to consensus on. ======================= PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in bytecode repository directory (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` is specified, including ``-OO``). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. Put in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate a PYC file will be incorporated into the bytecode file name. Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion with "0" while being so close to the interpreter version number. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, the bytecode file name of ``importlib.cpython-35.pyc`` would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as while as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed. Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own 0-2 optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['dead code elimination', 'constant folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter as this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== N/A Open Issues =========== Formatting of the optimization level in the file name ----------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-0.pyc`` * ``importlib.cpython-35.opt0.pyc`` * ``importlib.cpython-35.o0.pyc`` * ``importlib.cpython-35.O0.pyc`` * ``importlib.cpython-35.0.pyc`` * ``importlib.cpython-35-O0.pyc`` * ``importlib.O0.cpython-35.pyc`` * ``importlib.o0.cpython-35.pyc`` * ``importlib.0.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. Not specifying the optimization level when it is at 0 ----------------------------------------------------- It has been suggested that for the common case of when the optimizations are at level 0 that the entire part of the file name relating to the optimization level be left out. This would allow for file names of ``.pyc`` files to go unchanged, potentially leading to less backwards-compatibility issues (although Python 3.5 introduces a new magic number for bytecode so all bytecode files will have to be regenerated regardless of the outcome of this PEP). It would also allow a potentially redundant bit of information to be left out of the file name if an implementation of Python did not allow for optimizing bytecode. This would only occur, though, if the interpreter didn't support ``-O`` **and** didn't implement the ast module, else users could implement their own optimizations. Arguments against allow this special case is "explicit is better than implicit" and "special cases aren't special enough to break the rules". There are also currently no Python 3 interpreters that don't support ``-O``, so a potential Python 3 implementation which doesn't allow bytecode optimization is entirely theoretical at the moment. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_fro… ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py… ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

15 31

Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
by Isaac Schwabacher 11 Mar '15

11 Mar '15

On 15-03-08, nha pham wrote: > > We can optimize the TimSort algorithm by optimizing its binary insertion sort. > > The current version of binary insertion sort use this idea: > > Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. > > I suggest another idea: > > Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. > > For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I don't understand how this is an improvement, since with binary search the idea is that each comparison cuts the remaining list to search in half; i.e., each comparison yields one bit of information. Here, you're spending a comparison to cut the list to search at the element you just inserted, which is probably not right in the middle. If you miss the middle, you're getting on average less than a full bit of information from your comparison, so you're not reducing the remaining search space by as much as you would be if you just compared to the element in the middle of the list. > I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. For all that, though, experiment trumps theory... > Here is detail about algorithm and testing: https://github.com/nhapq/Optimize_binary_insertion_sort > > Sincerely. > > phqnha

4 7

Not documented special methods
by Serhiy Storchaka 11 Mar '15

11 Mar '15

There are many special names used in Python core and the stdlib, but absent in the documentation index [1]. If you see names that are defined or used in the module or area maintained by you, please add references to these names to the index and document them if it is needed. Repeat the lists here. Module level names used in pydoc: __author__ __credits__ __date__ __version__ Module level name used in doctest: __test__ Other module level names: __about__ (heapq only) __copyright__ (many modules) __cvsid__ (tarfile only) __docformat__ (doctest only) __email__ (test_with and test_keywordonlyarg only) __libmpdec_version__ (decimal only) __status__ (logging only) type attributes (mostly used in tests): __abstractmethods__ (used in abc, functools) __base__ __basicsize__ __dictoffset__ __flags__ (used in inspect, copyreg) __itemsize__ __weakrefoffset__ super() attributes: __self_class__ __thisclass__ Used in sqlite: __adapt__ __conform__ Used in ctypes: __ctype_be__ __ctype_le__ __ctypes_from_outparam__ Used in unittest: __unittest_expecting_failure__ __unittest_skip__ __unittest_skip_why__ float methods, for testing: __getformat__ __setformat__ Used in IDLE RPC: __attributes__ __methods__ Others: __alloc__ (bytearray method) __args__ (used in bdb) __build_class__ (builtins function, used in eval loop) __builtins__ (module attribute) __decimal_context__ (used in decimal) __exception__ (used in pdb) __getinitargs__ (used in pickle, datetime) __initializing__ (used in importlib) __isabstractmethod__ (function/method/descriptor attribute, used in abc, functools, types) __ltrace__ (used in eval loop, never set) __members__ (Enum attribute, used in many modules) __mp_main__ (used in multiprocessing) __new_member__ (Enum attribute, used in enum internally) __newobj__ (copyreg function, used in pickle, object.__reduce_ex__) __newobj_ex__ (copyreg function, used in pickle, object.__reduce_ex__) __objclass__ (descriptor/enum attribute, used in inspect, pydoc, doctest, multiprocessing) __prepare__ (metaclass method, used in builtins.__build_class__, types) __pycache__ (cache directory name) __return__ (used in pdb) __signature__ (used in inspect, never set) __sizeof__ (standard method, used in sys.getsizeof) __slotnames__ (used in object.__getstate__ for caching) __text_signature__ (function/method/descriptor attribute, used in inspect) __trunc__ (used in math.trunc, int, etc) __warningregistry__ (used in warnings) __weakref__ (used in weakref) __wrapped__ (used in inspect, functools, contextlib, asyncio) [1] http://bugs.python.org/issue23639

1 0

Re: [Python-Dev] str.lstrip bug?
by Isaac Schwabacher 10 Mar '15

10 Mar '15

On 15-03-10, Ethan Furman wrote: > On 03/10/2015 10:57 AM, Isaac Schwabacher wrote: > > On 15-03-10, Facundo Batista wrote: > >> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: > >> > >>> tiny➜ ~ python > >>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > >>> [GCC 4.8.1] on linux2 > >>> Type "help", "copyright", "credits" or "license" for more information. > >>> >>> a='device_info' > >>> >>> a.lstrip('device') > >>> '_info' > >>> >>> a.lstrip('device_') > >>> 'nfo' > >> > >> On one hand, this is the "development of python itself" list; this > >> issue was more aimed to the general python list, of you was sure that > >> this is a real bug, to the issue tracker. > >> > >> On the other hand, this is not a bug! If you pass a parameter to > >> lstrip it will (quoted from its help) "remove characters in chars > >> instead.", so the moment you pass "device_", it removes all those > >> characers from the left... note that the 'i' is removed twice. > > > > That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix). > > Nice way to bring it on-topic! > > And yes, that would be nice. Yes. When you are converting a code base from bash to python, you *really* don't want python to push back. > -- > ~Ethan~

1 0

Re: [Python-Dev] str.lstrip bug?
by Isaac Schwabacher 10 Mar '15

10 Mar '15

On 15-03-10, Facundo Batista wrote: > On Tue, Mar 10, 2015 at 2:27 PM, lou xiao <lox.xiao(a)gmail.com> wrote: > > > tiny➜ ~ python > > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > > [GCC 4.8.1] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> a='device_info' > >>>> a.lstrip('device') > > '_info' > >>>> a.lstrip('device_') > > 'nfo' > > On one hand, this is the "development of python itself" list; this > issue was more aimed to the general python list, of you was sure that > this is a real bug, to the issue tracker. > > On the other hand, this is not a bug! If you pass a parameter to > lstrip it will (quoted from its help) "remove characters in chars > instead.", so the moment you pass "device_", it removes all those > characers from the left... note that the 'i' is removed twice. That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix). ijs > Regards, > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > Twitter: @facundobatista > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ischwabacher%40wisc.edu

2 1