Mailman 3 April 2016 - Python-Dev

I hope this won't be my last comment here ~ yet it may well be...
by Burkhard Meier April 22, 2016

April 22, 2016

Please do allow me to share my humble experiences of being a software professional on a Windows platform. Almost 20 years. You know what; when I tried out 'sugar Linux' or Peppermint,,,the "admin' dude kicked me out 5 times in one sole eve, Maybe this is just *me*.. You know what: I did have my time with this *open source community*... I was just asking a sincere question. C'mon This was rather very ridiculous.

3 3

Summary of Python tracker Issues
by Python tracker April 22, 2016

April 22, 2016

ACTIVITY SUMMARY (2016-04-15 - 2016-04-22) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5491 ( +2) closed 33095 (+56) total 38586 (+58) Open issues with patches: 2384 Issues opened (41) ================== #26773: Shelve works inconsistently when carried over to child process http://bugs.python.org/issue26773 opened by Paul Ellenbogen #26774: Elide Py_atomic fences when WITH_THREAD is disabled? http://bugs.python.org/issue26774 opened by larry #26776: Determining the failure of C API call is ambiguous http://bugs.python.org/issue26776 opened by serhiy.storchaka #26779: pdb continue followed by an exception in the same frame shows http://bugs.python.org/issue26779 opened by Sriram Rajagopalan #26781: os.walk max_depth http://bugs.python.org/issue26781 opened by palaviv #26786: bdist_msi duplicates directories with names in ALL CAPS to a b http://bugs.python.org/issue26786 opened by Ivan.Pozdeev #26787: test_distutils fails when configured --with-lto http://bugs.python.org/issue26787 opened by gregory.p.smith #26788: test_gdb fails all tests on a profile-opt build configured --w http://bugs.python.org/issue26788 opened by gregory.p.smith #26789: Please do not log during shutdown http://bugs.python.org/issue26789 opened by smurfix #26790: bdist_msi package duplicates everything to a bogus location wh http://bugs.python.org/issue26790 opened by Ivan.Pozdeev #26791: shutil.move fails to move symlink (Invalid cross-device link) http://bugs.python.org/issue26791 opened by Unode #26792: docstrings of runpy.run_{module,path} are rather sparse http://bugs.python.org/issue26792 opened by Antony.Lee #26793: uuid causing thread issues when forking using os.fork py3.4+ http://bugs.python.org/issue26793 opened by Steven Adams #26794: curframe can be None in pdb.py http://bugs.python.org/issue26794 opened by Jacek.Pliszka #26796: BaseEventLoop.run_in_executor shouldn't specify max_workers fo http://bugs.python.org/issue26796 opened by Hans Lawrenz #26797: Segafault in _PyObject_Alloc http://bugs.python.org/issue26797 opened by yselivanov #26798: add BLAKE2 to hashlib http://bugs.python.org/issue26798 opened by Zooko.Wilcox-O'Hearn #26800: Don't accept bytearray as filenames part 2 http://bugs.python.org/issue26800 opened by pjenvey #26801: Fix shutil.get_terminal_size() to catch AttributeError http://bugs.python.org/issue26801 opened by ebarry #26803: syslog logging handler fails with address in unix abstract nam http://bugs.python.org/issue26803 opened by xdegaye #26804: Prioritize lowercase proxy variables in urllib.request http://bugs.python.org/issue26804 opened by frispete #26806: IDLE not displaying RecursionError tracebacks http://bugs.python.org/issue26806 opened by terry.reedy #26807: mock_open()().readline() fails at EOF http://bugs.python.org/issue26807 opened by rbcollins #26809: `string` exposes ChainMap from `collections` http://bugs.python.org/issue26809 opened by leewz #26810: inconsistent garbage collector behavior across platforms when http://bugs.python.org/issue26810 opened by unsec treedee #26811: segfault due to null pointer in tuple http://bugs.python.org/issue26811 opened by random832 #26812: ExtendedInterpolation drops user-defined 'vars' during _interp http://bugs.python.org/issue26812 opened by yab-arz #26814: [WIP] Add a new _PyObject_FastCall() function which avoids the http://bugs.python.org/issue26814 opened by haypo #26815: SIGBUS in test_ssl.test_dealloc_warn() on "AMD64 FreeBSD 10.0 http://bugs.python.org/issue26815 opened by haypo #26816: Make concurrent.futures.Executor an abc http://bugs.python.org/issue26816 opened by xiang.zhang #26817: Docs for StringIO should link to io.BytesIO http://bugs.python.org/issue26817 opened by guettli #26818: trace CLI doesn't respect -s option http://bugs.python.org/issue26818 opened by berker.peksag #26819: _ProactorReadPipeTransport pause_reading()/resume_reading() br http://bugs.python.org/issue26819 opened by Fulvio Esposito #26820: Prevent uses of format string based PyObject_Call* that do not http://bugs.python.org/issue26820 opened by josh.r #26822: itemgetter/attrgetter/methodcaller objects ignore keyword argu http://bugs.python.org/issue26822 opened by serhiy.storchaka #26823: Shrink recursive tracebacks http://bugs.python.org/issue26823 opened by ebarry #26824: Make some macros use Py_TYPE http://bugs.python.org/issue26824 opened by xiang.zhang #26826: Expose new copy_file_range() syscal in os module and use it to http://bugs.python.org/issue26826 opened by StyXman #26827: PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextent http://bugs.python.org/issue26827 opened by prinsherbert #26828: Implement __length_hint__() on map() and filter() to optimize http://bugs.python.org/issue26828 opened by haypo #26829: update docs: when creating classes a new dict is created for t http://bugs.python.org/issue26829 opened by ethan.furman Most recent 15 issues with no replies (15) ========================================== #26829: update docs: when creating classes a new dict is created for t http://bugs.python.org/issue26829 #26819: _ProactorReadPipeTransport pause_reading()/resume_reading() br http://bugs.python.org/issue26819 #26818: trace CLI doesn't respect -s option http://bugs.python.org/issue26818 #26817: Docs for StringIO should link to io.BytesIO http://bugs.python.org/issue26817 #26812: ExtendedInterpolation drops user-defined 'vars' during _interp http://bugs.python.org/issue26812 #26794: curframe can be None in pdb.py http://bugs.python.org/issue26794 #26792: docstrings of runpy.run_{module,path} are rather sparse http://bugs.python.org/issue26792 #26790: bdist_msi package duplicates everything to a bogus location wh http://bugs.python.org/issue26790 #26789: Please do not log during shutdown http://bugs.python.org/issue26789 #26786: bdist_msi duplicates directories with names in ALL CAPS to a b http://bugs.python.org/issue26786 #26779: pdb continue followed by an exception in the same frame shows http://bugs.python.org/issue26779 #26771: python-config.sh.in INCDIR does not match python version if ex http://bugs.python.org/issue26771 #26767: Inconsistant error messages for failed attribute modification http://bugs.python.org/issue26767 #26752: Mock(2.0.0).assert_has_calls() raise AssertionError in two sam http://bugs.python.org/issue26752 #26750: Mock autospec does not work with subclasses of property() http://bugs.python.org/issue26750 Most recent 15 issues waiting for review (15) ============================================= #26824: Make some macros use Py_TYPE http://bugs.python.org/issue26824 #26823: Shrink recursive tracebacks http://bugs.python.org/issue26823 #26822: itemgetter/attrgetter/methodcaller objects ignore keyword argu http://bugs.python.org/issue26822 #26818: trace CLI doesn't respect -s option http://bugs.python.org/issue26818 #26816: Make concurrent.futures.Executor an abc http://bugs.python.org/issue26816 #26814: [WIP] Add a new _PyObject_FastCall() function which avoids the http://bugs.python.org/issue26814 #26811: segfault due to null pointer in tuple http://bugs.python.org/issue26811 #26809: `string` exposes ChainMap from `collections` http://bugs.python.org/issue26809 #26804: Prioritize lowercase proxy variables in urllib.request http://bugs.python.org/issue26804 #26803: syslog logging handler fails with address in unix abstract nam http://bugs.python.org/issue26803 #26801: Fix shutil.get_terminal_size() to catch AttributeError http://bugs.python.org/issue26801 #26796: BaseEventLoop.run_in_executor shouldn't specify max_workers fo http://bugs.python.org/issue26796 #26787: test_distutils fails when configured --with-lto http://bugs.python.org/issue26787 #26786: bdist_msi duplicates directories with names in ALL CAPS to a b http://bugs.python.org/issue26786 #26781: os.walk max_depth http://bugs.python.org/issue26781 Top 10 most discussed issues (10) ================================= #26801: Fix shutil.get_terminal_size() to catch AttributeError http://bugs.python.org/issue26801 16 msgs #26814: [WIP] Add a new _PyObject_FastCall() function which avoids the http://bugs.python.org/issue26814 16 msgs #26824: Make some macros use Py_TYPE http://bugs.python.org/issue26824 15 msgs #26809: `string` exposes ChainMap from `collections` http://bugs.python.org/issue26809 14 msgs #26601: Use new madvise()'s MADV_FREE on the private heap http://bugs.python.org/issue26601 13 msgs #26803: syslog logging handler fails with address in unix abstract nam http://bugs.python.org/issue26803 13 msgs #26811: segfault due to null pointer in tuple http://bugs.python.org/issue26811 13 msgs #26793: uuid causing thread issues when forking using os.fork py3.4+ http://bugs.python.org/issue26793 12 msgs #26804: Prioritize lowercase proxy variables in urllib.request http://bugs.python.org/issue26804 10 msgs #26058: PEP 509: Add ma_version to PyDictObject http://bugs.python.org/issue26058 9 msgs Issues closed (52) ================== #4806: Function calls taking a generator as star argument can mask Ty http://bugs.python.org/issue4806 closed by martin.panter #7694: DeprecationWarnings in distutils are pointless http://bugs.python.org/issue7694 closed by berker.peksag #8978: "tarfile.ReadError: file could not be opened successfully" if http://bugs.python.org/issue8978 closed by lars.gustaebel #9317: Incorrect coverage file from trace test_pickle.py http://bugs.python.org/issue9317 closed by berker.peksag #10261: tarfile iterator without members caching http://bugs.python.org/issue10261 closed by lars.gustaebel #13876: Sporadic failure in test_socket: testRecvmsgEOF http://bugs.python.org/issue13876 closed by berker.peksag #15933: flaky test in test_datetime http://bugs.python.org/issue15933 closed by berker.peksag #17859: improve error message for saving ints to file http://bugs.python.org/issue17859 closed by serhiy.storchaka #18591: threading.Thread.run returning a result http://bugs.python.org/issue18591 closed by berker.peksag #20739: PEP 463 (except expression) implementation http://bugs.python.org/issue20739 closed by berker.peksag #21668: The select and time modules uses libm functions without linkin http://bugs.python.org/issue21668 closed by haypo #22625: When cross-compiling, don’t try to execute binaries http://bugs.python.org/issue22625 closed by martin.panter #22873: Re: SSLsocket.getpeercert - return ALL the fields of the certi http://bugs.python.org/issue22873 closed by berker.peksag #23029: test_warnings produces extra output in quiet mode http://bugs.python.org/issue23029 closed by berker.peksag #23251: mention in time.sleep() docs that it does not block other Pyth http://bugs.python.org/issue23251 closed by berker.peksag #24173: curses HOWTO/implementation disagreement http://bugs.python.org/issue24173 closed by berker.peksag #24838: tarfile.py: fix GNU and USTAR formats to properly handle paths http://bugs.python.org/issue24838 closed by lars.gustaebel #24922: assertWarnsRegex doesn't allow multiple warning messages http://bugs.python.org/issue24922 closed by berker.peksag #25314: Documentation: argparse's actions store_{true,false} default t http://bugs.python.org/issue25314 closed by martin.panter #25642: Setting maxsize breaks asyncio.JoinableQueue/Queue http://bugs.python.org/issue25642 closed by berker.peksag #25989: documentation version switcher is broken fro 2.6, 3.2, 3.3 http://bugs.python.org/issue25989 closed by berker.peksag #26535: Minor typo in the docs for struct.unpack http://bugs.python.org/issue26535 closed by martin.panter #26615: Missing entry in WRAPPER_ASSIGNMENTS in update_wrapper's doc http://bugs.python.org/issue26615 closed by berker.peksag #26657: Directory traversal with http.server and SimpleHTTPServer on w http://bugs.python.org/issue26657 closed by martin.panter #26659: slice() leaks memory when part of a cycle http://bugs.python.org/issue26659 closed by python-dev #26717: wsgiref.simple_server: mojibake with cp1252 bytes in PATH_INFO http://bugs.python.org/issue26717 closed by martin.panter #26720: memoryview from BufferedWriter becomes garbage http://bugs.python.org/issue26720 closed by martin.panter #26745: Redundant code in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue26745 closed by serhiy.storchaka #26751: Possible bug in sorting algorithm http://bugs.python.org/issue26751 closed by benjamin.peterson #26755: Update version{added,changed} docs in devguide http://bugs.python.org/issue26755 closed by berker.peksag #26760: Document PyFrameObject http://bugs.python.org/issue26760 closed by brett.cannon #26763: Update PEP-8 regarding binary operators http://bugs.python.org/issue26763 closed by gvanrossum #26766: The result type of bytearray formatting is not stable http://bugs.python.org/issue26766 closed by berker.peksag #26770: _Py_set_inheritable(): do nothing if the FD_CLOEXEC close is a http://bugs.python.org/issue26770 closed by haypo #26772: regex.ENHANCEMATCH crashes interpreter http://bugs.python.org/issue26772 closed by SilentGhost #26775: Improve test coverage on urllib.parse http://bugs.python.org/issue26775 closed by orsenthil #26777: test_asyncio: test_timeout_disable() fails randomly http://bugs.python.org/issue26777 closed by haypo #26778: More typo fixes http://bugs.python.org/issue26778 closed by serhiy.storchaka #26780: Illustrate both binary operator conventions in PEP-8 http://bugs.python.org/issue26780 closed by orsenthil #26782: subprocess.__all__ incomplete on Windows http://bugs.python.org/issue26782 closed by martin.panter #26783: test_os.WalkTests.test_walk_topdown don't test fwalk and Bytes http://bugs.python.org/issue26783 closed by serhiy.storchaka #26784: regular expression problem at umlaut handling http://bugs.python.org/issue26784 closed by serhiy.storchaka #26785: repr of -nan value should contain the sign http://bugs.python.org/issue26785 closed by mark.dickinson #26795: Fix PEP 344 Python version http://bugs.python.org/issue26795 closed by SilentGhost #26799: gdb support fails with "Invalid cast." http://bugs.python.org/issue26799 closed by haypo #26802: Avoid copy in call_function_var when no extra stack args are p http://bugs.python.org/issue26802 closed by serhiy.storchaka #26805: Refer to types.SimpleNamespace in namedtuple documentation http://bugs.python.org/issue26805 closed by paul.moore #26808: wsgiref.simple_server breaks unicode in URIs http://bugs.python.org/issue26808 closed by martin.panter #26813: Wrong Japanese translation of "Adverb" on Documentation http://bugs.python.org/issue26813 closed by benjamin.peterson #26821: array module "minimum size in bytes" table is wrong for int/lo http://bugs.python.org/issue26821 closed by georg.brandl #26825: Variable defined in exec(code) unreachable inside function cal http://bugs.python.org/issue26825 closed by eryksun #1612012: builtin compile() doc needs PyCF_DONT_IMPLY_DEDENT http://bugs.python.org/issue1612012 closed by berker.peksag

1 0

Modify PyMem_Malloc to use pymalloc for performance
by Victor Stinner April 22, 2016

April 22, 2016

Hi, There is an old discussion about the performance of PyMem_Malloc() memory allocator. CPython is stressing a lot memory allocators. Last time I made statistics, it was for the PEP 454: "For example, the Python test suites calls malloc() , realloc() or free() 270,000 times per second in average." https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator I proposed a simple change: modify PyMem_Malloc() to use the pymalloc allocator which is faster for allocation smaller than 512 bytes, or fallback to malloc() (which is the current internal allocator of PyMem_Malloc()). This tiny change makes Python up to 6% faster on some specific (macro) benchmarks, and it doesn't seem to make Python slower on any benchmark: http://bugs.python.org/issue26249#msg259445 Do you see any drawback of using pymalloc for PyMem_Malloc()? Does anyone recall the rationale to have two families to memory allocators? FYI Python has 3 families since 3.4: PyMem, PyObject but also PyMem_Raw! https://www.python.org/dev/peps/pep-0445/ -- Since pymalloc is only used for small memory allocations, I understand that small objects will not more be allocated on the heap memory, but only in pymalloc arenas which are allocated by mmap. The advantage of arenas is that it's possible to "punch holes" in the memory when a whole arena is freed, whereas the heap memory has the famous "fragmentation" issue because the heap is a single contiguous memory block. The libc malloc() uses mmap() for allocations larger than a threshold which is now dynamic, and initialized to 128 kB or 256 kB by default (I don't recall exactly the default value). Is there a risk of *higher* memory fragmentation if we start to use pymalloc for PyMem_Malloc()? Does someone know how to test it? Victor

4 19

Issue with DLL import library installation in Cygwin
by Erik Bray April 22, 2016

April 22, 2016

Hi all, I've been working on compiling/installing Python on Cygwin and have hit upon an odd issue in the Makefile that seems to have been around for as long as there's been Cygwin support in it. When building Python on Cygwin, both a libpython-X.Y.dll and a libpython-X.Y.dll.a are created. The latter is an "import library" consisting of stubs for functions in the DLL so that it can be linked to statically when building, for example, extension modules. The odd bit is that in the altbininstall target (see [1]) if the $(DLLLIBRARY) variable is defined then only it is installed, while $(LDLIBRARY) (which in this cases references the import library) is *not* installed, except in $(prefix)/lib/pythonX.Y/config, which is not normally on the linker search path, or even included by python-config --ldflags. Therefore static linking to libpython fails, unless the search path is explicitly modified, or a symlink is created from $(prefix)/lib/pythonX.Y/config/libpython.dll.a to $(prefix)/lib. In fact Cygwin's own package for Python manually creates the latter symlink in its install script. But it's not clear why Python's Makefile doesn't install this file in the first place. In other words, why not install $LDLIBRARY regardless? Thanks, Erik [1] https://hg.python.org/cpython/file/496e094f4734/Makefile.pre.in#l1097

1 0

obmalloc mmap/munmap thrashing
by Neil Schemenauer April 21, 2016

April 21, 2016

I was running Python 2.4.11 under strace and I noticed some odd looking system calls: mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9848681000 munmap(0x7f9848681000, 262144) = 0 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9848681000 munmap(0x7f9848681000, 262144) = 0 [... repeated a number of times ...] Looking at obmalloc.c, there doesn't seem to be any high/low watermark (hysteresis) associated with unallocating arenas. Is that true? If so, does it seem prudent to implement something to avoid this behavior? It seems potentially expensive if you program is running just at the threshold of needing another arena.

3 2

Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
by Ethan Furman April 21, 2016

April 21, 2016

On 04/09/2016 12:48 AM, Nick Coghlan wrote: > Considering the helper function usage, here's some examples in > combination with os.fsencode and os.fsdecode: > > # Status quo for binary/text path conversions > text_path = os.fsdecode(bytes_path) > bytes_path = os.fsencode(text_path) > > # Getting a text path from an arbitrary object > text_path = os.fspath(obj) # This doesn't scream "returns text!" > text_path = os.fspathname(obj) # This does > > # Getting a binary path from an arbitrary object > bytes_path = os.fsencode(os.fspath(obj)) > bytes_path = os.fsencode(os.fspathname(obj)) > > I'm starting to think the semantic nudge from the "name" suffix when > reading the code is worth the extra four characters when writing it > (keeping in mind that the whole point of this exercise is that most > folks *won't* be writing explicit conversions - the stdlib will handle > it on their behalf). > > I also think the more explicit name helps answer some of the type > signature questions that have arisen: > > 1. Does os.fspathname return rich Path objects? No, it returns names > as str objects > 2. Will file descriptors pass through os.fspathname? No, as they're > not names, they're numeric descriptors. > 3. Will bytes-like objects pass through os.fspathname? No, as they're > not names, they're encodings of names This worries me. I know the primary purpose of this change is to enable pathlib and os and the rest of the stdlib to work together, but consider . . . If adding a new attribute/method was as far as we went, new code (stdlib or otherwise) would look like: if isinstance(a_path_thingy, bytes): # because os can accept bytes pass elif isinstance(a_path_thingy, str): # but it's usually text pass elif hasattr(a_path_thingy, '__fspath__'): a_path_thingy = a_path_thingy.__fspath__() else: raise TypeError('not a valid path') # do something with the path If we add os.fspath(), but don't allow bytes to be returned from it, our above example looks more like: if isinstance(a_path_thingy, bytes): # because os can accept bytes pass else: a_path_thingy = os.fspath(a_path_thingy) # do something with the path Yes, it's better -- but it still requires a pre-check before calling os.fspath(). It is my contention that this is better: a_path_thingy = os.fspath(a_path_thingy) This raises two issues: 1) Part of the stdlib is the new scandir module, which can work with, and return, both bytes and text -- if __fspath__ can only hold text, DirEntry will not get the __fspath__ method added, and the pre-check, boiler-plate code will flourish; 2) pathlib.Path accepts bytes -- so what happens when a byte-derived Path is passed to os.fspath()? Is a TypeError raised? Do we guess and auto-convert with fsdecode()? I think the best answer is to - let __fspath__ hold bytes as well as text - let fspath() return bytes as well as text -- ~Ethan~

18 105

When the infamous Bier trunk hits ... where is our Python backup?
by Burkhard Meier April 21, 2016

April 21, 2016

Well, Just a thought.

4 4

PEP 509: Add a private version to dict (version 3)
by Victor Stinner April 20, 2016

April 20, 2016

Hi, Below if the third version of my PEP 509 (dict version). Changes since the version 2: * __setitem__() and update() now always increases the version: remove the micro-optimization on "dict[key] is new_value". Exception: version is not changed with dict.update() is called without argument. * be more explict on version++: explain that the operation must be atomic, and that dict methods are already atomic thanks to the GIL * Usage of the dict version: add Cython * "Guard against changing dict during iteration": don't guess if the new dict version can be used or not. Let's discuss that later. * rephrase/complete some sections * add links to new threads on python-dev I hope that I addressed all Jim's concerns about the version 2. Note: I also updated the implementation. The implementation now contains more tests for identical values and more tests on equal values. HTML version: https://www.python.org/dev/peps/pep-0509/ PEP: 509 Title: Add a private version to dict Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Add a new private version to the builtin ``dict`` type, incremented at each dictionary creation and at each dictionary change, to implement fast guards on namespaces. Rationale ========= In Python, the builtin ``dict`` type is used by many instructions. For example, the ``LOAD_GLOBAL`` instruction looks up a variable in the global namespace, or in the builtins namespace (two dict lookups). Python uses ``dict`` for the builtins namespace, globals namespace, type namespaces, instance namespaces, etc. The local namespace (function namespace) is usually optimized to an array, but it can be a dict too. Python is hard to optimize because almost everything is mutable: builtin functions, function code, global variables, local variables, ... can be modified at runtime. Implementing optimizations respecting the Python semantics requires to detect when "something changes": we will call these checks "guards". The speedup of optimizations depends on the speed of guard checks. This PEP proposes to add a private version to dictionaries to implement fast guards on namespaces. Dictionary lookups can be skipped if the version does not change which is the common case for most namespaces. The version is globally unique, so checking the version is also enough to check if the namespace dictionary was not replaced with a new dictionary. When the dictionary version does not change, the performance of a guard does not depend on the number of watched dictionary entries: the complexity is O(1). Example of optimization: copy the value of a global variable to function constants. This optimization requires a guard on the global variable to check if it was modified. If the global variable is not modified, the function uses the cached copy. If the global variable is modified, the function uses a regular lookup, and maybe also deoptimize the function (to remove the overhead of the guard check for next function calls). See the `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of guards to specialize functions and for a more general rationale on Python static optimizers. Guard example ============= Pseudo-code of an fast guard to check if a dictionary entry was modified (created, updated or deleted) using an hypothetical ``dict_get_version(dict)`` function:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.value = dict.get(key, UNSET) self.version = dict_get_version(dict) def check(self): """Return True if the dictionary entry did not change and the dictionary was not replaced.""" # read the version of the dictionary version = dict_get_version(self.dict) if version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary value = self.dict.get(self.key, UNSET) if value is self.value: # another key was modified: # cache the new dictionary version self.version = version return True # the key was modified return False Usage of the dict version ========================= Speedup method calls -------------------- Yury Selivanov wrote a `patch to optimize method calls <https://bugs.python.org/issue26110>`_. The patch depends on the `"implement per-opcode cache in ceval" <https://bugs.python.org/issue26219>`_ patch which requires dictionary versions to invalidate the cache if the globals dictionary or the builtins dictionary has been modified. The cache also requires that the dictionary version is globally unique. It is possible to define a function in a namespace and call it in a different namespace, using ``exec()`` with the *globals* parameter for example. In this case, the globals dictionary was replaced and the cache must also be invalidated. Specialized functions using guards ---------------------------------- The `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support specialized functions with guards. It allows to implement static optimizers for Python without breaking the Python semantics. The `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ of the `FAT Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project is an example of a static Python optimizer. It implements many optimizations which require guards on namespaces: * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on ``builtins.__dict__['len']`` and ``globals()['len']`` are required * Loop unrolling: to unroll the loop ``for i in range(...): ...``, guards on ``builtins.__dict__['range']`` and ``globals()['range']`` are required * etc. Pyjion ------ According of Brett Cannon, one of the two main developers of Pyjion, Pyjion can benefit from dictionary version to implement optimizations. `Pyjion <https://github.com/Microsoft/Pyjion>`_ is a JIT compiler for Python based upon CoreCLR (Microsoft .NET Core runtime). Cython ------ Cython can benefit from dictionary version to implement optimizations. `Cython <http://cython.org/>`_ is an optimising static compiler for both the Python programming language and the extended Cython programming language. Unladen Swallow --------------- Even if dictionary version was not explicitly mentioned, optimizing globals and builtins lookup was part of the Unladen Swallow plan: "Implement one of the several proposed schemes for speeding lookups of globals and builtins." (source: `Unladen Swallow ProjectPlan <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_). Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler implemented with LLVM. The project stopped in 2011: `Unladen Swallow Retrospective <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_. Changes ======= Add a ``ma_version_tag`` field to the ``PyDictObject`` structure with the C type ``PY_UINT64_T``, 64-bit unsigned integer. Add also a global dictionary version. Each time a dictionary is created, the global version is incremented and the dictionary version is initialized to the global version. The global version is also incremented and copied to the dictionary version at each dictionary change: * ``clear()`` if the dict is non-empty * ``pop(key)`` if the key exists * ``popitem()`` if the dict is non-empty * ``setdefault(key, value)`` if the key does not exist * ``__delitem__(key)`` if the key exists * ``__setitem__(key, value)`` always increases the version * ``update(...)`` if called with arguments The version increase must be atomic. In CPython, the Global Interpreter Lock (GIL) already protects ``dict`` methods to make them atomic. Example using an hypothetical ``dict_get_version(dict)`` function:: >>> d = {} >>> dict_get_version(d) 100 >>> d['key'] = 'value' >>> dict_get_version(d) 101 >>> d['key'] = 'new value' >>> dict_get_version(d) 102 >>> del d['key'] >>> dict_get_version(d) 103 ``dict.__setitem__(key, value)`` and ``dict.update(...)`` always increases the version, even if the new value is identical or is equal to the current value (even if ``(dict[key] is value) or (dict[key] == value)``). The field is called ``ma_version_tag``, rather than ``ma_version``, to suggest to compare it using ``version_tag == old_version_tag``, rather than ``version <= old_version`` which is wrong most of the time after an integer overflow. Backwards Compatibility ======================= Since the ``PyDictObject`` structure is not part of the stable ABI and the new dictionary version not exposed at the Python scope, changes are backward compatible. Implementation and Performance ============================== The `issue #26058: PEP 509: Add ma_version_tag to PyDictObject <https://bugs.python.org/issue26058>`_ contains a patch implementing this PEP. On pybench and timeit microbenchmarks, the patch does not seem to add any overhead on dictionary operations. For example, the following timeit micro-benchmarks takes 318 nanoseconds before and after the change:: python3.6 -m timeit 'd={1: 0}; d[2]=0; d[3]=0; d[4]=0; del d[1]; del d[2]; d.clear()' When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for a dictionary lookup, whereas a guard check only takes 3.8 ns. Moreover, a guard can watch for multiple keys. For example, for an optimization using 10 global variables in a function, 10 dictionary lookups costs 148 ns, whereas the guard still only costs 3.8 ns when the version does not change (39x as fast). The `fat module <http://fatoptimizer.readthedocs.org/en/latest/fat.html>`_ implements such guards: ``fat.GuardDict`` is based on the dictionary version. Integer overflow ================ The implementation uses the C type ``PY_UINT64_T`` to store the version: a 64 bits unsigned integer. The C code uses ``version++``. On integer overflow, the version is wrapped to ``0`` (and then continue to be incremented) according to the C standard. After an integer overflow, a guard can succeed whereas the watched dictionary key was modified. The bug only occurs at a guard check if there are exaclty ``2 ** 64`` dictionary creations or modifications since the previous guard check. If a dictionary is modified every nanosecond, ``2 ** 64`` modifications takes longer than 584 years. Using a 32-bit version, it only takes 4 seconds. That's why a 64-bit unsigned type is also used on 32-bit systems. A dictionary lookup at the C level takes 14.8 ns. A risk of a bug every 584 years is acceptable. Alternatives ============ Expose the version at Python level as a read-only __version__ property ---------------------------------------------------------------------- The first version of the PEP proposed to expose the dictionary version as a read-only ``__version__`` property at Python level, and also to add the property to ``collections.UserDict`` (since this type must mimick the ``dict`` API). There are multiple issues: * To be consistent and avoid bad surprises, the version must be added to all mapping types. Implementing a new mapping type would require extra work for no benefit, since the version is only required on the ``dict`` type in practice. * All Python implementations would have to implement this new property, it gives more work to other implementations, whereas they may not use the dictionary version at all. * Exposing the dictionary version at the Python level can lead the false assumption on performances. Checking ``dict.__version__`` at the Python level is not faster than a dictionary lookup. A dictionary lookup in Python has a cost of 48.7 ns and checking the version has a cost of 47.5 ns, the difference is only 1.2 ns (3%):: $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33' 10000000 loops, best of 3: 0.0487 usec per loop $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd.__version__ == 100' 10000000 loops, best of 3: 0.0475 usec per loop * The ``__version__`` can be wrapped on integer overflow. It is error prone: using ``dict.__version__ <= guard_version`` is wrong, ``dict.__version__ == guard_version`` must be used instead to reduce the risk of bug on integer overflow (even if the integer overflow is unlikely in practice). Mandatory bikeshedding on the property name: * ``__cache_token__``: name proposed by Nick Coghlan, name coming from `abc.get_cache_token() <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_. * ``__version__`` * ``__version_tag__`` * ``__timestamp__`` Add a version to each dict entry -------------------------------- A single version per dictionary requires to keep a strong reference to the value which can keep the value alive longer than expected. If we add also a version per dictionary entry, the guard can only store the entry version (a simple integer) to avoid the strong reference to the value: only strong references to the dictionary and to the key are needed. Changes: add a ``me_version_tag`` field to the ``PyDictKeyEntry`` structure, the field has the C type ``PY_UINT64_T``. When a key is created or modified, the entry version is set to the dictionary version which is incremented at any change (create, modify, delete). Pseudo-code of an fast guard to check if a dictionary key was modified using hypothetical ``dict_get_version(dict)`` and ``dict_get_entry_version(dict)`` functions:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.dict_version = dict_get_version(dict) self.entry_version = dict_get_entry_version(dict, key) def check(self): """Return True if the dictionary entry did not change and the dictionary was not replaced.""" # read the version of the dictionary dict_version = dict_get_version(self.dict) if dict_version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary to read the entry version entry_version = get_dict_key_version(dict, key) if entry_version == self.entry_version: # another key was modified: # cache the new dictionary version self.dict_version = dict_version self.entry_version = entry_version return True # the key was modified return False The main drawback of this option is the impact on the memory footprint. It increases the size of each dictionary entry, so the overhead depends on the number of buckets (dictionary entries, used or not used). For example, it increases the size of each dictionary entry by 8 bytes on 64-bit system. In Python, the memory footprint matters and the trend is to reduce it. Examples: * `PEP 393 -- Flexible String Representation <https://www.python.org/dev/peps/pep-0393/>`_ * `PEP 412 -- Key-Sharing Dictionary <https://www.python.org/dev/peps/pep-0412/>`_ Add a new dict subtype ---------------------- Add a new ``verdict`` type, subtype of ``dict``. When guards are needed, use the ``verdict`` for namespaces (module namespace, type namespace, instance namespace, etc.) instead of ``dict``. Leave the ``dict`` type unchanged to not add any overhead (CPU, memory footprint) when guards are not used. Technical issue: a lot of C code in the wild, including CPython core, expecting the exact ``dict`` type. Issues: * ``exec()`` requires a ``dict`` for globals and locals. A lot of code use ``globals={}``. It is not possible to cast the ``dict`` to a ``dict`` subtype because the caller expects the ``globals`` parameter to be modified (``dict`` is mutable). * C functions call directly ``PyDict_xxx()`` functions, instead of calling ``PyObject_xxx()`` if the object is a ``dict`` subtype * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some functions require the exact ``dict`` type. * ``Python/ceval.c`` does not completely supports dict subtypes for namespaces The ``exec()`` issue is a blocker issue. Other issues: * The garbage collector has a special code to "untrack" ``dict`` instances. If a ``dict`` subtype is used for namespaces, the garbage collector can be unable to break some reference cycles. * Some functions have a fast-path for ``dict`` which would not be taken for ``dict`` subtypes, and so it would make Python a little bit slower. Prior Art ========= Method cache and type version tag --------------------------------- In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It was merged into Python 2.6. The patch adds a "type attribute cache version tag" (``tp_version_tag``) and a "valid version tag" flag to types (the ``PyTypeObject`` structure). The type version tag is not exposed at the Python level. The version tag has the C type ``unsigned int``. The cache is a global hash table of 4096 entries, shared by all types. The cache is global to "make it fast, have a deterministic and low memory footprint, and be easy to invalidate". Each cache entry has a version tag. A global version tag is used to create the next version tag, it also has the C type ``unsigned int``. By default, a type has its "valid version tag" flag cleared to indicate that the version tag is invalid. When the first method of the type is cached, the version tag and the "valid version tag" flag are set. When a type is modified, the "valid version tag" flag of the type and its subclasses is cleared. Later, when a cache entry of these types is used, the entry is removed because its version tag is outdated. On integer overflow, the whole cache is cleared and the global version tag is reset to ``0``. See `Method cache (issue #1685986) <https://bugs.python.org/issue1685986>`_ and `Armin's method cache optimization updated for Python 2.6 (issue #1700288) <https://bugs.python.org/issue1700288>`_. Globals / builtins cache ------------------------ In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue #10401) <http://bugs.python.org/issue10401>`_ which adds a private ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``Py_ssize_t``. The patch adds a "global and builtin cache" to functions and frames, and changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the cache. The change on the ``PyDictObject`` structure is very similar to this PEP. Cached globals+builtins lookup ------------------------------ In 2006, Andrea Griffini proposed a patch implementing a `Cached globals+builtins lookup optimization <https://bugs.python.org/issue1616125>`_. The patch adds a private ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. Thread on python-dev: `About dictionary lookup caching <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_ (December 2006). Guard against changing dict during iteration -------------------------------------------- In 2013, Serhiy Storchaka proposed `Guard against changing dict during iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. This field is incremented when the dictionary is modified. PySizer ------- `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary entries. It uses a global process-wide counter for dictionaries, incremented each time that a dictionary is modified. The times are used to decide when child objects first appeared in their parent objects. Discussion ========== Thread on the mailing lists: * python-dev: `Updated PEP 509 <https://mail.python.org/pipermail/python-dev/2016-April/144250.html>`_ * python-dev: `RFC: PEP 509: Add a private version to dict <https://mail.python.org/pipermail/python-dev/2016-April/144137.html>`_ * python-dev: `PEP 509: Add a private version to dict <https://mail.python.org/pipermail/python-dev/2016-January/142685.html>`_ (january 2016) * python-ideas: `RFC: PEP: Add dict.__version__ <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_ (january 2016) Copyright ========= This document has been placed in the public domain.

1 2

Discussion on fspath: please wait for a PEP
by Victor Stinner April 20, 2016

April 20, 2016

Hi, I'm unable to count the number of threads about the fspath protocol. It's even more difficult to count the total number of emails. IMHO everyone had enough time to give him/her opinion. We even had multiple summaries :-) Can you please wait for a PEP? Brett Canon and Ethan Furman are working on a PEP. So please give them time to write it. The PEP should summarize the discussion and help a lot to make concrete progress on the design (avoid restarting to discuss the same points forever). I don't expect that more emails would add anything at the current state of the discussion. I think that we have enough other topics to discuss in the meanwhile ;-) FYI there is already an article about fspath/pathlib on LWN. Here is a free link until the article is freely accessible: "Python looks at paths" By Jake Edge (April 13, 2016) https://lwn.net/SubscriberLink/683350/4f52334af09653c8/ Victor

5 8

Bytes path
by Serhiy Storchaka April 20, 2016

April 20, 2016

What types should be accepted as bytes path? For now os.path is strict and accepts only bytes and bytes subclasses (even bytearray is not accepted) as bytes path. This is enough for working with low-level Posix paths and supporting backward compatibility. On other hand, most os functions is too permissive since 3.3 and accept any type that supports the buffer protocol as bytes path. Accepted even such meaningless objects as array('h'). Some functions (zipimport.zipimporter() in 3.x, _imp.load_dynamic() in 3.3+, builtin compile() etc in 3.4) accept even arbitrary iterables, e.g. [116, 101, 115, 116] (see http://bugs.python.org/issue26754). I think we should accept only bytes (and subclasses). Even bytearray is less acceptable since it is mutable and can't be used as a key in caches.

3 2