Mailman 3 September 2016 - Python-Dev

PEP 468 ready for pronouncement.
by Eric Snow 08 Sep '16

08 Sep '16

see: https://github.com/python/peps/blob/master/pep-0468.txt With the introduction of the compact dict implementation for CPython 3.6, PEP 468 becomes no more than a change to the language reference. I've adjusted the PEP to specify use of an ordered mapping rather than exactly OrderedDict. Thanks! -eric

2 1

PEP 529: Change Windows filesystem encoding to UTF-8
by Steve Dower 08 Sep '16

08 Sep '16

I'm about to be offline for a few days, so I wanted to get my current draft PEPs out for people can read and review. I don't believe there is a lot of change as a result of either PEP, but the impact of what change there is needs to be weighed against the benefits. We've already had some thorough discussion on this one and failed to reach agreement on whether we can make this change in 3.6 or if it needs a deprecation cycle that is more visible than the one we started in 3.3. In the latter case, we need to determine how visible that should be (i.e. warnings visible by default, visible for non-Windows platforms, value-dependent warnings/errors, etc.). IMHO, the argument about having the change be on-by-default or off-by-default is irrelevant until we decide on the deprecation issue, at which point it is obvious what the default should be. See https://bugs.python.org/issue27781 for the current proposed patch. I do need to update it in order to merge against default it seems (work for my upcoming flight). Cheers, Steve --- https://github.com/python/peps/blob/master/pep-0529.txt --- PEP: 529 Title: Change Windows filesystem encoding to UTF-8 Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Aug-2016 Post-History: 01-Sep-2016 Abstract ======== Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-16, and the ANSI APIs perform encoding and decoding using the active code page. This PEP proposes changing the default filesystem encoding on Windows to utf-8, and changing all filesystem functions to use the Unicode APIs for filesystem paths. This will not affect code that uses strings to represent paths, however those that use bytes for paths will now be able to correctly round-trip all valid paths in Windows filesystems. Currently, the conversions between Unicode (in the OS) and bytes (in Python) were lossy and would fail to round-trip characters outside of the user's active code page. Notably, this does not impact the encoding of the contents of files. These will continue to default to locale.getpreferredencoding (for text files) or plain bytes (for binary files). This only affects the encoding used when users pass a bytes object to Python where it is then passed to the operating system as a path name. Background ========== File system paths are almost universally represented as text with an encoding determined by the file system. In Python, we expose these paths via a number of interfaces, such as the ``os`` and ``io`` modules. Paths may be passed either direction across these interfaces, that is, from the filesystem to the application (for example, ``os.listdir()``), or from the application to the filesystem (for example, ``os.unlink()``). When paths are passed between the filesystem and the application, they are either passed through as a bytes blob or converted to/from str using ``os.fsencode()`` or ``sys.getfilesystemencoding()``. The result of encoding a string with ``sys.getfilesystemencoding()`` is a blob of bytes in the native format for the default file system. On Windows, the native format for the filesystem is utf-16-le. The recommended platform APIs for accessing the filesystem all accept and return text encoded in this format. However, prior to Windows NT (and possibly further back), the native format was a configurable machine option and a separate set of APIs existed to accept this format. The option (the "active code page") and these APIs (the "*A functions") still exist in recent versions of Windows for backwards compatibility, though new functionality often only has a utf-16-le API (the "*W functions"). In Python, str is recommended because it can correctly round-trip all characters used in paths (on POSIX with surrogateescape handling; on Windows because str maps to the native representation). On Windows bytes cannot round-trip all characters used in paths, as Python internally uses the *A functions and hence the encoding is "whatever the active code page is". Since the active code page cannot represent all Unicode characters, the conversion of a path into bytes can lose information without warning or any available indication. As a demonstration of this:: >>> open('test\uAB00.txt', 'wb').close() >>> import glob >>> glob.glob('test*') ['test\uab00.txt'] >>> glob.glob(b'test*') [b'test?.txt'] The Unicode character in the second call to glob has been replaced by a '?', which means passing the path back into the filesystem will result in a ``FileNotFoundError``. The same results may be observed with ``os.listdir()`` or any function that matches the return type to the parameter type. While one user-accessible fix is to use str everywhere, POSIX systems generally do not suffer from data loss when using bytes exclusively as the bytes are the canonical representation. Even if the encoding is "incorrect" by some standard, the file system will still map the bytes back to the file. Making use of this avoids the cost of decoding and reencoding, such that (theoretically, and only on POSIX), code such as this may be faster because of the use of `b'.'` compared to using `'.'`:: >>> for f in os.listdir(b'.'): ... os.stat(f) ... As a result, POSIX-focused library authors prefer to use bytes to represent paths. For some authors it is also a convenience, as their code may receive bytes already known to be encoded correctly, while others are attempting to simplify porting their code from Python 2. However, the correctness assumptions do not carry over to Windows where Unicode is the canonical representation, and errors may result. This potential data loss is why the use of bytes paths on Windows was deprecated in Python 3.3 - all of the above code snippets produce deprecation warnings on Windows. Proposal ======== Currently the default filesystem encoding is 'mbcs', which is a meta-encoder that uses the active code page. However, when bytes are passed to the filesystem they go through the *A APIs and the operating system handles encoding. In this case, paths are always encoded using the equivalent of 'mbcs:replace' - we have no ability to change this (though there is a user/machine configuration option to change the encoding from CP_ACP to CP_OEM, so it won't necessarily always match mbcs...) This proposal would remove all use of the *A APIs and only ever call the *W APIs. When Windows returns paths to Python as str, they will be decoded from utf-16-le and returned as text (in whatever the minimal representation is). When Windows returns paths to Python as bytes, they will be decoded from utf-16-le to utf-8 using surrogatepass (Windows does not validate surrogate pairs, so it is possible to have invalid surrogates in filenames). Equally, when paths are provided as bytes, they are decoded from utf-8 into utf-16-le and passed to the *W APIs. The use of utf-8 will not be configurable, with the possible exception of a "legacy mode" environment variable or X-flag. surrogateescape does not apply here, as the concern is not about retaining non-sensical bytes. Any path returned from the operating system will be valid Unicode, while bytes paths created by the user may raise a decoding error (currently these would raise ``OSError`` or a subclass). The choice of utf-8 bytes (as opposed to utf-16-le bytes) is to ensure the ability to round-trip without breaking the functionality of the ``os.path`` module, which assumes an ASCII-compatible encoding. Using utf-16-le as the encoding is more pure, but will cause more issues than are resolved. This change would also undeprecate the use of bytes paths on Windows. No change to the semantics of using bytes as a path is required - as before, they must be encoded with the encoding specified by ``sys.getfilesystemencoding()``. Specific Changes ================ Update sys.getfilesystemencoding -------------------------------- Remove the default value for ``Py_FileSystemDefaultEncoding`` and set it in ``initfsencoding()`` to utf-8, or if the legacy-mode switch is enabled to mbcs. Update the implementations of ``PyUnicode_DecodeFSDefaultAndSize`` and ``PyUnicode_EncodeFSDefault`` to use the standard utf-8 codec with surrogatepass error mode, or if the legacy-mode switch is enabled the code page codec with replace error mode. Update path_converter --------------------- Update the path converter to always decode bytes or buffer objects into text using ``PyUnicode_DecodeFSDefaultAndSize``. Change the ``narrow`` field from a ``char*`` string into a flag that indicates whether the original object was bytes. This is required for functions that need to return paths using the same type as was originally provided. Remove unused ANSI code ----------------------- Remove all code paths using the ``narrow`` field, as these will no longer be reachable by any caller. These are only used within ``posixmodule.c``. Other uses of paths should have use of bytes paths replaced with decoding and use of the *W APIs. Add legacy mode --------------- Add a legacy mode flag, enabled by the environment variable ``PYTHONLEGACYWINDOWSFSENCODING``. When this flag is set, the default filesystem encoding is set to mbcs rather than utf-8, and the error mode is set to 'replace' rather than 'strict'. The ``path_converter`` will continue to decode to wide characters and only *W APIs will be called, however, the bytes passed in and received from Python will be encoded the same as prior to this change. Undeprecate bytes paths on Windows ---------------------------------- Using bytes as paths on Windows is currently deprecated. We would announce that this is no longer the case, and that paths when encoded as bytes should use whatever is returned from ``sys.getfilesystemencoding()`` rather than the user's active code page. Rejected Alternatives ===================== Use strict mbcs decoding ------------------------ This is essentially the same as the proposed change, but instead of changing ``sys.getfilesystemencoding()`` to utf-8 it is changed to mbcs (which dynamically maps to the active code page). This approach allows the use of new functionality that is only available as *W APIs and also detection of encoding/decoding errors. For example, rather than silently replacing Unicode characters with '?', it would be possible to warn or fail the operation. Compared to the proposed fix, this could enable some new functionality but does not fix any of the problems described initially. New runtime errors may cause some problems to be more obvious and lead to fixes, provided library maintainers are interested in supporting Windows and adding a separate code path to treat filesystem paths as strings. Making the encoding mbcs without strict errors is equivalent to the legacy-mode switch being enabled by default. This is a possible course of action if there is significant breakage of actual code and a need to extend the deprecation period, but still a desire to have the simplifications to the CPython source. Make bytes paths an error on Windows ------------------------------------ By preventing the use of bytes paths on Windows completely we prevent users from hitting encoding issues. However, the motivation for this PEP is to increase the likelihood that code written on POSIX will also work correctly on Windows. This alternative would move the other direction and make such code completely incompatible. As this does not benefit users in any way, we reject it. Make bytes paths an error on all platforms ------------------------------------------ By deprecating and then disable the use of bytes paths on all platforms we prevent users from hitting encoding issues regardless of where the code was originally written. This would require a full deprecation cycle, as there are currently no warnings on platforms other than Windows. This is likely to be seen as a hostile action against Python developers in general, and as such is rejected at this time. Code that may break =================== The following code patterns may break or see different behaviour as a result of this change. Note that all of these examples produce deprecation warnings on Python 3.3 and later. Not managing encodings across boundaries ---------------------------------------- Code that does not manage encodings when crossing protocol boundaries may currently be working by chance, but could encounter issues when either encoding changes. For example:: filename = open('filename_in_mbcs.txt', 'rb').read() text = open(filename, 'r').read() To correct this code, the encoding of the bytes in ``filename`` should be specified, either when reading from the file or before using the value:: # Fix 1: Open file as text filename = open('filename_in_mbcs.txt', 'r', encoding='mbcs').read() text = open(filename, 'r').read() # Fix 2: Decode path filename = open('filename_in_mbcs.txt', 'rb').read() text = open(filename.decode('mbcs'), 'r').read() Explicitly using 'mbcs' ----------------------- Code that explicitly encodes text using 'mbcs' before passing to file system APIs. For example:: filename = open('files.txt', 'r').readline() text = open(filename.encode('mbcs'), 'r') To correct this code, the string should be passed without explicit encoding, or should use ``os.fsencode()``:: # Fix 1: Do not encode the string filename = open('files.txt', 'r').readline() text = open(filename, 'r') # Fix 2: Use correct encoding filename = open('files.txt', 'r').readline() text = open(os.fsencode(filename), 'r') Copyright ========= This document has been placed in the public domain.

7 15

hg push segfault
by Christian Heimes 08 Sep '16

08 Sep '16

Hi, About 10 minutes ago I got a couple of remote segfaults from hg.python.org. They occurred during push and pull operations: $ hg push pushing to ssh://hg@hg.python.org/cpython remote: bash: line 1: 25019 Segmentation fault HGPUSHER=christian.heimes /srv/hg/bin/hg-ssh /srv/hg/repos/* abort: no suitable response from remote hg! It's fine again now. Can somebody look into the matter, please? Christian

2 1

Re: [Python-Dev] cpython (3.5): supress coroutine warning when an exception is pending (#27968)
by Christian Heimes 08 Sep '16

08 Sep '16

On 2016-09-07 17:47, benjamin.peterson wrote: > https://hg.python.org/cpython/rev/234f758449f8 > changeset: 103223:234f758449f8 > branch: 3.5 > parent: 103213:7537ca1c2aaf > user: Benjamin Peterson <benjamin(a)python.org> > date: Wed Sep 07 08:46:59 2016 -0700 > summary: > supress coroutine warning when an exception is pending (#27968) > > files: > Objects/genobject.c | 27 +++++++++++++++------------ > 1 files changed, 15 insertions(+), 12 deletions(-) > > > diff --git a/Objects/genobject.c b/Objects/genobject.c > --- a/Objects/genobject.c > +++ b/Objects/genobject.c > @@ -21,7 +21,7 @@ > _PyGen_Finalize(PyObject *self) > { > PyGenObject *gen = (PyGenObject *)self; > - PyObject *res; > + PyObject *res = NULL; > PyObject *error_type, *error_value, *error_traceback; > > if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL) > @@ -33,23 +33,26 @@ > > /* If `gen` is a coroutine, and if it was never awaited on, > issue a RuntimeWarning. */ > - if (gen->gi_code != NULL > - && ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE > - && gen->gi_frame->f_lasti == -1 > - && !PyErr_Occurred() > - && PyErr_WarnFormat(PyExc_RuntimeWarning, 1, > - "coroutine '%.50S' was never awaited", > - gen->gi_qualname)) { > - res = NULL; /* oops, exception */ > + if (gen->gi_code != NULL && > + ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE && > + gen->gi_frame->f_lasti == -1) { > + if (!error_value) { > + PyErr_WarnFormat(PyExc_RuntimeWarning, 1, > + "coroutine '%.50S' was never awaited", > + gen->gi_qualname); > + } You don't check the return value of PyErr_WarnFormat(). It does not signal an exception in case warnings are turned into exceptions. Christian

2 1

Making PEP 3156 (asyncio) non-provisional
by Guido van Rossum 08 Sep '16

08 Sep '16

PEP 3156 and the asyncio module it defines have been provisional for the lifetime of Python 3.4 and 3.5. The module is now quite mature. I propose that we end the provisional period and make asyncio subject to the usual backwards compatibility rules: new features only appear in "minor" releases (e.g. 3.6, 3.7) and all changes must be backward compatible. There's some wiggle room though: in some cases we may decide that a given "feature" was really "undefined/undocumented behavior" and then we can treat it as a bug and fix it (== change the behavior) in a bugfix release (or during the 3.6 beta period). There are some worries that Twisted might request some incompatible changes in order to obtain better interoperability. I've sent an email to Amber Brown asking for a clarification. There's also the issue of starttls, a feature that we know we'd like to add but don't have ready for 3.6b1. I think the right approach there is to provide an add-on package on PyPI that implements a starttls-capable Transport class, and when that code is sufficiently battle-tested we can add it to the stdlib (hopefully by 3.7). Such a package might end up having to copy portions of the asyncio implementation and/or use internal/undocumented APIs; that's fine because it is only meant as a temporary measure, and we can make it clear that just because the starttls package uses a certain internal API that doesn't mean that API is now public. A big advantage of having the initial starttls implementation outside the stdlib is that its release schedule can be much more frequent than that of the stdlib (== every 6 months), and a security issue in the starttls package won't require all the heavy guns of doing a security release of all of CPython. -- --Guido van Rossum (python.org/~guido)

2 2

PEP 526 (variable annotations) accepted provisionally
by Guido van Rossum 08 Sep '16

08 Sep '16

I'm accepting PEP 526 provisionally. I am personally confident that this PEP is adding a useful new feature to the language: annotations that can be used by a wide variety of tools, whether off-line type checkers or frameworks that add runtime checking (e.g. traits or traitlets). The provisional status reflects the understanding that minor details of the proposed syntax and its runtime effects may still have to change based on experience during the 3.6 life cycle. (For example, maybe we end up not liking ClassVar, or maybe we'll decide we'll want to support `x, y, z: T` after all.) There's been some quite contentious discussion about the PEP, on and off python-dev, regarding how the mere presence of annotation syntax in the language will change the way people will see the language. My own experience using mypy and PyCharm has been quite different: annotations are a valuable addition for large code bases, and it's worth the effort to add them to large legacy code bases (think millions of lines of Python 2.7 code that needs to move to Python 3 by 2020). The effect of this has been that engineers using Python are happier and more confident that their code works than before, have an easier time spelunking code they don't know, and are less afraid of big refactorings (where conversion to Python 3 can be seen as the ultimate refactoring). I should blog about our experience at Dropbox; I hope the Zulip open source folks (not at Dropbox) will also blog about their experience. In the meantime you can read Daniel F. Moisset's three-part blog about adding annotations to pycodestyle (formerly pep8) here: http://www.machinalis.com/blog/a-day-with-mypy-part-1/ If you want to see a large open source code base that's annotated for mypy (with 97% coverage), I recommend looking at Zulip: https://github.com/zulip/zulip Finally, some of us are starting a new (informational) PEP to set expectations for how type checkers should make use of the annotation syntax standardized by PEP 484 and PEP 526. This is going to take more time, and new collaborators are welcome here: https://github.com/ilevkivskyi/peps/blob/new-pep/pep-0555.txt. (Mark, I really hope you'll accept the invitation to participate. Your experience would be most welcome.) -- --Guido van Rossum (python.org/~guido)

4 3

Make "global after use" a SyntaxError
by Ivan Levkivskyi 07 Sep '16

07 Sep '16

Hi all, The documentation at https://docs.python.org/3/reference/simple_stmts.html says that: "Names listed in a global statement must not be used in the same code block textually preceding that global statement" But then later: "CPython implementation detail: The current implementation does not enforce the two restrictions, but programs should not abuse this freedom, as future implementations may enforce them..." Code like this def f(): x = 1 global x gives SyntaxWarning for several releases, maybe it is time to make it a SyntaxError? (I have opened an issue for this http://bugs.python.org/issue27999 I will submit a patch soon). -- Ivan

3 2

RFC: PEP 509: Add a private version to dict
by Victor Stinner 07 Sep '16

07 Sep '16

Hi, I updated my PEP 509 to make the dictionary version globally unique. With *two* use cases of this PEP (Yury's method call patch and my FAT Python project), I think that the PEP is now ready to be accepted. Globally unique identifier is a requirement for Yury's patch optimizing method calls ( https://bugs.python.org/issue26110 ). It allows to check for free if the dictionary was replaced. I also renamed the ma_version field to ma_version_tag. HTML version: https://www.python.org/dev/peps/pep-0509/ Victor PEP: 509 Title: Add a private version to dict Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Add a new private version to the builtin ``dict`` type, incremented at each dictionary creation and at each dictionary change, to implement fast guards on namespaces. Rationale ========= In Python, the builtin ``dict`` type is used by many instructions. For example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the global namespace, or in the builtins namespace (two dict lookups). Python uses ``dict`` for the builtins namespace, globals namespace, type namespaces, instance namespaces, etc. The local namespace (namespace of a function) is usually optimized to an array, but it can be a dict too. Python is hard to optimize because almost everything is mutable: builtin functions, function code, global variables, local variables, ... can be modified at runtime. Implementing optimizations respecting the Python semantics requires to detect when "something changes": we will call these checks "guards". The speedup of optimizations depends on the speed of guard checks. This PEP proposes to add a version to dictionaries to implement fast guards on namespaces. Dictionary lookups can be skipped if the version does not change which is the common case for most namespaces. Since the version is globally unique, the version is also enough to check if the namespace dictionary was not replaced with a new dictionary. The performance of a guard does not depend on the number of watched dictionary entries, complexity of O(1), if the dictionary version does not change. Example of optimization: copy the value of a global variable to function constants. This optimization requires a guard on the global variable to check if it was modified. If the variable is modified, the variable must be loaded at runtime when the function is called, instead of using the constant. See the `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of guards to specialize functions and for the rationale on Python static optimizers. Guard example ============= Pseudo-code of an fast guard to check if a dictionary entry was modified (created, updated or deleted) using an hypothetical ``dict_get_version(dict)`` function:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.value = dict.get(key, UNSET) self.version = dict_get_version(dict) def check(self): """Return True if the dictionary entry did not changed and the dictionary was not replaced.""" # read the version of the dict structure version = dict_get_version(self.dict) if version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary value = self.dict.get(self.key, UNSET) if value is self.value: # another key was modified: # cache the new dictionary version self.version = version return True # the key was modified return False Usage of the dict version ========================= Speedup method calls 1.2x ------------------------- Yury Selivanov wrote a `patch to optimize method calls <https://bugs.python.org/issue26110>`_. The patch depends on the `implement per-opcode cache in ceval <https://bugs.python.org/issue26219>`_ patch which requires dictionary versions to invalidate the cache if the globals dictionary or the builtins dictionary has been modified. The cache also requires that the dictionary version is globally unique. It is possible to define a function in a namespace and call it in a different namespace: using ``exec()`` with the *globals* parameter for example. In this case, the globals dictionary was changed and the cache must be invalidated. Specialized functions using guards ---------------------------------- The `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support specialized functions with guards. It allows to implement static optimizers for Python without breaking the Python semantics. Example of a static Python optimizer: the `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ of the `FAT Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project implements many optimizations which require guards on namespaces. Examples: * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on ``builtins.__dict__['len']`` and ``globals()['len']`` are required * Loop unrolling: to unroll the loop ``for i in range(...): ...``, guards on ``builtins.__dict__['range']`` and ``globals()['range']`` are required Pyjion ------ According of Brett Cannon, one of the two main developers of Pyjion, Pyjion can also benefit from dictionary version to implement optimizations. Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET Core runtime). Unladen Swallow --------------- Even if dictionary version was not explicitly mentioned, optimizing globals and builtins lookup was part of the Unladen Swallow plan: "Implement one of the several proposed schemes for speeding lookups of globals and builtins." Source: `Unladen Swallow ProjectPlan <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_. Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler implemented with LLVM. The project stopped in 2011: `Unladen Swallow Retrospective <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_. Changes ======= Add a ``ma_version_tag`` field to the ``PyDictObject`` structure with the C type ``PY_INT64_T``, 64-bit unsigned integer. Add also a global dictionary version. Each time a dictionary is created, the global version is incremented and the dictionary version is initialized to the global version. The global version is also incremented and copied to the dictionary version at each dictionary change: * ``clear()`` if the dict was non-empty * ``pop(key)`` if the key exists * ``popitem()`` if the dict is non-empty * ``setdefault(key, value)`` if the `key` does not exist * ``__detitem__(key)`` if the key exists * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value is not ``dict[key]`` * ``update(...)`` if new values are different than existing values: values are compared by identity, not by their content; the version can be incremented multiple times The ``PyDictObject`` structure is not part of the stable ABI. The field is called ``ma_version_tag`` rather than ``ma_version`` to suggest to compare it using ``version_tag == old_version_tag`` rather than ``version <= old_version`` which makes the integer overflow much likely. Example using an hypothetical ``dict_get_version(dict)`` function:: >>> d = {} >>> dict_get_version(d) 100 >>> d['key'] = 'value' >>> dict_get_version(d) 101 >>> d['key'] = 'new value' >>> dict_get_version(d) 102 >>> del d['key'] >>> dict_get_version(d) 103 The version is not incremented if an existing key is set to the same value. For efficiency, values are compared by their identity: ``new_value is old_value``, not by their content: ``new_value == old_value``. Example:: >>> d = {} >>> value = object() >>> d['key'] = value >>> dict_get_version(d) 40 >>> d['key'] = value >>> dict_get_version(d) 40 .. note:: CPython uses some singleton like integers in the range [-5; 257], empty tuple, empty strings, Unicode strings of a single character in the range [U+0000; U+00FF], etc. When a key is set twice to the same singleton, the version is not modified. Implementation and Performance ============================== The `issue #26058: PEP 509: Add ma_version_tag to PyDictObject <https://bugs.python.org/issue26058>`_ contains a patch implementing this PEP. On pybench and timeit microbenchmarks, the patch does not seem to add any overhead on dictionary operations. When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for a dictionary lookup, whereas a guard check only takes 3.8 ns. Moreover, a guard can watch for multiple keys. For example, for an optimization using 10 global variables in a function, 10 dictionary lookups costs 148 ns, whereas the guard still only costs 3.8 ns when the version does not change (39x as fast). The `fat module <http://fatoptimizer.readthedocs.org/en/latest/fat.html>`_ implements such guards: ``fat.GuardDict`` is based on the dictionary version. Integer overflow ================ The implementation uses the C type ``PY_UINT64_T`` to store the version: a 64 bits unsigned integer. The C code uses ``version++``. On integer overflow, the version is wrapped to ``0`` (and then continue to be incremented) according to the C standard. After an integer overflow, a guard can succeed whereas the watched dictionary key was modified. The bug only occurs at a guard check if there are exaclty ``2 ** 64`` dictionary creations or modifications since the previous guard check. If a dictionary is modified every nanosecond, ``2 ** 64`` modifications takes longer than 584 years. Using a 32-bit version, it only takes 4 seconds. That's why a 64-bit unsigned type is also used on 32-bit systems. A dictionary lookup at the C level takes 14.8 ns. A risk of a bug every 584 years is acceptable. Alternatives ============ Expose the version at Python level as a read-only __version__ property ---------------------------------------------------------------------- The first version of the PEP proposed to expose the dictionary version as a read-only ``__version__`` property at Python level, and also to add the property to ``collections.UserDict`` (since this type must mimick the ``dict`` API). There are multiple issues: * To be consistent and avoid bad surprises, the version must be added to all mapping types. Implementing a new mapping type would require extra work for no benefit, since the version is only required on the ``dict`` type in practice. * All Python implementations must implement this new property, it gives more work to other implementations, whereas they may not use the dictionary version at all. * Exposing the dictionary version at Python level can lead the false assumption on performances. Checking ``dict.__version__`` at the Python level is not faster than a dictionary lookup. A dictionary lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5 ns, the difference is only 1.2 ns (3%):: $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33' 10000000 loops, best of 3: 0.0487 usec per loop $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd.__version__ == 100' 10000000 loops, best of 3: 0.0475 usec per loop * The ``__version__`` can be wrapped on integer overflow. It is error prone: using ``dict.__version__ <= guard_version`` is wrong, ``dict.__version__ == guard_version`` must be used instead to reduce the risk of bug on integer overflow (even if the integer overflow is unlikely in practice). Mandatory bikeshedding on the property name: * ``__cache_token__``: name proposed by Nick Coghlan, name coming from `abc.get_cache_token() <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_. * ``__version__`` * ``__timestamp__`` Add a version to each dict entry -------------------------------- A single version per dictionary requires to keep a strong reference to the value which can keep the value alive longer than expected. If we add also a version per dictionary entry, the guard can only store the entry version to avoid the strong reference to the value (only strong references to the dictionary and to the key are needed). Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure, the field has the C type ``PY_INT64_T``. When a key is created or modified, the entry version is set to the dictionary version which is incremented at any change (create, modify, delete). Pseudo-code of an fast guard to check if a dictionary key was modified using hypothetical ``dict_get_version(dict)`` ``dict_get_entry_version(dict)`` functions:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.dict_version = dict_get_version(dict) self.entry_version = dict_get_entry_version(dict, key) def check(self): """Return True if the dictionary entry did not changed and the dictionary was not replaced.""" # read the version of the dict structure dict_version = dict_get_version(self.dict) if dict_version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary entry_version = get_dict_key_version(dict, key) if entry_version == self.entry_version: # another key was modified: # cache the new dictionary version self.dict_version = dict_version return True # the key was modified return False The main drawback of this option is the impact on the memory footprint. It increases the size of each dictionary entry, so the overhead depends on the number of buckets (dictionary entries, used or unused yet). For example, it increases the size of each dictionary entry by 8 bytes on 64-bit system. In Python, the memory footprint matters and the trend is to reduce it. Examples: * `PEP 393 -- Flexible String Representation <https://www.python.org/dev/peps/pep-0393/>`_ * `PEP 412 -- Key-Sharing Dictionary <https://www.python.org/dev/peps/pep-0412/>`_ Add a new dict subtype ---------------------- Add a new ``verdict`` type, subtype of ``dict``. When guards are needed, use the ``verdict`` for namespaces (module namespace, type namespace, instance namespace, etc.) instead of ``dict``. Leave the ``dict`` type unchanged to not add any overhead (memory footprint) when guards are not needed. Technical issue: a lot of C code in the wild, including CPython core, expecting the exact ``dict`` type. Issues: * ``exec()`` requires a ``dict`` for globals and locals. A lot of code use ``globals={}``. It is not possible to cast the ``dict`` to a ``dict`` subtype because the caller expects the ``globals`` parameter to be modified (``dict`` is mutable). * Functions call directly ``PyDict_xxx()`` functions, instead of calling ``PyObject_xxx()`` if the object is a ``dict`` subtype * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some functions require the exact ``dict`` type. * ``Python/ceval.c`` does not completely supports dict subtypes for namespaces The ``exec()`` issue is a blocker issue. Other issues: * The garbage collector has a special code to "untrack" ``dict`` instances. If a ``dict`` subtype is used for namespaces, the garbage collector can be unable to break some reference cycles. * Some functions have a fast-path for ``dict`` which would not be taken for ``dict`` subtypes, and so it would make Python a little bit slower. Prior Art ========= Method cache and type version tag --------------------------------- In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It was merged into Python 2.6. The patch adds a "type attribute cache version tag" (``tp_version_tag``) and a "valid version tag" flag to types (the ``PyTypeObject`` structure). The type version tag is not available at the Python level. The version tag has the C type ``unsigned int``. The cache is a global hash table of 4096 entries, shared by all types. The cache is global to "make it fast, have a deterministic and low memory footprint, and be easy to invalidate". Each cache entry has a version tag. A global version tag is used to create the next version tag, it also has the C type ``unsigned int``. By default, a type has its "valid version tag" flag cleared to indicate that the version tag is invalid. When the first method of the type is cached, the version tag and the "valid version tag" flag are set. When a type is modified, the "valid version tag" flag of the type and its subclasses is cleared. Later, when a cache entry of these types is used, the entry is removed because its version tag is outdated. On integer overflow, the whole cache is cleared and the global version tag is reset to ``0``. See `Method cache (issue #1685986) <https://bugs.python.org/issue1685986>`_ and `Armin's method cache optimization updated for Python 2.6 (issue #1700288) <https://bugs.python.org/issue1700288>`_. Globals / builtins cache ------------------------ In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue #10401) <http://bugs.python.org/issue10401>`_ which adds a private ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``Py_ssize_t``. The patch adds a "global and builtin cache" to functions and frames, and changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the cache. The change on the ``PyDictObject`` structure is very similar to this PEP. Cached globals+builtins lookup ------------------------------ In 2006, Andrea Griffini proposed a patch implementing a `Cached globals+builtins lookup optimization <https://bugs.python.org/issue1616125>`_. The patch adds a private ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. Thread on python-dev: `About dictionary lookup caching <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_. Guard against changing dict during iteration -------------------------------------------- In 2013, Serhiy Storchaka proposed `Guard against changing dict during iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. This field is incremented when the dictionary is modified, and so is very similar to the proposed dictionary version. Sadly, the dictionary version proposed in this PEP doesn't help to detect dictionary mutation. The dictionary version changes when values are replaced, whereas modifying dictionary values while iterating on dictionary keys is legit in Python. PySizer ------- `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary entries. It uses a global process-wide counter for dictionaries, incremented each time that a dictionary is modified. The times are used to decide when child objects first appeared in their parent objects. Discussion ========== Thread on the mailing lists: * python-dev: `PEP 509: Add a private version to dict <https://mail.python.org/pipermail/python-dev/2016-January/142685.html>`_ (january 2016) * python-ideas: `RFC: PEP: Add dict.__version__ <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_ (january 2016) Copyright ========= This document has been placed in the public domain.

15 37

Commits to migrated repos no longer sent to Python-checkins
by Emanuel Barry 07 Sep '16

07 Sep '16

The repos which used to send to Python-checkins no longer do so since their respective migrations (devguide, peps). I don't know who's responsible for that, so I figured I'd post here. -Emanuel

3 4

A Pseudo-Post-Mortem (not dead yet) on my Multi-Core Python Project.
by Eric Snow 07 Sep '16

07 Sep '16

I'm not anticipating much discussion on this, but wanted to present a summary of my notes from the project I proposed last year and have since tabled. http://ericsnowcurrently.blogspot.com/2016/09/solving-mutli-core-python.html -eric

3 2