Mailman 3 April 2016 - Python-Dev

Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
by Ethan Furman April 21, 2016

April 21, 2016

On 04/09/2016 12:48 AM, Nick Coghlan wrote: > Considering the helper function usage, here's some examples in > combination with os.fsencode and os.fsdecode: > > # Status quo for binary/text path conversions > text_path = os.fsdecode(bytes_path) > bytes_path = os.fsencode(text_path) > > # Getting a text path from an arbitrary object > text_path = os.fspath(obj) # This doesn't scream "returns text!" > text_path = os.fspathname(obj) # This does > > # Getting a binary path from an arbitrary object > bytes_path = os.fsencode(os.fspath(obj)) > bytes_path = os.fsencode(os.fspathname(obj)) > > I'm starting to think the semantic nudge from the "name" suffix when > reading the code is worth the extra four characters when writing it > (keeping in mind that the whole point of this exercise is that most > folks *won't* be writing explicit conversions - the stdlib will handle > it on their behalf). > > I also think the more explicit name helps answer some of the type > signature questions that have arisen: > > 1. Does os.fspathname return rich Path objects? No, it returns names > as str objects > 2. Will file descriptors pass through os.fspathname? No, as they're > not names, they're numeric descriptors. > 3. Will bytes-like objects pass through os.fspathname? No, as they're > not names, they're encodings of names This worries me. I know the primary purpose of this change is to enable pathlib and os and the rest of the stdlib to work together, but consider . . . If adding a new attribute/method was as far as we went, new code (stdlib or otherwise) would look like: if isinstance(a_path_thingy, bytes): # because os can accept bytes pass elif isinstance(a_path_thingy, str): # but it's usually text pass elif hasattr(a_path_thingy, '__fspath__'): a_path_thingy = a_path_thingy.__fspath__() else: raise TypeError('not a valid path') # do something with the path If we add os.fspath(), but don't allow bytes to be returned from it, our above example looks more like: if isinstance(a_path_thingy, bytes): # because os can accept bytes pass else: a_path_thingy = os.fspath(a_path_thingy) # do something with the path Yes, it's better -- but it still requires a pre-check before calling os.fspath(). It is my contention that this is better: a_path_thingy = os.fspath(a_path_thingy) This raises two issues: 1) Part of the stdlib is the new scandir module, which can work with, and return, both bytes and text -- if __fspath__ can only hold text, DirEntry will not get the __fspath__ method added, and the pre-check, boiler-plate code will flourish; 2) pathlib.Path accepts bytes -- so what happens when a byte-derived Path is passed to os.fspath()? Is a TypeError raised? Do we guess and auto-convert with fsdecode()? I think the best answer is to - let __fspath__ hold bytes as well as text - let fspath() return bytes as well as text -- ~Ethan~

18 105

When the infamous Bier trunk hits ... where is our Python backup?
by Burkhard Meier April 21, 2016

April 21, 2016

Well, Just a thought.

4 4

PEP 509: Add a private version to dict (version 3)
by Victor Stinner April 20, 2016

April 20, 2016

Hi, Below if the third version of my PEP 509 (dict version). Changes since the version 2: * __setitem__() and update() now always increases the version: remove the micro-optimization on "dict[key] is new_value". Exception: version is not changed with dict.update() is called without argument. * be more explict on version++: explain that the operation must be atomic, and that dict methods are already atomic thanks to the GIL * Usage of the dict version: add Cython * "Guard against changing dict during iteration": don't guess if the new dict version can be used or not. Let's discuss that later. * rephrase/complete some sections * add links to new threads on python-dev I hope that I addressed all Jim's concerns about the version 2. Note: I also updated the implementation. The implementation now contains more tests for identical values and more tests on equal values. HTML version: https://www.python.org/dev/peps/pep-0509/ PEP: 509 Title: Add a private version to dict Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Add a new private version to the builtin ``dict`` type, incremented at each dictionary creation and at each dictionary change, to implement fast guards on namespaces. Rationale ========= In Python, the builtin ``dict`` type is used by many instructions. For example, the ``LOAD_GLOBAL`` instruction looks up a variable in the global namespace, or in the builtins namespace (two dict lookups). Python uses ``dict`` for the builtins namespace, globals namespace, type namespaces, instance namespaces, etc. The local namespace (function namespace) is usually optimized to an array, but it can be a dict too. Python is hard to optimize because almost everything is mutable: builtin functions, function code, global variables, local variables, ... can be modified at runtime. Implementing optimizations respecting the Python semantics requires to detect when "something changes": we will call these checks "guards". The speedup of optimizations depends on the speed of guard checks. This PEP proposes to add a private version to dictionaries to implement fast guards on namespaces. Dictionary lookups can be skipped if the version does not change which is the common case for most namespaces. The version is globally unique, so checking the version is also enough to check if the namespace dictionary was not replaced with a new dictionary. When the dictionary version does not change, the performance of a guard does not depend on the number of watched dictionary entries: the complexity is O(1). Example of optimization: copy the value of a global variable to function constants. This optimization requires a guard on the global variable to check if it was modified. If the global variable is not modified, the function uses the cached copy. If the global variable is modified, the function uses a regular lookup, and maybe also deoptimize the function (to remove the overhead of the guard check for next function calls). See the `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of guards to specialize functions and for a more general rationale on Python static optimizers. Guard example ============= Pseudo-code of an fast guard to check if a dictionary entry was modified (created, updated or deleted) using an hypothetical ``dict_get_version(dict)`` function:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.value = dict.get(key, UNSET) self.version = dict_get_version(dict) def check(self): """Return True if the dictionary entry did not change and the dictionary was not replaced.""" # read the version of the dictionary version = dict_get_version(self.dict) if version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary value = self.dict.get(self.key, UNSET) if value is self.value: # another key was modified: # cache the new dictionary version self.version = version return True # the key was modified return False Usage of the dict version ========================= Speedup method calls -------------------- Yury Selivanov wrote a `patch to optimize method calls <https://bugs.python.org/issue26110>`_. The patch depends on the `"implement per-opcode cache in ceval" <https://bugs.python.org/issue26219>`_ patch which requires dictionary versions to invalidate the cache if the globals dictionary or the builtins dictionary has been modified. The cache also requires that the dictionary version is globally unique. It is possible to define a function in a namespace and call it in a different namespace, using ``exec()`` with the *globals* parameter for example. In this case, the globals dictionary was replaced and the cache must also be invalidated. Specialized functions using guards ---------------------------------- The `PEP 510 -- Specialized functions with guards <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support specialized functions with guards. It allows to implement static optimizers for Python without breaking the Python semantics. The `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ of the `FAT Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project is an example of a static Python optimizer. It implements many optimizations which require guards on namespaces: * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on ``builtins.__dict__['len']`` and ``globals()['len']`` are required * Loop unrolling: to unroll the loop ``for i in range(...): ...``, guards on ``builtins.__dict__['range']`` and ``globals()['range']`` are required * etc. Pyjion ------ According of Brett Cannon, one of the two main developers of Pyjion, Pyjion can benefit from dictionary version to implement optimizations. `Pyjion <https://github.com/Microsoft/Pyjion>`_ is a JIT compiler for Python based upon CoreCLR (Microsoft .NET Core runtime). Cython ------ Cython can benefit from dictionary version to implement optimizations. `Cython <http://cython.org/>`_ is an optimising static compiler for both the Python programming language and the extended Cython programming language. Unladen Swallow --------------- Even if dictionary version was not explicitly mentioned, optimizing globals and builtins lookup was part of the Unladen Swallow plan: "Implement one of the several proposed schemes for speeding lookups of globals and builtins." (source: `Unladen Swallow ProjectPlan <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_). Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler implemented with LLVM. The project stopped in 2011: `Unladen Swallow Retrospective <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_. Changes ======= Add a ``ma_version_tag`` field to the ``PyDictObject`` structure with the C type ``PY_UINT64_T``, 64-bit unsigned integer. Add also a global dictionary version. Each time a dictionary is created, the global version is incremented and the dictionary version is initialized to the global version. The global version is also incremented and copied to the dictionary version at each dictionary change: * ``clear()`` if the dict is non-empty * ``pop(key)`` if the key exists * ``popitem()`` if the dict is non-empty * ``setdefault(key, value)`` if the key does not exist * ``__delitem__(key)`` if the key exists * ``__setitem__(key, value)`` always increases the version * ``update(...)`` if called with arguments The version increase must be atomic. In CPython, the Global Interpreter Lock (GIL) already protects ``dict`` methods to make them atomic. Example using an hypothetical ``dict_get_version(dict)`` function:: >>> d = {} >>> dict_get_version(d) 100 >>> d['key'] = 'value' >>> dict_get_version(d) 101 >>> d['key'] = 'new value' >>> dict_get_version(d) 102 >>> del d['key'] >>> dict_get_version(d) 103 ``dict.__setitem__(key, value)`` and ``dict.update(...)`` always increases the version, even if the new value is identical or is equal to the current value (even if ``(dict[key] is value) or (dict[key] == value)``). The field is called ``ma_version_tag``, rather than ``ma_version``, to suggest to compare it using ``version_tag == old_version_tag``, rather than ``version <= old_version`` which is wrong most of the time after an integer overflow. Backwards Compatibility ======================= Since the ``PyDictObject`` structure is not part of the stable ABI and the new dictionary version not exposed at the Python scope, changes are backward compatible. Implementation and Performance ============================== The `issue #26058: PEP 509: Add ma_version_tag to PyDictObject <https://bugs.python.org/issue26058>`_ contains a patch implementing this PEP. On pybench and timeit microbenchmarks, the patch does not seem to add any overhead on dictionary operations. For example, the following timeit micro-benchmarks takes 318 nanoseconds before and after the change:: python3.6 -m timeit 'd={1: 0}; d[2]=0; d[3]=0; d[4]=0; del d[1]; del d[2]; d.clear()' When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for a dictionary lookup, whereas a guard check only takes 3.8 ns. Moreover, a guard can watch for multiple keys. For example, for an optimization using 10 global variables in a function, 10 dictionary lookups costs 148 ns, whereas the guard still only costs 3.8 ns when the version does not change (39x as fast). The `fat module <http://fatoptimizer.readthedocs.org/en/latest/fat.html>`_ implements such guards: ``fat.GuardDict`` is based on the dictionary version. Integer overflow ================ The implementation uses the C type ``PY_UINT64_T`` to store the version: a 64 bits unsigned integer. The C code uses ``version++``. On integer overflow, the version is wrapped to ``0`` (and then continue to be incremented) according to the C standard. After an integer overflow, a guard can succeed whereas the watched dictionary key was modified. The bug only occurs at a guard check if there are exaclty ``2 ** 64`` dictionary creations or modifications since the previous guard check. If a dictionary is modified every nanosecond, ``2 ** 64`` modifications takes longer than 584 years. Using a 32-bit version, it only takes 4 seconds. That's why a 64-bit unsigned type is also used on 32-bit systems. A dictionary lookup at the C level takes 14.8 ns. A risk of a bug every 584 years is acceptable. Alternatives ============ Expose the version at Python level as a read-only __version__ property ---------------------------------------------------------------------- The first version of the PEP proposed to expose the dictionary version as a read-only ``__version__`` property at Python level, and also to add the property to ``collections.UserDict`` (since this type must mimick the ``dict`` API). There are multiple issues: * To be consistent and avoid bad surprises, the version must be added to all mapping types. Implementing a new mapping type would require extra work for no benefit, since the version is only required on the ``dict`` type in practice. * All Python implementations would have to implement this new property, it gives more work to other implementations, whereas they may not use the dictionary version at all. * Exposing the dictionary version at the Python level can lead the false assumption on performances. Checking ``dict.__version__`` at the Python level is not faster than a dictionary lookup. A dictionary lookup in Python has a cost of 48.7 ns and checking the version has a cost of 47.5 ns, the difference is only 1.2 ns (3%):: $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33' 10000000 loops, best of 3: 0.0487 usec per loop $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd.__version__ == 100' 10000000 loops, best of 3: 0.0475 usec per loop * The ``__version__`` can be wrapped on integer overflow. It is error prone: using ``dict.__version__ <= guard_version`` is wrong, ``dict.__version__ == guard_version`` must be used instead to reduce the risk of bug on integer overflow (even if the integer overflow is unlikely in practice). Mandatory bikeshedding on the property name: * ``__cache_token__``: name proposed by Nick Coghlan, name coming from `abc.get_cache_token() <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_. * ``__version__`` * ``__version_tag__`` * ``__timestamp__`` Add a version to each dict entry -------------------------------- A single version per dictionary requires to keep a strong reference to the value which can keep the value alive longer than expected. If we add also a version per dictionary entry, the guard can only store the entry version (a simple integer) to avoid the strong reference to the value: only strong references to the dictionary and to the key are needed. Changes: add a ``me_version_tag`` field to the ``PyDictKeyEntry`` structure, the field has the C type ``PY_UINT64_T``. When a key is created or modified, the entry version is set to the dictionary version which is incremented at any change (create, modify, delete). Pseudo-code of an fast guard to check if a dictionary key was modified using hypothetical ``dict_get_version(dict)`` and ``dict_get_entry_version(dict)`` functions:: UNSET = object() class GuardDictKey: def __init__(self, dict, key): self.dict = dict self.key = key self.dict_version = dict_get_version(dict) self.entry_version = dict_get_entry_version(dict, key) def check(self): """Return True if the dictionary entry did not change and the dictionary was not replaced.""" # read the version of the dictionary dict_version = dict_get_version(self.dict) if dict_version == self.version: # Fast-path: dictionary lookup avoided return True # lookup in the dictionary to read the entry version entry_version = get_dict_key_version(dict, key) if entry_version == self.entry_version: # another key was modified: # cache the new dictionary version self.dict_version = dict_version self.entry_version = entry_version return True # the key was modified return False The main drawback of this option is the impact on the memory footprint. It increases the size of each dictionary entry, so the overhead depends on the number of buckets (dictionary entries, used or not used). For example, it increases the size of each dictionary entry by 8 bytes on 64-bit system. In Python, the memory footprint matters and the trend is to reduce it. Examples: * `PEP 393 -- Flexible String Representation <https://www.python.org/dev/peps/pep-0393/>`_ * `PEP 412 -- Key-Sharing Dictionary <https://www.python.org/dev/peps/pep-0412/>`_ Add a new dict subtype ---------------------- Add a new ``verdict`` type, subtype of ``dict``. When guards are needed, use the ``verdict`` for namespaces (module namespace, type namespace, instance namespace, etc.) instead of ``dict``. Leave the ``dict`` type unchanged to not add any overhead (CPU, memory footprint) when guards are not used. Technical issue: a lot of C code in the wild, including CPython core, expecting the exact ``dict`` type. Issues: * ``exec()`` requires a ``dict`` for globals and locals. A lot of code use ``globals={}``. It is not possible to cast the ``dict`` to a ``dict`` subtype because the caller expects the ``globals`` parameter to be modified (``dict`` is mutable). * C functions call directly ``PyDict_xxx()`` functions, instead of calling ``PyObject_xxx()`` if the object is a ``dict`` subtype * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some functions require the exact ``dict`` type. * ``Python/ceval.c`` does not completely supports dict subtypes for namespaces The ``exec()`` issue is a blocker issue. Other issues: * The garbage collector has a special code to "untrack" ``dict`` instances. If a ``dict`` subtype is used for namespaces, the garbage collector can be unable to break some reference cycles. * Some functions have a fast-path for ``dict`` which would not be taken for ``dict`` subtypes, and so it would make Python a little bit slower. Prior Art ========= Method cache and type version tag --------------------------------- In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It was merged into Python 2.6. The patch adds a "type attribute cache version tag" (``tp_version_tag``) and a "valid version tag" flag to types (the ``PyTypeObject`` structure). The type version tag is not exposed at the Python level. The version tag has the C type ``unsigned int``. The cache is a global hash table of 4096 entries, shared by all types. The cache is global to "make it fast, have a deterministic and low memory footprint, and be easy to invalidate". Each cache entry has a version tag. A global version tag is used to create the next version tag, it also has the C type ``unsigned int``. By default, a type has its "valid version tag" flag cleared to indicate that the version tag is invalid. When the first method of the type is cached, the version tag and the "valid version tag" flag are set. When a type is modified, the "valid version tag" flag of the type and its subclasses is cleared. Later, when a cache entry of these types is used, the entry is removed because its version tag is outdated. On integer overflow, the whole cache is cleared and the global version tag is reset to ``0``. See `Method cache (issue #1685986) <https://bugs.python.org/issue1685986>`_ and `Armin's method cache optimization updated for Python 2.6 (issue #1700288) <https://bugs.python.org/issue1700288>`_. Globals / builtins cache ------------------------ In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue #10401) <http://bugs.python.org/issue10401>`_ which adds a private ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``Py_ssize_t``. The patch adds a "global and builtin cache" to functions and frames, and changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the cache. The change on the ``PyDictObject`` structure is very similar to this PEP. Cached globals+builtins lookup ------------------------------ In 2006, Andrea Griffini proposed a patch implementing a `Cached globals+builtins lookup optimization <https://bugs.python.org/issue1616125>`_. The patch adds a private ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. Thread on python-dev: `About dictionary lookup caching <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_ (December 2006). Guard against changing dict during iteration -------------------------------------------- In 2013, Serhiy Storchaka proposed `Guard against changing dict during iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict`` type), the field has the C type ``size_t``. This field is incremented when the dictionary is modified. PySizer ------- `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary entries. It uses a global process-wide counter for dictionaries, incremented each time that a dictionary is modified. The times are used to decide when child objects first appeared in their parent objects. Discussion ========== Thread on the mailing lists: * python-dev: `Updated PEP 509 <https://mail.python.org/pipermail/python-dev/2016-April/144250.html>`_ * python-dev: `RFC: PEP 509: Add a private version to dict <https://mail.python.org/pipermail/python-dev/2016-April/144137.html>`_ * python-dev: `PEP 509: Add a private version to dict <https://mail.python.org/pipermail/python-dev/2016-January/142685.html>`_ (january 2016) * python-ideas: `RFC: PEP: Add dict.__version__ <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_ (january 2016) Copyright ========= This document has been placed in the public domain.

1 2

Discussion on fspath: please wait for a PEP
by Victor Stinner April 20, 2016

April 20, 2016

Hi, I'm unable to count the number of threads about the fspath protocol. It's even more difficult to count the total number of emails. IMHO everyone had enough time to give him/her opinion. We even had multiple summaries :-) Can you please wait for a PEP? Brett Canon and Ethan Furman are working on a PEP. So please give them time to write it. The PEP should summarize the discussion and help a lot to make concrete progress on the design (avoid restarting to discuss the same points forever). I don't expect that more emails would add anything at the current state of the discussion. I think that we have enough other topics to discuss in the meanwhile ;-) FYI there is already an article about fspath/pathlib on LWN. Here is a free link until the article is freely accessible: "Python looks at paths" By Jake Edge (April 13, 2016) https://lwn.net/SubscriberLink/683350/4f52334af09653c8/ Victor

5 8

Bytes path
by Serhiy Storchaka April 20, 2016

April 20, 2016

What types should be accepted as bytes path? For now os.path is strict and accepts only bytes and bytes subclasses (even bytearray is not accepted) as bytes path. This is enough for working with low-level Posix paths and supporting backward compatibility. On other hand, most os functions is too permissive since 3.3 and accept any type that supports the buffer protocol as bytes path. Accepted even such meaningless objects as array('h'). Some functions (zipimport.zipimporter() in 3.x, _imp.load_dynamic() in 3.3+, builtin compile() etc in 3.4) accept even arbitrary iterables, e.g. [116, 101, 115, 116] (see http://bugs.python.org/issue26754). I think we should accept only bytes (and subclasses). Even bytearray is less acceptable since it is mutable and can't be used as a key in caches.

3 2

Dependent packages not listed on PyPI
by Andreas Maier April 19, 2016

April 19, 2016

Hi, I have a package "pywbem" which in its setup script specifies a number of dependent packages via "install_requires". I should also say that it extends setuptools/distutils with its own additional keywords, e.g. it adds a "develop_requires", but I believe (hope) that is irrelevant for my problem. In pywbem 0.8.3, the dependencies are: args = { ..., 'install_requires': [ 'six', 'ply', ], ..., } and when running on Python 2.x, an additional one is added, dependent on the OS platform and bit size: if sys.version_info[0] == 2: if platform.system() == 'Windows': if platform.architecture()[0] == '64bit': m2crypto_req = 'M2CryptoWin64>=0.21' else: m2crypto_req = 'M2CryptoWin32>=0.21' else: m2crypto_req = 'M2Crypto>=0.24' args['install_requires'] += [ m2crypto_req, ] The problem is that the pywbem package on PyPI does not show these dependencies: https://pypi.python.org/pypi/pywbem/0.8.3 I wonder whether this is the reason for a particular installation problem we have seen (https://github.com/pywbem/pywbem/issues/113). I do see other projects on PyPI, that show the dependencies they specify in their setup scripts, on their PyPI package page in a "*Requires Distributions*" section: * https://pypi.python.org/pypi/bandit/0.17.3 * https://pypi.python.org/pypi/json-spec/0.9.14 Many others also do not have their dependencies shown, including six, pbr, PyYAML, lxml, to name just a few. So far, I was unable to find out what the presence or absence of that information is related to, in the source of the project. Here are my questions: 1. What causes the "Requires Distributions" section on a PyPI package page to show up there? 2. Is it important to show up there (e.g. for some tools)? Andy

2 1

My first post here ~ do you need more Python core developers on Windows?
by Burkhard Meier April 19, 2016

April 19, 2016

Hi, I just subscribed to the "Python-Dev" mailing list and the 'Welcome" reply asked me to introduce myself. My name is Burkhard Meier and I wrote the "Python GUI Programming Cookbook" published by Packt. It is available on Amazon and PacktPub.com. Maybe I can become more involved in the Python community as a Python developer on Windows . Kind regards, Burkhard Meier

3 3

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
by Wes Turner April 18, 2016

April 18, 2016

On Apr 18, 2016 3:19 PM, "Ethan Furman" <ethan(a)stoneleaf.us> wrote: > > On 04/18/2016 12:54 PM, Wes Turner wrote: > >> Don't we *have* to always support bytes because other programs can >> create filenames containing bytes? > > > Yes, but not every function has to support bytes. Because there's no function overloading in Python, we then must have explicit typing conditionals. I haven't the time to dig through and compare this with the other fine solutions presented; is there a reason that a proxy/facade PrimitiveType wouldn't solve for this? class TextThing: __init__(self, data): self.data = data self.type_ = type(data) __getattr__(self, key): return getattr(self.data, key) > > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

1 0

Re: [Python-Dev] [Python-ideas] pep 7 line break suggestion differs from pep 8
by Chris Angelico April 18, 2016

April 18, 2016

On Tue, Apr 19, 2016 at 5:33 AM, Joseph Jevnik <joejev(a)gmail.com> wrote: > I saw that there was recently a change to pep 8 to suggest adding a line > break before a binary operator. Pep 7 suggests the opposite: > >> When you break a long expression at a binary operator, the operator goes >> at the end of the previous line, e.g.: > >> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && >> type->tp_dictoffset == b_size && >> (size_t)t_size == b_size + sizeof(PyObject *)) >> return 0; /* "Forgive" adding a __dict__ only */ > > I imagine that some of the reasons for making the change in pep 8 for > readability reasons will also > translate to C; maybe pep 7 should also be updated. I would agree with this. Passing it directly to python-dev as that's where the key decision makers are. ChrisA

2 1

[Question][Asyncio] Process + Threads + asyncio... has sense?
by cr0hn April 18, 2016

April 18, 2016

Hi all, It's the first time I write in this list. Sorry if it's not the best place for this question. After I read the Asyncio's documentation, PEPs, Guido/Jesse/David Beazley articles/talks, etc, I developed a PoC library that mixes: Process + Threads + Asyncio Tasks, doing an scheme like this diagram: main -> Process 1 -> Thread 1.1 -> Task 1.1.1 -> Task 1.1.2 -> Task 1.1.3 -> Thread 1.2 -> Task 1.2.1 -> Task 1.2.2 -> Task 1.2.3 Process 2 -> Thread 2.1 -> Task 2.1.1 -> Task 2.1.2 -> Task 2.1.3 -> Thread 2.2 -> Task 2.2.1 -> Task 2.2.2 -> Task 2.2.3 In my local tests, this approach appear to improve (and simplify) the concurrency/parallelism for some tasks but, before release the library at github, I don't know if my aproach is wrong and I would appreciate your opinion. Thank you very much for your time. Regards! -- Daniel García a.k.a. cr0hn - Security researcher and pentester @ggdaniel http://www.cr0hn.com/me/

2 2

Re: [Python-Dev] Updated PEP 509
by Jim J. Jewett April 18, 2016

April 18, 2016

On Sat, Apr 16, 2016 at 5:01 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > * I mentionned that version++ must be atomic, and that in the case of > CPython, it's done by the GIL Better; if those methods *already* hold the GIL, it is worth saying "already", to indicate that the change is not expensive. > * I removed the dict[key]=value; dict[key]=value. It's really a > micro-optimization. I also fear that Raymond will complain because it > adds an if in the hot code of dict, and the dict type is very > important for Python performance. That is an acceptable answer. Though I really do prefer explicitly *refusing to promise* either way when the replacement/replaced objects are ==. dicts (and other collections) already assume sensible ==, even explicitly allowing self-matches of objects that are not equal to themselves. I don't like the idea of making new promises that violate (or rely on violations of) that sensible == assumption. -jJ

2 1

Re: [Python-Dev] Wordcode: new regular bytecode using 16-bit units
by Demur Rumed April 17, 2016

April 17, 2016

The outstanding bug with this patch right now is a regression in line numbers causing the test for http://bugs.python.org/issue9936 to fail. I've tried to debug it without success

2 1

pathlib - current status of discussions
by Ethan Furman April 17, 2016

April 17, 2016

name: ---- We are down to two choices: - __fspath__, or - __fspathname__ The final choice I suspect will be affected by the choice to allow (or not) bytes. method or attribute: ------------------- method built-in: -------- Almost - we'll put it in the os module add to str: ---------- No, not all strings are paths. add to C API: ------------ Yes. Possible names include PyUnicode_FromFSPath and PyObject_Path -- again, the choice of bytes inclusion will affect the final choice of name. add a Path ABC: -------------- undecided Sticking points: --------------- Do we allow bytes to be returned from os.fspath()? If yes, then do we allow bytes from __fspath__()? -- ~Ethan~

19 79

PEP 8 updated on whether to break before or after a binary update
by Guido van Rossum April 16, 2016

April 16, 2016

After a fruitful discussion on python-ideas I've decided that it's fine to break lines *before* a binary operator. It looks better and Knuth recommends it. The head of the python-ideas discussion: https://mail.python.org/pipermail/python-ideas/2016-April/039752.html See also the discussion in the tracker: http://bugs.python.org/issue26763 Here's the diff I applied: https://hg.python.org/peps/rev/3857909d7956 The talk by Brandon Rhodes where Knuth is referenced ([3] below): http://rhodesmill.org/brandon/slides/2012-11-pyconca/#laying-down-the-law The key section in PEP 8 that was updated (apart from fixing up references): Should a line break before or after a binary operator? ------------------------------------------------------ For decades the recommended style has been to break after binary operators. However, recent reseach unearthed recommendations by Donald Knuth to break *before* binary operators, in his writings about typesetting [3]_. Therefore it is permissible to break before or after a binary operator, as long as the convention is consistent locally. For new code Knuth's style is suggested. Some examples of code breaking before binary Boolean operators:: class Rectangle(Blob): def __init__(self, width, height, color='black', emphasis=None, highlight=0): if (width == 0 and height == 0 and color == 'red' and emphasis == 'strong' or highlight > 100): raise ValueError("sorry, you lose") if (width == 0 and height == 0 and (color == 'red' or emphasis is None)): raise ValueError("I don't think so -- values are %s, %s" % (width, height)) Blob.__init__(self, width, height, color, emphasis, highlight) -- --Guido van Rossum (python.org/~guido)

12 13

Summary of Python tracker Issues
by Python tracker April 15, 2016

April 15, 2016

ACTIVITY SUMMARY (2016-04-08 - 2016-04-15) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5489 (+12) closed 33039 (+46) total 38528 (+58) Open issues with patches: 2381 Issues opened (45) ================== #11205: Evaluation order of dictionary display is different from refer http://bugs.python.org/issue11205 reopened by ncoghlan #25609: Add a ContextManager ABC and type http://bugs.python.org/issue25609 reopened by brett.cannon #25731: Assigning and deleting __new__ attr on the class does not allo http://bugs.python.org/issue25731 reopened by barry #26673: Tkinter error when opening IDLE configuration menu http://bugs.python.org/issue26673 reopened by terry.reedy #26716: EINTR handling in fcntl http://bugs.python.org/issue26716 opened by Jack Zhou #26717: wsgiref.simple_server: mojibake with cp1252 bytes in PATH_INFO http://bugs.python.org/issue26717 opened by Anthony Sottile #26720: memoryview from BufferedWriter becomes garbage http://bugs.python.org/issue26720 opened by martin.panter #26721: Avoid socketserver.StreamRequestHandler.wfile doing partial wr http://bugs.python.org/issue26721 opened by martin.panter #26724: Serialize dict with non-string keys to JSON — unexpected res http://bugs.python.org/issue26724 opened by anton-ryzhov #26726: Incomplete Internationalization in Argparse Module http://bugs.python.org/issue26726 opened by IronGrid #26728: make pdb.set_trace() accept debugger commands as arguments and http://bugs.python.org/issue26728 opened by irdb #26729: Incorrect __text_signature__ for sorted http://bugs.python.org/issue26729 opened by eriknw #26730: SpooledTemporaryFile doesn't correctly preserve data for text http://bugs.python.org/issue26730 opened by James Hennessy #26731: subprocess on windows leaks stdout/stderr handle to child proc http://bugs.python.org/issue26731 opened by saifujinaro #26732: multiprocessing sentinel resource leak http://bugs.python.org/issue26732 opened by quick-b #26733: staticmethod and classmethod are ignored when disassemble clas http://bugs.python.org/issue26733 opened by xiang.zhang #26736: Use HTTPS protocol in links http://bugs.python.org/issue26736 opened by serhiy.storchaka #26739: idle: Errno 10035 a non-blocking socket operation could not be http://bugs.python.org/issue26739 opened by MICHAEL JACOBSON #26740: tarfile: accessing (listing and extracting) tarball fails with http://bugs.python.org/issue26740 opened by Tomas Tomecek #26741: subprocess.Popen should emit a ResourceWarning in destructor i http://bugs.python.org/issue26741 opened by haypo #26742: imports in test_warnings changes warnings.filters http://bugs.python.org/issue26742 opened by haypo #26743: Unable to import random with python2.7 on power pc based machi http://bugs.python.org/issue26743 opened by ragreddy #26744: print() function hangs on MS-Windows 10 http://bugs.python.org/issue26744 opened by Ma Lin #26745: Redundant code in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue26745 opened by xiang.zhang #26746: struct.pack(): trailing padding bytes on x64 http://bugs.python.org/issue26746 opened by skrah #26750: Mock autospec does not work with subclasses of property() http://bugs.python.org/issue26750 opened by amaury.forgeotdarc #26751: Possible bug in sorting algorithm http://bugs.python.org/issue26751 opened by David.Manowitz #26752: Mock(2.0.0).assert_has_calls() raise AssertionError in two sam http://bugs.python.org/issue26752 opened by jekin000 #26753: Obmalloc lock LOCK_INIT and LOCK_FINI are never used http://bugs.python.org/issue26753 opened by larry #26754: PyUnicode_FSDecoder() accepts arbitrary iterable http://bugs.python.org/issue26754 opened by serhiy.storchaka #26755: Update version{added,changed} docs in devguide http://bugs.python.org/issue26755 opened by berker.peksag #26756: fileinput handling of unicode errors from standard input http://bugs.python.org/issue26756 opened by jmb236 #26757: test_urllib2net.test_http_basic() timeout after 15 min on http://bugs.python.org/issue26757 opened by haypo #26758: Unnecessary format string handling for no argument slot wrappe http://bugs.python.org/issue26758 opened by josh.r #26759: PyBytes_FromObject accepts arbitrary iterable http://bugs.python.org/issue26759 opened by serhiy.storchaka #26760: Document PyFrameObject http://bugs.python.org/issue26760 opened by brett.cannon #26762: test_multiprocessing_spawn leaves processes running in backgro http://bugs.python.org/issue26762 opened by martin.panter #26763: Update PEP-8 regarding binary operators http://bugs.python.org/issue26763 opened by IanLee1521 #26764: SystemError in bytes.__rmod__ http://bugs.python.org/issue26764 opened by serhiy.storchaka #26765: Factor out common bytes and bytearray implementation http://bugs.python.org/issue26765 opened by serhiy.storchaka #26766: The result type of bytearray formatting is not stable http://bugs.python.org/issue26766 opened by berker.peksag #26767: Inconsistant error messages for failed attribute modification http://bugs.python.org/issue26767 opened by serhiy.storchaka #26769: Python 2.7: make private file descriptors non inheritable http://bugs.python.org/issue26769 opened by haypo #26770: _Py_set_inheritable(): do nothing if the FD_CLOEXEC close is a http://bugs.python.org/issue26770 opened by haypo #26771: python-config.sh.in INCDIR does not match python version if ex http://bugs.python.org/issue26771 opened by benzea Most recent 15 issues with no replies (15) ========================================== #26771: python-config.sh.in INCDIR does not match python version if ex http://bugs.python.org/issue26771 #26769: Python 2.7: make private file descriptors non inheritable http://bugs.python.org/issue26769 #26767: Inconsistant error messages for failed attribute modification http://bugs.python.org/issue26767 #26765: Factor out common bytes and bytearray implementation http://bugs.python.org/issue26765 #26760: Document PyFrameObject http://bugs.python.org/issue26760 #26758: Unnecessary format string handling for no argument slot wrappe http://bugs.python.org/issue26758 #26752: Mock(2.0.0).assert_has_calls() raise AssertionError in two sam http://bugs.python.org/issue26752 #26750: Mock autospec does not work with subclasses of property() http://bugs.python.org/issue26750 #26739: idle: Errno 10035 a non-blocking socket operation could not be http://bugs.python.org/issue26739 #26728: make pdb.set_trace() accept debugger commands as arguments and http://bugs.python.org/issue26728 #26726: Incomplete Internationalization in Argparse Module http://bugs.python.org/issue26726 #26700: Make digest_size a class variable http://bugs.python.org/issue26700 #26697: tkFileDialog crash on askopenfilename Python 2.7 64-bit Win7 http://bugs.python.org/issue26697 #26696: Document collections.abc.ByteString http://bugs.python.org/issue26696 #26695: pickle and _pickle accelerator have different behavior when un http://bugs.python.org/issue26695 Most recent 15 issues waiting for review (15) ============================================= #26770: _Py_set_inheritable(): do nothing if the FD_CLOEXEC close is a http://bugs.python.org/issue26770 #26769: Python 2.7: make private file descriptors non inheritable http://bugs.python.org/issue26769 #26766: The result type of bytearray formatting is not stable http://bugs.python.org/issue26766 #26765: Factor out common bytes and bytearray implementation http://bugs.python.org/issue26765 #26764: SystemError in bytes.__rmod__ http://bugs.python.org/issue26764 #26763: Update PEP-8 regarding binary operators http://bugs.python.org/issue26763 #26755: Update version{added,changed} docs in devguide http://bugs.python.org/issue26755 #26750: Mock autospec does not work with subclasses of property() http://bugs.python.org/issue26750 #26745: Redundant code in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue26745 #26742: imports in test_warnings changes warnings.filters http://bugs.python.org/issue26742 #26741: subprocess.Popen should emit a ResourceWarning in destructor i http://bugs.python.org/issue26741 #26736: Use HTTPS protocol in links http://bugs.python.org/issue26736 #26733: staticmethod and classmethod are ignored when disassemble clas http://bugs.python.org/issue26733 #26730: SpooledTemporaryFile doesn't correctly preserve data for text http://bugs.python.org/issue26730 #26729: Incorrect __text_signature__ for sorted http://bugs.python.org/issue26729 Top 10 most discussed issues (10) ================================= #26743: Unable to import random with python2.7 on power pc based machi http://bugs.python.org/issue26743 20 msgs #26766: The result type of bytearray formatting is not stable http://bugs.python.org/issue26766 12 msgs #25702: Link Time Optimizations support for GCC and CLANG http://bugs.python.org/issue25702 10 msgs #25910: Fixing links in documentation http://bugs.python.org/issue25910 10 msgs #26647: ceval: use Wordcode, 16-bit bytecode http://bugs.python.org/issue26647 10 msgs #26716: EINTR handling in fcntl http://bugs.python.org/issue26716 9 msgs #26729: Incorrect __text_signature__ for sorted http://bugs.python.org/issue26729 9 msgs #26763: Update PEP-8 regarding binary operators http://bugs.python.org/issue26763 9 msgs #26359: CPython build options for out-of-the box performance http://bugs.python.org/issue26359 8 msgs #26601: Use new madvise()'s MADV_FREE on the private heap http://bugs.python.org/issue26601 8 msgs Issues closed (48) ================== #13410: String formatting bug in interactive mode http://bugs.python.org/issue13410 closed by serhiy.storchaka #13952: mimetypes doesn't recognize .csv http://bugs.python.org/issue13952 closed by berker.peksag #14784: Re-importing _warnings changes warnings.filters http://bugs.python.org/issue14784 closed by martin.panter #15984: Wrong documentation for PyUnicode_FromObject() and PyUnicode_F http://bugs.python.org/issue15984 closed by martin.panter #16329: mimetypes does not support webm type http://bugs.python.org/issue16329 closed by berker.peksag #17264: Update Building C and C++ Extensions with distutils documentat http://bugs.python.org/issue17264 closed by berker.peksag #17339: bytes() TypeError message is misleadingly narrow http://bugs.python.org/issue17339 closed by serhiy.storchaka #18461: X Error in tkinter http://bugs.python.org/issue18461 closed by serhiy.storchaka #21069: test_fileno of test_urllibnet intermittently fails http://bugs.python.org/issue21069 closed by martin.panter #22659: SyntaxError in the configure_ctypes http://bugs.python.org/issue22659 closed by berker.peksag #23397: PEP 431 implementation http://bugs.python.org/issue23397 closed by berker.peksag #24951: Idle test_configdialog fails on Fedora 23, 3.6 http://bugs.python.org/issue24951 closed by terry.reedy #25339: sys.stdout.errors is set to "surrogateescape" http://bugs.python.org/issue25339 closed by serhiy.storchaka #25496: tarfile: Default value for compresslevel is not documented http://bugs.python.org/issue25496 closed by martin.panter #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 closed by martin.panter #26057: Avoid nonneeded use of PyUnicode_FromObject() http://bugs.python.org/issue26057 closed by serhiy.storchaka #26257: Eliminate buffer_tests.py http://bugs.python.org/issue26257 closed by martin.panter #26404: socketserver context manager http://bugs.python.org/issue26404 closed by martin.panter #26585: Use html.escape to replace _quote_html in http.server http://bugs.python.org/issue26585 closed by martin.panter #26587: Possible duplicate entries in sys.path if .pth files are used http://bugs.python.org/issue26587 closed by brett.cannon #26609: Wrong request target in test_httpservers.py http://bugs.python.org/issue26609 closed by martin.panter #26610: test_venv.test_with_pip() fails when ctypes is missing http://bugs.python.org/issue26610 closed by haypo #26623: JSON encode: more informative error http://bugs.python.org/issue26623 closed by serhiy.storchaka #26624: Windows hangs in call to CRT setlocale() http://bugs.python.org/issue26624 closed by python-dev #26639: Tools/i18n/pygettext.py: replace deprecated imp module with im http://bugs.python.org/issue26639 closed by haypo #26668: Remove Lib/test/test_importlib/regrtest.py? http://bugs.python.org/issue26668 closed by brett.cannon #26685: Raise errors from socket.close() http://bugs.python.org/issue26685 closed by martin.panter #26687: Use Py_RETURN_NONE in sqlite3 module http://bugs.python.org/issue26687 closed by berker.peksag #26699: locale.str docstring is incorrect: "Convert float to integer" http://bugs.python.org/issue26699 closed by orsenthil #26706: Update OpenSSL version in readme http://bugs.python.org/issue26706 closed by python-dev #26712: Unify (r)split(), (l/r)strip() method tests http://bugs.python.org/issue26712 closed by martin.panter #26714: telnetlib.Telnet should act as a context manager http://bugs.python.org/issue26714 closed by SilentGhost #26715: can not deactivate venv (deactivate.bat) if the venv was activ http://bugs.python.org/issue26715 closed by zach.ware #26718: super.__init__ leaks memory if called multiple times http://bugs.python.org/issue26718 closed by brett.cannon #26719: More efficient formatting of ints and floats in json http://bugs.python.org/issue26719 closed by serhiy.storchaka #26722: Fold compare operators on constants (peephole) http://bugs.python.org/issue26722 closed by ncoghlan #26723: Add an option to skip _decimal module http://bugs.python.org/issue26723 closed by skrah #26725: list() destroys map object data http://bugs.python.org/issue26725 closed by ned.deily #26727: ctypes.util.find_msvcrt() does not work in python 3.5.1 http://bugs.python.org/issue26727 closed by steve.dower #26734: Repeated mmap\munmap calls during temporary allocation http://bugs.python.org/issue26734 closed by pitrou #26735: os.urandom(2500) fails on Solaris 11.3 http://bugs.python.org/issue26735 closed by haypo #26737: csv.DictReader throws generic error when fieldnames is accesse http://bugs.python.org/issue26737 closed by serhiy.storchaka #26738: listname.strip() does not work right if the name ends with an http://bugs.python.org/issue26738 closed by SilentGhost #26747: types.InstanceType only for old style class only in 2.7 http://bugs.python.org/issue26747 closed by berker.peksag #26748: enum.Enum is False-y http://bugs.python.org/issue26748 closed by ethan.furman #26749: Update devguide to include Fedora's DNF http://bugs.python.org/issue26749 closed by berker.peksag #26761: winsound module very unstable in Windows 10 http://bugs.python.org/issue26761 closed by zach.ware #26768: Fix instructions at WindowsCompilers for MSVC/SDKs http://bugs.python.org/issue26768 closed by berker.peksag

1 0

Should secrets include a fallback for hmac.compare_digest?
by Steven D'Aprano April 15, 2016

April 15, 2016

Now that PEP 506 has been approved, I've checked in the secrets module, but an implementation question has come up regarding compare_digest. Currently, the module tries to import hmac.compare_digest, and if that fails, then it falls back to a Python version. But since compare_digest has been available since 3.3, I'm now questioning whether the fallback is useful at all. Perhaps for alternate Python implementations? So, two questions: - should secrets include a fallback? - if so, what is the preferred way of doing this? # option 1: fallback if compare_digest is missing try: from hmac import compare_digest except ImportError: def compare_digest(a, b): ... # option 2: "C accelerator idiom" def compare_digest(a, b): ... try: from hmac import compare_digest except ImportError: pass Option 1 is closer to how I would write hybrid 2/3 code, but option 2 is how PEP 399 suggests it should be written. https://www.python.org/dev/peps/pep-0399/ Currently, hmac imports compare_digest from _operator. There's no Python version in operator either. Should there be? -- Steve

4 5

PEP 506 secrets module
by Steven D'Aprano April 15, 2016

April 15, 2016

Hi, As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement. https://www.python.org/dev/peps/pep-0506/ There is code and tests here: https://bitbucket.org/sdaprano/secrets or you can run hg clone https://sdaprano@bitbucket.org/sdaprano/secrets The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4. -- Steve

13 28

Most 3.x buildbots are green again, please don't break them and watch them!
by Victor Stinner April 15, 2016

April 15, 2016

Hi, Last months, most 3.x buildbots failed randomly. Some of them were always failing. I spent some time to fix almost all Windows and Linux buildbots. There were a lot of different issues. So please try to not break buildbots again and remind to watch them sometimes: http://buildbot.python.org/all/waterfall?category=3.x.stable&category=3.x.u… Next weeks, I will try to backport some fixes to Python 3.5 (if needed) to make these buildbots more stable too. Python 2.7 buildbots are also in a sad state (ex: test_marshal segfaults on Windows, see issue #25264). But it's not easy to get a Windows with the right compiler to develop on Python 2.7 on Windows. -- Maybe it's time to move more 3.x buildbots to the "stable" category? http://buildbot.python.org/all/waterfall?category=3.x.stable By the way, I don't understand why "AMD64 OpenIndiana 3.x" is considered as stable since it's failing with multiple issues since many months and nobody is working on these failures. I suggest to move this buildbot back to the unstable category. -- We have many offline buildbots. What's the status of these buildbots? Should we expect that they come back soon? Or would it be possible to hide them? It would help to check the status of all buildbots. -- Failing buildbots: - AMD64 FreeBSD CURRENT 3.x: http://bugs.python.org/issue26566 -- I installed a fresh FreeBSD CURRENT in a VM and I'm unable to reproduce failures. Maybe the buildbot slave is oudated and FreeBSD must be upgraded? - AMD64 OpenIndiana 3.x, x86 OpenIndiana 3.x: test_socket failures on sendfile. Sorry but I'm not really interested by this OS. - PPC64 AIX 3.x: failing tests: test_httplib, test_httpservers, test_socket, test_distutils, test_asyncio, (...); random timeout failure in test_eintr, etc. I don't have access to AIX and I'm not interested to acquire an AIX license, nor to install it. I'm not sure that it's useful to have an AIX buildbot and no core developer have access to AIX, and nobody is working on AIX failures. Maybe HP wants to help us to support AIX? (Provide manpower, access to AIX servers, or something like that.) - x86 OpenBSD 3.x: 5 tests failed, test_crypt test_socket test_ssl test_strptime test_time. This OS needs some love ;-) - the 4 ICC buildbots are failing with stack overflow, segfault, etc. Again, I'm not sure that these buildbots are useful since it looks like we don't support this compiler yet. Or does it help to work on supporting this compiler? Who is working on ICC support? -- FYI I also made some enhancements on regrtest (our test runner for the test suite), mostly to debug failures: - display the duration of tests taking longer than 30 seconds - new timestamp prefix, used to debug buildbot hangs - when parallel tests are interrupted, display progress on waiting for completion - add timeout to main process when using -jN: it should help to debug buildbot hang - "Run tests in parallel using 3 child processes" or "Run tests sequentially" message which helps to understand how tests are running. There is the -j1 trap which has no effect: tests are still run sequentially. By the way, I proposed to really use subprocesses when -j1 is used: http://bugs.python.org/issue25285 The default timeout changed from 1 hour to 15 min, it's the maximum duration to run a single test file (ex: test_os.py). On my Linux box, running the whole test suite in parallel (10 child processes for my 4 CPU cores with hyperthreading) with Python compiled in debug mode (slow) takes 4 min 37 sec. Tell me if the default timeout is too low. It can be configured per buildbot if needed (TESTTIMEOUT env var). -- By the way, I'm always surprised by the huge difference of time needed to run a build on the different slaves: from a few minutes to more than 3 hours. The fatest Windows slave takes 28 minutes (run tests in parallel using 4 child processes), whereas the 3 others (run tests sequentially and) take between 2 hours and more than 3 hours! Why running tests on Windows takes so long? Maybe we should make sure that no buildbot run tests sequentially, because it creates a lot of annoying side effects (even if sometimes it helps to find tricky bugs, sometimes bugs restricted to the tests themself) and because a lot of time simply wait a few seconds. So running mutliple tests in parallel don't burn your CPU, it's just faster. IMHO the risk of random timeout failures is low compared to the speedup. -- The most interesting bug was a deadlock in locale.setlocale() on Windows 7: the bug made the buildbot to hang "sometimes" (randomly). Jeremy Kloth identified the bug, but Steve Dower noticed us that it's already fixed in Visual Studio 2015 Update 1: so please update VS if it's not the case yet. Steve added a post-build test to check if the ucrtbase/ucrtbased DLL has the known bug. => http://bugs.python.org/issue26624 Victor

10 11

Re: [Python-Dev] MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

On Thu, Apr 14, 2016 at 8:27 PM, Guido van Rossum <guido(a)python.org> wrote: > Great analysis! What might stand in the way of adoption is concern for > bytecode manipulation libraries that would have to be changed. What > might encourage adoption would be a benchmark showing this saves a lot > of time. > > Personally I'm expecting it won't make much of a difference for real > programs since almost always the cost of creating the function is > dwarfed by the (total) cost of running it. But Python does create a > lot of functions, and there's also lambdas. This change alone is very unlikely to have a measurable performance impact. The intention is to clean up ceval.c/compile.c a bit, nothing more. If many other opcodes were somehow slimmed down in the similar fashion, then we might (or might not) see perf gains. For example, most slot dispatch opcodes can be compressed into a single opcode+slot index with inlined dispatch logic, instead of each one individually calling C API functions... > There's also talk of switching to wordcode, in a different thread. > Maybe the idea would be easier to introduce there? (Bytecode libraries > would have to change anyways, so the additional concern for this > change would be minimal.) Wordcode can benefit from this change, because it guarantees single-byte MAKE_FUNCTION oparg. I think that Python should make bytecode explicitly unstable and subject to change with any major release. The potential for a faster Python interpreter (or simple JIT) is huge; requiring bytecode compatibility will slow down any progress in this area.

4 3

Re: [Python-Dev] MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

On Thu, Apr 14, 2016 at 8:32 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > > Would you like to work on a patch to implement that change? I'll work on a patch. Should I post it to bugs.python.org? > Since Python 3.6 may get a new bytecode format (wordcode, see the > other thread on this mlailing list), I think that it's ok to change > MAKE_FUNCTION in the same release. Wordcode looks like pure win from (projected) 25% bytecode size reduction alone.

2 1

Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!
by Victor Stinner April 14, 2016

April 14, 2016

Le 14 avr. 2016 11:16 AM, "Serhiy Storchaka" <storchaka(a)gmail.com> a écrit : > A desirable but nonexistent feature is to write emails to authors of commits that broke buildbots. How hard to implement this? Yeah I also had this idea since many years but buildbots were quite unstable. Maybe we should be more strict to consider a buildbot as stable? I propose to experiment sending notifications of failure to the authors of changes *and* to a new mailing list. I would subscribe to such list. An even safer starting point would be to only start with the mailing list. FYI I'm connected to the #python-dev IRC channel which already contain these notifications. But I agree that mails are better. > What are you think about backporting recent regrtest to 2.7? Most needed features to me are the -m and -G options. Regrtest changed a lot in python 3.6 (new test.libregrtest library). I suggest to start from python 3.5. For -m: if it doesn't need to modify the unittest module, I agree. I don't know -G option. > Would be nice to add a feature for running every test in separate subprocess. This will isolate the effect of failed tests. See my email :-) I proposed to modify -j1 to run tests in subrpocesses. I even mentionned my issue. I suggest to use -jN on all buildbot, at least -j1. Maybe -j2 is even better since many tests are waiting on IO or simple sleep. Victor

2 1

MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

MAKE_FUNCTION opcode is complex due to the way it receives input arguments: 1) default args, individually; 2) default kwonly args, individual name-value pairs; 3) a tuple of parameter names (single constant); 4) annotation values, individually; 5) code object; 6) qualname. The counts for 1,2,4 are packed into oparg bitfields, making oparg large. My suggestion is to pre-package 1-4 before calling MAKE_FUNCTION, i.e. explicitly emit BUILD_TUPLE for defaults args and BUILD_MAPs for keyword defaults and annotations. Then, MAKE_FUNCTION will become a dramatically simpler 5 argument opcode, taking 1) default args tuple (optional); 2) default keyword only args dict (optional); 3) annotations dict (optional); 4) code object; 5) qualname. These arguments correspond exactly to __annotations__, __kwdefaults__, __defaults__, __code__ and __qualname__ attributes. For optional args, oparg bits should indicate individual arg presence. (This also saves None checks in opcode implementation.) If we add another optional argument (and oparg bit) for __closure__ attribute, then separate MAKE_CLOSURE opcode becomes unnecessary. Default args tuple is likely to be a constant and can be packaged whole, compensating for the extra size of explicit BUILD_* instructions. Compare the current implementation: https://github.com/python/cpython/blob/master/Python/ceval.c#L3262 with this provisional implementation (untested): TARGET(MAKE_FUNCTION) { PyObject *qualname = POP(); PyObject *codeobj = POP(); PyFunctionObject *func; func = (PyFunctionObject *)PyFunction_NewWithQualName( codeobj, f->f_globals, qualname); Py_DECREF(codeobj); Py_DECREF(qualname); if (func == NULL) goto error; /* NB: Py_None is not an acceptable value for these. */ if (oparg & 0x08) func->func_closure = POP(); if (oparg & 0x04) func->func_annotations = POP(); if (oparg & 0x02) func->func_kwdefaults = POP(); if (oparg & 0x01) func->func_defaults = POP(); PUSH((PyObject *)func); DISPATCH(); } compile.c also gets a bit simpler, but not much. What do you think?

3 3

Not receiving bug tracker emails
by Martin Panter April 14, 2016

April 14, 2016

For the last ~36 hours I have stopped receiving emails for messages posted in the bug tracker. Is anyone else having this problem? Has anything changed recently? I have had it set to send to my gmail.com address since the beginning. At the moment the last bug message email is <https://bugs.python.org/issue19959#msg262569> with “Date: Mon, 28 Mar 2016 12:19:49 +0000”. I have checked spam and they are not going there. Earlier this year I had to set up a rule to avoid lots of tracker emails suddenly going to spam. I suspect there was something about the emails that Google doesn’t like (though I don’t understand the technical details). Maybe this has recently gotten worse at the Google end?

6 16

Re: [Python-Dev] pathlib - current status of discussions
by Random832 April 14, 2016

April 14, 2016

1 0

Re: [Python-Dev] pathlib - current status of discussions
by Random832 April 13, 2016

April 13, 2016

On Apr 13, 2016 19:06, Brett Cannon <brett(a)python.org> wrote: > On Wed, 13 Apr 2016 at 15:46 Nikolaus Rath <Nikolaus(a)rath.org> wrote: >> When passing an object that is of type str and has a __fspath__ >> attribute, all approaches return the value of __fspath__(). >> >> However, when passing something of type bytes, the second approach >> returns the object, while the third returns the value of __fspath__(). >> >> Is this intentional? I think a __fspath__ attribute should always be >> preferred. > > > It's very much intentional. If we define __fspath__() to only return strings but still want to minimize boilerplate of allowing bytes to simply pass through without checking a path argument to see if it is bytes then approach #2 is warranted. But if __fspath__() can return bytes then approach #3 allows for it. Er, the difference comes in when the object passed to os.fspath is a subclass of bytes that, itself, has a __fspath__ method (which may return a str). It's unlikely to occur in the wild, but is a semantic difference between this case and all other objects with __fspath__ methods.

1 0