Mailman 3 January 2018 - Python-Dev

Unique loader per module
by Barry Warsaw Jan. 23, 2018

Jan. 23, 2018

We have what I think is one last design question for importlib.resources. https://gitlab.com/python-devs/importlib_resources/issues/49 The problem is that the ResourceReader ABC omits the package from the function signatures, so that on a compatible loader, you only need to specify the resource you are interested in. This is fine for file loaders because every package will have a unique loader instance associated with it, so it will know which package the requested resource is homed to. But AFAICT, there is no specification or requirement in Python that every module/package have a unique loader instance. In fact, it’s not unreasonable given some of the text in PEP 302 to think that loader instances can be shared. The PEP says "In many cases the finder and loader can be one and the same object: finder.find_module() would just return self” and you aren’t going to typically have a unique finder per module, so that would implied a shared loader per finder. We even have an existence proof in the zip importer: >>> import test.test_importlib.zipdata02 >>> import sys, os >>> sys.path.insert(0, os.path.join(os.path.dirname(test.test_importlib.zipdata02.__file__), 'ziptestdata.zip')) >>> import ziptestdata.two >>> import ziptestdata.one >>> ziptestdata.one.__spec__.loader == ziptestdata.two.__spec__.loader True The issue above proposes two solutions. The first is to change the ABC so that it includes the package argument in the ABC method signatures. That way, a shared loader will know which package the requested resource is relative to. Brett doesn’t like this, for several reasons (quoting): 1. redundant API in all cases where the loader is unique to the module 2. the memory savings of sharing a loader is small 3. it's implementation complexity/overhead for an optimization case. The second solution, and the one Brett prefers, is to reimplement zip importer to not use a shared loader. This may not be that difficult, if for example we were to use a delegate loader wrapping a shared loader. The bigger problem IMHO is two-fold: 1. It would be backward incompatible. If there’s any code out there expecting a shared loader in zipimport, it would break 2. More problematic is that we’d have to impose an additional requirement on loaders - that they always be unique per module, contradicting the advice in PEP 302 The reason for this is third party finder/loaders. Sure, we can fix zipimport but any third party finder/loaders could have the same problem, and they’d be within their rights given the current specification. We’d have to prohibit that, or at least say that any third party finder/loaders that shared their loader can’t implement ResourceReader (which would be the practical effect anyway). I think that would be a shame. So while I agree with Brett that it’s uglier, and that once we decide we’re essentially locked into the API, I don’t see a whole lot of options. Thoughts, feedback, suggestions are welcome. -Barry

3 4

Slipping Python 3.5.5rc1 and 3.4.8rc1 because of a Travis CI issue--can someone make Travis CI happy?
by Larry Hastings Jan. 23, 2018

Jan. 23, 2018

I have three PRs for Python 3.5.5rc1: https://github.com/python/cpython/pull/4656 https://github.com/python/cpython/pull/5197 https://github.com/python/cpython/pull/5201 I can't merge them because Travis CI is unhappy. All three CI tests fail in the same way, reporting this error: The command "pyenv global system 3.5" failed and exited with 1 during . Since Travis CI is a "required check", Github won't let me merge the PR. Yes I could manually merge the patches by hand but I'm hoping it doesn't come to that. I'm slipping 3.4.8rc1 because I prefer to do both releases at once. I'm hoping this problem will be resolved quickly; if we only slip the RCs by a day or two I won't slip the final releases (in about two weeks). PLS SND HALP, //arry/

7 7

Unexpected bytecode difference
by Alexander Belopolsky Jan. 22, 2018

Jan. 22, 2018

I have encountered the following difference between Python 3 and 2: (py3) >>> compile('xxx', '<>', 'eval').co_code b'e\x00S\x00' (py2) >>> compile('xxx', '<>', 'eval').co_code 'e\x00\x00S' Note that 'S' (the code for RETURN_VALUE) and a zero byte are swapped in Python 2 compared to Python 3. Is this change documented somewhere?

4 6

Drop support for old unsupported FreeBSD and Linux kernels?
by Victor Stinner Jan. 22, 2018

Jan. 22, 2018

Hi, I'm working on a exhaustive list of platforms supported by Python: http://vstinner.readthedocs.io/cpython.html#supported-platforms I noticed that the extended support phase of Windows Vista is expired, so I proposed to drop Vista support: "Drop support of Windows Vista in Python 3.7" https://bugs.python.org/issue32592 https://github.com/python/cpython/pull/5231 Python has an explicit policy for Windows support, extract of the PEP 11: "CPython’s Windows support now follows [Microsoft product support lifecycle]. A new feature release X.Y.0 will support all Windows releases whose extended support phase is not yet expired. Subsequent bug fix releases will support the same Windows releases as the original feature release (even if the extended support phase has ended)." For Linux and FreeBSD, we have no explicit rule. CPython code base still contains code for FreeBSD 4... but FreeBSD 4 support ended longer than 10 years ago (January 31, 2007). Maybe it's time to drop support of these old platforms to cleanup the CPython code base to ease its maintainance. I proposed: "Drop FreeBSD 9 and older support:" https://bugs.python.org/issue32593 https://github.com/python/cpython/pull/5232 FreeBSD 9 supported ended 1 year ago (December 2016). FreeBSD support: https://www.freebsd.org/security/ https://www.freebsd.org/security/unsupported.html CPython still has compatibility code for Linux 2.6, whereas the support of Linux 2.6.x ended in August 2011, longer than 6 years ago. Should we also drop support for old Linux kernels? If yes, which ones? The Linux kernel has LTS version, the oldest is Linux 3.2 (support will end in May, 2018). Linux kernel support: https://www.kernel.org/category/releases.html Note: I'm only talking about changing the future Python 3.7. We should have the same support policy than for Windows. If Python 3.x.0 supports a platform, this support should be kept in the whole lifetime of the 3.x cycle (until it's end-of-line). Victor

9 11

Concerns about method overriding and subclassing with dataclasses
by Ethan Smith Jan. 22, 2018

Jan. 22, 2018

Hello all, I've recently been experimenting with dataclasses. They totally rock! A lot of the boilerplate for the AST I've designed in Python is automatically taken care of, it's really great! However, I have a few concerns about the implementation. In a few cases I want to override the repr of the AST nodes. I wrote a __repr__ and ran the code but lo and behold I got a type error. I couldn't override it. I quickly learned that one needs to pass a keyword to the dataclass decorator to tell it *not* to auto generate methods you override. I have two usability concerns with the current implementation. I emailed Eric about the first, and he said I should ask for thoughts here. The second I found after a couple of days sitting on this message. The first is that needing both a keyword and method is duplicative and unnecessary. Eric agreed it was a hassle, but felt it was justified considering someone may accidentally override a dataclass method. I disagree with this point of view as dataclasses are billed as providing automatic methods. Overriding via method definition is very natural and idiomatic. I don't really see how someone could accidentally override a dataclass method if methods were not generated by the dataclass decorator that are already defined in the class at definition time. The second concern, which I came across more recently, is if I have a base class, and dataclasses inherit from this base class, inherited __repr__ & co are silently overridden by dataclass. This is both unexpected, and also means I need to pass a repr=False to each subclass' decorator to get correct behavior, which somewhat defeats the utility of subclassing. Im not as sure a whole lot can be done about this though. I appreciate any thoughts folks have related to this. Cheers, ~>Ethan Smith

11 35

PEP 567 v3
by Yury Selivanov Jan. 22, 2018

Jan. 22, 2018

Hi, This is a third version of PEP 567. Changes from v2: 1. PyThreadState now references Context objects directly (instead of referencing _ContextData). This fixes out of sync Context.get() and ContextVar.get(). 2. Added a new Context.copy() method. 3. Renamed Token.old_val property to Token.old_value 4. ContextVar.reset(token) now raises a ValueError if the token was created in a different Context. 5. All areas of the PEP were updated to be more precise. Context is *no longer* defined as a read-only or an immutable mapping; ContextVar.get() behaviour is fully defined; the immutability is only mentioned in the Implementation section to avoid confusion; etc. 6. Added a new Examples section. The reference implementation has been updated to include all these changes. The only open question I personally have is whether ContextVar.reset() should be idempotent or not. Maybe we should be strict and raise an error if a user tries to reset a variable more than once with the same token object? Other than that, I'm pretty happy with this version. Big thanks to everybody helping with the PEP! PEP: 567 Title: Context Variables Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Dec-2017 Python-Version: 3.7 Post-History: 12-Dec-2017, 28-Dec-2017, 16-Jan-2018 Abstract ======== This PEP proposes a new ``contextvars`` module and a set of new CPython C APIs to support context variables. This concept is similar to thread-local storage (TLS), but, unlike TLS, it also allows correctly keeping track of values per asynchronous task, e.g. ``asyncio.Task``. This proposal is a simplified version of :pep:`550`. The key difference is that this PEP is concerned only with solving the case for asynchronous tasks, not for generators. There are no proposed modifications to any built-in types or to the interpreter. This proposal is not strictly related to Python Context Managers. Although it does provide a mechanism that can be used by Context Managers to store their state. Rationale ========= Thread-local variables are insufficient for asynchronous tasks that execute concurrently in the same OS thread. Any context manager that saves and restores a context value using ``threading.local()`` will have its context values bleed to other code unexpectedly when used in async/await code. A few examples where having a working context local storage for asynchronous code is desirable: * Context managers like ``decimal`` contexts and ``numpy.errstate``. * Request-related data, such as security tokens and request data in web applications, language context for ``gettext``, etc. * Profiling, tracing, and logging in large code bases. Introduction ============ The PEP proposes a new mechanism for managing context variables. The key classes involved in this mechanism are ``contextvars.Context`` and ``contextvars.ContextVar``. The PEP also proposes some policies for using the mechanism around asynchronous tasks. The proposed mechanism for accessing context variables uses the ``ContextVar`` class. A module (such as ``decimal``) that wishes to use the new mechanism should: * declare a module-global variable holding a ``ContextVar`` to serve as a key; * access the current value via the ``get()`` method on the key variable; * modify the current value via the ``set()`` method on the key variable. The notion of "current value" deserves special consideration: different asynchronous tasks that exist and execute concurrently may have different values for the same key. This idea is well-known from thread-local storage but in this case the locality of the value is not necessarily bound to a thread. Instead, there is the notion of the "current ``Context``" which is stored in thread-local storage. Manipulation of the current context is the responsibility of the task framework, e.g. asyncio. A ``Context`` is a mapping of ``ContextVar`` objects to their values. The ``Context`` itself exposes the ``abc.Mapping`` interface (not ``abc.MutableMapping``!), so it cannot be modified directly. To set a new value for a context variable in a ``Context`` object, the user needs to: * make the ``Context`` object "current" using the ``Context.run()`` method; * use ``ContextVar.set()`` to set a new value for the context variable. The ``ContextVar.get()`` method looks for the variable in the current ``Context`` object using ``self`` as a key. It is not possible to get a direct reference to the current ``Context`` object, but it is possible to obtain a shallow copy of it using the ``contextvars.copy_context()`` function. This ensures that the caller of ``Context.run()`` is the sole owner of its ``Context`` object. Specification ============= A new standard library module ``contextvars`` is added with the following APIs: 1. ``copy_context() -> Context`` function is used to get a copy of the current ``Context`` object for the current OS thread. 2. ``ContextVar`` class to declare and access context variables. 3. ``Context`` class encapsulates context state. Every OS thread stores a reference to its current ``Context`` instance. It is not possible to control that reference directly. Instead, the ``Context.run(callable, *args, **kwargs)`` method is used to run Python code in another context. contextvars.ContextVar ---------------------- The ``ContextVar`` class has the following constructor signature: ``ContextVar(name, *, default=_NO_DEFAULT)``. The ``name`` parameter is used for introspection and debug purposes, and is exposed as a read-only ``ContextVar.name`` attribute. The ``default`` parameter is optional. Example:: # Declare a context variable 'var' with the default value 42. var = ContextVar('var', default=42) (The ``_NO_DEFAULT`` is an internal sentinel object used to detect if the default value was provided.) ``ContextVar.get(default=_NO_DEFAULT)`` returns a value for the context variable for the current ``Context``:: # Get the value of `var`. var.get() If there is no value for the variable in the current context, ``ContextVar.get()`` will: * return the value of the *default* argument of the ``get()`` method, if provided; or * return the default value for the context variable, if provided; or * raise a ``LookupError``. ``ContextVar.set(value) -> Token`` is used to set a new value for the context variable in the current ``Context``:: # Set the variable 'var' to 1 in the current context. var.set(1) ``ContextVar.reset(token)`` is used to reset the variable in the current context to the value it had before the ``set()`` operation that created the ``token`` (or to remove the variable if it was not set):: assert var.get(None) is None token = var.set(1) try: ... finally: var.reset(token) assert var.get(None) is None ``ContextVar.reset()`` method is idempotent and can be called multiple times on the same Token object: second and later calls will be no-ops. The method raises a ``ValueError`` if: * called with a token object created by another variable; or * the current ``Context`` object does not match the one where the token object was created. contextvars.Token ----------------- ``contextvars.Token`` is an opaque object that should be used to restore the ``ContextVar`` to its previous value, or to remove it from the context if the variable was not set before. It can be created only by calling ``ContextVar.set()``. For debug and introspection purposes it has: * a read-only attribute ``Token.var`` pointing to the variable that created the token; * a read-only attribute ``Token.old_value`` set to the value the variable had before the ``set()`` call, or to ``Token.MISSING`` if the variable wasn't set before. contextvars.Context ------------------- ``Context`` object is a mapping of context variables to values. ``Context()`` creates an empty context. To get a copy of the current ``Context`` for the current OS thread, use the ``contextvars.copy_context()`` method:: ctx = contextvars.copy_context() To run Python code in some ``Context``, use ``Context.run()`` method:: ctx.run(function) Any changes to any context variables that ``function`` causes will be contained in the ``ctx`` context:: var = ContextVar('var') var.set('spam') def function(): assert var.get() == 'spam' assert ctx[var] == 'spam' var.set('ham') assert var.get() == 'ham' assert ctx[var] == 'ham' ctx = copy_context() # Any changes that 'function' makes to 'var' will stay # isolated in the 'ctx'. ctx.run(function) assert var.get() == 'spam' assert ctx[var] == 'ham' ``Context.run()`` raises a ``RuntimeError`` when called on the same context object from more than one OS thread, or when called recursively. ``Context.copy()`` returns a shallow copy of the context object. ``Context`` objects implement the ``collections.abc.Mapping`` ABC. This can be used to introspect contexts:: ctx = contextvars.copy_context() # Print all context variables and their values in 'ctx': print(ctx.items()) # Print the value of 'some_variable' in context 'ctx': print(ctx[some_variable]) Note that all Mapping methods, including ``Context.__getitem__`` and ``Context.get``, ignore default values for context variables (i.e. ``ContextVar.default``). This means that for a variable *var* that was created with a default value and was not set in the *context*: * ``context[var]`` raises a ``KeyError``, * ``var in context`` returns ``False``, * the variable isn't included in ``context.items()``, etc. asyncio ------- ``asyncio`` uses ``Loop.call_soon()``, ``Loop.call_later()``, and ``Loop.call_at()`` to schedule the asynchronous execution of a function. ``asyncio.Task`` uses ``call_soon()`` to run the wrapped coroutine. We modify ``Loop.call_{at,later,soon}`` and ``Future.add_done_callback()`` to accept the new optional *context* keyword-only argument, which defaults to the current context:: def call_soon(self, callback, *args, context=None): if context is None: context = contextvars.copy_context() # ... some time later context.run(callback, *args) Tasks in asyncio need to maintain their own context that they inherit from the point they were created at. ``asyncio.Task`` is modified as follows:: class Task: def __init__(self, coro): ... # Get the current context snapshot. self._context = contextvars.copy_context() self._loop.call_soon(self._step, context=self._context) def _step(self, exc=None): ... # Every advance of the wrapped coroutine is done in # the task's context. self._loop.call_soon(self._step, context=self._context) ... Implementation ============== This section explains high-level implementation details in pseudo-code. Some optimizations are omitted to keep this section short and clear. The ``Context`` mapping is implemented using an immutable dictionary. This allows for a O(1) implementation of the ``copy_context()`` function. The reference implementation implements the immutable dictionary using Hash Array Mapped Tries (HAMT); see :pep:`550` for analysis of HAMT performance [1]_. For the purposes of this section, we implement an immutable dictionary using a copy-on-write approach and built-in dict type:: class _ContextData: def __init__(self): self._mapping = dict() def get(self, key): return self._mapping[key] def set(self, key, value): copy = _ContextData() copy._mapping = self._mapping.copy() copy._mapping[key] = value return copy def delete(self, key): copy = _ContextData() copy._mapping = self._mapping.copy() del copy._mapping[key] return copy Every OS thread has a reference to the current ``Context`` object:: class PyThreadState: context: Context ``contextvars.Context`` is a wrapper around ``_ContextData``:: class Context(collections.abc.Mapping): _data: _ContextData _prev_context: Optional[Context] def __init__(self): self._data = _ContextData() self._prev_context = None def run(self, callable, *args, **kwargs): if self._prev_context is not None: raise RuntimeError( f'cannot enter context: {self} is already entered') ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() self._prev_context = ts.context try: ts.context = self return callable(*args, **kwargs) finally: ts.context = self._prev_context self._prev_context = None def copy(self): new = Context() new._data = self._data return new # Mapping API methods are implemented by delegating # `get()` and other Mapping methods to `self._data`. ``contextvars.copy_context()`` is implemented as follows:: def copy_context(): ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() return ts.context.copy() ``contextvars.ContextVar`` interacts with ``PyThreadState.context`` directly:: class ContextVar: def __init__(self, name, *, default=_NO_DEFAULT): self._name = name self._default = default @property def name(self): return self._name def get(self, default=_NO_DEFAULT): ts: PyThreadState = PyThreadState_Get() if ts.context is not None: try: return ts.context[self] except KeyError: pass if default is not _NO_DEFAULT: return default if self._default is not _NO_DEFAULT: return self._default raise LookupError def set(self, value): ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() data: _ContextData = ts.context._data try: old_value = data.get(self) except KeyError: old_value = Token.MISSING updated_data = data.set(self, value) ts.context._data = updated_data return Token(ts.context, self, old_value) def reset(self, token): if token._var is not self: raise ValueError( "Token was created by a different ContextVar") ts: PyThreadState = PyThreadState_Get() if token._ctx is not ts.context: raise ValueError( "Token was created in a different Context") if token._used: return if token._old_value is Token.MISSING: ts.context._data = data.delete(token._var) else: ts.context._data = data.set(token._var, token._old_value) token._used = True Note that the in the reference implementation, ``ContextVar.get()`` has an internal cache for the most recent value, which allows to bypass a hash lookup. This is similar to the optimization the ``decimal`` module implements to retrieve its context from ``PyThreadState_GetDict()``. See :pep:`550` which explains the implementation of the cache in great detail. The ``Token`` class is implemented as follows:: class Token: MISSING = object() def __init__(self, ctx, var, old_value): self._ctx = ctx self._var = var self._old_value = old_value self._used = False @property def var(self): return self._var @property def old_value(self): return self._old_value Summary of the New APIs ======================= Python API ---------- 1. A new ``contextvars`` module with ``ContextVar``, ``Context``, and ``Token`` classes, and a ``copy_context()`` function. 2. ``asyncio.Loop.call_at()``, ``asyncio.Loop.call_later()``, ``asyncio.Loop.call_soon()``, and ``asyncio.Future.add_done_callback()`` run callback functions in the context they were called in. A new *context* keyword-only parameter can be used to specify a custom context. 3. ``asyncio.Task`` is modified internally to maintain its own context. C API ----- 1. ``PyContextVar * PyContextVar_New(char *name, PyObject *default)``: create a ``ContextVar`` object. The *default* argument can be ``NULL``, which means that the variable has no default value. 2. ``int PyContextVar_Get(PyContextVar *, PyObject *default_value, PyObject **value)``: return ``-1`` if an error occurs during the lookup, ``0`` otherwise. If a value for the context variable is found, it will be set to the ``value`` pointer. Otherwise, ``value`` will be set to ``default_value`` when it is not ``NULL``. If ``default_value`` is ``NULL``, ``value`` will be set to the default value of the variable, which can be ``NULL`` too. ``value`` is always a new reference. 3. ``PyContextToken * PyContextVar_Set(PyContextVar *, PyObject *)``: set the value of the variable in the current context. 4. ``PyContextVar_Reset(PyContextVar *, PyContextToken *)``: reset the value of the context variable. 5. ``PyContext * PyContext_New()``: create a new empty context. 6. ``PyContext * PyContext_Copy()``: get a copy of the current context. 7. ``int PyContext_Enter(PyContext *)`` and ``int PyContext_Exit(PyContext *)`` allow to set and restore the context for the current OS thread. It is required to always restore the previous context:: PyContext *old_ctx = PyContext_Copy(); if (old_ctx == NULL) goto error; if (PyContext_Enter(new_ctx)) goto error; // run some code if (PyContext_Exit(old_ctx)) goto error; Design Considerations ===================== Why contextvars.Token and not ContextVar.unset()? ------------------------------------------------- The Token API allows to get around having a ``ContextVar.unset()`` method, which is incompatible with chained contexts design of :pep:`550`. Future compatibility with :pep:`550` is desired (at least for Python 3.7) in case there is demand to support context variables in generators and asynchronous generators. The Token API also offers better usability: the user does not have to special-case absence of a value. Compare:: token = cv.get() try: cv.set(blah) # code finally: cv.reset(token) with:: _deleted = object() old = cv.get(default=_deleted) try: cv.set(blah) # code finally: if old is _deleted: cv.unset() else: cv.set(old) Rejected Ideas ============== Replication of threading.local() interface ------------------------------------------ Please refer to :pep:`550` where this topic is covered in detail: [2]_. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Libraries that use ``threading.local()`` to store context-related values, currently work correctly only for synchronous code. Switching them to use the proposed API will keep their behavior for synchronous code unmodified, but will automatically enable support for asynchronous code. Examples ======== Converting code that uses threading.local() ------------------------------------------- A typical code that uses ``threading.local()`` usually looks like the following snippet:: class mylocal(threading.local): # Subclass threading.local to specify a default value. value = 'spam' mylocal = mylocal() # To set a new value: mylocal.value = 'new value' # To read the current value: mylocal.value Such code can be converted to use the ``contextvars`` module:: mylocal = contextvars.ContextVar('mylocal', 'spam') # To set a new value: mylocal.set('new value') # To read the current value: mylocal.get() Offloading execution to other threads ------------------------------------- It is possible to run code in a separate OS thread using a copy of the current thread context:: executor = ThreadPoolExecutor() current_context = contextvars.copy_context() executor.submit( lambda: current_context.run(some_function)) Reference Implementation ======================== The reference implementation can be found here: [3]_. References ========== .. [1] https://www.python.org/dev/peps/pep-0550/#appendix-hamt-performance-analysis .. [2] https://www.python.org/dev/peps/pep-0550/#replication-of-threading-local-in… .. [3] https://github.com/python/cpython/pull/5027 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

8 21

sys.settrace does not produce events for C functions
by Pablo Galindo Salgado Jan. 21, 2018

Jan. 21, 2018

The docs for sys.settrace mention that: >> event is a string: 'call', 'line', 'return', 'exception', >> 'c_call', 'c_return', or 'c_exception' But in the code for ceval.c the only point where call_trace is invoked with PyTrace_C_CALL or PyTrace_C_RETURN is under the C_TRACE macro. In this macro this line prevents any function set up using sys.settrace to call call_trace with the mentioned arguments: if (tstate->use_tracing && tstate->c_profilefunc) Notice that from the code of PyEval_SetTrace and PyEval_SetProfile, only the later sets tstate->c_profilefunc and therefore only functions installed using sys.setprofile will recieve a c_call for the event. Xiang Zhan has suggested me to ask here what is the best course of action: 1) Document this behavior. 2) Fix the code. This question is related to this issue: https://bugs.python.org/issue17799 Thanks everyone for your time! Pablo

2 1

Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
by Christian Heimes Jan. 20, 2018

Jan. 20, 2018

Hi, I'm still working on a ssl module PEP for 3.7 [1], but it's probably not going to be finished before beta 1 deadline. I have a bunch of fixes and improvements for the ssl module in queue, most of them require OpenSSL 1.0.2 features. The features are also present and working properly since LibreSSL 2.5.3 If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land bunch of useful goodies like proper hostname verification [2], proper fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate and PrivateKey types (support loading certs and keys from file and memory) [4], and simplified cipher suite configuration [5]. I can finally clean up _ssl.c during the beta phase, too. OpenSSL 1.0.1 is out of support since December 2016, 0.9.8 since 2015. These versions haven't received any security updates for more than a year! All major Linux and BSD distributions have at least 1.0.2 [6]. The only relevant exception is Ubuntu 14.04 LTS, because Travis CI is running 14.04. PR 3562 [7] contains a PoC to compile a custom build of OpenSSL on Travis. Builds are cached. Regards, Christian [1] https://github.com/tiran/peps/blob/sslmodule37/pep-9999.txt [2] https://bugs.python.org/issue31399 [3] https://bugs.python.org/issue32185 [4] https://bugs.python.org/issue18369 [5] https://bugs.python.org/issue31429 [6] https://gist.github.com/tiran/c5409bbd60a5f082f654d967add8cc79 [7] https://github.com/python/cpython/pull/3462

16 35

Exposing different versions of a system call in Python
by Pablo Galindo Salgado Jan. 20, 2018

Jan. 20, 2018

Hello everyone, In today's episode of exposing useful Linux system calls I am exposing preadv2 in this PR: https://github.com/python/cpython/pull/5239 as requested in this issue: https://bugs.python.org/issue31368 As njsmith has commented in the PR, preadv2 only exists because regular preadv was missing a flags argument, and in C the only way to add an argument is to make a new function. In Python we have already exposed preadv2 and a possible solution would be add a optional argument that passes the new parametera to preadv and calls preadv2 if this happens. On the other side, we have pipe and pipe2 as an example of exposing two versions when this situation happens. The question is: What is preferable, exposing both functions or augment the old one? Thank you everyone for your time!

5 4

Summary of Python tracker Issues
by Python tracker Jan. 19, 2018

Jan. 19, 2018

ACTIVITY SUMMARY (2018-01-12 - 2018-01-19) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6400 (+31) closed 37949 (+28) total 44349 (+59) Open issues with patches: 2489 Issues opened (46) ================== #32493: UUID Module - FreeBSD build failure https://bugs.python.org/issue32493 reopened by vstinner #32540: venv docs - doesn't match behavior https://bugs.python.org/issue32540 opened by jason.coombs #32541: cgi.FieldStorage constructor assumes all lines terminate with https://bugs.python.org/issue32541 opened by Ian Craggs #32542: memory not freed, aka memory leak continues... https://bugs.python.org/issue32542 opened by Michael.Felt #32545: Unable to install Python 3.7.0a4 on Windows 10 - Error 0x80070 https://bugs.python.org/issue32545 opened by mwr256 #32546: Unusual TypeError with dataclass decorator https://bugs.python.org/issue32546 opened by rhettinger #32547: csv.DictWriter emits strange errors if fieldnames is an iterat https://bugs.python.org/issue32547 opened by bendotc #32548: IDLE: Add links for email and docs to help_about https://bugs.python.org/issue32548 opened by csabella #32549: Travis: Test with OpenSSL 1.1.0 https://bugs.python.org/issue32549 opened by christian.heimes #32550: STORE_ANNOTATION bytecode is unnecessary and can be removed. https://bugs.python.org/issue32550 opened by Mark.Shannon #32551: Zipfile & directory execution in 3.5.4 also adds the parent di https://bugs.python.org/issue32551 opened by nedbat #32552: Improve text for file arguments in argparse.ArgumentDefaultsHe https://bugs.python.org/issue32552 opened by elypter #32553: venv says to use python3 which does not exist in 3.6.4 https://bugs.python.org/issue32553 opened by Paul Watson #32554: random.seed(tuple) uses the randomized hash function and so is https://bugs.python.org/issue32554 opened by johnnyd #32555: Encoding issues with the locale encoding https://bugs.python.org/issue32555 opened by vstinner #32556: support bytes paths in nt _getdiskusage, _getvolumepathname, a https://bugs.python.org/issue32556 opened by eryksun #32557: allow shutil.disk_usage to take a file path https://bugs.python.org/issue32557 opened by eryksun #32560: inherit the py launcher's STARTUPINFO https://bugs.python.org/issue32560 opened by eryksun #32561: Add API to io objects for non-blocking reads/writes https://bugs.python.org/issue32561 opened by njs #32562: Support fspath protocol in AF_UNIX sockaddr resolution https://bugs.python.org/issue32562 opened by njs #32563: -Werror=declaration-after-statement expat build failure on Pyt https://bugs.python.org/issue32563 opened by ncoghlan #32565: Document the version of adding opcodes https://bugs.python.org/issue32565 opened by serhiy.storchaka #32566: Not able to open Python IDLE https://bugs.python.org/issue32566 opened by Kiran #32567: Venv’s config file (pyvenv.cfg) should be compatible with Co https://bugs.python.org/issue32567 opened by uranusjr #32568: Fix handling of sizehint=-1 in select.epoll() https://bugs.python.org/issue32568 opened by taleinat #32571: Speed up and clean up getting optional attributes in C code https://bugs.python.org/issue32571 opened by serhiy.storchaka #32572: Add the ftplib option, overrides the IP address. https://bugs.python.org/issue32572 opened by studioes #32573: sys.argv documentation should include caveat for embedded envi https://bugs.python.org/issue32573 opened by pgacv2 #32574: asyncio.Queue, put() leaks memory if the queue is full https://bugs.python.org/issue32574 opened by Mordis #32579: UUID module fix, uuid1 python module function https://bugs.python.org/issue32579 opened by David CARLIER2 #32580: Fallback to dev_urandom doesn't work when py_getrandom returns https://bugs.python.org/issue32580 opened by jernejs #32581: A bug of the write funtion of ConfigParser.py https://bugs.python.org/issue32581 opened by jiangjinhu666 #32582: chr raises OverflowError https://bugs.python.org/issue32582 opened by ukl #32583: Crash during decoding using UTF-16/32 and custom error handler https://bugs.python.org/issue32583 opened by sibiryakov #32584: Uninitialized free_extra in code_dealloc https://bugs.python.org/issue32584 opened by jeethu #32585: Add ttk::spinbox to tkinter.ttk https://bugs.python.org/issue32585 opened by Alan Moore #32587: Make REG_MULTI_SZ support PendingFileRenameOperations https://bugs.python.org/issue32587 opened by nanonyme #32589: Statistics as a result from timeit https://bugs.python.org/issue32589 opened by MGilch #32590: Proposal: add an "ensure(arg)" builtin for parameter validatio https://bugs.python.org/issue32590 opened by ncoghlan #32591: Deprecate sys.set_coroutine_wrapper and replace it with more f https://bugs.python.org/issue32591 opened by njs #32592: Drop support of Windows Vista in Python 3.7 https://bugs.python.org/issue32592 opened by vstinner #32593: Drop support of FreeBSD 9 and older in Python 3.7 https://bugs.python.org/issue32593 opened by vstinner #32594: File object 'name' attribute inconsistent type and not obvious https://bugs.python.org/issue32594 opened by skip.montanaro #32596: Lazy import concurrent.futures.process and thread https://bugs.python.org/issue32596 opened by inada.naoki #32597: Bad detection of clang https://bugs.python.org/issue32597 opened by bapt #32598: Use autoconf to detect OpenSSL and libssl features https://bugs.python.org/issue32598 opened by christian.heimes Most recent 15 issues with no replies (15) ========================================== #32598: Use autoconf to detect OpenSSL and libssl features https://bugs.python.org/issue32598 #32597: Bad detection of clang https://bugs.python.org/issue32597 #32596: Lazy import concurrent.futures.process and thread https://bugs.python.org/issue32596 #32593: Drop support of FreeBSD 9 and older in Python 3.7 https://bugs.python.org/issue32593 #32585: Add ttk::spinbox to tkinter.ttk https://bugs.python.org/issue32585 #32584: Uninitialized free_extra in code_dealloc https://bugs.python.org/issue32584 #32583: Crash during decoding using UTF-16/32 and custom error handler https://bugs.python.org/issue32583 #32579: UUID module fix, uuid1 python module function https://bugs.python.org/issue32579 #32574: asyncio.Queue, put() leaks memory if the queue is full https://bugs.python.org/issue32574 #32572: Add the ftplib option, overrides the IP address. https://bugs.python.org/issue32572 #32567: Venv’s config file (pyvenv.cfg) should be compatible with Co https://bugs.python.org/issue32567 #32565: Document the version of adding opcodes https://bugs.python.org/issue32565 #32562: Support fspath protocol in AF_UNIX sockaddr resolution https://bugs.python.org/issue32562 #32557: allow shutil.disk_usage to take a file path https://bugs.python.org/issue32557 #32556: support bytes paths in nt _getdiskusage, _getvolumepathname, a https://bugs.python.org/issue32556 Most recent 15 issues waiting for review (15) ============================================= #32598: Use autoconf to detect OpenSSL and libssl features https://bugs.python.org/issue32598 #32596: Lazy import concurrent.futures.process and thread https://bugs.python.org/issue32596 #32593: Drop support of FreeBSD 9 and older in Python 3.7 https://bugs.python.org/issue32593 #32592: Drop support of Windows Vista in Python 3.7 https://bugs.python.org/issue32592 #32589: Statistics as a result from timeit https://bugs.python.org/issue32589 #32585: Add ttk::spinbox to tkinter.ttk https://bugs.python.org/issue32585 #32582: chr raises OverflowError https://bugs.python.org/issue32582 #32580: Fallback to dev_urandom doesn't work when py_getrandom returns https://bugs.python.org/issue32580 #32574: asyncio.Queue, put() leaks memory if the queue is full https://bugs.python.org/issue32574 #32572: Add the ftplib option, overrides the IP address. https://bugs.python.org/issue32572 #32571: Speed up and clean up getting optional attributes in C code https://bugs.python.org/issue32571 #32565: Document the version of adding opcodes https://bugs.python.org/issue32565 #32563: -Werror=declaration-after-statement expat build failure on Pyt https://bugs.python.org/issue32563 #32555: Encoding issues with the locale encoding https://bugs.python.org/issue32555 #32550: STORE_ANNOTATION bytecode is unnecessary and can be removed. https://bugs.python.org/issue32550 Top 10 most discussed issues (10) ================================= #32534: Speed-up list.insert: use memmove() https://bugs.python.org/issue32534 22 msgs #31900: localeconv() should decode numeric fields from LC_NUMERIC enco https://bugs.python.org/issue31900 19 msgs #32551: Zipfile & directory execution in 3.5.4 also adds the parent di https://bugs.python.org/issue32551 9 msgs #32553: venv says to use python3 which does not exist in 3.6.4 https://bugs.python.org/issue32553 9 msgs #30693: tarfile add uses random order https://bugs.python.org/issue30693 8 msgs #32561: Add API to io objects for non-blocking reads/writes https://bugs.python.org/issue32561 8 msgs #32550: STORE_ANNOTATION bytecode is unnecessary and can be removed. https://bugs.python.org/issue32550 7 msgs #32594: File object 'name' attribute inconsistent type and not obvious https://bugs.python.org/issue32594 7 msgs #29708: support reproducible Python builds https://bugs.python.org/issue29708 6 msgs #32493: UUID Module - FreeBSD build failure https://bugs.python.org/issue32493 6 msgs Issues closed (26) ================== #15221: os.path.is*() may return False if path can't be accessed https://bugs.python.org/issue15221 closed by Mariatta #26330: shutil.disk_usage() on Windows can't properly handle unicode https://bugs.python.org/issue26330 closed by Mariatta #29240: PEP 540: Add a new UTF-8 mode https://bugs.python.org/issue29240 closed by vstinner #29476: Simplify set_add_entry() https://bugs.python.org/issue29476 closed by rhettinger #29911: Uninstall command line in Windows registry does not uninstall https://bugs.python.org/issue29911 closed by steve.dower #32346: Speed up slot lookup for class creation https://bugs.python.org/issue32346 closed by pitrou #32403: date, time and datetime alternate constructors should take fas https://bugs.python.org/issue32403 closed by belopolsky #32404: fromtimestamp does not call __new__ in datetime subclasses https://bugs.python.org/issue32404 closed by belopolsky #32516: Add a shared library mechanism for win32 https://bugs.python.org/issue32516 closed by steve.dower #32529: Call readinto in shutil.copyfileobj https://bugs.python.org/issue32529 closed by YoSTEALTH #32537: multiprocessing.pool.Pool.starmap_async - wrong parameter name https://bugs.python.org/issue32537 closed by pablogsal #32539: os.listdir(...) on deep path on windows in python2.7 fails wit https://bugs.python.org/issue32539 closed by serhiy.storchaka #32543: odd floor division behavior https://bugs.python.org/issue32543 closed by serhiy.storchaka #32544: Speed up hasattr(o, name) and getattr(o, name, default) https://bugs.python.org/issue32544 closed by inada.naoki #32558: Socketserver documentation : error in server example https://bugs.python.org/issue32558 closed by tcolombo #32559: logging - public function to get level from name https://bugs.python.org/issue32559 closed by vinay.sajip #32564: Syntax error on using variable async https://bugs.python.org/issue32564 closed by Andrew Olefira #32569: Blake2 module, memory clearance update https://bugs.python.org/issue32569 closed by David Carlier #32570: Python crash 0xc00000fd Windows10 x64 - "fail without words https://bugs.python.org/issue32570 closed by eryksun #32575: IDLE cannot locate certain SyntaxErrors raised by f-string exp https://bugs.python.org/issue32575 closed by terry.reedy #32576: concurrent.futures.thread deadlock due to Queue in weakref cal https://bugs.python.org/issue32576 closed by pitrou #32577: Pip creates entry point commands in the wrong place when invok https://bugs.python.org/issue32577 closed by uranusjr #32578: x86-64 Sierra 3.6: test_asyncio fails with timeout after 15 mi https://bugs.python.org/issue32578 closed by vstinner #32586: urllib2 HOWTO URLError example minor error https://bugs.python.org/issue32586 closed by Mariatta #32588: Move _findvs into its own extension module https://bugs.python.org/issue32588 closed by steve.dower #32595: Deque with iterable object as one object https://bugs.python.org/issue32595 closed by rhettinger

1 0