Mailman 3 January 2018 - Python-Dev

Summary of Python tracker Issues
by Python tracker Jan. 12, 2018

Jan. 12, 2018

ACTIVITY SUMMARY (2018-01-05 - 2018-01-12) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6369 ( -8) closed 37921 (+50) total 44290 (+42) Open issues with patches: 2470 Issues opened (38) ================== #20104: expose posix_spawn(p) https://bugs.python.org/issue20104 reopened by gregory.p.smith #23749: asyncio missing wrap_socket (starttls) https://bugs.python.org/issue23749 reopened by vstinner #29137: Fix fpectl-induced ABI breakage https://bugs.python.org/issue29137 reopened by doko #31804: multiprocessing calls flush on sys.stdout at exit even if it i https://bugs.python.org/issue31804 reopened by Pox TheGreat #31993: pickle.dump allocates unnecessary temporary bytes / str https://bugs.python.org/issue31993 reopened by serhiy.storchaka #32206: Run modules with pdb https://bugs.python.org/issue32206 reopened by ncoghlan #32498: urllib.parse.unquote raises incorrect errormessage when string https://bugs.python.org/issue32498 opened by stein-k #32500: PySequence_Length() raises TypeError on dict type https://bugs.python.org/issue32500 opened by mgorny #32501: Documentation for dir([object]) https://bugs.python.org/issue32501 opened by Vladislavs Burakovs #32502: uuid1() broken on macos high sierra https://bugs.python.org/issue32502 opened by anpetral #32503: Avoid creating small frames in pickle protocol 4 https://bugs.python.org/issue32503 opened by serhiy.storchaka #32505: dataclasses: make field() with no annotation an error https://bugs.python.org/issue32505 opened by eric.smith #32506: dataclasses: no need for OrderedDict now that dict guarantees https://bugs.python.org/issue32506 opened by eric.smith #32509: doctest syntax ambiguity between continuation line and ellipsi https://bugs.python.org/issue32509 opened by jason.coombs #32511: Thread primitives do not report the OS-level error on failure https://bugs.python.org/issue32511 opened by zwol #32512: Add an option to profile to run library module as a script https://bugs.python.org/issue32512 opened by mariocj89 #32513: dataclasses: make it easier to use user-supplied special metho https://bugs.python.org/issue32513 opened by eric.smith #32514: 0x80070002 - The system cannot find the file specified https://bugs.python.org/issue32514 opened by Beatty0111 #32515: Add an option to trace to run module as a script https://bugs.python.org/issue32515 opened by mariocj89 #32516: Add a shared library mechanism for win32 https://bugs.python.org/issue32516 opened by xoviat #32517: test_read_pty_output() of test_asyncio hangs on macOS 10.13.2 https://bugs.python.org/issue32517 opened by vstinner #32519: venv API docs - symlinks default incorrect https://bugs.python.org/issue32519 opened by jason.coombs #32521: NIS module fails to build due to the removal of interfaces rel https://bugs.python.org/issue32521 opened by cstratak #32522: Add precision argument to datetime.now https://bugs.python.org/issue32522 opened by p-ganssle #32523: inconsistent spacing in changelog.html https://bugs.python.org/issue32523 opened by ned.deily #32524: Python 2.7 leaks a packages __init__.py module object on Synta https://bugs.python.org/issue32524 opened by Segev Finer #32526: Closing async generator while it is running does not raise an https://bugs.python.org/issue32526 opened by achimnol #32528: Change base class for futures.CancelledError https://bugs.python.org/issue32528 opened by socketpair #32529: Call readinto in shutil.copyfileobj https://bugs.python.org/issue32529 opened by YoSTEALTH #32530: How ro fix the chm encoding in Non western european codepage(c https://bugs.python.org/issue32530 opened by Nim #32531: gdb.execute can not put string value. https://bugs.python.org/issue32531 opened by callmekohei #32532: improve sys.settrace and sys.setprofile documentation https://bugs.python.org/issue32532 opened by xiang.zhang #32533: SSLSocket read/write thread-unsafety https://bugs.python.org/issue32533 opened by Alexey Baldin #32534: Speed-up list.insert https://bugs.python.org/issue32534 opened by jeethu #32536: ast and tokenize disagree about line number https://bugs.python.org/issue32536 opened by Mark.Shannon #32537: multiprocessing.pool.Pool.starmap_async - wrong parameter name https://bugs.python.org/issue32537 opened by devnull #32538: Multiprocessing Manager on 3D list - no change of the list pos https://bugs.python.org/issue32538 opened by John_81 #32539: os.listdir(...) on deep path on windows in python2.7 fails wit https://bugs.python.org/issue32539 opened by Anthony Sottile Most recent 15 issues with no replies (15) ========================================== #32539: os.listdir(...) on deep path on windows in python2.7 fails wit https://bugs.python.org/issue32539 #32538: Multiprocessing Manager on 3D list - no change of the list pos https://bugs.python.org/issue32538 #32537: multiprocessing.pool.Pool.starmap_async - wrong parameter name https://bugs.python.org/issue32537 #32536: ast and tokenize disagree about line number https://bugs.python.org/issue32536 #32532: improve sys.settrace and sys.setprofile documentation https://bugs.python.org/issue32532 #32531: gdb.execute can not put string value. https://bugs.python.org/issue32531 #32530: How ro fix the chm encoding in Non western european codepage(c https://bugs.python.org/issue32530 #32524: Python 2.7 leaks a packages __init__.py module object on Synta https://bugs.python.org/issue32524 #32519: venv API docs - symlinks default incorrect https://bugs.python.org/issue32519 #32515: Add an option to trace to run module as a script https://bugs.python.org/issue32515 #32513: dataclasses: make it easier to use user-supplied special metho https://bugs.python.org/issue32513 #32511: Thread primitives do not report the OS-level error on failure https://bugs.python.org/issue32511 #32505: dataclasses: make field() with no annotation an error https://bugs.python.org/issue32505 #32496: lib2to3 fails to parse a ** of a conditional expression https://bugs.python.org/issue32496 #32494: interface to gdbm_count https://bugs.python.org/issue32494 Most recent 15 issues waiting for review (15) ============================================= #32534: Speed-up list.insert https://bugs.python.org/issue32534 #32529: Call readinto in shutil.copyfileobj https://bugs.python.org/issue32529 #32524: Python 2.7 leaks a packages __init__.py module object on Synta https://bugs.python.org/issue32524 #32521: NIS module fails to build due to the removal of interfaces rel https://bugs.python.org/issue32521 #32515: Add an option to trace to run module as a script https://bugs.python.org/issue32515 #32512: Add an option to profile to run library module as a script https://bugs.python.org/issue32512 #32506: dataclasses: no need for OrderedDict now that dict guarantees https://bugs.python.org/issue32506 #32503: Avoid creating small frames in pickle protocol 4 https://bugs.python.org/issue32503 #32497: datetime.strptime creates tz naive object from value containin https://bugs.python.org/issue32497 #32492: C Fast path for namedtuple's property/itemgetter pair https://bugs.python.org/issue32492 #32477: Move jumps optimization from the peepholer to the compiler https://bugs.python.org/issue32477 #32476: Add concat functionality to ElementTree xpath find https://bugs.python.org/issue32476 #32475: Add ability to query number of buffered bytes available on buf https://bugs.python.org/issue32475 #32471: Add an UML class diagram to the collections.abc module documen https://bugs.python.org/issue32471 #32469: Generator and coroutine repr could be more helpful https://bugs.python.org/issue32469 Top 10 most discussed issues (10) ================================= #32522: Add precision argument to datetime.now https://bugs.python.org/issue32522 19 msgs #32509: doctest syntax ambiguity between continuation line and ellipsi https://bugs.python.org/issue32509 11 msgs #31993: pickle.dump allocates unnecessary temporary bytes / str https://bugs.python.org/issue31993 10 msgs #32521: NIS module fails to build due to the removal of interfaces rel https://bugs.python.org/issue32521 10 msgs #32500: PySequence_Length() raises TypeError on dict type https://bugs.python.org/issue32500 8 msgs #9325: Add an option to pdb/trace/profile to run library module as a https://bugs.python.org/issue9325 7 msgs #32346: Speed up slot lookup for class creation https://bugs.python.org/issue32346 7 msgs #32528: Change base class for futures.CancelledError https://bugs.python.org/issue32528 7 msgs #31804: multiprocessing calls flush on sys.stdout at exit even if it i https://bugs.python.org/issue31804 6 msgs #32248: Port importlib_resources (module and ABC) to Python 3.7 https://bugs.python.org/issue32248 6 msgs Issues closed (51) ================== #2518: smtpd.py to handle huge email https://bugs.python.org/issue2518 closed by barry #3802: smtpd.py __getaddr insufficient handling https://bugs.python.org/issue3802 closed by barry #8503: smtpd SMTPServer does not allow domain filtering https://bugs.python.org/issue8503 closed by barry #11260: smtpd-as-a-script feature should be documented and should use https://bugs.python.org/issue11260 closed by barry #12815: Coverage of smtpd.py https://bugs.python.org/issue12815 closed by barry #12816: smtpd uses library outside of the standard libraries https://bugs.python.org/issue12816 closed by barry #14261: Cleanup in smtpd module https://bugs.python.org/issue14261 closed by barry #16462: smtpd should return greeting https://bugs.python.org/issue16462 closed by barry #17607: missed peephole optimization (unnecessary jump at end of funct https://bugs.python.org/issue17607 closed by serhiy.storchaka #19678: smtpd.py: channel should be passed to process_message https://bugs.python.org/issue19678 closed by barry #19679: smtpd.py (SMTPChannel): implement enhanced status codes https://bugs.python.org/issue19679 closed by barry #19806: smtpd crashes when a multi-byte UTF-8 sequence is split betwee https://bugs.python.org/issue19806 closed by barry #22071: Remove long-time deprecated attributes from smtpd https://bugs.python.org/issue22071 closed by barry #22158: RFC 6531 (SMTPUTF8) support in smtpd.PureProxy https://bugs.python.org/issue22158 closed by barry #22159: smtpd.PureProxy and smtpd.DebuggingServer do not work with dec https://bugs.python.org/issue22159 closed by barry #24340: co_stacksize estimate can be highly off https://bugs.python.org/issue24340 closed by serhiy.storchaka #25546: python 3.5 installation problem; Error 0x80240017: Failed to e https://bugs.python.org/issue25546 closed by steve.dower #25954: Python 3.5.1 installer fails on Windows 7 https://bugs.python.org/issue25954 closed by steve.dower #26036: Unnecessary arguments on smtpd.SMTPServer https://bugs.python.org/issue26036 closed by barry #28416: defining persistent_id in _pickle.Pickler subclass causes refe https://bugs.python.org/issue28416 closed by serhiy.storchaka #28747: Expose SSL_CTX_set_cert_verify_callback https://bugs.python.org/issue28747 closed by steve.dower #28888: Installer fails when newer version of CRT is pending installat https://bugs.python.org/issue28888 closed by steve.dower #29409: Implement PEP 529 for io.FileIO https://bugs.python.org/issue29409 closed by steve.dower #30121: Windows: subprocess debug assertion on failure to execute the https://bugs.python.org/issue30121 closed by Segev Finer #30579: Allow traceback objects to be instantiated/mutated/annotated https://bugs.python.org/issue30579 closed by ncoghlan #31113: Stack overflow with large program https://bugs.python.org/issue31113 closed by serhiy.storchaka #31145: PriorityQueue.put() fails with TypeError if priority_number in https://bugs.python.org/issue31145 closed by rhettinger #31975: Add a default filter for DeprecationWarning in __main__ https://bugs.python.org/issue31975 closed by ncoghlan #32267: strptime misparses offsets with microsecond format https://bugs.python.org/issue32267 closed by belopolsky #32278: Allow dataclasses.make_dataclass() to omit type information https://bugs.python.org/issue32278 closed by eric.smith #32279: Pass keyword arguments from dataclasses.make_dataclass() to @d https://bugs.python.org/issue32279 closed by eric.smith #32320: Add default value support to collections.namedtuple() https://bugs.python.org/issue32320 closed by rhettinger #32427: Rename and expose dataclasses._MISSING https://bugs.python.org/issue32427 closed by eric.smith #32428: dataclasses: make it an error to have initialized non-fields i https://bugs.python.org/issue32428 closed by eric.smith #32448: subscriptable https://bugs.python.org/issue32448 closed by terry.reedy #32449: MappingView must inherit from Collection instead of Sized https://bugs.python.org/issue32449 closed by rhettinger #32450: non-descriptive variable name https://bugs.python.org/issue32450 closed by benjamin.peterson #32467: dict_values isn't considered a Collection nor a Container https://bugs.python.org/issue32467 closed by rhettinger #32473: Readibility of ABCMeta._dump_registry() https://bugs.python.org/issue32473 closed by inada.naoki #32486: tail optimization for 'yield from' https://bugs.python.org/issue32486 closed by benjamin.peterson #32493: UUID Module - FreeBSD build failure https://bugs.python.org/issue32493 closed by pitrou #32499: Add dataclasses.is_dataclass(obj) https://bugs.python.org/issue32499 closed by eric.smith #32504: Wheel failed include data_files https://bugs.python.org/issue32504 closed by benjamin.peterson #32507: Change Windows install to applocal UCRT https://bugs.python.org/issue32507 closed by steve.dower #32508: Problem while reading back from a list of lists https://bugs.python.org/issue32508 closed by steven.daprano #32510: Broken comparisons (probably caused by wrong caching of values https://bugs.python.org/issue32510 closed by steven.daprano #32518: HTTPServer can't deal with persistent connection properly https://bugs.python.org/issue32518 closed by r.david.murray #32520: error writing to file in binary mode - python 3.6.3 https://bugs.python.org/issue32520 closed by serhiy.storchaka #32525: Empty tuples are not optimized as constant expressions https://bugs.python.org/issue32525 closed by brett.cannon #32527: windows 7 python 3.6 : after some security updates, import ibm https://bugs.python.org/issue32527 closed by christian.heimes #32535: msvcr140.dll has been replaced with vcruntime140.dll https://bugs.python.org/issue32535 closed by steve.dower

1 0

Bug report in audioop module.
by Moses Egypt Jan. 11, 2018

Jan. 11, 2018

I was told to post this here when I asked what to do on the python reddit. This is the issue: https://bugs.python.org/issue32004 It has received no response since I posted it two months ago, so I figured I didn't fill something out correctly to get it put on someone's tracker. This is my first bug report, so please let me know if there is anything I need to do get it appointed to the correct person next time. Thanks.

3 2

PEP 567 v2
by Yury Selivanov Jan. 10, 2018

Jan. 10, 2018

This is a second version of PEP 567. A few things have changed: 1. I now have a reference implementation: https://github.com/python/cpython/pull/5027 2. The C API was updated to match the implementation. 3. The get_context() function was renamed to copy_context() to better reflect what it is really doing. 4. Few clarifications/edits here and there to address earlier feedback. Yury PEP: 567 Title: Context Variables Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Dec-2017 Python-Version: 3.7 Post-History: 12-Dec-2017, 28-Dec-2017 Abstract ======== This PEP proposes a new ``contextvars`` module and a set of new CPython C APIs to support context variables. This concept is similar to thread-local storage (TLS), but, unlike TLS, it also allows correctly keeping track of values per asynchronous task, e.g. ``asyncio.Task``. This proposal is a simplified version of :pep:`550`. The key difference is that this PEP is concerned only with solving the case for asynchronous tasks, not for generators. There are no proposed modifications to any built-in types or to the interpreter. This proposal is not strictly related to Python Context Managers. Although it does provide a mechanism that can be used by Context Managers to store their state. Rationale ========= Thread-local variables are insufficient for asynchronous tasks that execute concurrently in the same OS thread. Any context manager that saves and restores a context value using ``threading.local()`` will have its context values bleed to other code unexpectedly when used in async/await code. A few examples where having a working context local storage for asynchronous code is desirable: * Context managers like ``decimal`` contexts and ``numpy.errstate``. * Request-related data, such as security tokens and request data in web applications, language context for ``gettext``, etc. * Profiling, tracing, and logging in large code bases. Introduction ============ The PEP proposes a new mechanism for managing context variables. The key classes involved in this mechanism are ``contextvars.Context`` and ``contextvars.ContextVar``. The PEP also proposes some policies for using the mechanism around asynchronous tasks. The proposed mechanism for accessing context variables uses the ``ContextVar`` class. A module (such as ``decimal``) that wishes to store a context variable should: * declare a module-global variable holding a ``ContextVar`` to serve as a key; * access the current value via the ``get()`` method on the key variable; * modify the current value via the ``set()`` method on the key variable. The notion of "current value" deserves special consideration: different asynchronous tasks that exist and execute concurrently may have different values for the same key. This idea is well-known from thread-local storage but in this case the locality of the value is not necessarily bound to a thread. Instead, there is the notion of the "current ``Context``" which is stored in thread-local storage, and is accessed via ``contextvars.copy_context()`` function. Manipulation of the current ``Context`` is the responsibility of the task framework, e.g. asyncio. A ``Context`` is conceptually a read-only mapping, implemented using an immutable dictionary. The ``ContextVar.get()`` method does a lookup in the current ``Context`` with ``self`` as a key, raising a ``LookupError`` or returning a default value specified in the constructor. The ``ContextVar.set(value)`` method clones the current ``Context``, assigns the ``value`` to it with ``self`` as a key, and sets the new ``Context`` as the new current ``Context``. Specification ============= A new standard library module ``contextvars`` is added with the following APIs: 1. ``copy_context() -> Context`` function is used to get a copy of the current ``Context`` object for the current OS thread. 2. ``ContextVar`` class to declare and access context variables. 3. ``Context`` class encapsulates context state. Every OS thread stores a reference to its current ``Context`` instance. It is not possible to control that reference manually. Instead, the ``Context.run(callable, *args, **kwargs)`` method is used to run Python code in another context. contextvars.ContextVar ---------------------- The ``ContextVar`` class has the following constructor signature: ``ContextVar(name, *, default=_NO_DEFAULT)``. The ``name`` parameter is used only for introspection and debug purposes, and is exposed as a read-only ``ContextVar.name`` attribute. The ``default`` parameter is optional. Example:: # Declare a context variable 'var' with the default value 42. var = ContextVar('var', default=42) (The ``_NO_DEFAULT`` is an internal sentinel object used to detect if the default value was provided.) ``ContextVar.get()`` returns a value for context variable from the current ``Context``:: # Get the value of `var`. var.get() ``ContextVar.set(value) -> Token`` is used to set a new value for the context variable in the current ``Context``:: # Set the variable 'var' to 1 in the current context. var.set(1) ``ContextVar.reset(token)`` is used to reset the variable in the current context to the value it had before the ``set()`` operation that created the ``token``:: assert var.get(None) is None token = var.set(1) try: ... finally: var.reset(token) assert var.get(None) is None ``ContextVar.reset()`` method is idempotent and can be called multiple times on the same Token object: second and later calls will be no-ops. contextvars.Token ----------------- ``contextvars.Token`` is an opaque object that should be used to restore the ``ContextVar`` to its previous value, or remove it from the context if the variable was not set before. It can be created only by calling ``ContextVar.set()``. For debug and introspection purposes it has: * a read-only attribute ``Token.var`` pointing to the variable that created the token; * a read-only attribute ``Token.old_value`` set to the value the variable had before the ``set()`` call, or to ``Token.MISSING`` if the variable wasn't set before. Having the ``ContextVar.set()`` method returning a ``Token`` object and the ``ContextVar.reset(token)`` method, allows context variables to be removed from the context if they were not in it before the ``set()`` call. contextvars.Context ------------------- ``Context`` object is a mapping of context variables to values. ``Context()`` creates an empty context. To get a copy of the current ``Context`` for the current OS thread, use the ``contextvars.copy_context()`` method:: ctx = contextvars.copy_context() To run Python code in some ``Context``, use ``Context.run()`` method:: ctx.run(function) Any changes to any context variables that ``function`` causes will be contained in the ``ctx`` context:: var = ContextVar('var') var.set('spam') def function(): assert var.get() == 'spam' var.set('ham') assert var.get() == 'ham' ctx = copy_context() # Any changes that 'function' makes to 'var' will stay # isolated in the 'ctx'. ctx.run(function) assert var.get() == 'spam' Any changes to the context will be contained in the ``Context`` object on which ``run()`` is called on. ``Context.run()`` is used to control in which context asyncio callbacks and Tasks are executed. It can also be used to run some code in a different thread in the context of the current thread:: executor = ThreadPoolExecutor() current_context = contextvars.copy_context() executor.submit( lambda: current_context.run(some_function)) ``Context`` objects implement the ``collections.abc.Mapping`` ABC. This can be used to introspect context objects:: ctx = contextvars.copy_context() # Print all context variables and their values in 'ctx': print(ctx.items()) # Print the value of 'some_variable' in context 'ctx': print(ctx[some_variable]) asyncio ------- ``asyncio`` uses ``Loop.call_soon()``, ``Loop.call_later()``, and ``Loop.call_at()`` to schedule the asynchronous execution of a function. ``asyncio.Task`` uses ``call_soon()`` to run the wrapped coroutine. We modify ``Loop.call_{at,later,soon}`` and ``Future.add_done_callback()`` to accept the new optional *context* keyword-only argument, which defaults to the current context:: def call_soon(self, callback, *args, context=None): if context is None: context = contextvars.copy_context() # ... some time later context.run(callback, *args) Tasks in asyncio need to maintain their own context that they inherit from the point they were created at. ``asyncio.Task`` is modified as follows:: class Task: def __init__(self, coro): ... # Get the current context snapshot. self._context = contextvars.copy_context() self._loop.call_soon(self._step, context=self._context) def _step(self, exc=None): ... # Every advance of the wrapped coroutine is done in # the task's context. self._loop.call_soon(self._step, context=self._context) ... C API ----- 1. ``PyContextVar * PyContextVar_New(char *name, PyObject *default)``: create a ``ContextVar`` object. 2. ``int PyContextVar_Get(PyContextVar *, PyObject *default_value, PyObject **value)``: return ``-1`` if an error occurs during the lookup, ``0`` otherwise. If a value for the context variable is found, it will be set to the ``value`` pointer. Otherwise, ``value`` will be set to ``default_value`` when it is not ``NULL``. If ``default_value`` is ``NULL``, ``value`` will be set to the default value of the variable, which can be ``NULL`` too. ``value`` is always a borrowed reference. 3. ``PyContextToken * PyContextVar_Set(PyContextVar *, PyObject *)``: set the value of the variable in the current context. 4. ``PyContextVar_Reset(PyContextVar *, PyContextToken *)``: reset the value of the context variable. 5. ``PyContext * PyContext_New()``: create a new empty context. 6. ``PyContext * PyContext_Copy()``: get a copy of the current context. 7. ``int PyContext_Enter(PyContext *)`` and ``int PyContext_Exit(PyContext *)`` allow to set and restore the context for the current OS thread. It is required to always restore the previous context:: PyContext *old_ctx = PyContext_Copy(); if (old_ctx == NULL) goto error; if (PyContext_Enter(new_ctx)) goto error; // run some code if (PyContext_Exit(old_ctx)) goto error; Implementation ============== This section explains high-level implementation details in pseudo-code. Some optimizations are omitted to keep this section short and clear. For the purposes of this section, we implement an immutable dictionary using ``dict.copy()``:: class _ContextData: def __init__(self): self._mapping = dict() def get(self, key): return self._mapping[key] def set(self, key, value): copy = _ContextData() copy._mapping = self._mapping.copy() copy._mapping[key] = value return copy def delete(self, key): copy = _ContextData() copy._mapping = self._mapping.copy() del copy._mapping[key] return copy Every OS thread has a reference to the current ``_ContextData``. ``PyThreadState`` is updated with a new ``context_data`` field that points to a ``_ContextData`` object:: class PyThreadState: context_data: _ContextData ``contextvars.copy_context()`` is implemented as follows:: def copy_context(): ts : PyThreadState = PyThreadState_Get() if ts.context_data is None: ts.context_data = _ContextData() ctx = Context() ctx._data = ts.context_data return ctx ``contextvars.Context`` is a wrapper around ``_ContextData``:: class Context(collections.abc.Mapping): def __init__(self): self._data = _ContextData() def run(self, callable, *args, **kwargs): ts : PyThreadState = PyThreadState_Get() saved_data : _ContextData = ts.context_data try: ts.context_data = self._data return callable(*args, **kwargs) finally: self._data = ts.context_data ts.context_data = saved_data # Mapping API methods are implemented by delegating # `get()` and other Mapping calls to `self._data`. ``contextvars.ContextVar`` interacts with ``PyThreadState.context_data`` directly:: class ContextVar: def __init__(self, name, *, default=_NO_DEFAULT): self._name = name self._default = default @property def name(self): return self._name def get(self, default=_NO_DEFAULT): ts : PyThreadState = PyThreadState_Get() data : _ContextData = ts.context_data try: return data.get(self) except KeyError: pass if default is not _NO_DEFAULT: return default if self._default is not _NO_DEFAULT: return self._default raise LookupError def set(self, value): ts : PyThreadState = PyThreadState_Get() data : _ContextData = ts.context_data try: old_value = data.get(self) except KeyError: old_value = Token.MISSING ts.context_data = data.set(self, value) return Token(self, old_value) def reset(self, token): if token._used: return if token._old_value is Token.MISSING: ts.context_data = data.delete(token._var) else: ts.context_data = data.set(token._var, token._old_value) token._used = True class Token: MISSING = object() def __init__(self, var, old_value): self._var = var self._old_value = old_value self._used = False @property def var(self): return self._var @property def old_value(self): return self._old_value Implementation Notes ==================== * The internal immutable dictionary for ``Context`` is implemented using Hash Array Mapped Tries (HAMT). They allow for O(log N) ``set`` operation, and for O(1) ``copy_context()`` function, where *N* is the number of items in the dictionary. For a detailed analysis of HAMT performance please refer to :pep:`550` [1]_. * ``ContextVar.get()`` has an internal cache for the most recent value, which allows to bypass a hash lookup. This is similar to the optimization the ``decimal`` module implements to retrieve its context from ``PyThreadState_GetDict()``. See :pep:`550` which explains the implementation of the cache in a great detail. Summary of the New APIs ======================= * A new ``contextvars`` module with ``ContextVar``, ``Context``, and ``Token`` classes, and a ``copy_context()`` function. * ``asyncio.Loop.call_at()``, ``asyncio.Loop.call_later()``, ``asyncio.Loop.call_soon()``, and ``asyncio.Future.add_done_callback()`` run callback functions in the context they were called in. A new *context* keyword-only parameter can be used to specify a custom context. * ``asyncio.Task`` is modified internally to maintain its own context. Design Considerations ===================== Why contextvars.Token and not ContextVar.unset()? ------------------------------------------------- The Token API allows to get around having a ``ContextVar.unset()`` method, which is incompatible with chained contexts design of :pep:`550`. Future compatibility with :pep:`550` is desired (at least for Python 3.7) in case there is demand to support context variables in generators and asynchronous generators. The Token API also offers better usability: the user does not have to special-case absence of a value. Compare:: token = cv.get() try: cv.set(blah) # code finally: cv.reset(token) with:: _deleted = object() old = cv.get(default=_deleted) try: cv.set(blah) # code finally: if old is _deleted: cv.unset() else: cv.set(old) Rejected Ideas ============== Replication of threading.local() interface ------------------------------------------ Please refer to :pep:`550` where this topic is covered in detail: [2]_. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Libraries that use ``threading.local()`` to store context-related values, currently work correctly only for synchronous code. Switching them to use the proposed API will keep their behavior for synchronous code unmodified, but will automatically enable support for asynchronous code. Reference Implementation ======================== The reference implementation can be found here: [3]_. References ========== .. [1] https://www.python.org/dev/peps/pep-0550/#appendix-hamt-performance-analysis .. [2] https://www.python.org/dev/peps/pep-0550/#replication-of-threading-local-in… .. [3] https://github.com/python/cpython/pull/5027 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

10 60

[RELEASE] Python 3.7.0a4 is now available for testing
by Ned Deily Jan. 9, 2018

Jan. 9, 2018

Python 3.7.0a4 is the last of four planned alpha releases of Python 3.7, the next feature release of Python. During the alpha phase, Python 3.7 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The next preview release, 3.7.0b1, is planned for 2018-01-29. You can find Python 3.7.0a4 and more information here: https://www.python.org/downloads/release/python-370a4/ -- Ned Deily nad(a)python.org -- []

3 2

subprocess not escaping "^" on Windows
by Christian Tismer Jan. 9, 2018

Jan. 9, 2018

Hi Guys, yes I know there was a lengthy thread on python-dev in 2014 called "subprocess shell=True on Windows doesn't escape ^ character". But in the end, I still don't understand why subprocess does escape the double quote when shell=True but not other special characters like "^"? Yes I know that certain characters are escaped under certain Windows versions and others are not. And it is not trivial to make that work correctly in all cases. But I think if we support some escaping at all, then we should also support all special cases. Or what sense should an escape make if it works sometimes and sometimes not? The user would have to know which cases work and which not. But I thought we want to remove exactly that burden from him? ----- As a side note: In most cases where shell=True is found, people seem to need evaluation of the PATH variable. To my understanding, >>> from subprocess import call >>> call(("ls",)) works in Linux, but (with dir) not in Windows. But that is misleading because "dir" is a builtin command but "ls" is not. The same holds for "del" (Windows) and "rm" (Linux). So I thought that using shell=True was a good Thing on windows, but actually it is the start of all evil. Using regular commands like "git" works fine on Windows and Linux without the shell=True parameter. Perhaps it would be a good thing to emulate the builtin programs in python by some shell=True replacement (emulate_shell=True?) to match the normal user expectations without using the shell? Cheers - Chris -- Christian Tismer-Sperling :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

6 13

Best Python API for exposing posix_spawn
by Pablo Galindo Salgado Jan. 9, 2018

Jan. 9, 2018

Hi, I'm currently working on exposing posix_spawn in the posix module (and by extension in the os module). You can find the initial implementation in this PR: https://github.com/python/cpython/pull/5109 As pointed out by Gregory P. Smith, some changes are needed in the way the file_actions arguments is passed from Python. For context, posix_spawn has the following declaration: int posix_spawn(pid_t *pid, const char *path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *attrp, char *const argv[], char *const envp[]); Here, file_actions is an object that represents a list of file actions (open, close or dup2) that is populated using helper functions on the C API. The question is: what is the best way to deal with this argument? Following Gregory's comment on the PR I understand that he is proposing to have three objects in the os module representing each action and pass a sequence of these objects to the Python API. What I am not sure about this is that there is no previous example of such classes in the os module for other similar APIs and therefore I am not sure if there is a better approach. Thanks you very much for your time! Pablo Galindo

7 13

PEP 568: how we could extend PEP 567 to handle generator contexts
by Nathaniel Smith Jan. 8, 2018

Jan. 8, 2018

See below for a first pass version of PEP 568, which builds on PEP 567 to add context isolation for generators. Basically, as far as features go: PEP 567 + PEP 568 = PEP 550 The actual API details are all different though. To reiterate, *this is not currently a proposal for adding to Python*. It probably will become one for 3.8, because Yury and I still think this is a good idea. But for *right now*, it's just a hypothetical "what if?" to help guide the PEP 567 discussion, because if we want to avoid accidentally ruling out the option of making an extension like this later, it helps to know how such an extension would work. There's a rendered version at: https://www.python.org/dev/peps/pep-0568 (very slightly out of date compared to the version pasted below) -n ------ PEP: 568 Title: Generator-sensitivity for Context Variables Author: Nathaniel J. Smith <njs(a)pobox.com> Status: Deferred Type: Standards Track Content-Type: text/x-rst Created: 04-Jan-2018 Python-Version: 3.8 Post-History: 2018-01-07 Abstract ======== Context variables provide a generic mechanism for tracking dynamic, context-local state, similar to thread-local storage but generalized to cope work with other kinds of thread-like contexts, such as asyncio Tasks. PEP 550 proposed a mechanism for context-local state that was also sensitive to generator context, but this was pretty complicated, so the BDFL requested it be simplified. The result was PEP 567, which is targeted for inclusion in 3.7. This PEP then extends PEP 567's machinery to add generator context sensitivity. This PEP is starting out in the "deferred" status, because there isn't enough time to give it proper consideration before the 3.7 feature freeze. The only goal *right now* is to understand what would be required to add generator context sensitivity in 3.8, so that we can avoid shipping something in 3.7 that would rule it out by accident. (Ruling it out on purpose can wait until 3.8 ;-).) Rationale ========= [Currently the point of this PEP is just to understand *how* this would work, with discussion of *whether* it's a good idea deferred until after the 3.7 feature freeze. So rationale is TBD.] High-level summary ================== Instead of holding a single ``Context``, the threadstate now holds a ``ChainMap`` of ``Context``\s. ``ContextVar.get`` and ``ContextVar.set`` are backed by the ``ChainMap``. Generators and async generators each have an associated ``Context`` that they push onto the ``ChainMap`` while they're running to isolate their context-local changes from their callers, though this can be overridden in cases like ``(a)contextlib.contextmanager`` where "leaking" context changes from the generator into its caller is desireable. Specification ============= Review of PEP 567 ----------------- Let's start by reviewing how PEP 567 works, and then in the next section we'll describe the differences. In PEP 567, a ``Context`` is a ``Mapping`` from ``ContextVar`` objects to arbitrary values. In our pseudo-code here we'll pretend that it uses a ``dict`` for backing storage. (The real implementation uses a HAMT, which is semantically equivalent to a ``dict`` but with different performance trade-offs.):: class Context(collections.abc.Mapping): def __init__(self): self._data = {} self._in_use = False def __getitem__(self, key): return self._data[key] def __iter__(self): return iter(self._data) def __len__(self): return len(self._data) At any given moment, the threadstate holds a current ``Context`` (initialized to an empty ``Context`` when the threadstate is created); we can use ``Context.run`` to temporarily switch the current ``Context``:: # Context.run def run(self, fn, *args, **kwargs): if self._in_use: raise RuntimeError("Context already in use") tstate = get_thread_state() old_context = tstate.current_context tstate.current_context = self self._in_use = True try: return fn(*args, **kwargs) finally: state.current_context = old_context self._in_use = False We can fetch a shallow copy of the current ``Context`` by calling ``copy_context``; this is commonly used when spawning a new task, so that the child task can inherit context from its parent:: def copy_context(): tstate = get_thread_state() new_context = Context() new_context._data = dict(tstate.current_context) return new_context In practice, what end users generally work with is ``ContextVar`` objects, which also provide the only way to mutate a ``Context``. They work with a utility class ``Token``, which can be used to restore a ``ContextVar`` to its previous value:: class Token: MISSING = sentinel_value() # Note: constructor is private def __init__(self, context, var, old_value): self._context = context self.var = var self.old_value = old_values # XX: PEP 567 currently makes this a method on ContextVar, but # I'm going to propose it switch to this API because it's simpler. def reset(self): # XX: should we allow token reuse? # XX: should we allow tokens to be used if the saved # context is no longer active? if self.old_value is self.MISSING: del self._context._data[self.context_var] else: self._context._data[self.context_var] = self.old_value # XX: the handling of defaults here uses the simplified proposal from # https://mail.python.org/pipermail/python-dev/2018-January/151596.html # This can be updated to whatever we settle on, it was just less # typing this way :-) class ContextVar: def __init__(self, name, *, default=None): self.name = name self.default = default def get(self): context = get_thread_state().current_context return context.get(self, self.default) def set(self, new_value): context = get_thread_state().current_context token = Token(context, self, context.get(self, Token.MISSING)) context._data[self] = new_value return token Changes from PEP 567 to this PEP -------------------------------- In general, ``Context`` remains the same. However, now instead of holding a single ``Context`` object, the threadstate stores a stack of them. This stack acts just like a ``collections.ChainMap``, so we'll use that in our pseudocode. ``Context.run`` then becomes:: # Context.run def run(self, fn, *args, **kwargs): if self._in_use: raise RuntimeError("Context already in use") tstate = get_thread_state() old_context_stack = tstate.current_context_stack tstate.current_context_stack = ChainMap([self]) # changed self._in_use = True try: return fn(*args, **kwargs) finally: state.current_context_stack = old_context_stack self._in_use = False Aside from some updated variables names (e.g., ``tstate.current_context`` → ``tstate.current_context_stack``), the only change here is on the marked line, which now wraps the context in a ``ChainMap`` before stashing it in the threadstate. We also add a ``Context.push`` method, which is almost exactly like ``Context.run``, except that it temporarily pushes the ``Context`` onto the existing stack, instead of temporarily replacing the whole stack:: # Context.push def push(self, fn, *args, **kwargs): if self._in_use: raise RuntimeError("Context already in use") tstate = get_thread_state() tstate.current_context_stack.maps.insert(0, self) # different from run self._in_use = True try: return fn(*args, **kwargs) finally: tstate.current_context_stack.maps.pop(0) # different from run self._in_use = False In most cases, we don't expect ``push`` to be used directly; instead, it will be used implicitly by generators. Specifically, every generator object and async generator object gains a new attribute ``.context``. When an (async) generator object is created, this attribute is initialized to an empty ``Context``, like: ``self.context = Context()``. This is a mutable attribute; it can be changed by user code. But it has type ``Optional[Context]`` and this is enforced: trying to set it to anything that isn't a ``Context`` object or ``None`` will raise an error. Whenever we enter an generator via ``__next__``, ``send``, ``throw``, or ``close``, or enter an async generator by calling one of those methods on its ``__anext__``, ``asend``, ``athrow``, or ``aclose`` coroutines, then its ``.context`` attribute is checked, and if non-``None``, is automatically pushed:: # GeneratorType.__next__ def __next__(self): if self.context is not None: return self.context.push(self.__real_next__) else: return self.__real_next__() While we don't expect people to use ``Context.push`` often, making it a public API preserves the principle that a generator can always be rewritten as an explicit iterator class with equivalent semantics. Also, we modify ``contextlib.(async)contextmanager`` to always set its (async) generator objects' ``.context`` attribute to ``None``:: # contextlib._GeneratorContextManagerBase.__init__ def __init__(self, func, args, kwds): self.gen = func(*args, **kwds) self.gen.context = None # added ... This makes sure that code like this continues to work as expected:: @contextmanager def decimal_precision(prec): with decimal.localcontext() as ctx: ctx.prec = prec yield with decimal_precision(2): ... The general idea here is that by default, every generator object gets its own local context, but if users want to explicitly get some other behavior then they can do that. Otherwise, things mostly work as before, except that we go through and swap everything to use the threadstate ``ChainMap`` instead of the threadstate ``Context``. In full detail: The ``copy_context`` function now returns a flattened copy of the "effective" context. (As an optimization, the implementation might choose to do this flattening lazily, but if so this will be made invisible to the user.) Compared to our previous implementation above, the only change here is that ``tstate.current_context`` has been replaced with ``tstate.current_context_stack``:: def copy_context() -> Context: tstate = get_thread_state() new_context = Context() new_context._data = dict(tstate.current_context_stack) return new_context ``Token`` is unchanged, and the changes to ``ContextVar.get`` are trivial:: # ContextVar.get def get(self): context_stack = get_thread_state().current_context_stack return context_stack.get(self, self.default) ``ContextVar.set`` is a little more interesting: instead of going through the ``ChainMap`` machinery like everything else, it always mutates the top ``Context`` in the stack, and – crucially! – sets up the returned ``Token`` to restore *its* state later. This allows us to avoid accidentally "promoting" values between different levels in the stack, as would happen if we did ``old = var.get(); ...; var.set(old)``:: # ContextVar.set def set(self, new_value): top_context = get_thread_state().current_context_stack.maps[0] # Note: top_context.get, NOT current_context_stack.get token = Token(top_context, self, top_context.get(self, Token.MISSING)) top_context._data[self] = new_value return token And finally, to allow for introspection of the full context stack, we provide a new function ``contextvars.get_context_stack``:: def get_context_stack() -> List[Context]: return list(get_thread_state().current_context_stack.maps) That's all. Comparison to PEP 550 ===================== The main difference from PEP 550 is that it reified what we're calling "contexts" and "context stacks" as two different concrete types (``LocalContext`` and ``ExecutionContext`` respectively). This led to lots of confusion about what the differences were, and which object should be used in which places. This proposal simplifies things by only reifying the ``Context``, which is "just a dict", and makes the "context stack" an unnamed feature of the interpreter's runtime state – though it is still possible to introspect it using ``get_context_stack``, for debugging and other purposes. Implementation notes ==================== ``Context`` will continue to use a HAMT-based mapping structure under the hood instead of ``dict``, since we expect that calls to ``copy_context`` are much more common than ``ContextVar.set``. In almost all cases, ``copy_context`` will find that there's only one ``Context`` in the stack (because it's rare for generators to spawn new tasks), and can simply re-use its HAMT directly, copy-on-write style; in other cases HAMTs are cheap to merge and if necessary this can be done lazily. Rather than using an actual ``ChainMap`` object, we'll represent the context stack using some appropriate structure – the most appropriate options are probably either a bare ``list`` with the "top" of the stack being the end of the list so we can use ``push``\/``pop``, or else an intrusive linked list (``PyThreadState`` → ``Context`` → ``Context`` → ...), with the "top" of the stack at the beginning of the list to allow efficient push/pop. A critical optimization in PEP 567 is the caching of values inside ``ContextVar``. Switching from a single context to a context stack makes this a little bit more complicated, but not too much. Currently, we invalidate the cache whenever the threadstate's current ``Context`` changes (on thread switch, and when entering/exiting ``Context.run``). The simplest approach here would be to invalidate the cache whenever stack changes (on thread switch, when entering/exiting ``Context.run``, and when entering/leaving ``Context.push``). The main effect of this is that iterating a generator will invalidate the cache. It seems unlikely that this will cause serious problems, but if it does, then I think it can be avoided with a cleverer cache key that recognizes that pushing and then popping a ``Context`` returns the threadstate to its previous state. (Idea: store the cache key for a particular stack configuration in the topmost ``Context``.) It seems unavoidable in this design that uncached ``get`` will be O(n), where n is the size of the context stack. However, n will generally be very small – it's roughly the number of nested generators, so usually n=1, and it will be extremely rare to see n greater than, say, 5. At worst, n is bounded by the recursion limit. In addition, we can expect that in most cases of deep generator recursion, most of the ``Context``\s in the stack will be empty, and thus can be skipped extremely quickly during lookup. And for repeated lookups the caching mechanism will kick in. So it's probably possible to construct some extreme case where this causes performance problems, but ordinary code should be essentially unaffected. Copyright ========= This document has been placed in the public domain. .. Local Variables: indent-tabs-mode: nil coding: utf-8 End: -- Nathaniel J. Smith -- https://vorpus.org

1 0

Re: [Python-Dev] subprocess not escaping "^" on Windows
by Random832 Jan. 8, 2018

Jan. 8, 2018

On Sun, Jan 7, 2018, at 19:50, Steve Dower wrote: > Considering there is no cross-platform compatibility here anyway, I > don’t think it’s that bad an option to let users do their own escaping, > especially since those who are successfully using this feature already > do. I don't really think we should give up on cross-platform compatibility that easily. There are a number of constructs supported with the same syntax by both cmd and unix shells (pipes and redirection, mainly) that people may want to use.

1 0

Whatever happened to 'nonlocal x = y'?
by Nathaniel Smith Jan. 6, 2018

Jan. 6, 2018

PEP 3104 says: """ A shorthand form is also permitted, in which nonlocal is prepended to an assignment or augmented assignment: nonlocal x = 3 The above has exactly the same meaning as nonlocal x; x = 3. (Guido supports a similar form of the global statement [24].) """ The PEP metadata says it was approved and implemented in 3.0, yet this part never seems to have been implemented: Python 3.7.0a3+ (heads/master:53f9135667, Dec 29 2017, 19:08:19) [GCC 7.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def f(): ... x = 1 ... def g(): ... nonlocal x = 2 File "<stdin>", line 4 nonlocal x = 2 ^ SyntaxError: invalid syntax Was this just an oversight, or did it get rejected at some point and no-one remembered to update that PEP? -n -- Nathaniel J. Smith -- https://vorpus.org

6 10

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?
by Random832 Jan. 6, 2018

Jan. 6, 2018

On Fri, Jan 5, 2018, at 14:07, Guido van Rossum wrote: > Yeah, but I've changed my mind on this -- I think it's needless added > complexity that helps save one line of code in very few use cases. And you > don't really think the PEP endorses `nonlocal foo += 1` do you? The PEP itself is very clear about supporting "augmented assignment", even defining the grammar differently for this case (such that you can't do it with multiple names in a single statement). The grammar shown also supports "nonlocal foo = bar = baz", though the text doesn't mention this form.

1 0