Mailman 3 November 2017 - Python-Dev

Comments on PEP 563 (Postponed Evaluation of Annotations)
by Mark Shannon Nov. 20, 2017

Nov. 20, 2017

Hi, Overall I am strongly in favour of this PEP. It pretty much cures all the ongoing pain of using PEP 3017 annotations for type hints. There is one thing I don't like however, and that is treating strings as if the quotes weren't there. While this seems like a superficial simplification to make transition easier, it introduces inconsistency and will ultimately make both implementing and using type hints harder. Having the treatment of strings depend on their depth in the AST seems confusing and unnecessary: "List[int]" becomes 'List[int]' # quotes removed but List["int"] becomes 'List["int"]' # quoted retained Also, T = "My unparseable annotation" def f()->T: pass would remain legal, but def f()->"My unparseable annotation" would become illegal. The change in behaviour between the above two code snippets is already confusing enough without making one of them a SyntaxError. Using annotations for purposes other than type hinting is legal and has been for quite a while. Also, PEP 484 type-hints are not the only type system in the Python ecosystem. Cython has a long history of using static type hints. For tools other than MyPy, the inconsistent quoting is onerous and will require double-quoting to prevent a parse error. For example def foo()->"unsigned int": ... will become illegal and require the cumbersome def foo()->'"unsigned int"': ... Cheers, Mark.

4 3

Show DeprecationWarning in debug mode?
by Victor Stinner Nov. 20, 2017

Nov. 20, 2017

Hi, I noticed that Python not only hides DeprecationWarning, but also PendingDeprecationWarning and ImportWarning by default. While I understand why we decided to hide these warnings to users for a Python compiled in release mode, why are they hidden in Python debug builds? I'm asking the question because in debug mode, Python shows ResourceWarning warnings (whereas these warnings are hidden in release mode). Why only displaying ResourceWarning, but not other warnings in debug mode? At least, with the Python 3.7 new "developer mode" (-X dev), now you can be totally annoyed^W^W appreciate *all* these warnings :-) Example: ------------------ $ cat x.py import warnings warnings.warn('Resource warning', ResourceWarning) warnings.warn('Deprecation warning', DeprecationWarning) # Release build: ignore all :-( $ python3 x.py # Debug build: ignore deprecation :-| $ ./python x.py x.py:2: ResourceWarning: Resource warning warnings.warn('Resource warning', ResourceWarning) # Developer mode: show all :-) $ ./python -X dev x.py x.py:2: ResourceWarning: Resource warning warnings.warn('Resource warning', ResourceWarning) x.py:3: DeprecationWarning: Deprecation warning warnings.warn('Deprecation warning', DeprecationWarning) ------------------ Or maybe we should start adding new modes like -X all-warnings-except-PendingDeprecationWarning, -X I-really-really-love-warnings and -X warnings-hater, as Barry proposed? Victor

6 7

PEP 563: Postponed Evaluation of Annotations
by Lukasz Langa Nov. 20, 2017

Nov. 20, 2017

Based on positive feedback on python-ideas back in September, I'm publishing the second draft for consideration on python-dev. I hope you like it! A nicely formatted rendering is available here: https://www.python.org/dev/peps/pep-0563/ (Just make sure you're looking at the version that has "Post-History: 1-Nov-2017" as you might have an older version cached by your browser, or you might be seeing a newer version if you get to this e-mail in the future.) - Ł PEP: 563 Title: Postponed Evaluation of Annotations Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Sep-2017 Python-Version: 3.7 Post-History: 1-Nov-2017 Resolution: Abstract ======== PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. PEP 484 introduced a standard meaning to annotations: type hints. PEP 526 defined variable annotations, explicitly tying them with the type hinting use case. This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in ``__annotations__`` in string form. This change is going to be introduced gradually, starting with a new ``__future__`` import in Python 3.7. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Just like default values, annotations are evaluated at function definition time. This creates a number of issues for the type hinting use case: * forward references: when a type hint contains names that have not been defined yet, that definition needs to be expressed as a string literal; * type hints are executed at module import time, which is not computationally free. Postponing the evaluation of annotations solves both problems. Non-goals --------- Just like in PEP 484 and PEP 526, it should be emphasized that **Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.** This PEP is meant to solve the problem of forward references in type annotations. There are still cases outside of annotations where forward references will require usage of string literals. Those are listed in a later section of this document. Annotations without forced evaluation enable opportunities to improve the syntax of type hints. This idea will require its own separate PEP and is not discussed further in this document. Non-typing usage of annotations ------------------------------- While annotations are still available for arbitrary use besides type checking, it is worth mentioning that the design of this PEP, as well as its precursors (PEP 484 and PEP 526), is predominantly motivated by the type hinting use case. With Python 3.7, PEP 484 graduates from provisional status. Other enhancements to the Python programming language like PEP 544, PEP 557, or PEP 560, are already being built on this basis as they depend on type annotations and the ``typing`` module as defined by PEP 484. With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated. Implementation ============== In Python 4.0, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective ``__annotations__`` dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. If an annotation was already a string, this string is preserved verbatim. In other cases, the string form is obtained from the AST during the compilation step, which means that the string form preserved might not preserve the exact formatting of the source. Annotations need to be syntactically valid Python expressions, also when passed as literal strings (i.e. ``compile(literal, '', 'eval')``). Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by ``typing.get_type_hints()``). Note that as per PEP 526, local variable annotations are not evaluated at all since they are not accessible outside of the function's closure. Enabling the future behavior in Python 3.7 ------------------------------------------ The functionality described above can be enabled starting from Python 3.7 using the following special import:: from __future__ import annotations Resolving Type Hints at Runtime =============================== To resolve an annotation at runtime from its string form to the result of the enclosed expression, user code needs to evaluate the string. For code that uses type hints, the ``typing.get_type_hints(obj, globalns=None, localns=None)`` function correctly evaluates expressions back from its string form. Note that all valid code currently using ``__annotations__`` should already be doing that since a type annotation can be expressed as a string literal. For code which uses annotations for other purposes, a regular ``eval(ann, globals, locals)`` call is enough to resolve the annotation. In both cases it's important to consider how globals and locals affect the postponed evaluation. An annotation is no longer evaluated at the time of definition and, more importantly, *in the same scope* it was defined. Consequently, using local state in annotations is no longer possible in general. As for globals, the module where the annotation was defined is the correct context for postponed evaluation. The ``get_type_hints()`` function automatically resolves the correct value of ``globalns`` for functions and classes. It also automatically provides the correct ``localns`` for classes. When running ``eval()``, the value of globals can be gathered in the following way: * function objects hold a reference to their respective globals in an attribute called ``__globals__``; * classes hold the name of the module they were defined in, this can be used to retrieve the respective globals:: cls_globals = vars(sys.modules[SomeClass.__module__]) Note that this needs to be repeated for base classes to evaluate all ``__annotations__``. * modules should use their own ``__dict__``. The value of ``localns`` cannot be reliably retrieved for functions because in all likelihood the stack frame at the time of the call no longer exists. For classes, ``localns`` can be composed by chaining vars of the given class and its base classes (in the method resolution order). Since slots can only be filled after the class was defined, we don't need to consult them for this purpose. Runtime annotation resolution and class decorators -------------------------------------------------- Metaclasses and class decorators that need to resolve annotations for the current class will fail for annotations that use the name of the current class. Example:: def class_decorator(cls): annotations = get_type_hints(cls) # raises NameError on 'C' print(f'Annotations for {cls}: {annotations}') return cls @class_decorator class C: singleton: 'C' = None This was already true before this PEP. The class decorator acts on the class before it's assigned a name in the current definition scope. Runtime annotation resolution and ``TYPE_CHECKING`` --------------------------------------------------- Sometimes there's code that must be seen by a type checker but should not be executed. For such situations the ``typing`` module defines a constant, ``TYPE_CHECKING``, that is considered ``True`` during type checking but ``False`` at runtime. Example:: import typing if typing.TYPE_CHECKING: import expensive_mod def a_func(arg: expensive_mod.SomeClass) -> None: a_var: expensive_mod.SomeClass = arg ... This approach is also useful when handling import cycles. Trying to resolve annotations of ``a_func`` at runtime using ``typing.get_type_hints()`` will fail since the name ``expensive_mod`` is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime). This was already true before this PEP. Backwards Compatibility ======================= This is a backwards incompatible change. Applications depending on arbitrary objects to be directly present in annotations will break if they are not using ``typing.get_type_hints()`` or ``eval()``. Annotations that depend on locals at the time of the function definition will not be resolvable later. Example:: def generate(): A = Optional[int] class C: field: A = 1 def method(self, arg: A) -> None: ... return C X = generate() Trying to resolve annotations of ``X`` later by using ``get_type_hints(X)`` will fail because ``A`` and its enclosing scope no longer exists. Python will make no attempt to disallow such annotations since they can often still be successfully statically analyzed, which is the predominant use case for annotations. Annotations using nested classes and their respective state are still valid. They can use local names or the fully qualified name. Example:: class C: field = 'c_field' def method(self, arg: C.field) -> None: # this is OK ... class D: field2 = 'd_field' def method(self, arg: C.field) -> C.D.field2: # this is OK ... def method(self, arg: field) -> D.field2: # this is OK ... In the presence of an annotation that isn't a syntactically valid expression, SyntaxError is raised at compile time. However, since names aren't resolved at that time, no attempt is made to validate whether used names are correct or not. Deprecation policy ------------------ Starting with Python 3.7, a ``__future__`` import is required to use the described functionality. No warnings are raised. In Python 3.8 a ``PendingDeprecationWarning`` is raised by the compiler in the presence of type annotations in modules without the ``__future__`` import. Starting with Python 3.9 the warning becomes a ``DeprecationWarning``. In Python 4.0 this will become the default behavior. Use of annotations incompatible with this PEP is no longer supported. Forward References ================== Deliberately using a name before it was defined in the module is called a forward reference. For the purpose of this section, we'll call any name imported or defined within a ``if TYPE_CHECKING:`` block a forward reference, too. This PEP addresses the issue of forward references in *type annotations*. The use of string literals will no longer be required in this case. However, there are APIs in the ``typing`` module that use other syntactic constructs of the language, and those will still require working around forward references with string literals. The list includes: * type definitions:: T = TypeVar('T', bound='UserId') UserId = NewType('UserId', 'SomeType') Employee = NamedTuple('Employee', [('name', str), ('id', 'UserId')]) * aliases:: Alias = Optional['SomeType'] AnotherAlias = Union['SomeType', 'OtherType'] * casting:: cast('SomeType', value) * base classes:: class C(Tuple['SomeType', 'OtherType']): ... Depending on the specific case, some of the cases listed above might be worked around by placing the usage in a ``if TYPE_CHECKING:`` block. This will not work for any code that needs to be available at runtime, notably for base classes and casting. For named tuples, using the new class definition syntax introduced in Python 3.6 solves the issue. In general, fixing the issue for *all* forward references requires changing how module instantiation is performed in Python, from the current single-pass top-down model. This would be a major change in the language and is out of scope for this PEP. Rejected Ideas ============== Keep the ability to use function local state when defining annotations ---------------------------------------------------------------------- With postponed evaluation, this would require keeping a reference to the frame in which an annotation got created. This could be achieved for example by storing all annotations as lambdas instead of strings. This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again. Note that in the case of nested classes, the functionality to get the effective "globals" and "locals" at definition time is provided by ``typing.get_type_hints()``. If a function generates a class or a function with annotations that have to use local variables, it can populate the given generated object's ``__annotations__`` dictionary directly, without relying on the compiler. Disallow local state usage for classes, too ------------------------------------------- This PEP originally proposed limiting names within annotations to only allow names from the model-level scope, including for classes. The author argued this makes name resolution unambiguous, including in cases of conflicts between local names and module-level names. This idea was ultimately rejected in case of nested classes. Instead, ``typing.get_type_hints()`` got modified to populate the local namespace correctly if class-level annotations are needed. The reasons for rejecting the idea were that it goes against the intuition of how scoping works in Python, and would break enough existing type annotations to make the transition cumbersome. Finally, local scope access is required for class decorators to be able to evaluate type annotations. This is because class decorators are applied before the class receives its name in the outer scope. Introduce a new dictionary for the string literal form instead -------------------------------------------------------------- Yury Selivanov shared the following idea: 1. Add a new special attribute to functions: ``__annotations_text__``. 2. Make ``__annotations__`` a lazy dynamic mapping, evaluating expressions from the corresponding key in ``__annotations_text__`` just-in-time. This idea is supposed to solve the backwards compatibility issue, removing the need for a new ``__future__`` import. Sadly, this is not enough. Postponed evaluation changes which state the annotation has access to. While postponed evaluation fixes the forward reference problem, it also makes it impossible to access function-level locals anymore. This alone is a source of backwards incompatibility which justifies a deprecation period. A ``__future__`` import is an obvious and explicit indicator of opting in for the new functionality. It also makes it trivial for external tools to recognize the difference between a Python files using the old or the new approach. In the former case, that tool would recognize that local state access is allowed, whereas in the latter case it would recognize that forward references are allowed. Finally, just-in-time evaluation in ``__annotations__`` is an unnecessary step if ``get_type_hints()`` is used later. Drop annotations with -O ------------------------ There are two reasons this is not satisfying for the purpose of this PEP. First, this only addresses runtime cost, not forward references, those still cannot be safely used in source code. A library maintainer would never be able to use forward references since that would force the library users to use this new hypothetical -O switch. Second, this throws the baby out with the bath water. Now *no* runtime annotation use can be performed. PEP 557 is one example of a recent development where evaluating type annotations at runtime is useful. All that being said, a granular -O option to drop annotations is a possibility in the future, as it's conceptually compatible with existing -O behavior (dropping docstrings and assert statements). This PEP does not invalidate the idea. Prior discussion ================ In PEP 484 ---------- The forward reference problem was discussed when PEP 484 was originally drafted, leading to the following statement in the document: A compromise is possible where a ``__future__`` import could enable turning *all* annotations in a given module into string literals, as follows:: from __future__ import annotations class ImSet: def add(self, a: ImSet) -> List[ImSet]: ... assert ImSet.add.__annotations__ == { 'a': 'ImSet', 'return': 'List[ImSet]' } Such a ``__future__`` import statement may be proposed in a separate PEP. python/typing#400 ----------------- The problem was discussed at length on the typing module's GitHub project, under `Issue 400 <https://github.com/python/typing/issues/400>`_. The problem statement there includes critique of generic types requiring imports from ``typing``. This tends to be confusing to beginners: Why this:: from typing import List, Set def dir(o: object = ...) -> List[str]: ... def add_friends(friends: Set[Friend]) -> None: ... But not this:: def dir(o: object = ...) -> list[str]: ... def add_friends(friends: set[Friend]) -> None ... Why this:: up_to_ten = list(range(10)) friends = set() But not this:: from typing import List, Set up_to_ten = List[int](range(10)) friends = Set[Friend]() While typing usability is an interesting problem, it is out of scope of this PEP. Specifically, any extensions of the typing syntax standardized in PEP 484 will require their own respective PEPs and approval. Issue 400 ultimately suggests postponing evaluation of annotations and keeping them as strings in ``__annotations__``, just like this PEP specifies. This idea was received well. Ivan Levkivskyi supported using the ``__future__`` import and suggested unparsing the AST in ``compile.c``. Jukka Lehtosalo pointed out that there are some cases of forward references where types are used outside of annotations and postponed evaluation will not help those. For those cases using the string literal notation would still be required. Those cases are discussed briefly in the "Forward References" section of this PEP. The biggest controversy on the issue was Guido van Rossum's concern that untokenizing annotation expressions back to their string form has no precedent in the Python programming language and feels like a hacky workaround. He said: One thing that comes to mind is that it's a very random change to the language. It might be useful to have a more compact way to indicate deferred execution of expressions (using less syntax than ``lambda:``). But why would the use case of type annotations be so all-important to change the language to do it there first (rather than proposing a more general solution), given that there's already a solution for this particular use case that requires very minimal syntax? Eventually, Ethan Smith and schollii voiced that feedback gathered during PyCon US suggests that the state of forward references needs fixing. Guido van Rossum suggested coming back to the ``__future__`` idea, pointing out that to prevent abuse, it's important for the annotations to be kept both syntactically valid and evaluating correctly at runtime. First draft discussion on python-ideas -------------------------------------- Discussion happened largely in two threads, `the original announcement <https://mail.python.org/pipermail/python-ideas/2017-September/thread.html#4…>`_ and a follow-up called `PEP 563 and expensive backwards compatibility <https://mail.python.org/pipermail/python-ideas/2017-September/thread.html#4…>`_. The PEP received rather warm feedback (4 strongly in favor, 2 in favor with concerns, 2 against). The biggest voice of concern on the former thread being Steven D'Aprano's review stating that the problem definition of the PEP doesn't justify breaking backwards compatibility. In this response Steven seemed mostly concerned about Python no longer supporting evaluation of annotations that depended on local function/class state. A few people voiced concerns that there are libraries using annotations for non-typing purposes. However, none of the named libraries would be invalidated by this PEP. They do require adapting to the new requirement to call ``eval()`` on the annotation with the correct ``globals`` and ``locals`` set. This detail about ``globals`` and ``locals`` having to be correct was picked up by a number of commenters. Nick Coghlan benchmarked turning annotations into lambdas instead of strings, sadly this proved to be much slower at runtime than the current situation. The latter thread was started by Jim J. Jewett who stressed that the ability to properly evaluate annotations is an important requirement and backwards compatibility in that regard is valuable. After some discussion he admitted that side effects in annotations are a code smell and modal support to either perform or not perform evaluation is a messy solution. His biggest concern remained loss of functionality stemming from the evaluation restrictions on global and local scope. Nick Coghlan pointed out that some of those evaluation restrictions from the PEP could be lifted by a clever implementation of an evaluation helper, which could solve self-referencing classes even in the form of a class decorator. He suggested the PEP should provide this helper function in the standard library. Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and Ivan Levkivskyi. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

19 63

Make the stable API-ABI usable
by Victor Stinner Nov. 20, 2017

Nov. 20, 2017

Hi, tl; dr I propose to extend the existing "stable API" to make it almost as complete as the current API. For example, add back PyTuple_GET_ITEM() to be stable API, but it becomes a function call rather than a macro. The final question is if it's not too late to iterate on an implementation of this idea for Python 3.7? Knowing that the stable API doesn't affect the "current API" at all, since the "new C API" (extended stable API) would only be accessible using an *opt-in* flag. Since I failed to find time to write a proper PEP (sorry about that), and the incoming deadline for Python 3.7 features is getting closer, I decided to dump my dump into this email. Sorry about the raw formatting. It's a third turn of feedback. The first one was done in python-ideas last July, the second was at the CPython sprint last September at Instagram. Both were positive. https://mail.python.org/pipermail/python-ideas/2017-July/046399.html The C API of CPython is amazing. It made CPython as powerful as we know it today. Without the C API, Python wouldn't have numpy. Without numpy, Python wouldn't be as popular as it is nowadays. Ok, enough for the introduction. The C API is awful. I hate it so much! The C API is old, error prone, too big, and expose every single CPython implementation detail... The C API is likely the first reason why faster implementation of Python (PyPy?) are not as popular as they should be. The fact that the C API is so widely used prevents many evolutions of CPython. CPython is stuck by its own (public) C API. PyPy developers spent a lot of time to make cffi great and advertize it all around the world. While it's a great project, the fact is that the C API *is* still used and the motivation to rewrite old code written with the C API to cffi, Cython or whateer else is too low. Maybe because the carrot is not big enough. Let's make a bigger carrot! (bigger than PyPy amazing performances? I'm not sure that it's doable :-( we will see) Using Cython, it's simpler to write C extensions and Cython code is more "portable" on different Python versions, since the C API evolved the last 10 years. For example, Python 3 renamed PyString with PyBytes and dropped the PyInt type. Writing a C extension working on Python 2 and Python 3 requires to "pollute" the code with many #ifdef. Cython helps to reduce them (or even avoid them? sorry, I don't know well Cython). The C *API* is tidely linked to the *ABI*. I tried to explain it with an example in my article: https://vstinner.github.io/new-python-c-api.html A known ABI issue is that it's not possible to load C extensions compiled in "release mode" on a Python compiled in "debug mode". The debug mode changes the API in a subtle way which changes the ABI and so makes C extensions incompatible. This issue prevents many people to use a Python debug build, whereas debug builds are very helpful to detect bugs earlier. Another issue is that C extensions must be recompiled for each Python release (3.5, 3.6, 3.7, etc.). Linux vendors like Red Hat cannot provide a single binary for multiple Python versions which prevents to upgade the "system" Python in the lifecycle of a distribution major version (or at least, it makes things much more complicated in term of packaging, it multiply packages for each Python version...). . . . Don't worry (be happy!), I'm not writing this email to complain, but to propose a solution. Aha! I propose to modify the API step by step to add more functions to the "stable ABI" (PEP 384) to be able to compile more and more C extensions using the "stable API" (please try to follow, first it was B, now it's P... seriously, the difference between the ABI and the API is subtle, to simplify let's say that it's the same thing, ok? :-)). I wrote a draft PEP, but I never found time to update it after the two rounds of feedbacks (python-ideas and the sprint) to write a proper PEP. So I will only give a link to my draft, sorry! https://github.com/vstinner/misc/blob/master/python/pep_c_api.rst In short, the plan is to add back the PyTuple_GET_ITEM() *API* to the "stable API" but change its implementation to a *function call* rather than the existing macro, so the compiled C extension will use a function call and so don't rely on the ABI anymore. My plan is to have two main milestones: (1) Python 3.7: Extend the *existing* opt-in "stable API" which requires to compile C extensions in a special mode. Add maybe an option in distutils to ease the compilation of a C extension with the "stable API"? (2) In Python 3.8, --if the project is successful and the performance overhead is acceptable compared the advantages of having C extensions working on multiple Python verisons--, make the "stable API (without implementation details)" the default, but add a new opt-in option to give access to the "full API (with implementation details)" for debuggers and other people who understand what they do (like Cython?). Note: currently, the "stable API" is accessible using Py_LIMITED_API define, and the "full API" is accessible using Py_BUILD_CORE define. No define gives the current C API. My problem is more on the concrete implementation: * Need to provide two different API using the same filenames (like: #include "Python.h") * Need to extend distutils to have a flag to compile a C extension with one specific API (define Py_LIMITED_API or Py_BUILD_CORE?) * Need to test many C extensions and check how many extensions are broken My plan for Python 3.7 is to not touch the current API at all. There is no risk of backward incompatibility. You should only get issues if you opt-in for the new API without implementation details. Final note: Nothing new under the sun: PyPy already implemented my "idea"! Where the idea is a C API without macros; PyTuple_GET_ITEM() is already a function call in PyPy ;-) Final question: Is it acceptable to iterate on many small changes on the C API implement this idea in Python 3.7? Maybe only write partial implementation, and finish it in Python 3.8? Victor

6 12

__future__ imports and breaking code (was: PEP 563: Postponed Evaluation of Annotations)
by Koos Zevenhoven Nov. 19, 2017

Nov. 19, 2017

Previously, I expressed some concerns about PEP 563 regarding what should happen when a string is used as an annotation. Since my point here is more general, I'm starting yet another thread. For a lot of existing type-annotated code, adding "from __tuture__ import annotations" [1] *doesn't break anything*. But that doesn't seem right. The whole point of __future__ imports is to break things. Maybe the __future__ import will not give a 100% equivalent functionality to what will be in Python 4 by default, but anyway, it's Python 4 that should break as little as possible. This leaves the breaking business to the future import, if necessary. If someone cares enough to add the future import that avoids needing string annotations for forward references, it shouldn't be such a big deal to get a warning if there's a string annotation left. But the person upgrading to Python 4 (or whatever they might be upgrading) will have a lot less motivation to figure out what went wrong. Then again, code that works in both Python 3 and 4 could still have the future import. But that would defeat the purpose of Python 4 as a clean and high-performance dynamic language. —Koos [1] As defined in the PEP 563 draft: https://mail.python.org/pipermail/python-dev/2017-November/150062.html -- + Koos Zevenhoven + http://twitter.com/k7hoven +

1 0

unittest isolation and warnings
by Christian Tismer Nov. 18, 2017

Nov. 18, 2017

Hi guys, when writing tests, I suddenly discovered that unittest is not isolated to warnings. Example: One of my tests emits warnings when a certain condition is met. Instead of reporting the error immediately, it uses warnings, and at the end of the test, an error is produced if there were warnings. if hasattr(__main__, "__warningregistry__"): raise RuntimeError("There are errors, see above.") By chance, I discovered that an error was suddenly triggered without a warning. That must mean the warning existed already from another test as a left-over. My question: Is that known, and is that intended? To what extent are the test cases isolated from each other? I do admit that my usage of warnings is somewhat special. But it is very convenient to report many errors on remote servers. Cheers -- Chris -- Christian Tismer :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

3 3

Clarify the compatibility policy of lib2to3.
by Dong-hee Na Nov. 17, 2017

Nov. 17, 2017

Hi, Few days ago, I submitted a patch(https://github.com/python/cpython/pull/4417) which updates2to3 converts `operator.isCallable(obj)` to `isinstance(obj, collections.abc.Callable)`. This was Serhiy Storchaka’s idea(https://bugs.python.org/issue32046) and I agree with his idea since `callable` is not available in all 3.x versions. However, some people would like to clarify the policy of 2to3. That means that 2to3 is compatible with the branch that is currently maintained? or it is converted to specific Python 3.X codes with the specific version of 2to3? Cordially, Dong-hee — Dong-hee Na Chungnam National University | Computer Science & Engineering Tel: +82 010-3353-9127 Email: donghee.na92(a)gmail.com Linkedin: https://www.linkedin.com/in/dong-hee-na-2b713b49/ Github: https://github.com/corona10

2 1

Python possible vulnerabilities in concurrency
by Stephen Michell Nov. 17, 2017

Nov. 17, 2017

I am looking for one or two experts to discuss with me how Python concurrency features fit together, and possible vulnerabilities associated with that. TR 24772 lists 5 vulnerabilities associated with 1. activating threads, tasks or pico-threads 2. Directed termination of threads, tasks or pico-threads 3. Premature termination of threads, tasks or pico-threads 4. Concurrent access to data shared between threads, tasks or pico-threads, and 5. Lock protocol errors for concurrent entities I need to document how these appear (or don’t appear) in Python. The writeups would possibly swamp this email reflector, so I am looking for a small number of people to review these sections of our language-independent document and discuss with me how these are handled in Python. I have a good background in these issues, but no relevant experience with Python. Please contact me at stephen.michell(a)maurya.on.ca <mailto:stephen.michell@maurya.on.ca> to respond directly. Thank you …stephen michell Convenor ISO/IEC/JTC 1/SC 22/WG 23 Programming Language Vulnerabilities Working Group

9 10

Summary of Python tracker Issues
by Python tracker Nov. 17, 2017

Nov. 17, 2017

ACTIVITY SUMMARY (2017-11-10 - 2017-11-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6261 (+14) closed 37553 (+46) total 43814 (+60) Open issues with patches: 2409 Issues opened (39) ================== #27494: 2to3 parser failure caused by a comma after a generator expres https://bugs.python.org/issue27494 reopened by serhiy.storchaka #31963: AMD64 Debian PGO 3.x buildbot: compilation failed with an inte https://bugs.python.org/issue31963 reopened by vstinner #32004: Allow specifying code packing order in audioop adpcm functions https://bugs.python.org/issue32004 opened by MosesofEgypt #32005: multiprocessing.Array misleading error message in slice assign https://bugs.python.org/issue32005 opened by steven.daprano #32006: multiprocessing.Array 'c' code is not documented https://bugs.python.org/issue32006 opened by steven.daprano #32007: nis module fails to build against glibc-2.26 https://bugs.python.org/issue32007 opened by floppymaster #32008: Example suggest to use a TLSv1 socket https://bugs.python.org/issue32008 opened by kroeckx #32014: multiprocessing Server's shutdown method useless send message https://bugs.python.org/issue32014 opened by stevezh #32016: Python 3.6.3 venv FAILURE https://bugs.python.org/issue32016 opened by nihon2000 #32017: profile.Profile() has no method enable() https://bugs.python.org/issue32017 opened by pitrou #32019: Interactive shell doesn't work with readline bracketed paste https://bugs.python.org/issue32019 opened by Aaron.Meurer #32021: Brotli encoding is not recognized by mimetypes https://bugs.python.org/issue32021 opened by Andrey #32022: Python problem - == RESTART: Shell ===== https://bugs.python.org/issue32022 opened by shamon51 #32024: Nominal decorator function call syntax is inconsistent with re https://bugs.python.org/issue32024 opened by ncoghlan #32025: Add time.thread_time() https://bugs.python.org/issue32025 opened by pitrou #32026: Memory leaks in Python on Windows https://bugs.python.org/issue32026 opened by pjna #32028: Syntactically wrong suggestions by the new custom print statem https://bugs.python.org/issue32028 opened by mdraw #32030: PEP 432: Rewrite Py_Main() https://bugs.python.org/issue32030 opened by vstinner #32031: Do not use the canonical path in pydoc test_mixed_case_module_ https://bugs.python.org/issue32031 opened by xdegaye #32033: The pwd module implementation incorrectly sets some attributes https://bugs.python.org/issue32033 opened by xdegaye #32035: Documentation of zipfile.ZipFile().writestr() fails to mention https://bugs.python.org/issue32035 opened by Daniel5148 #32038: Add API to intercept socket.close() https://bugs.python.org/issue32038 opened by yselivanov #32039: timeit documentation should describe caveats https://bugs.python.org/issue32039 opened by barry #32041: Cannot cast '\0' to c_void_p https://bugs.python.org/issue32041 opened by Ilya.Kulakov #32042: Option for comparing values instead of reprs in doctest https://bugs.python.org/issue32042 opened by Tomáš Petříček #32043: Add a new -X dev option: "developer mode" https://bugs.python.org/issue32043 opened by vstinner #32045: Does json.dumps have a memory leak? https://bugs.python.org/issue32045 opened by rohandsa #32046: 2to3 fix for operator.isCallable() https://bugs.python.org/issue32046 opened by serhiy.storchaka #32047: asyncio: enable debug mode when -X dev is used https://bugs.python.org/issue32047 opened by vstinner #32049: 2.7.14 does not uninstall cleanly if installation was run as S https://bugs.python.org/issue32049 opened by niemalsnever #32050: Deprecated python3 -x option https://bugs.python.org/issue32050 opened by vstinner #32051: Possible issue in multiprocessing doc https://bugs.python.org/issue32051 opened by 1a1a11a #32052: Provide access to buffer of asyncio.StreamReader https://bugs.python.org/issue32052 opened by Bruce Merry #32054: Creating RPM on Python 2 works, but Python 3 fails because of https://bugs.python.org/issue32054 opened by pgacv2 #32055: Reconsider comparison chaining for containment tests https://bugs.python.org/issue32055 opened by ncoghlan #32056: bug in Lib/wave.py https://bugs.python.org/issue32056 opened by BT123 #32059: detect_modules() in setup.py must also search the sysroot path https://bugs.python.org/issue32059 opened by xdegaye #32060: Should an ABC fail if no abstract methods are defined? https://bugs.python.org/issue32060 opened by Alex Corcoles #32063: test_multiprocessing_forkserver failed with OSError: [Errno 48 https://bugs.python.org/issue32063 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #32063: test_multiprocessing_forkserver failed with OSError: [Errno 48 https://bugs.python.org/issue32063 #32056: bug in Lib/wave.py https://bugs.python.org/issue32056 #32054: Creating RPM on Python 2 works, but Python 3 fails because of https://bugs.python.org/issue32054 #32052: Provide access to buffer of asyncio.StreamReader https://bugs.python.org/issue32052 #32049: 2.7.14 does not uninstall cleanly if installation was run as S https://bugs.python.org/issue32049 #32046: 2to3 fix for operator.isCallable() https://bugs.python.org/issue32046 #32035: Documentation of zipfile.ZipFile().writestr() fails to mention https://bugs.python.org/issue32035 #32019: Interactive shell doesn't work with readline bracketed paste https://bugs.python.org/issue32019 #32017: profile.Profile() has no method enable() https://bugs.python.org/issue32017 #32016: Python 3.6.3 venv FAILURE https://bugs.python.org/issue32016 #32014: multiprocessing Server's shutdown method useless send message https://bugs.python.org/issue32014 #32008: Example suggest to use a TLSv1 socket https://bugs.python.org/issue32008 #32006: multiprocessing.Array 'c' code is not documented https://bugs.python.org/issue32006 #32005: multiprocessing.Array misleading error message in slice assign https://bugs.python.org/issue32005 #32004: Allow specifying code packing order in audioop adpcm functions https://bugs.python.org/issue32004 Most recent 15 issues waiting for review (15) ============================================= #32056: bug in Lib/wave.py https://bugs.python.org/issue32056 #32050: Deprecated python3 -x option https://bugs.python.org/issue32050 #32047: asyncio: enable debug mode when -X dev is used https://bugs.python.org/issue32047 #32046: 2to3 fix for operator.isCallable() https://bugs.python.org/issue32046 #32043: Add a new -X dev option: "developer mode" https://bugs.python.org/issue32043 #32031: Do not use the canonical path in pydoc test_mixed_case_module_ https://bugs.python.org/issue32031 #32030: PEP 432: Rewrite Py_Main() https://bugs.python.org/issue32030 #32025: Add time.thread_time() https://bugs.python.org/issue32025 #32002: test_c_locale_coercion fails when the default LC_CTYPE != "C" https://bugs.python.org/issue32002 #31997: SSL lib does not handle trailing dot (period) in hostname or c https://bugs.python.org/issue31997 #31993: pickle.dump allocates unnecessary temporary bytes / str https://bugs.python.org/issue31993 #31985: Deprecate openfp() in aifc, sunau and wave https://bugs.python.org/issue31985 #31978: make it simpler to round fractions https://bugs.python.org/issue31978 #31971: idle_test: failures on x86 Windows7 3.x https://bugs.python.org/issue31971 #31968: exec(): method's default arguments from dict-inherited globals https://bugs.python.org/issue31968 Top 10 most discussed issues (10) ================================= #32038: Add API to intercept socket.close() https://bugs.python.org/issue32038 19 msgs #31975: Add a default filter for DeprecationWarning in __main__ https://bugs.python.org/issue31975 18 msgs #31993: pickle.dump allocates unnecessary temporary bytes / str https://bugs.python.org/issue31993 10 msgs #32021: Brotli encoding is not recognized by mimetypes https://bugs.python.org/issue32021 10 msgs #31701: faulthandler dumps 'Windows fatal exception: code 0xe06d7363' https://bugs.python.org/issue31701 8 msgs #32025: Add time.thread_time() https://bugs.python.org/issue32025 7 msgs #32030: PEP 432: Rewrite Py_Main() https://bugs.python.org/issue32030 7 msgs #32050: Deprecated python3 -x option https://bugs.python.org/issue32050 6 msgs #31356: Add context manager to temporarily disable GC https://bugs.python.org/issue31356 5 msgs #32022: Python problem - == RESTART: Shell ===== https://bugs.python.org/issue32022 5 msgs Issues closed (46) ================== #15606: re.VERBOSE whitespace behavior not completely documented https://bugs.python.org/issue15606 closed by serhiy.storchaka #24555: Python logic error when deal with re and muti-threading https://bugs.python.org/issue24555 closed by serhiy.storchaka #24896: It is undocumented that re.UNICODE and re.LOCALE affect re.IGN https://bugs.python.org/issue24896 closed by serhiy.storchaka #29180: skip tests that raise PermissionError in test_os (non-root use https://bugs.python.org/issue29180 closed by xdegaye #29181: skip tests that raise PermissionError in test_tarfile (non-roo https://bugs.python.org/issue29181 closed by xdegaye #30133: Strings that end with properly escaped backslashes cause error https://bugs.python.org/issue30133 closed by serhiy.storchaka #30143: Using collections ABC from collections.abc rather than collect https://bugs.python.org/issue30143 closed by serhiy.storchaka #30148: Pathological regex behaviour https://bugs.python.org/issue30148 closed by serhiy.storchaka #30349: Preparation for advanced set syntax in regular expressions https://bugs.python.org/issue30349 closed by serhiy.storchaka #30399: Get rid of trailing comma in the repr() of BaseException https://bugs.python.org/issue30399 closed by serhiy.storchaka #30696: infinite loop in PyRun_InteractiveLoopFlags() https://bugs.python.org/issue30696 closed by xdegaye #30950: Convert round() to Arument Clinic https://bugs.python.org/issue30950 closed by serhiy.storchaka #31691: Include missing info on required build steps and how to build https://bugs.python.org/issue31691 closed by steve.dower #31702: Allow to specify the number of rounds for SHA-* hashing in cry https://bugs.python.org/issue31702 closed by serhiy.storchaka #31824: Missing default argument detail in documentation of StreamRead https://bugs.python.org/issue31824 closed by berker.peksag #31867: Duplicated keys in MIME type_map with different values https://bugs.python.org/issue31867 closed by asvetlov #31948: [EASY] Broken MSDN links in msilib docs https://bugs.python.org/issue31948 closed by Mariatta #31949: Bugs in PyTraceBack_Print() https://bugs.python.org/issue31949 closed by serhiy.storchaka #31976: Segfault when closing BufferedWriter from a different thread https://bugs.python.org/issue31976 closed by pitrou #31979: Simplify converting non-ASCII strings to int, float and comple https://bugs.python.org/issue31979 closed by serhiy.storchaka #31994: json encoder exception could be better https://bugs.python.org/issue31994 closed by serhiy.storchaka #31995: Set operations documentation error https://bugs.python.org/issue31995 closed by rhettinger #32001: @lru_cache needs to be called with () https://bugs.python.org/issue32001 closed by rhettinger #32009: seg fault when using Cntrl-q keymap to exit app https://bugs.python.org/issue32009 closed by martin.panter #32010: Multiple get "itemgetter" https://bugs.python.org/issue32010 closed by rhettinger #32011: Restore loading of TYPE_INT64 in marshal https://bugs.python.org/issue32011 closed by serhiy.storchaka #32012: Disallow ambiguous syntax f(x for x in [1],) https://bugs.python.org/issue32012 closed by ncoghlan #32013: _pickle: Py_DECREF seems to be missing from a failure case in https://bugs.python.org/issue32013 closed by serhiy.storchaka #32015: Asyncio looping during simultaneously socket read/write and re https://bugs.python.org/issue32015 closed by asvetlov #32018: inspect.signature does not respect PEP 8 https://bugs.python.org/issue32018 closed by yselivanov #32020: arraymodule: Missing Py_DECREF in failure case of make_array() https://bugs.python.org/issue32020 closed by serhiy.storchaka #32023: Always require parentheses for genexps in base class lists https://bugs.python.org/issue32023 closed by serhiy.storchaka #32027: argparse allow_abbrev option also controls short flag combinat https://bugs.python.org/issue32027 closed by berker.peksag #32029: cgi: TypeError when no argument string is found https://bugs.python.org/issue32029 closed by berker.peksag #32032: Module-level pickle tests test only default implementation https://bugs.python.org/issue32032 closed by serhiy.storchaka #32034: Error when unpickling IncompleteReadError https://bugs.python.org/issue32034 closed by yselivanov #32036: error mixing positional and non-positional arguments with `arg https://bugs.python.org/issue32036 closed by r.david.murray #32037: Pickle 32-bit integers with protocol 0 as INT instead of LONG https://bugs.python.org/issue32037 closed by serhiy.storchaka #32040: Sorting pahtlib.Paths does give the same order as sorting the https://bugs.python.org/issue32040 closed by r.david.murray #32044: pip3 install 3.6.3 crashes on startup; 3.5.4 works; on OS X 10 https://bugs.python.org/issue32044 closed by berker.peksag #32048: Misprint in the unittest.mock documentation. https://bugs.python.org/issue32048 closed by Артемий Родионов #32053: Inconsistent use of tabs and spaces in indentation not always https://bugs.python.org/issue32053 closed by skrah #32057: time.sleep(n) interrupted by signal https://bugs.python.org/issue32057 closed by Bogdan Popa #32058: Faulty behaviour in email.utils.parseaddr if square brackets i https://bugs.python.org/issue32058 closed by r.david.murray #32061: test_httpservers.test_undecodable_filename() fails with [Errno https://bugs.python.org/issue32061 closed by vstinner #32062: test_venv fails if zlib is not available https://bugs.python.org/issue32062 closed by vstinner

1 0

PEP 560: bases classes / confusion
by Jim J. Jewett Nov. 16, 2017

Nov. 16, 2017

(1) I found the following (particularly "bases classes") very confusing: """ If an object that is not a class object appears in the bases of a class definition, then ``__mro_entries__`` is searched on it. If found, it is called with the original tuple of bases as an argument. The result of the call must be a tuple, that is unpacked in the bases classes in place of this object. (If the tuple is empty, this means that the original bases is simply discarded.) """ Based on the following GenericAlias/NewList/Tokens example, I think I now I understand what you mean, and would have had somewhat less difficulty if it were expressed as: """ When an object that is not a class object appears in the (tuple of) bases of a class definition, then attribute ``__mro_entries__`` is searched on that non-class object. If ``__mro_entries__`` found, it is called with the entire original tuple of bases as an argument. The result of the call must be a tuple, which is unpacked and replaces only the non-class object in the tuple of bases. (If the tuple is empty, this means that the original bases is simply discarded.) """ Note that this makes some assumptions about the __mro_entries__ signature that I wasn't quite sure about from the example. So building on that: class ABList(A, NewList[int], B): I *think* the following will happen: "NewList[int]" will be evaluated, and __class_getitem__ called, so that the bases tuple will be (A, GenericAlias(NewList, int), B) # (A) I *think* __mro_entries__ gets called with the full tuple, # instead of just the object it is found on. # (B) I *think* it is called on the results of evaluating # the terms within the tuple, instead of the original # string representation. _tmp = __mro_entries__(A, GenericAlias(NewList, int), B) # (C) I *think* __mro_entries__ returns a replacement for # just the single object, even though it was called on # the whole tuple, without knowing which object it # represents. bases = (A, _tmp, B) # (D) If there are two non-class objects, I *think* the # second one gets the same arguments as the first, # rather than an intermediate tuple with the first such # object already substituted out. -jJ

6 9