Mailman 3 May 2022 - Typing-sig

PyCon 2022 Typing Summit Recap
by Shannon Zhu 10 Jul '22

10 Jul '22

Hello everyone, I’m excited to share some resources from the 2022 Typing Summit! The summit was held last week, on 4/28 at PyCon in Salt Lake City. It was in-person this year without support for live streaming, so we know a lot of you couldn’t fly out but are still interested in catching up with the topics discussed. We took some best-effort video and notes for each of the presentations and discussions, and some of the speakers have also uploaded their slides and speaker notes to share. You can find links to video and slides per topic (where available), as well as paraphrased audience-member notes and action-items here: https://docs.google.com/document/d/1_CBM9SeBrFTFULtwAzGmxlBRhqzpIArQ65cWa2W… Please thank Matthew Rahtz for personally taking and cutting video of every topic. They are linked individually in the doc above, and you can find them all in one place here: https://drive.google.com/drive/folders/1IVV0URv7_CCxiOvY6GC6adB72d5LKcvI<https://drive.google.com/drive/folders/1IVV0URv7_CCxiOvY6GC6adB72d5LKcvI> We will be uploading a full-length video of the typing summit to PyCon’s YouTube channel in the coming days. Note: Unfortunately due to battery life and other technical limitations, the video slides are hard to read, and we missed the last bit of the panel discussion. If anyone else at the summit had taken notes or wanted to share takeaways or further questions, please feel free to respond to this thread. 😊 Thanks again to all of the presenters for the work you’ve put in, and thanks to everyone who made it out in person to listen and discuss! It’s exciting to see so many typing related features already going into 3.11 and all the potential for future changes to improve the typing experience in Python. Best, Shannon & Alfonso

6 5

Almost accepting PEP 681 – Data Class Transforms
by Petr Viktorin 06 Jun '22

06 Jun '22

Hello, As an initial implementation that will be improved in the future, the specification in PEP 681 is fine. Feel free to add the decorator to Python 3.11 at your convenience. However, the PEP includes several worrying recommendations like: - we recommend that the maintainers of attrs move away from the legacy semantics and adopt auto_attribs behaviors by default. - We chose not to support this feature and recommend that attrs users avoid converters. - Attrs users should use the dataclass-standard eq and order parameter names instead. These are probably meant as recommendations from typing-sig, but an accepted PEP represents consensus of the entire Python community. A typing PEP is not an appropriate place to make recommendations like this, especially without reaching out to the maintainer of attrs. As far as I know,the attrs and pydantic libraries are using the reference implementation, but their authors weren't consulted on the PEP itself. Could you either change the wording (e.g. say that the unsupported features need bespoke type-checker functionality for proper type checking), or work with attrs to make the same recommendations in its documentation? Happy typing, — Petr, on behalf of the Steering Council

9 11

Abstract attributes for Mixins
by Thomas Kehrenberg 02 Jun '22

02 Jun '22

When working with mixins, I often wish I could specify abstract attributes – to have the contract “if the class provides these attributes, then this mixin provides this functionality.” Currently, it’s possible to write the following: class ProvidesLength(ABC): # my mixin data: list[float] def __len__(self) -> int: return len(self.data) @dataclass class A(ProvidesLength): data: list[float] = field(default_factory=list) class B(ProvidesLength): def __init__(self, data: list[float] | None = None): self.data = [] if data is None else data class C(ProvidesLength): @property def data(self) -> list[float]: # incompatible with supertype return [0.0, 1.0, 2.0] However, type checkers will not warn me if the concrete class doesn’t specify the `data` attribute in the above, because type checkers treat the type declaration `data: list[float]` like a variable definition. Ideally, there’d be some way to say that an attribute is abstract, like we can do with methods. It might look like this: from typing import ABSTRACT class ProvidesLength(ABC): data: list[float] = ABSTRACT def __len__(self) -> int: return len(self.data) but I’m not attached to any syntax. I know that I can use abstract properties to get something similar: class LengthMixin(ABC): @property @abstractmethod def data(self) -> list[float]: ... def __len__(self) -> int: return len(self.data) @dataclass class A(LengthMixin): data_: list[float] = field(default_factory=list) @property def data(self) -> list[float]: return self.data_ class B(LengthMixin): def __init__(self, data: list[float] | None = None): self._data = [] if data is None else data @property def data(self) -> list[float]: return self._data But this is just very verbose and doesn’t play that well with dataclasses (I can’t use a dataclass field to implement the abstract property). Another thing that actually works with pyright is to use a Protocol instead of an ABC: class ProvidesLength(Protocol): data: list[float] def __len__(self) -> int: return len(self.data) class A(ProvidesLength): # pyright complains about missing `data` data2: list[float] But Protocols do not feel like the appropriate concept to use for mixins. Besides, it doesn’t work in other type checkers I’ve tried (mypy and pycharm). Other possible syntaxes for abstract attributes: class Mixin(ABC): a: Abstract[list[float]] # wrapping in Abstract[] b: list[float] = ... # ellipses as value Best, Thomas

3 2

Generic type alias documentation?
by Paul Bryan 01 Jun '22

01 Jun '22

My Google-fu skills may be waning. I can't seem to find documentation for the new style generic type aliases. Example: tuple[int, ...] I can find the older-style typing.Tuple documentation. Are the new generic type aliases documented somewhere in the Python docs site?

4 5

typing.override decorator
by Shannon Zhu 31 May '22

31 May '22

Hi everyone! Pyre has received several user requests over the last few months to support @override functionality in the Python type system. I’ve compiled some of details here to see what you all think (I hope the formatting works over the mailing list): Motivation: One core functionality of the type checker is to flag when refactors or changes break pre-existing semantic structures in the code, so users can identify and make fixes across their project without doing a manual audit of their code. Unlike most other language type systems, Python does not provide a way to identify call sites that need to be changed to stay consistent when an overridden function API changes. This makes refactoring code dangerous and increases the cost of re-architectures and codemods. Take for example, a simple inheritance structure: class Base: def foo(self, input: int) -> int: return input class Child(Base): def foo(self, input: int) -> int: return input def callsite(input: Child) -> None: input.foo(1) If an overridden method on the superclass is renamed or deleted, the type checker will only identify and flag call sites that deal with objects of the base class directly, but still happily accept any call sites dealing with subclasses. class Base: def new_foo(self, input: int, flag: bool) -> int: # new semantics / side effects return input class Child(Base): # this method now silently does something fundamentally different - defines a new function unrelated to what it used to override def foo(self, input: int) -> int: return input def callsite(input: Child) -> None: input.foo(1) # no indication this API changed While this will run when executed, the type checker omits to mention the fact that a core piece of semantics have changed: the subclass method that was formerly overriding a superclass method is now, silently, defining a brand new method on the class and fooling call-sites into thinking nothing has changed with the old API. Explicitly marking a method as intending to override something in the parent would help us flag these problems in a complicated inheritance system. We’ve seen runtime bug reports usually caused by an engineer renaming a parent class method and forgetting to rename that method in all child classes. This blows up at runtime and causes some user frustration because this pattern is caught in type systems of many other languages (see below). Syntax & Semantics: A method decorated with @typing.override will throw a type error if it is not actively overriding a compatible method in a (non-Protocol) parent class. from typing import override class Base: def foo(self) -> int: return 1 def bar(self, x: str) -> str: return x class Child(Base): @override def foo(self) -> int: return 2 @override def bar(self, y: int) -> int: # type error: signature mismatch return y @override def baz() -> int: # type error: no overridden method return 1 Note: Method decorators @staticmethod and @classmethod should always be applied last - in other words, @override should wrap the method more closely (should only matter for runtime). from typing import override class Base: @staticmethod def foo() -> int: return 1 @classmethod def bar(cls, x: str) -> str: return x class Child(Base): @staticmethod @override def foo() -> int: return 2 @override def bar(self, x: str) -> str: # type error: signature mismatch return x Adoption: The proposed syntax and semantics above imply that the use of the @override decorator will be purely opt-in. Alternatively, type checkers may want to offer an option for users to flag methods that are actively overriding a method in the parent class but are missing the decorator. There are a couple ways this may be possible: 1. Class Inheritance: Denote a class as “enforcing explicit overrides” (syntax suggestions welcome) and error on any subclass methods that override parent methods but are not decorated with @override. This is what the PyPi runtime library (see below) provides via inheritance of a helper EnforceOverrides class. It would affect downstream dependencies that may subclass your logic but not be type checked together, and may be confusing 1. Everywhere: Make explicit annotation of @override a core expectation of well-annotated code, enforcing its usage in strictly typed modules across the board. This is probably too invasive. Precedent in other languages: A non-exhaustive list of other languages that provide the same feature: * Hack has <<__Override>><https://docs.hhvm.com/hack/attributes/predefined-attributes#__override> * Java has @Override<https://docs.oracle.com/javase/tutorial/java/IandI/override.html> * Scala has override<https://www.javatpoint.com/scala-method-overriding> * Kotlin has override<https://kotlinlang.org/docs/inheritance.html#overriding-methods> * Swift has override<https://docs.swift.org/swift-book/LanguageGuide/Inheritance.html#ID198> * C++ has override<https://en.cppreference.com/w/cpp/language/override> * JS/Flow discourage classes (in favor of functions) Comparison to third-party runtime enforcement library: There already exists a library<https://pypi.org/project/overrides/> that enforces @overrides (sic) and @final at runtime. PEP 591<https://peps.python.org/pep-0591/> added a final decorator to the typing module. The override component of the runtime library is not supported statically at all, which has added some confusion around the mix/matched support where the runtime library is imported and used. The ways in which the current static typing.override proposal differs from the runtime library are: 1. Proposed spelling is override rather than overrides 2. May not enforce an option where methods in classes inheriting from EnforceOverrides that are missing this decorator will throw type errors* *The runtime library provides two motivation examples for EnforceOverrides unrelated to feature adoption that I do not find motivating for the type checking use case: 1. “A method that is added to a superclass is shadowed by an existing method with the same name in a subclass.” The type checker will throw errors already here, unless the name, all parameters types and names (and arity), and return type are all perfectly compatible. That seems extremely unlikely unless the methods are in fact intended to do the same thing, which suggests this is not flagging a real error. 1. “A method of a superclass that is overridden by a subclass but the signature of the overridden method is incompatible with that of the inherited one.” Similarly, the type checker will already throw incompatible override errors to flag this. Looking for feedback on whether this would make a good addition to the typing module, as well as thoughts on some of the open questions regarding an “EnforceOverrides” equivalent or any other concerns. Thanks! Shannon

8 8

Opened and changed typing issues week 2022-W21
by Typing Bot 30 May '22

30 May '22

No issues or pull requests with the label 'topic: feature' were opened or updated last week in the typing repository on GitHub. All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Unbounded TypeVar on variable's type info
by Nicolas Haller 28 May '22

28 May '22

Hello, I have a use case that I'm not sure if this is something the current typing system could handle (and if it should). Basically, I'd like to use generics on a variable's type info to constrain the variable a bit more. My prime example would be to constrain a dict value based on the dict key, something like that: U = TypeVar("U", bound=Item) _CHECKS_TO_APPLY: dict[Type[U], list[Type[Check[U]]]] In that example, _CHECKS_TO_APPLY stores what kind of "Check" is applicable on a type "Item". The "Check" type is a hierarchy of specialized generic classes taking the type "Item" they can check as a generic parameter. I put a longer example there: https://gist.github.com/Gaasmann/beeebf526c163280cff796b842ee800a Right now, I'm not sure if what I'm doing is valid. In any case, mypy isn't happy: generic_typing.py:61: error: Type variable "generic_typing.U" is unbound generic_typing.py:61: note: (Hint: Use "Generic[U]" or "Protocol[U]" base class to bind "U" inside a class) generic_typing.py:61: note: (Hint: Use "U" in function signature to bind "U" inside a function) Do you think this use case makes sense, and if this is something the system should be able to handle? Thanks, -- Nicolas

2 2

Opened and changed typing issues week 2022-W20
by Typing Bot 23 May '22

23 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were updated last week: #182 Define a JSON type opened by @brettcannon https://github.com/python/typing/issues/182 #548 Higher-Kinded TypeVars opened by @tek https://github.com/python/typing/issues/548 #1183 Annotate sync/async code via Generic | generic TypeVar / Type aliases in Generic opened by @Bobronium https://github.com/python/typing/issues/1183 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

typing_extensions is now a separate repo
by Jelle Zijlstra 19 May '22

19 May '22

The typing_extensions module is now developed in its own repo, https://github.com/python/typing_extensions. Previously, it was part of https://github.com/python/typing, but that repo is used for several different purposes and typing_extensions stood out as unlike the others. Please watch the new repo if you're interested in typing_extensions development. The typing repo still hosts the https://github.com/python/typing/discussions forum, the https://typing.readthedocs.io/en/latest/, and issues proposing bigger changes to the type system.

1 0

Stubs: Supporting __all__ += [...]
by Sebastian Rittau 17 May '22

17 May '22

It would be useful if type checkers would support the `+=` operator for `__all__`, at least in stubs. For example, have a look at this abomination from typing.pyi: if sys.version_info >= (3, 11): __all__ = [ "Annotated", # snipped about 60 items "TypeGuard", ] elif sys.version_info >= (3, 10): __all__ = [ "Annotated", # snipped about 60 items again "TypeGuard", ] elif sys.version_info >= (3, 9): __all__ = [ # mostly the same items again ] elif sys.version_info >= (3, 8): __all__ = [ # mostly the same items again ] elif sys.version_info >= (3, 7): __all__ = [ # mostly the same items again ] else: __all__ = [ # mostly the same items again ] While this is an extreme case, adding items to modules is of course a common occurrence, and the current duplication is hard to read and error-prone. This would much better be written as: __all__ = [ # lots of items ] if sys.version_info >= (3, 7): __all__ += [SomeItem, AnotherItem] # etc. for all other versions This makes is immediately clear which items were added in which Python version. This leads to a few questions: * Do type checkers maybe already support this? * Would this be hard to implement? * How could removals be handled? - Sebastian

3 2