Mailman 3 November 2020 - Typing-sig

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 23 Dec '22

23 Dec '22

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

16 34

Explicit Type Aliases
by Shannon Zhu 17 Aug '22

17 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

User-defined type guards
by Eric Traut 18 Jan '21

18 Jan '21

I occasionally receive questions from pyright users about extending type narrowing to handle cases that are not possible with today’s Python type system. Typescript provides a useful facility called [user-defined type guards]( https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defin…) that allows users to extend the notion of type narrowing. It would be straightforward to add this idea to Python. Do others think this would be useful, and is there any interest in standardizing this? Here’s a concrete proposal. We could add a new generic type called `TypeGuard` in the `typing` module. It would be a subclass of bool, defined simply as: ```python class TypeGuard(bool, Generic[_T]): pass ``` If any function or method accepts one parameter and returns a `TypeGuard` type, a type checker would assume that the argument passed to this function can be narrowed to the type specified in the `TypeGuard` type argument. Here are a few examples: ```python from typing import Any, Generic, List, Tuple, TypeGuard, TypeVar _T = TypeVar("_T") def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]: return TypeGuard(len(val) == 2) def foo(names: Tuple[str, ...]): if is_two_element_tuple(names): reveal_type(names) # Tuple[str, str] def is_string_list(val: List[object]) -> TypeGuard[List[str]]: return TypeGuard(all(isinstance(x, str) for x in val)) def bar(stuff: List[Any]): if is_string_list(stuff): reveal_type(stuff) # List[str] ``` Thoughts? Suggestions? -Eric --- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.

12 59

Variadic generics PEP draft
by Matthew Rahtz 16 Dec '20

16 Dec '20

Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew

4 10

Should a single TypeVar be allowed in a function signature?
by Rebecca Chen 13 Dec '20

13 Dec '20

Hi all, I was hoping to get some opinions on whether something like the following is valid, and if so, how it should be interpreted: from typing import MutableSequence, TypeVar T = TypeVar('T') def f(seq: MutableSequence[T]) -> int: return len(seq) (This is, of course, *not* inside a generic class parameterized on `T`.) Currently, pytype emits an [invalid-annotation] error for this piece of code, with the explanation that `T` appears only once in `f`. Neither mypy nor pyre emits an error. As far as I know, how a single TypeVar should be treated is not covered in any of the typing PEPs. pytype's reasons for making this an error are: * The error messaging helps users avoid mistakes caused by misunderstanding how to use a TypeVar. * pytype's behavior when encountering a single TypeVar (especially when a bound or constraints come into play) is ill-defined, so we don't want anyone depending on it. The reasons I've seen for not making it an error are: * The TypeVar provides a way to annotate `seq` as a sequence of anything, while still allowing usage errors to be caught in the body of `f`. `Any` swallows all errors, and `object` does not work for invariant containers. * There are potential future uses for a single TypeVar. One that came up in the tensor typing call this morning is using a ListVariadic as a placeholder to represent an unused part of an annotation. * PEP 484 does not explicitly say that this should be an error, so it makes sense to err on the side of allowing it. * In spirit, `T` is an unused variable; flagging such things is not the job of a type checker. Thoughts? Best, Rebecca

4 10

Spelling the type of a runtime type annotation object
by David Foster 04 Dec '20

04 Dec '20

I am excited about the idea of being able to match JSON data received by Python web applications at runtime using type annotations to describe the expected shape of the data. JSON data is typically in the form of structured dictionaries (TypedDicts) containing Lists and primitives (str, float, bool, None). Consider an example web application that wants to provide a service to draw shapes, perhaps on a connected physical billboard. The service has a '/draw_shape' endpoint which takes a JSON object (a Shape TypedDict) describing a shape to draw: from bottle import HTTPResponse, request, route from typing import Literal, TypedDict, Union class Point2D(TypedDict): x: float y: float class Circle(TypedDict): type: Literal['circle'] center: Point2D # has a nested TypedDict! radius: float class Rect(TypedDict): type: Literal['rect'] x: float y: float width: float height: float Shape = Union[Circle, Rect] # a Tagged Union / Discriminated Union @route('/draw_shape') def draw_shape_endpoint() -> None: request_json = request.json # type: object if isinstance(request_json, Shape): draw_shape(request_json) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request Now, it's not actually possible in draw_shape_endpoint() to use isinstance() to perform a runtime type check against a complex type annotation object such as a Shape (which is a Union[Circle, Rect]), a Circle/Rect (which is a TypedDict subtype), or a Literal[...]. Guido mentioned in thread [1] that it would be problematic to extend/alter isinstance() to match complex type annotations for a number of reasons. Notably: * the recursive checks would make isinstance() a speed trap in certain cases, and * the subtype rules used by isinstance() aren't convenient for matching JSON when working with {int, float, bool} types. - In fact the subtype rules used by isinstance() aren't even consistent with the subtype rules used by typecheckers conforming to PEP 484 ("Type Hints") for such types, and the rules currently used by typecheckers are generally more friendly to matching JSON. So I wrote my own function trycast() [2] that can do the kind of matching that I want. With it, I can write: @route('/draw_shape') def draw_shape_endpoint() -> None: if (shape := trycast(Shape, request.json)) is not None: draw_shape(shape) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request The idea behind trycast(...) is that it is similar to the existing cast(...) function, but actually performs a runtime check to see whether its value argument actually conforms to its type annotation argument, using typechecker subtype rules from PEP 484 ("Type Hints"). trycast() works beautifully at runtime. There's just one problem: I can't actually spell the types in the function signature for trycast(...)! You might think it would be: T = TypeVar('T') def trycast(type: Type[T], value: object) -> Optional[T]: ... However typecheckers won't currently allow me to pass a runtime value like `Union[int, str]` as an argument for parameter `type` (which requires a `Type[T]`) because `isinstance(Union[int, str], type) == False`! (Jukka explains this in thread [3].) So it seems I need to be able to spell "an object that can be used where a type annotation is expected", but there's currently no syntax for that. **So I'd like to propose that Python typecheckers make it possible to spell "an object that can be used where a type annotation is expected".** Having the existence of such a spelling would make it possible to type-annotate several new kinds of valuable metaprogramming functions. Consider the following uses: * A function that can check whether an object conforms to the shape described by a type annotation. This is similar to the idea of an isinstance check, but includes cases that isinstance cannot handle, like isinstance(x, Union[int, str]) or isinstance(x, SomeTypedDict). def conforms_to(typelike: TypeAnnotation, value: object) -> bool: ... ^^^^^^^^^^^^^^ tentative spelling * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or None if it doesn't: def trycast(typelike: TypeAnnotation[T], value: object) -> Optional[T]: ... Notice that Optional here is accepting a T member that is bound by TypeAnnotation[T]. * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or return a custom sentinel if it does not: def trycast(typelike: TypeAnnotation[T], value: object, failure: F) -> Union[T, F] Notice that the Union here is accepting a T member that is bound by TypeAnnotation[T] in addition to the bound created by the F TypeVar. There are two main approaches I can see for adding the capability to spell "an object that can be used where a type annotation is expected": 1. Widen the interpretation of Type[T] to cover not just cases where `isinstance(T, type)` but also allow Type[T] to match any type annotation object (like `Union[str, int]`). 2. Introduce an alternate distinct spelling, such as `TypeAnnotation[T]` (which I use in the examples above), that is in addition to the existing spelling for `Type[T]`. - Logically `TypeAnnotation[T]` would be a supertype of `Type[T]` since all instances of `type` are (I believe) valid type annotations in their own right. **Reactions to each of these approaches? Comments on tradeoffs?** If folks think the concept of this new spelling is overall a good idea and I can get some clarity around the above questions, I'd be willing to make a stab at implementing an initial version in mypy. (If using approach 2 where a new distinct spelling is added, I'd probably opt to put it in the mypy_extensions package for the time being.) [1]: https://mail.python.org/archives/list/python-ideas@python.org/message/AZ2JB… [2]: https://github.com/davidfstr/trycast [3]: https://github.com/python/mypy/issues/9003#issuecomment-646690595 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy

3 4

Defaults for type vars
by Sebastian Rittau 26 Nov '20

26 Nov '20

This was previously discussed in https://github.com/python/typing/issues/307. It would often be useful to be able to specify a default type for type vars. A few examples: * Counter is most likely used with ints in the vast majority of cases. Being able to write "Counter" instead of "Counter[int]" makes sense to me. * Currently, when marking a previously non-generic type as generic, it will break existing type annotations. Having a default value works around this. To use Counter as an example again: Currently it is non-generic, but making it generic would break existing annotations. Adding a default of "int" would work around this problem. * Another example from the issue above: Using a default of None can prevent overloads, as in the following example: "def nullcontext(enter_result: _T = ...) -> ContextManager[_T]: ...". Without the default we need an explicit overload for the case where no argument is provided, because otherwise the type checker can't know the type of _T. While this is not a big problem in stubs, I find overloads quite obnoxious in implementations. While of course not completely comparable, I find myself using generic defaults quite often when working with TypeScript. A few examples as anectodal evidence: * We have functions that fetch JSON from an API. They are generic over the return type and default to a general JSON type, but can be overwritten if the response shape is known. (Because we control the API.) * We have a few cases, where an object is generic over a list of strings. Take a class wrapping HTML <select> elements. Sometimes all the possible options are known to the application, and have semantic meaning. (For example for choosing a view.) So it makes sense to declare a class like "class Select[Literal["foo", "bar"]]" (using Python syntax). It's then possible to have "Select.value" have the type "Literal["foo", "bar"]". In other cases, the possible options are unknown (taken from a database, for example). In that case, using "class Select[str]" would make sense. The latter is also a sensible default. * Generally I tend to use more type variables per type in TypeScript than in Python, partially due to the fact that some of them can be set to sensible defaults. While it would be obnoxious to always define four generics everytime you use a type, in typeshed I only have to define one of them with the others having sensible defaults. What do other people think about this and how would we proceed if this is deemed useful? - Sebastian

3 2

Literal for constants
by Andrew Svetlov 25 Nov '20

25 Nov '20

In Python there is a wide-spread idiom of using specific object instances as markers: MARKER = object() def f(arg) -> if arg is MARKER: do_something_special() else: process_arg_as_string(arg) When I add type hints my initial thought was decorating MARKER as constant literal: from typing import Final, Literal, Union MARKER: Final[object] = object() def f(arg: Union[str, Literal[MARKER]]) -> None: pass MARKER is declared as a final value, it cannot be changed from the typing perspective and IMHO satisfies requirements for Literal. Sure, it doesn't work with the current mypy but maybe the proposal makes sense? -- Thanks, Andrew Svetlov

4 4

Tensor typing meeting 12/10/2020
by Matthew Rahtz 25 Nov '20

25 Nov '20

Hey all, Our next tensor typing meeting will be next week. WHEN? Monday 12th October, 9:30am San Francisco Time, 5:30pm London time. WHERE? http://meet.google.com/fft-dzjq-ksu WHAT? We have a relatively sparse agenda this time. I'll lead a discussion, but if anyone else would like to present/discuss something, let me know! 1. Status updates 2. Discussion: Use of ... (Ellipsis) for unspecified parts of shapes Cheers! Matthew

3 13

Re: Tensor typing meeting 12/10/2020
by Lucio Fernandez-Arjona 20 Nov '20

20 Nov '20

hi Matthew, sorry about the access problem. it should be fixed now, please let me know if it's not: https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… I created a cheatsheet with a comparison of the star syntax and the previous version of your draft PEP: https://docs.google.com/document/d/1u5N8_x5v5CzsXf8XffR-uuM5uuYW2bZVoEZzmcA…. I will update it with the new Apply/Map version of the draft PEP. That Typed Scheme paper was an interesting one, thanks for sharing. It's good to learn from what other languages are doing. This week I looked at the tensor typing extension for Kotlin from Facebook, but unfortunately it didn't seem to contain variadics. About the new Apply function, do you think that `Apply[Tuple, Ts]` works better than `Tuple[Expand[Ts]]`? I find the latter more intuitive. Almost as having the word "expand" where the star would be. Maybe it's because I never used the old python `apply`, and I associate "apply" more with `pandas.apply` which is (without axis parameters) an element-wise application. This element-wise application I would tend to associate with `Apply[Tuple, Ts] = Tuple[T1], Tuple[T2],...`. A potential advantage of `Expand` is to allow `Tuple[int, Expand[Ts]]` which `Apply` does not allow. Cheers, Luk

1 0

Typing-sig November 2020