Mailman 3 December 2020 - Typing-sig

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 24 Dec '22

24 Dec '22

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

16 34

Explicit Type Aliases
by Shannon Zhu 18 Aug '22

18 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

Review of PEP 646 (Variadic Generics)
by Guido van Rossum 15 Mar '21

15 Mar '21

I have read the proposed PEP about variadic generics (PEP 646) and I like it enough that I want to sponsor it and want to help getting it over the finish line (we have to get the Steering Council to understand enough of it that they'll delegate approval to me :-). For reference, here's the PR that proposes to add PEP 646: https://github.com/python/peps/pull/1740 And here's the original Google Doc: https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… A good review starts by briefly summarizing the proposal being reviewed, so here's the proposal in my own words. **Motivation A:** We want to create generic types that take an arbitrary number of type parameters, like Tuple. For example, Tensors where each dimension is a "type". There is a demonstration of this without variadics, but it requires defining types `Tensor1[T1]`, `Tensor2[T1, T2]`, etc.: https://github.com/deepmind/tensor_annotations. We want just `Tensor[T1]`, `Tensor[T1, T2]`, etc., for any number of parameters. **Motivation B:** The type of functions like map() and zip() cannot be expressed using the existing type system. The simplest example would be the type of ``` def foo(*args): return args a = foo(42, "abc") # Should have type Tuple[int, str] ``` **Proposal:** Introduce a new kind of type variable that can be instantiated with an arbitrary number of types, some new syntax, and a new type operator: ``` Ts = TypeVarTuple("Ts") # NEW T = TypeVar("T") def f(*args: *Ts) -> Tuple[*Ts]: ... class C(Generic[*Ts]): ... Callable[[*Ts], T] Tuple[*Ts] Map[SomeType, Ts] # SomeType is a generic of one parameter ``` In most cases the form `*Ts` may be preceded and/or followed by any number of non-variadic types, e.g., `Tuple[int, int, *Ts, str]`. In cases where it's unambiguous, multiple variadic type variables are also allowed, e.g., `Tuple[*Ts1, *Ts2]`. For older Python versions, `Expand[Ts]` would mean the same as `*Ts`. So now let me go on with my (generally favorable) review. (I left many detailed editorial comments in the Google Doc -- I will not repeat those here.) I like the proposal a lot, and I am glad that we now have (apparently) a working prototype in Pyre. This has been on our wish list since at least 2016 -- much early discussion happened in https://github.com/python/typing/issues/193 and at various meetings at PyCon and at the Bay Area Typing Meetups (links in the PEP). The proposed syntax has cycled through endless variations, and I am fine with the current proposal, even though it is still slightly clunky. There's also https://github.com/python/typing/issues/513, which is specifically about array types. There are probably other motivating applications that the PEP doesn't mention, for example certain decorator types (I doubt that all of these are taken care by PEP 612, ParamSpec). I wonder why the proposal left out `Union[*Ts]`. This would seem useful, e.g. to type this function: ``` def f(*args): return random.choice(args) ``` which could be typed naturally as follows: ``` def f(*args: *Ts) -> Union[*Ts]: return random.choice(args) ``` I'm not sure that `Tensor[T1, T2, ...]` is the be-all and end-all of tensor types (e.g. where would you put the data type of numpy arrays?) but maybe that can be handled by just adding one non-variadic type variable (a complete example would be nice though). There are also proposals for integer generics, which deserve their own PEP (presumably aiming at Python 3.11). Eric Traut proposed an extension that would allow defining variadic subtypes of Sequence which behave similar to Tuple (where `Tuple[int, int]` is a subtype of `Tuple[int, ...]` which is a subtype of `Sequence[int]`), but I'm not sure we would need that a lot -- we could always add that later. The introduction of a prefix `*` operator requires new syntax in a few cases. While `Callable[[*Ts], T]` is already valid (the parser interprets this as sequence unpacking), `Tuple[*Ts]` is not, and neither is `def f(*a: *Ts)`. For `Tuple[*Ts]` we can piggy-back on PEP 637 (keyword indexing, which adds this as well), but for the `def` example we'll need to add something new specifically for this PEP. I think that's fine -- we can give it runtime semantics that turns `*Ts` into `(*Ts,)`, which is similar to the other places: at runtime it iterates over the argument, producing a tuple. In all cases we need to support `Expand[Ts]` as well for backwards compatibility with Python 3.9 and before. The `Map[]` operator is, as I said, fairly clunky. In the past various other syntaxes have been proposed. In particular, @sixolet p <https://github.com/python/typing/issues/513>roposed a syntax that would allow defining `zip()` as follows: ``` def zip(*args: Iterable[Ts]) -> Iterator[Tuple[Ts, ...]]: ... ``` Compare this to what it would look using the current proposal: ``` def zip(*args: *Map[Iterable, Ts]) -> Iterator[Ts]): ... # Note that Iterator[Ts] is the same as Iterator[Tuple[*Ts]] ``` Sixolet's syntax made the iteration over the elements of Ts implicit, which is slightly shorter, and doesn't require "higher-order type functions" (is there an official name for that?), but also slightly more cryptic, and created yet another use for the ellipsis: `Tuple[Ts, ...]` is not quite analogous to `Tuple[T, ...]`, since the latter is *homogeneous* while the former is still heterogeneous. The new notation uses an explicit `Map[]` operator, which is similar to the choice we made in PEP 612 for `Concatenate[]`. (Speaking of this choice, we could drop the `*` prefix and rely purely on `Expand[]`, but that feels unnecessarily verbose, and we'll get most of the needed syntax for free with PEP 637, assuming it's accepted.) All in all my recommendation for this PEP is: clean up the text based on the GDoc feedback, add `Union[*Ts]`, and submit to the Steering Council. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

11 80

User-defined type guards
by Eric Traut 18 Jan '21

18 Jan '21

I occasionally receive questions from pyright users about extending type narrowing to handle cases that are not possible with today’s Python type system. Typescript provides a useful facility called [user-defined type guards]( https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defin…) that allows users to extend the notion of type narrowing. It would be straightforward to add this idea to Python. Do others think this would be useful, and is there any interest in standardizing this? Here’s a concrete proposal. We could add a new generic type called `TypeGuard` in the `typing` module. It would be a subclass of bool, defined simply as: ```python class TypeGuard(bool, Generic[_T]): pass ``` If any function or method accepts one parameter and returns a `TypeGuard` type, a type checker would assume that the argument passed to this function can be narrowed to the type specified in the `TypeGuard` type argument. Here are a few examples: ```python from typing import Any, Generic, List, Tuple, TypeGuard, TypeVar _T = TypeVar("_T") def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]: return TypeGuard(len(val) == 2) def foo(names: Tuple[str, ...]): if is_two_element_tuple(names): reveal_type(names) # Tuple[str, str] def is_string_list(val: List[object]) -> TypeGuard[List[str]]: return TypeGuard(all(isinstance(x, str) for x in val)) def bar(stuff: List[Any]): if is_string_list(stuff): reveal_type(stuff) # List[str] ``` Thoughts? Suggestions? -Eric --- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.

12 59

Variadic generics PEP draft
by Matthew Rahtz 16 Dec '20

16 Dec '20

Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew

4 10

Should a single TypeVar be allowed in a function signature?
by Rebecca Chen 13 Dec '20

13 Dec '20

Hi all, I was hoping to get some opinions on whether something like the following is valid, and if so, how it should be interpreted: from typing import MutableSequence, TypeVar T = TypeVar('T') def f(seq: MutableSequence[T]) -> int: return len(seq) (This is, of course, *not* inside a generic class parameterized on `T`.) Currently, pytype emits an [invalid-annotation] error for this piece of code, with the explanation that `T` appears only once in `f`. Neither mypy nor pyre emits an error. As far as I know, how a single TypeVar should be treated is not covered in any of the typing PEPs. pytype's reasons for making this an error are: * The error messaging helps users avoid mistakes caused by misunderstanding how to use a TypeVar. * pytype's behavior when encountering a single TypeVar (especially when a bound or constraints come into play) is ill-defined, so we don't want anyone depending on it. The reasons I've seen for not making it an error are: * The TypeVar provides a way to annotate `seq` as a sequence of anything, while still allowing usage errors to be caught in the body of `f`. `Any` swallows all errors, and `object` does not work for invariant containers. * There are potential future uses for a single TypeVar. One that came up in the tensor typing call this morning is using a ListVariadic as a placeholder to represent an unused part of an annotation. * PEP 484 does not explicitly say that this should be an error, so it makes sense to err on the side of allowing it. * In spirit, `T` is an unused variable; flagging such things is not the job of a type checker. Thoughts? Best, Rebecca

4 10

Type annotation for slice
by Hao Zhang 12 Dec '20

12 Dec '20

(it is formerly discussed in https://discuss.python.org/t/is-it-a-good-practice-to-pass-tuple-of-slice-t… ) Currently we cannot write type annotation for slice like `slice[str, str]`. And it is useful when write something like `a["C1":"C2"]` , which is used in pandas already. And I also want to write something like `a[(1,2):4,(5,6):9]` which type could be `tuple[slice[tuple[int, int], int, None], ...]` So I think it would be better if slice support type annotation

3 9

Release: tensor typing stubs (+ next tensor typing meeting)
by Matthew Rahtz 09 Dec '20

09 Dec '20

Hey all, We've finally managed to open-source our library for shape annotations of tensors! It allows you to write code like import tensor_annotations.tensorflow as ttf from tensor_annotations.axes import Batch, Time def get_batch() -> ttf.Tensor2[Time, Batch]: ... def train(batch: ttf.Tensor2[Batch, Time]): ... batch = get_batch() train(batch) and run e.g. Mypy on it to detect errors like test.py:6: error: Argument 1 to "train" has incompatible type "Tensor2[Time, Batch]"; expected "Tensor2[Batch, Time]" Check it out at https://github.com/deepmind/tensor_annotations. We'd love to hear your feedback, so give it a try and let us know what you think! It would be great to discuss folks' experiences with it at the next tensor typing meeting, but even if we were to do it at the earliest next week, that's starting to feel a little too close to the holiday season. So let's wait until the new year - tentatively aiming for *Monday the 11th of January *for our next meeting. Until then - happy holidays, all! Matthew

1 0

Spelling the type of a runtime type annotation object
by David Foster 04 Dec '20

04 Dec '20

I am excited about the idea of being able to match JSON data received by Python web applications at runtime using type annotations to describe the expected shape of the data. JSON data is typically in the form of structured dictionaries (TypedDicts) containing Lists and primitives (str, float, bool, None). Consider an example web application that wants to provide a service to draw shapes, perhaps on a connected physical billboard. The service has a '/draw_shape' endpoint which takes a JSON object (a Shape TypedDict) describing a shape to draw: from bottle import HTTPResponse, request, route from typing import Literal, TypedDict, Union class Point2D(TypedDict): x: float y: float class Circle(TypedDict): type: Literal['circle'] center: Point2D # has a nested TypedDict! radius: float class Rect(TypedDict): type: Literal['rect'] x: float y: float width: float height: float Shape = Union[Circle, Rect] # a Tagged Union / Discriminated Union @route('/draw_shape') def draw_shape_endpoint() -> None: request_json = request.json # type: object if isinstance(request_json, Shape): draw_shape(request_json) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request Now, it's not actually possible in draw_shape_endpoint() to use isinstance() to perform a runtime type check against a complex type annotation object such as a Shape (which is a Union[Circle, Rect]), a Circle/Rect (which is a TypedDict subtype), or a Literal[...]. Guido mentioned in thread [1] that it would be problematic to extend/alter isinstance() to match complex type annotations for a number of reasons. Notably: * the recursive checks would make isinstance() a speed trap in certain cases, and * the subtype rules used by isinstance() aren't convenient for matching JSON when working with {int, float, bool} types. - In fact the subtype rules used by isinstance() aren't even consistent with the subtype rules used by typecheckers conforming to PEP 484 ("Type Hints") for such types, and the rules currently used by typecheckers are generally more friendly to matching JSON. So I wrote my own function trycast() [2] that can do the kind of matching that I want. With it, I can write: @route('/draw_shape') def draw_shape_endpoint() -> None: if (shape := trycast(Shape, request.json)) is not None: draw_shape(shape) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request The idea behind trycast(...) is that it is similar to the existing cast(...) function, but actually performs a runtime check to see whether its value argument actually conforms to its type annotation argument, using typechecker subtype rules from PEP 484 ("Type Hints"). trycast() works beautifully at runtime. There's just one problem: I can't actually spell the types in the function signature for trycast(...)! You might think it would be: T = TypeVar('T') def trycast(type: Type[T], value: object) -> Optional[T]: ... However typecheckers won't currently allow me to pass a runtime value like `Union[int, str]` as an argument for parameter `type` (which requires a `Type[T]`) because `isinstance(Union[int, str], type) == False`! (Jukka explains this in thread [3].) So it seems I need to be able to spell "an object that can be used where a type annotation is expected", but there's currently no syntax for that. **So I'd like to propose that Python typecheckers make it possible to spell "an object that can be used where a type annotation is expected".** Having the existence of such a spelling would make it possible to type-annotate several new kinds of valuable metaprogramming functions. Consider the following uses: * A function that can check whether an object conforms to the shape described by a type annotation. This is similar to the idea of an isinstance check, but includes cases that isinstance cannot handle, like isinstance(x, Union[int, str]) or isinstance(x, SomeTypedDict). def conforms_to(typelike: TypeAnnotation, value: object) -> bool: ... ^^^^^^^^^^^^^^ tentative spelling * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or None if it doesn't: def trycast(typelike: TypeAnnotation[T], value: object) -> Optional[T]: ... Notice that Optional here is accepting a T member that is bound by TypeAnnotation[T]. * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or return a custom sentinel if it does not: def trycast(typelike: TypeAnnotation[T], value: object, failure: F) -> Union[T, F] Notice that the Union here is accepting a T member that is bound by TypeAnnotation[T] in addition to the bound created by the F TypeVar. There are two main approaches I can see for adding the capability to spell "an object that can be used where a type annotation is expected": 1. Widen the interpretation of Type[T] to cover not just cases where `isinstance(T, type)` but also allow Type[T] to match any type annotation object (like `Union[str, int]`). 2. Introduce an alternate distinct spelling, such as `TypeAnnotation[T]` (which I use in the examples above), that is in addition to the existing spelling for `Type[T]`. - Logically `TypeAnnotation[T]` would be a supertype of `Type[T]` since all instances of `type` are (I believe) valid type annotations in their own right. **Reactions to each of these approaches? Comments on tradeoffs?** If folks think the concept of this new spelling is overall a good idea and I can get some clarity around the above questions, I'd be willing to make a stab at implementing an initial version in mypy. (If using approach 2 where a new distinct spelling is added, I'd probably opt to put it in the mypy_extensions package for the time being.) [1]: https://mail.python.org/archives/list/python-ideas@python.org/message/AZ2JB… [2]: https://github.com/davidfstr/trycast [3]: https://github.com/python/mypy/issues/9003#issuecomment-646690595 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy

3 4

Next tensor typing meeting
by Matthew Rahtz 02 Dec '20

02 Dec '20

Hey everyone, We'd delayed the last tensor typing meeting until the 7th of December, but as it turns out we still don't have anything to discuss :( I was *hoping *we'd be able to share something about the release of our tensor typing stubs, but as it turned out there were still another 2 steps of approval to get in the Google open-sourcing process, and they've been taking forever. I'll give it another week just in case, but otherwise send out an email next week to confirm when our first meeting of the new year will be. Otherwise, status updates on our end: - Still working on the PEP draft for variadic generics <https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs…>; in particular, trying out some different options for how the star syntax should work (thanks to suggestions from Luk!) and how that enables simplification of the rest of the proposal. I'm getting closer to feeling happy with the revisions, so will send around a mail soon asking for comments on an official second draft. Until next time, happy start of December! Best, Matthew

1 0

Typing-sig December 2020