Mailman 3 January 2021 - Typing-sig

Runtime Access to Type Parameters
by Saul Shanabrook 07 Jun '23

07 Jun '23

I would like to propose being able to access, at runtime, the type parameters in classmethods and in constructors. I wrote up a draft PEP on this to capture the thinking here: https://github.com/saulshanabrook/peps/blob/master/pep-9999.md It seems easier to add them to classmethods, so even that addition alone would be very welcome! It could be added to Guido's existing draft PR for pep 585, by making a small change in the getattr function (https://github.com/python/cpython/pull/18239/files#diff-a34ae826f869897e56e…), to first do a `__getattribute__` and check if the result is a descriptor (has a `__get__` attribute), and if so passing the generic class itself in as the class, instead of the origin class. I wrote up a little example of that in the PEP. To make it available in the constructor, I proposed adding a global function `get_parameters` to `typing` which returns the parameters in the constructor. I also provide a little mock implementation of that. Note that work is only for custom classes which inherit from `typing.Generic` not for any built in collections.

5 6

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 24 Dec '22

24 Dec '22

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

16 34

Explicit Type Aliases
by Shannon Zhu 18 Aug '22

18 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

Review of PEP 646 (Variadic Generics)
by Guido van Rossum 15 Mar '21

15 Mar '21

I have read the proposed PEP about variadic generics (PEP 646) and I like it enough that I want to sponsor it and want to help getting it over the finish line (we have to get the Steering Council to understand enough of it that they'll delegate approval to me :-). For reference, here's the PR that proposes to add PEP 646: https://github.com/python/peps/pull/1740 And here's the original Google Doc: https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs… A good review starts by briefly summarizing the proposal being reviewed, so here's the proposal in my own words. **Motivation A:** We want to create generic types that take an arbitrary number of type parameters, like Tuple. For example, Tensors where each dimension is a "type". There is a demonstration of this without variadics, but it requires defining types `Tensor1[T1]`, `Tensor2[T1, T2]`, etc.: https://github.com/deepmind/tensor_annotations. We want just `Tensor[T1]`, `Tensor[T1, T2]`, etc., for any number of parameters. **Motivation B:** The type of functions like map() and zip() cannot be expressed using the existing type system. The simplest example would be the type of ``` def foo(*args): return args a = foo(42, "abc") # Should have type Tuple[int, str] ``` **Proposal:** Introduce a new kind of type variable that can be instantiated with an arbitrary number of types, some new syntax, and a new type operator: ``` Ts = TypeVarTuple("Ts") # NEW T = TypeVar("T") def f(*args: *Ts) -> Tuple[*Ts]: ... class C(Generic[*Ts]): ... Callable[[*Ts], T] Tuple[*Ts] Map[SomeType, Ts] # SomeType is a generic of one parameter ``` In most cases the form `*Ts` may be preceded and/or followed by any number of non-variadic types, e.g., `Tuple[int, int, *Ts, str]`. In cases where it's unambiguous, multiple variadic type variables are also allowed, e.g., `Tuple[*Ts1, *Ts2]`. For older Python versions, `Expand[Ts]` would mean the same as `*Ts`. So now let me go on with my (generally favorable) review. (I left many detailed editorial comments in the Google Doc -- I will not repeat those here.) I like the proposal a lot, and I am glad that we now have (apparently) a working prototype in Pyre. This has been on our wish list since at least 2016 -- much early discussion happened in https://github.com/python/typing/issues/193 and at various meetings at PyCon and at the Bay Area Typing Meetups (links in the PEP). The proposed syntax has cycled through endless variations, and I am fine with the current proposal, even though it is still slightly clunky. There's also https://github.com/python/typing/issues/513, which is specifically about array types. There are probably other motivating applications that the PEP doesn't mention, for example certain decorator types (I doubt that all of these are taken care by PEP 612, ParamSpec). I wonder why the proposal left out `Union[*Ts]`. This would seem useful, e.g. to type this function: ``` def f(*args): return random.choice(args) ``` which could be typed naturally as follows: ``` def f(*args: *Ts) -> Union[*Ts]: return random.choice(args) ``` I'm not sure that `Tensor[T1, T2, ...]` is the be-all and end-all of tensor types (e.g. where would you put the data type of numpy arrays?) but maybe that can be handled by just adding one non-variadic type variable (a complete example would be nice though). There are also proposals for integer generics, which deserve their own PEP (presumably aiming at Python 3.11). Eric Traut proposed an extension that would allow defining variadic subtypes of Sequence which behave similar to Tuple (where `Tuple[int, int]` is a subtype of `Tuple[int, ...]` which is a subtype of `Sequence[int]`), but I'm not sure we would need that a lot -- we could always add that later. The introduction of a prefix `*` operator requires new syntax in a few cases. While `Callable[[*Ts], T]` is already valid (the parser interprets this as sequence unpacking), `Tuple[*Ts]` is not, and neither is `def f(*a: *Ts)`. For `Tuple[*Ts]` we can piggy-back on PEP 637 (keyword indexing, which adds this as well), but for the `def` example we'll need to add something new specifically for this PEP. I think that's fine -- we can give it runtime semantics that turns `*Ts` into `(*Ts,)`, which is similar to the other places: at runtime it iterates over the argument, producing a tuple. In all cases we need to support `Expand[Ts]` as well for backwards compatibility with Python 3.9 and before. The `Map[]` operator is, as I said, fairly clunky. In the past various other syntaxes have been proposed. In particular, @sixolet p <https://github.com/python/typing/issues/513>roposed a syntax that would allow defining `zip()` as follows: ``` def zip(*args: Iterable[Ts]) -> Iterator[Tuple[Ts, ...]]: ... ``` Compare this to what it would look using the current proposal: ``` def zip(*args: *Map[Iterable, Ts]) -> Iterator[Ts]): ... # Note that Iterator[Ts] is the same as Iterator[Tuple[*Ts]] ``` Sixolet's syntax made the iteration over the elements of Ts implicit, which is slightly shorter, and doesn't require "higher-order type functions" (is there an official name for that?), but also slightly more cryptic, and created yet another use for the ellipsis: `Tuple[Ts, ...]` is not quite analogous to `Tuple[T, ...]`, since the latter is *homogeneous* while the former is still heterogeneous. The new notation uses an explicit `Map[]` operator, which is similar to the choice we made in PEP 612 for `Concatenate[]`. (Speaking of this choice, we could drop the `*` prefix and rely purely on `Expand[]`, but that feels unnecessarily verbose, and we'll get most of the needed syntax for free with PEP 637, assuming it's accepted.) All in all my recommendation for this PEP is: clean up the text based on the GDoc feedback, add `Union[*Ts]`, and submit to the Steering Council. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

11 80

Alternative notation for Callable
by mark00bell＠gmail.com 07 Mar '21

07 Mar '21

PEP 604 introduced a new notation for specifying Unions, with `X | Y` being an alias for `typing.Union[X, Y]`. While thinking about other notations that could be useful, I would like to suggest also having `[ArgType1, ArgType2, ArgType3] -> ReturnType` as an alternative for `typing.Callable[[ArgType1, ArgType2, ArgType3], ReturnType]` This would then mean that, for example, we could annotate the `map` function as: `def map(func: [X] -> Y, items: Iterable[X]) -> Iterable[Y]:` Which would make this very similar to the annotation used to define a function that can be passed into `map`: ` def f(x: X) -> Y:` What do people think of this proposal? (I originally submitted this https://discuss.python.org/t/additional-type-annotation-for-callable/6800 here but was told that this mailing list may be more appropriate, please let me know if there is somewhere more appropriate to ask this)

5 4

TypeForm: Type Hint for Typing Special Forms and Regular Types
by David Foster 05 Mar '21

05 Mar '21

I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email: https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQ… Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically. For further background on why this feature is being introduced and prior discussions, please see: (1) the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICT… (2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david(a)dafoster.net> Sponsor: TODO Discussions-To: typing-sig(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` And here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns `True`: ``` def isassignable(value: object, form: TypeForm) -> bool: ... ``` With the introduction of `TypeGuard` in PEP 647[^TypeGuardPep] the above function can be enhanced to return a `TypeGuard` instead of a regular `bool`: [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` Without `TypeForm` the next-best type that could be given to the `form` parameter in the above examples would be `object` which would improperly allow values like `1` to be passed in. More importantly, there would be no way to express the relationship between the parameter type of `form` and the function's return type. NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICT… Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is a type variable or as `TypeForm` with no argument. The `T` in `TypeForm[T]` is always an invariant type variable with no bound: * Attempting to use a type variable `T` declared with a `bound=...` or marked with `covariant=True` or `contravariant=True` in a `TypeForm[T]` should be rejected by typecheckers. * Attempting to use a literal type-form `*form*` as an argument to `TypeForm[*form*]`, such as `TypeForm[int]`, should be rejected by typecheckers. (See §"TypeForm[*form*] or Literal[*form*]" in Rejected Ideas for rationale.) The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[_T]` where `_T` is a freshly created type variable. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: TypeForm) -> bool: ... def is_type(form: TypeForm) -> bool: ... ``` Using TypeForm Types -------------------- Type-form types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') U = TypeVar('U') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm[U]]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to infer a TypeForm variable type for a bare assignment, use the walrus operator instead of the assignment operator: ``` NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm ``` Or explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- A particular literal value is said to *inhabit* a type if it is described by that type and can be assigned to a variable of that type. The type `TypeForm` is inhabited by exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` *and* as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is not valid in one of the above locations is not an inhabitant of `TypeForm`. For example the special forms `Final`, `Final[int]`, `NoReturn`, and `ClassVar[int]` are not inhabitants. Incomplete forms like a bare `Optional` or `Union` are also not inhabitants. Example of inhabitants include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]` or `Union[int, str]` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms A few peculiar cases permitted by the above include: ``` STR_FORM: TypeForm = str STR_OR_SUBCLASS_FORM: TypeForm = Type[str] SOME_FORM: TypeForm = TypeForm ``` Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. The latter are inhabitants of `TypeForm`. Additionally the "top-level" form for a particular variable type can itself be a forward reference such as in: ``` class Node: value: object next: 'Node' # 'Node' is normalized to ForwardRef('Node') at runtime ``` Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) subcomponents. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `Type[C]` is consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type variable. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Overloading ----------- `TypeForm` can be used in overloads. For example: ``` T = TypeVar('T') F = TypeVar('F') @overload def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... @overload def trycast(form: TypeForm[T], value: object, failure: F) -> Union[T, F]: ... ``` Interactions with Type[C] ------------------------- A `TypeForm` and a `Type` cannot simultaneously constrain the same type variable within a generic function definition: ``` def bad_func(form: TypeForm[T], tp: Type[T]): ... # invalid ``` A group of type variables where some variables are constrained by `TypeForm`s, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Interactions with cast ---------------------- It is possible to cast a value to a `TypeForm` or `TypeForm[T]` type: ``` OPTIONAL_STR_OBJECT: object = Optional[str] OPTIONAL_STR = cast(TypeForm, OPTIONAL_STR_OBJECT) ``` Type Inference -------------- A runtime object corresponding to a special form should be inferred as having type `TypeForm` unless it can instead be inferred to have type `Type[C]` for some `C`: ``` T = TypeVar('T') def identity(value: T) -> T: return value A_TYPE = identity(str) # inferred type is: Type[str] A_FORM = identity(Optional[str]) # inferred type is: TypeForm ``` Backwards Compatibility ======================= Previously the type `object` would be inferred for runtime objects corresponding to a special form like `Optional[str]` rather than `TypeForm`. But since `TypeForm` is consistent with `object` no unexpected typechecking failures will be introduced by this change. Ongoing Maintenance =================== As new kinds of typing special forms are introduced by upcoming PEPs (usually to the `typing` module), it will be necessary for typecheckers to define whether or not the new forms should be considered inhabitants of `TypeForm` or not, based on the general rules in §"Values of type TypeForm" above. Therefore it is recommended that any new PEP introducing a new typing special form should define whether it inhabits `TypeForm` or not. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[*form*] or Literal[*form*] ----------------------------------- This PEP only defines syntax for a bare `TypeForm` and a `TypeForm[T]` parameterized by a type variable. In particular it does not define syntax like that allows referring to a specific known type-form value via a syntax like `TypeForm[*form*]`: ``` NULL_TYPE := type(None) # has type TypeForm[_T] rather than TypeForm[type(None)] UNION_STR_TYPE := Union[Str] # has type TypeForm[_T] rather than TypeForm[Union[str]] ``` In addition similar syntax like `Literal[*form*]` also does not work. Neither of those syntaxes seem to be especially useful; type forms are not typically used as sentinel values. It is possible that a future PEP may revisit such syntaxes. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

5 27

Marking individual TypedDict items as required
by David Foster 22 Feb '21

22 Feb '21

I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email. For further background on why this feature is being introduced and prior discussions, please see: * the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/PYHMQEL7… -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces new syntax `Required[...]` that can be used on individual items of a `total=False` TypedDict to mark them as required. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax to mark *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different function: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` It is an error to use `Required[...]` on an item of a TypedDict whose *totality* is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `Required[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict that contains some keys that are required and some that are potentially-missing, you should define a single TypedDict with `total=False` and mark those items that are required by wrapping the value type with `Required[...]`. For example: ``` class Movie(TypedDict, total=False): # use total=False title: Required[str] # wrap the str type with Required[...] release_year: Required[int] # wrap the int type with Required[...] directors: List[str] # do not mark potentially-missing items writers: List[str] # do not mark potentially-missing items ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over the `Optional[*type*]` syntax for marking such item values, to avoid using both `Required[...]` and `Optional[...]` within the same TypedDict definition: Yes (Python 3.10+): ``` class Dog(TypedDict, total=False): name: Required[str] owner: str|None ``` Avoid (Python 3.10+), Okay (Python 3.5 - 3.9): ``` class Dog(TypedDict, total=False): name: Required[str] owner: Optional[str] # ick; avoid using Optional and Required at same time ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required item opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means an optional item req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means opposite-of-normal-totality item req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * NotRequired -- in negative form; two words * OptionalItem, OptionalKey -- two words * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked Open Issues =========== §"How to Teach This" recommends the use of syntax `T|None` over `Optional[T]` in TypedDict definitions where `Required[...]` is being used to avoid confusion. But in Python versions earlier than 3.10 the `T|None` syntax is not available. Is there an alternative workable recommendation that would be worth making for such Python versions? References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

7 31

Per-field syntax for defining potentially missing fields in TypedDict
by Tuukka Mustonen 08 Feb '21

08 Feb '21

I believe I was suggested to post this here (instead of at https://github.com/python/mypy/issues/9867). TLDR; Allow `Omittable[...]` annotation, to mark fields that may not exist. Current way of doing this, with `total=False`, is verbose and forces to break the natural grouping of data. As I understand, `total=False` was discussed/added in https://github.com/python/mypy/issues/2632. There's not much debate about the syntax there - the examples/suggestions there are `MayExist[...]`, `NotRequired[...]`, `Checked[...]`, `OptionalItem[...]` or (the other way around) `Required[...]`. However, none of these were considered good enough, so `total=False` got chosen instead. How about `Omittable[...]`? It's only +1 char to `Optional`, which would be perfect, but of course cannot be used here. Longer explanation for why we should need this below, or view the https://github.com/python/mypy/issues/9867 which has Markdown formatting. Though, Guido already said that "There is not much disagreement on the problem". --- We can define optional fields for `TypedDict` as in: class MyDictBase(TypedDict): required: str class MyDict(MyDict): optional: str Compare this to some other languages (like Typescript): interface Data { required: string; optional?: string; } The problem is with syntax here - the Python version is less intuitive, more verbose (for short declarations) and it breaks the "flow" of fields. **1. Less intuitive** When you think about: { "required1": "...", "optional1": "...", "required2": "..." } Why do you need to split up the required and optional fields of one single structure into 2 classes? It's just... not nice. **2. More verbose** While I understand that Python syntax saves characters when there are lots of optional fields, which would be represented like this: class MyDict(TypedDict): optional1: Checked[str] optional2: Checked[str] optional3: Checked[str] ... Omitting `Checked[]` we save 9 chars per field and the result looks nicer as: class MyDict(TypedDict, total=False): optional1: str optional2: str optional3: str ... However, there is certain limit, when this starts to make sense. If you only have a few fields, the drawback of having to break the data into two classes and splitting up fields from their context (see point 3 below) outweighs the benefit of saving a few characters. Furthermore, the problem of verbosity is oftentimes solved by `?` or similar in other languages: interface Data { required: string; optional1?: string; optional2?: string; optional3?: string; ... } This adds only single character per field, and is completely acceptable. Like we are now getting `|` for `Union`, maybe we'll get `?` some day, and it will solve the problem of verbosity. Even without, I would be glad to add some visual clutter to avoid having to split up my structure into two different sections. **3. Breaks the "flow" of fields** While the order of items in a dictionary-like structure is insignificant to a machine (though we got ordered dicts by default in 3.6+), it often has meaning to a human. For example, a structure in our application looks something like: { "schema": "some-schema", "v": 1, "msg_id": "...", "flow_id": "(O) ...", "category": "...", "type": "(O) ...", "created_on": "...", "created_by": "...", "updated_on": "(O) ...", "updated_by": "(O) ..." } As you can see, there are sort of field "groups" there. This grouping/ordering must be broken when defining a TypedDict for the structure: class StructBase(TypedDict): schema: str v: int msg_id: str category: str created_on: str created_by: str class Struct(StructBase, total=False): flow_id: str type: str updated_on: str updated_by: str The result is much more verbose, and more difficult to follow.

10 38

Enabling support for generic TypedDicts
by Samodya Abey 27 Jan '21

27 Jan '21

Hi all, I wanted to bring into your attention about enabling support for generic TypedDicts. I am coming in from the conversation at: [WIP] Adding support for generic TypedDicts (https://github.com/microsoft/pyright/pull/1390) The TypedDict PEP 589 does not necessarily talk about generics. And the runtime TypedDict implementation does not support it either. Traceback (most recent call last): File "testing.py", line 4, in <module> class TD(Generic[T], TypedDict): File "/usr/lib/python3.9/typing.py", line 1894, in __new__ raise TypeError('cannot inherit from both a TypedDict type ' TypeError: cannot inherit from both a TypedDict type and a non-TypedDict base class Here I would like to discuss here if the implementation is done this way for a good reason (may be due to structural subtyping complications) and if there might still be a way to move forward and add generic support for TypedDicts. As you can see below there is a natural syntax for it (in both ways) and have backward compatibility. from typing import TypedDict, TypeVar, Generic, ListT = TypeVar("T") class TD(Generic[T], TypedDict): f1: List[T] R = TypeVar("R") FileCachedData = TypedDict('FileCachedData', {'mtime': float, 'size': int, 'value': R}) def test_td(aa: TD[T], bb: TD[T]): return aa["f1"], bb["f1"] td1 = test_td({"f1": ["foo"]}, {"f1": ["bar"]})reveal_type(td1) # info: Type of "td1" is "tuple[List[str], List[str]]" test_td({"f1": ["foo"]}, {"f1": [1]}) # error class TD2(TD[T]): f2: T td21: TD2[str] = { 'f1':["far"], 'f2':"boo" }reveal_type(td21["f1"]) # info: Type of "td21["f1"]" is "List[str]" td22 = TD2( f1=["far"], f2="boo" )reveal_type(td22) # info: Type of "td22" is "TD2[str]" td23: TD2[str] = { 'f1':[0], 'f2':"boo" } # error Thank you, Samodya

2 2

@overloads and arguments without defaults following arguments with defaults
by Sebastian Rittau 26 Jan '21

26 Jan '21

When working with overloads, it would often make sense if an argument without default could follow an argument with defaults. Take this simplified example: @overload def foo(arg1: str = ..., arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ... This is invalid syntax. Currently, the workaround is fairly cumbersome, redundant, and hard to read (especially in non-toy examples): @overload def foo(arg1: str, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., *, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ... Would it make sense to relax Python's syntax to allow this case? And what would that mean for non-overloaded functions using this syntax? - Sebastian

5 7

Typing-sig January 2021