Mailman 3 February 2021 - Typing-sig

Annotating an Enum class
by Eric Traut 23 Feb '21

23 Feb '21

I have a question about annotations for classes that derive from `Enum`. Instances of these classes are allowed to have custom instance variables that can be initialized by a custom `__init__` or `__new__` method. Refer to [this documentation](https://docs.python.org/3/library/enum.html#when-to-use-new-… for details. My question is, how would these instance variables be annotated in a type stub? How would a type checker differentiate between one of these instance variables and a class variable that represents a member of the enumeration (and therefore is typed as an instance of the enum class)? The metaclass apparently treats assignments within the class definition as enumeration values. Consider the following example: ```python class Color(Enum): # Members of the enumeration RED = ((1, 0, 0), "#FF0000") YELLOW = ((0, 1, 1), "#00FFFF") GREEN = ((0, 1, 0), "#00FF00") # Instance variables for instances of this enumeration components: Tuple[float, float, float] html_encoding: str # Custom init method def __init__(self, components: Tuple[float, float, float], html_encoding: str) -> None: self._value_ = html_value self.components = components self.html_encoding = html_encoding print(Color.RED.value) # "#FF0000" print(Color.RED.components) # (1, 0, 0) ``` How would I annotate this in a type stub? I would have thought the following: ```python class Color(Enum): RED = ... YELLOW = ... GREEN = ... components: Tuple[float, float, float] html_encoding: str ``` However, mypy interprets all five of the variables (including `components` and `html_encoding`) as members of the enumeration. ```python reveal_type(Color.RED) # Revealed type is 'Literal[test.Color.RED]?' reveal_type(Color.components) # Revealed type is 'Literal[test.Color.components]?' reveal_type(Color.html_encoding) # Revealed type is 'Literal[test.Color.html_encoding]?' ``` Furthermore, if I look at the way that enums are annotated in typeshed, they don't appear to use assignments to indicate enum elements. For example, `uuid.pyi` includes the following declaration. ```python class SafeUUID(Enum): safe: int unsafe: int unknown: None ``` So typeshed stubs appear to assume that variables with type annotations within a `Enum` class definition should be interpreted as members of the enumeration, not as instance variables within enum instances. I searched the typeshed and mypy issues and couldn't find any discussion of this topic. Any thoughts or suggestions? -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.

4 5

Marking individual TypedDict items as required
by David Foster 21 Feb '21

21 Feb '21

I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email. For further background on why this feature is being introduced and prior discussions, please see: * the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/PYHMQEL7… -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces new syntax `Required[...]` that can be used on individual items of a `total=False` TypedDict to mark them as required. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax to mark *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different function: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` It is an error to use `Required[...]` on an item of a TypedDict whose *totality* is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `Required[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict that contains some keys that are required and some that are potentially-missing, you should define a single TypedDict with `total=False` and mark those items that are required by wrapping the value type with `Required[...]`. For example: ``` class Movie(TypedDict, total=False): # use total=False title: Required[str] # wrap the str type with Required[...] release_year: Required[int] # wrap the int type with Required[...] directors: List[str] # do not mark potentially-missing items writers: List[str] # do not mark potentially-missing items ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over the `Optional[*type*]` syntax for marking such item values, to avoid using both `Required[...]` and `Optional[...]` within the same TypedDict definition: Yes (Python 3.10+): ``` class Dog(TypedDict, total=False): name: Required[str] owner: str|None ``` Avoid (Python 3.10+), Okay (Python 3.5 - 3.9): ``` class Dog(TypedDict, total=False): name: Required[str] owner: Optional[str] # ick; avoid using Optional and Required at same time ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required item opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means an optional item req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means opposite-of-normal-totality item req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * NotRequired -- in negative form; two words * OptionalItem, OptionalKey -- two words * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked Open Issues =========== §"How to Teach This" recommends the use of syntax `T|None` over `Optional[T]` in TypedDict definitions where `Required[...]` is being used to avoid confusion. But in Python versions earlier than 3.10 the `T|None` syntax is not available. Is there an alternative workable recommendation that would be worth making for such Python versions? References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

7 31

Dumb Question? Type for Coroutine Function?
by Paul Bryan 18 Feb '21

18 Feb '21

I see a type for a coroutine (collections.abc.Coroutine), but I'm looking for a type to represent a coroutine function, i.e. asyncio.iscoroutinefunction(val) == True. If course, I can use (am currently using) collections.abc.Callable, but am hoping for something more explicit. Wondering why I'm not finding it. What am I missing?

2 2

Implementing type support for PEP 634
by Eric Traut 18 Feb '21

18 Feb '21

I’ve implemented support for PEP 634 (Structural Pattern Matching) in Pyright. If you’re interested, you can install the latest version of Pyright (1.1.112) and play with it. I figured I’d share some of the challenges and choices I faced with this feature in the hopes that this might be helpful to contributors to the other type checkers and as feedback to the PEP authors. *PEP completeness* First, I want to complement the authors of this PEP and its companions (635 and 636). They’re very well written. PEP 634 contemplates and answers all of the questions I had about the intended behaviors and edge cases. I can tell that a lot of thought went into this design. The only area where I think this design falls a bit short is with respect to its interaction with static type checking. The authors of the PEP apparently chose to ignore (or didn’t consider) a few implications for type checking. I’ll provide more details below. The good news is that these situations arise only in cases that I anticipate will be used infrequently. The common use cases for structural pattern matching work well with static type checking. *Soft keywords and parse recovery* The keyword `match` is documented to be a “soft” keyword. That preserves backward compatibility with existing code, but it presents challenges for tools that need to do parse recovery. The parser in the Python interpreter doesn’t need to concern itself with parse recovery because it simply reports the first syntax error it finds and then bails. Tools like type checkers, language servers, linters, reformatters, etc. need to handle syntax errors gracefully and recover the parse as quickly as possible to minimize cascading errors. The `match` keyword is especially challenging because it can be followed by an arbitrary number of tokens that are legal for both match statements and other expressions. For example, `match(1, 2, 3)` and `match[1, 2, 3]` are legal token sequences for both match statements and call/index expressions. That means a parser can’t do a single-token lookahead and determine whether the developer intended a match statement or an expression. The parser needs to apply grammar rules and consume tokens until it sees (or doesn’t see) a colon token after the expression. This works fine if there are no syntax errors between the `match` token and the colon (or end of line). If syntax errors are present, parse recovery becomes difficult. If a developer forgets a colon on a match statement or forgets to close a parentheses pair in the subject expression, they’re unfortunately going to see a lot of red squiggles in their editor, and the error message won't always be clear because the parser can’t determine the developer’s intent. I highlight this as a consideration for future extensions to the Python grammar. It’s preferable that new soft keywords are used in contexts where the parser can distinguish between a keyword and an identifier based on the next token. *Code flow model* From a type checking perspective, I chose to model match statements as a series of if/elif statements. That’s effectively how it works at runtime. This correctly models variable assignments performed as part of pattern matching, within the guard condition expression, and the “case” code block. For example, if a variable is assigned a value in some case blocks but not others, it is potentially unbound at the end of the match statement, and the type checker should report an error if it is subsequently referenced. This same approach should work correctly if a match statement is located within a loop (or nested loops) that modify variables that affect the type of the subject expression. ```python def func(subj: object): match subj: case 1 as x: y = 3 case 2 if (x := 1): # y is not assigned in this code path pass case 3: return case _ as x: y = 1 print(x) # No error print(y) # Error: "y" is possibly unbound ``` The above code example also demonstrates that when a wildcard pattern is present in one of the case statements, the code flow logic can assume that there will be no “fall through” (i.e. no patterns matched). If the list of case statements cannot be statically proven to be comprehensive, the code flow logic should assume that a “fall through” can occur. *Capture variable assignments* PEP 634 indicates that capture variables may be left unbound if a pattern doesn’t match, but it also allows implementations to assign values as part of the pattern evaluation. I chose to implement pattern matching in terms of type narrowing. The subject expression’s type is evaluated, and that type is narrowed based on the pattern. The resulting narrowed type is then assigned back to the pattern and any captured variables contained therein. If the entire pattern (or portions of the pattern) are not a possible match based on the subject expression’s type, captured variables receive a `Never` type. (`Never` is a type used within Pyright that is semantically equivalent to an empty union.) ```python def func(subj: int): match subj: case (3 as x) | (3.14 as y): reveal_type(x) # Literal[3] reveal_type(y) # Never case list() as z: reveal_type(z) # Never ``` *Type narrowing based on patterns* As I mentioned above, I use type narrowing to match the subject expression’s type to a pattern. This is applied recursively for sub-patterns. As with if/else statements, the type can be narrowed for both the positive (“if”) and negative (“else”) cases. Negative cases correctly accommodate “or” patterns, which are evaluated in a particular order. In this example, the type `Literal[3] | bool | None` is narrowed to just `Literal[False]` based on the elements of “or” pattern that precede `x`. ```python def func(subj: Literal[3] | bool | None): match subj: case 3 | True | None | x: reveal_type(x) # Literal[False] ``` *Type narrowing for literal patterns* Literal patterns allow for more aggressive type narrowing. For example: ```python def func(subj: int | str): match subj: case (1 | 'abc') as x: reveal_type(x) # Literal[1, 'abc'] case y: reveal_type(y) # int | str ``` *Type narrowing for sequence patterns* Sequence patterns allow for more aggressive type narrowing when the subject expression’s type is a tuple. Here is an example: ```python def func(subj: tuple[int, str, complex, float] | tuple[int, str]): match subj: case [x, *y, z]: reveal_type(x) # int reveal_type(y) # tuple[str, complex] | tuple[()] reveal_type(z) # float | str ``` *Type narrowing for mapping patterns* If the subject expression is a TypedDict, Pyright uses the additional information about value types for pattern matching and type narrowing. For other generic `Mapping` types, it uses the key and value type arguments for pattern matching and type narrowing. ```python class Movie(TypedDict): title: str release_year: int def func(subj: dict[int | Literal["title"], complex] | Movie): match subj: case {3: a1}: reveal_type(a1) # complex case {"title": b1}: reveal_type(b1) # complex | str case {"director": c1}: reveal_type(c1) # Never ``` *Type inference for captured variables* In many cases, types for captured variables (used in capture patterns, “as” patterns, star patterns, and double star patterns) can be properly inferred, as demonstrated in the samples above. If a capture variable has been declared with an annotated type elsewhere within the scope, Pyright will verify that the inferred type is assignable to that declared type. ```python def func(subj: tuple[int, str]): x: str match subj: case [x]: # (No error because pattern will never match subject expression type) pass case [x, y]: # Expression of type "int" cannot be assigned to declared type "str" pass ``` *Type checking challenges* There are a few parts of this spec that present challenges for static type checking. _Value patterns_: The spec indicates that equality (i.e. the `__eq__` operator) is used in value pattern matching. A static type checker is typically able to determine the type of the specified value, but that doesn’t guarantee that the type of the captured pattern matches the type. Most `__eq__` methods are annotated to accept any `object` as an argument. That means a type checker should conservatively assume that any `object` will potentially match the pattern. In my current implementation, I assume that the pattern will match only if the subject expression type is compatible with the value type. For example, if the value type is an `int` but the subject expression type is `list[float]`, the type checker assumes that the pattern match will fail. However, if the subject expression type is `object` (from which `int` derives) or `float` (which is a subtype of `int`), the pattern is considered “potentially matchable”. I think that’s a reasonable assumption in most cases, but I may need to revisit this assumption once pattern matching is adopted more broadly. I understand why value pattern matching was added in its general form, but I wish the spec would constrain its use a bit more so as to avoid this (potentially bad) assumption. _Class patterns_: There are three challenges with class patterns: First, positional arguments use a mapping defined in the `__match_args__` attribute. The spec indicates this attribute can be a list or a tuple. From a type-checking perspective, it would be better if the spec mandated the use of a tuple. Only tuple types retain the types of their individual elements, so a tuple type with string literal elements can be used to specify the names of the attributes associated with positional arguments. A list will, at best, have the type `list[str]`, which is not useful for determining the names of the attributes contained within the list. If the type checker cannot determine the literal string values within `__match_args__`, then type safety for positional arguments cannot be guaranteed by the type checker. Second, types for variables captured by arguments suffer from the same problem as variable patterns. Class pattern arguments are matched using the `__eq__` operator. Here there’s even a greater challenge than with value patterns because there is no subject type to narrow in this case. Take for example the pattern `Foo(a=3 as x, b='' as y)`. If the type of attribute `a` is `int`, then it’s probably safe to infer that the type of `x` is `Literal[3]`, but a more conservative approach would infer the type `object` because the `__eq__` method for class `int` allows any `object` to be passed as an argument. As with value patterns, I’ve chosen to adopt a more aggressive type narrowing approach here. I think this will be generally safe and will serve users better. The third challenge with class patterns is with generic classes. There’s insufficient information in the pattern to infer the type arguments for a generic type. For example, if the subject expression’s type is `object` and the pattern is `list() as x`, the inferred type of variable `x` is `list`, but the type argument cannot be determined. `Any` must be assumed in cases where the subject expression type doesn't provide enough information to get this information from type narrowing. In my implementation, I use the special variant of `Any` called `Unknown` that Pyright uses to track types that are not explicitly declared as `Any`. The problem here is that the presence of an `Unknown` will trigger errors when Pyright is used in “strict” mode, and I don’t see a good workaround for users other than to ensure that the type of the subject expression is already narrow enough to prevent this from occurring. They may need to choose between strict type checking and class pattern matching in some cases. *Test cases* If you would like to borrow test cases from the Pyright source base, you’re welcome to do so. You can find them here: https://github.com/microsoft/pyright/blob/master/packages/pyright-internal/… All of the match test cases are in files of the form “match<number>.py”. -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.

3 4

Void type
by ocket 8888 18 Feb '21

18 Feb '21

I think it'd be nice to have a "void" type added to `typing` that would indicate to type checkers that a function or method cannot be used in legal assignments. Functionally, it's not much different than a `None` return type, but the difference would be from typing import Any, Void def none() -> None: pass def void() -> Void: pass a: Any = none() #passes type checking a = void() #fails type checking Also, it's not subscriptable, nor legal in any non-return-value context. Like: #fails type checking - param can't be Void def print_it(o: Void) -> int: print(o) return 0 I had a PR (https://github.com/python/cpython/pull/24557) but Guido van Rossum closed it, said it should be discussed here first.

8 10

Next tensor typing meeting: this Monday!
by Matthew Rahtz 14 Feb '21

14 Feb '21

Hi everyone, Our next tensor typing meeting will be *Monday the 8th* of February at *6pm London time/10am San Francisco time*. Our agenda this time is a short one: 1. In-person discussion of PEP 646 updates 2. Discussion of typing features in NumPy 1.20 As always, we'll be meeting at https://meet.google.com/fft-dzjq-ksu. See you there! Best, Matthew

2 4

Re: Marking individual TypedDict items as required
by syastrov＠gmail.com 13 Feb '21

13 Feb '21

Another idea: instead of introducing NotRequired, allow negating the type Required using something like this: key: -Required[int] Benefits: - one word - less imports - shorter than NotRequired Downsides: - introduces use of new operator applied to this situation which may or may not coincide with the operator that may be used in the future as a shorthand for marking keys as being required or not.

1 0

Re: Marking individual TypedDict items as required
by syastrov＠gmail.com 13 Feb '21

13 Feb '21

I’m glad this is being taken up and the PEP looks good to me. I second the idea of adding NotRequired (Or NonRequired) as well, as I’m glad this is being taken up and the PEP looks good to me. I second the idea of adding NotRequired (Or NonRequired) as well, as personally I have seen more cases of dicts where there are many required keys and few non-required ones (was going to say “optional” :). Use of required keys when possible provides a more precise type, so I would prefer to only single out the non-required keys to avoid (if we only had Required) accidentally having other keys being non-required because I forgot to specify that. In other words, I’d like to make it take more effort to make keys non-required (but not so much so that I feel that the current situation with subclassing and flipping the total parameter is ideal). TypeScript’s :? syntax is essentially doing the same thing as NotRequired would, by making it (albeit very slightly) more work to define a non-required key.

1 0

Type hinting **kwargs
by Paul Bryan 07 Feb '21

07 Feb '21

I originally posted this on python-ideas; Guido suggested moving it here. I'm encountering a situation where it would be far better for me to use **kwargs (multiple functions taking the same set of parameters) to avoid repeating multiple function definitions, explicitly enumerating all parameters for each function definition. Therefore, I'd like to have some meaningful type hint for **kwargs. First, I thought TypedDict would be a great fit, until I found this passage in PEP 589: > These features were left out from this PEP, but they are potential > extensions to be added in the future: > ... > TypedDict can't be used for specifying the type of a **kwargs > argument. This would allow restricting the allowed keyword arguments > and their types. According to PEP 484, using a TypedDict type as the > type of **kwargs means that the TypedDict is valid as the value of > arbitrary keyword arguments, but it doesn't restrict which keyword > arguments should be allowed. The syntax **kwargs: Expand[T] has been > proposed for this [11]. > ... > [11] https://github.com/python/mypy/issues/4441 In the mypy issue discussion, there seems to be two issues: 1. Ambiguity of whether the type hint applies to the **kwargs dict vs. each value in the dict. I think this could be a red herring; kwargs presents as a dict and I think TypedDict can unambiguously represent the structure of that dict. I don't see how it should be different than any other dict parameter being defined as a function parameter. 2. A question of whether one needs to assume that for a TypedDict, the type validator should assume total=False. I think what keys are required should be explicitly determined by the TypedDict (required keys inferred from total). I suspect the issue could be that mypy may not be able to determine this during static type analysis? In summary, **kwargs is a dict; TypedDict can specify its structure; static type checker limitations should not necessarily restrict its use in this way. Therefore, I propose a short PEP be drafted to codify the validity of using TypedDict to specify the type of a **kwargs argument. Guido noted in a response: > I will have to admit that in PEP 484 we screwed up with `*args` and > `**kwds` -- we thought that these would always be homogeneous and we > thought we'd save the user some time by letting them write `*args: > int` instead of `*args: Tuple[int, ...]` (and similar for `**kwds`), > but this use case and a few others have shown that we would have been > better off with the more verbose syntax. It's too late to change, > alas. PEP 646 is one example where we had to twist ourselves into > knots to make something work for `*args`. Is there anything in PEP 646 that would prevent **kwargs being annotated as I'm suggesting? I'm not seeing anything obvious. Paul

3 3

Per-field syntax for defining potentially missing fields in TypedDict
by Tuukka Mustonen 07 Feb '21

07 Feb '21

I believe I was suggested to post this here (instead of at https://github.com/python/mypy/issues/9867). TLDR; Allow `Omittable[...]` annotation, to mark fields that may not exist. Current way of doing this, with `total=False`, is verbose and forces to break the natural grouping of data. As I understand, `total=False` was discussed/added in https://github.com/python/mypy/issues/2632. There's not much debate about the syntax there - the examples/suggestions there are `MayExist[...]`, `NotRequired[...]`, `Checked[...]`, `OptionalItem[...]` or (the other way around) `Required[...]`. However, none of these were considered good enough, so `total=False` got chosen instead. How about `Omittable[...]`? It's only +1 char to `Optional`, which would be perfect, but of course cannot be used here. Longer explanation for why we should need this below, or view the https://github.com/python/mypy/issues/9867 which has Markdown formatting. Though, Guido already said that "There is not much disagreement on the problem". --- We can define optional fields for `TypedDict` as in: class MyDictBase(TypedDict): required: str class MyDict(MyDict): optional: str Compare this to some other languages (like Typescript): interface Data { required: string; optional?: string; } The problem is with syntax here - the Python version is less intuitive, more verbose (for short declarations) and it breaks the "flow" of fields. **1. Less intuitive** When you think about: { "required1": "...", "optional1": "...", "required2": "..." } Why do you need to split up the required and optional fields of one single structure into 2 classes? It's just... not nice. **2. More verbose** While I understand that Python syntax saves characters when there are lots of optional fields, which would be represented like this: class MyDict(TypedDict): optional1: Checked[str] optional2: Checked[str] optional3: Checked[str] ... Omitting `Checked[]` we save 9 chars per field and the result looks nicer as: class MyDict(TypedDict, total=False): optional1: str optional2: str optional3: str ... However, there is certain limit, when this starts to make sense. If you only have a few fields, the drawback of having to break the data into two classes and splitting up fields from their context (see point 3 below) outweighs the benefit of saving a few characters. Furthermore, the problem of verbosity is oftentimes solved by `?` or similar in other languages: interface Data { required: string; optional1?: string; optional2?: string; optional3?: string; ... } This adds only single character per field, and is completely acceptable. Like we are now getting `|` for `Union`, maybe we'll get `?` some day, and it will solve the problem of verbosity. Even without, I would be glad to add some visual clutter to avoid having to split up my structure into two different sections. **3. Breaks the "flow" of fields** While the order of items in a dictionary-like structure is insignificant to a machine (though we got ordered dicts by default in 3.6+), it often has meaning to a human. For example, a structure in our application looks something like: { "schema": "some-schema", "v": 1, "msg_id": "...", "flow_id": "(O) ...", "category": "...", "type": "(O) ...", "created_on": "...", "created_by": "...", "updated_on": "(O) ...", "updated_by": "(O) ..." } As you can see, there are sort of field "groups" there. This grouping/ordering must be broken when defining a TypedDict for the structure: class StructBase(TypedDict): schema: str v: int msg_id: str category: str created_on: str created_by: str class Struct(StructBase, total=False): flow_id: str type: str updated_on: str updated_by: str The result is much more verbose, and more difficult to follow.

10 38