I went ahead and revised the PEP for Required[...] to support NotRequired[...] as well. See copy at the bottom of this email. Supporting NotRequired[...] nicely simplifies the "How to Teach This" section and integrates well elsewhere. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: `Required[...]` which can be used on individual items of a `total=False` TypedDict to mark them as required, and `NotRequired[...]` which can be used on individual items of a regular (`total=True`) TypedDict to mark them as potentially-missing. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, or as potentially-missing in a regular `total=True` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict): title: str release_year: int directors: NotRequired[List[str]] # mark potentially-missing key writers: NotRequired[List[str]] # mark potentially-missing key ``` or: ``` class Movie(TypedDict, total=False): title: Required[str] # mark required key release_year: Required[int] # mark required key directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different purpose: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as `Required[...]`. Nevertheless it is common for folks wanting to extend a regular (`total=True`) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the `NotRequired[...]` form for that case: ``` class Movie(TypedDict): # version 1 of a JSON API title: str release_year: int # version 2 of a JSON API, # added as potentially-missing keys for backward compatibility directors: NotRequired[List[str]] writers: NotRequired[List[str]] ``` For other alternative syntaxes considered please see ยง"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Additionally the `typing.NotRequired` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key. It may only be used in TypedDict definitions where `total=True`, which is the default totality: ``` class Movie(TypedDict): # implicitly total=True title: str release_year: int directors: NotRequired[List[str]] writers: NotRequired[List[str]] ``` It is an error to use `Required[...]` on an item of a TypedDict whose totality is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `NotRequired[...]` on an item of a TypedDict whose totality is false: ``` class Cell(TypedDict, total=False): value: NotRequired[object] # error: NotRequired[...] is only usable when total=True ``` It is an error to use `Required[...]` or `NotRequired[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, you should define a single TypedDict and mark those few keys that are potentially-missing by wrapping the value type with `NotRequired[...]`. For example: ``` class Movie(TypedDict): title: str release_year: int directors: NotRequired[List[str]] # mark potentially-missing key writers: NotRequired[List[str]] # mark potentially-missing key ``` If on the other hand most keys are potentially-missing and a few are required, you should instead define a single TypedDict with `total=False` and mark those few keys that are required by wrapping the value type with `Required[...]`. For example: ``` class Node(TypedDict, total=False): value: Required[object] # mark required key label: str weight: float ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over `Optional[*type*]` for marking such item values, to avoid using `Required[...]` or `NotRequired[...]` alongside `Optional[...]` within the same TypedDict definition: Yes: ``` from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] ``` Avoid (unless Python 3.5-3.6): ``` class Dog(TypedDict): name: str owner: NotRequired[Optional[str]] # ick; avoid using both Optional and NotRequired ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required` and `NotRequired`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]` and `NotRequired[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript's `undefined` type, perhaps called `Missing`: ``` class MyThing(TypedDict): req1: int opt1: str|Missing req2: float ``` Such a `Missing` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: ``` class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 ``` ``` def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing ``` Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of `...|Missing`, equivalent to `Union[..., Missing]`, doesn't align well with what a union normally means: `Union[...]` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include `Final[...]` and `ClassVar[...]`, which the proposal for `Required[...]` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of `Union[..., Missing]` doesn't align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using `isinstance` checks: ``` class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) ``` However if we were to allow `Union[..., Missing]` you'd either have to eliminate the `Missing` case with `hasattr` for object attributes: ``` class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) ``` or a check against `locals()` for local variables: ``` def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? ``` or a check via other means, such as against `globals()` for global variables: ``` warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? ``` Weird and inconsistent. `Missing` is not really a value at all; it's an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a `Union[..., Missing]`-style syntax would be difficult. [^pyright-no-union] [^pyright-no-union]: https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new `Missing` type-level constant would be very close to introducing a new `Missing` value-level constant at runtime, creating a second null-like runtime value in addition to `None`. Having two different null-like constants in Python (`None` and `Missing`) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants `null` and `undefined`. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Change Optional to mean "optional item" in certain contexts instead of "nullable" --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of `Optional` inside the TypedDict to mean "optional item" rather than its usual meaning of "nullable": ``` class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] ``` or: ``` class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] ``` This would add more confusion for users because it would mean that in *some* contexts the meaning of `Optional[...]` is different than in other contexts. Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * OptionalItem, OptionalKey -- two words; too easy to confuse with optional * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<