I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email. For further background on why this feature is being introduced and prior discussions, please see: * the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/PYHMQEL7K... -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces new syntax `Required[...]` that can be used on individual items of a `total=False` TypedDict to mark them as required. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax to mark *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different function: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` It is an error to use `Required[...]` on an item of a TypedDict whose *totality* is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `Required[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict that contains some keys that are required and some that are potentially-missing, you should define a single TypedDict with `total=False` and mark those items that are required by wrapping the value type with `Required[...]`. For example: ``` class Movie(TypedDict, total=False): # use total=False title: Required[str] # wrap the str type with Required[...] release_year: Required[int] # wrap the int type with Required[...] directors: List[str] # do not mark potentially-missing items writers: List[str] # do not mark potentially-missing items ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over the `Optional[*type*]` syntax for marking such item values, to avoid using both `Required[...]` and `Optional[...]` within the same TypedDict definition: Yes (Python 3.10+): ``` class Dog(TypedDict, total=False): name: Required[str] owner: str|None ``` Avoid (Python 3.10+), Okay (Python 3.5 - 3.9): ``` class Dog(TypedDict, total=False): name: Required[str] owner: Optional[str] # ick; avoid using Optional and Required at same time ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required item opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means an optional item req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means opposite-of-normal-totality item req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * NotRequired -- in negative form; two words * OptionalItem, OptionalKey -- two words * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked Open Issues =========== §"How to Teach This" recommends the use of syntax `T|None` over `Optional[T]` in TypedDict definitions where `Required[...]` is being used to avoid confusion. But in Python versions earlier than 3.10 the `T|None` syntax is not available. Is there an alternative workable recommendation that would be worth making for such Python versions? References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<