Marking individual TypedDict items as required
I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email. For further background on why this feature is being introduced and prior discussions, please see: * the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/PYHMQEL7K... -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces new syntax `Required[...]` that can be used on individual items of a `total=False` TypedDict to mark them as required. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax to mark *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different function: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` It is an error to use `Required[...]` on an item of a TypedDict whose *totality* is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `Required[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict that contains some keys that are required and some that are potentially-missing, you should define a single TypedDict with `total=False` and mark those items that are required by wrapping the value type with `Required[...]`. For example: ``` class Movie(TypedDict, total=False): # use total=False title: Required[str] # wrap the str type with Required[...] release_year: Required[int] # wrap the int type with Required[...] directors: List[str] # do not mark potentially-missing items writers: List[str] # do not mark potentially-missing items ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over the `Optional[*type*]` syntax for marking such item values, to avoid using both `Required[...]` and `Optional[...]` within the same TypedDict definition: Yes (Python 3.10+): ``` class Dog(TypedDict, total=False): name: Required[str] owner: str|None ``` Avoid (Python 3.10+), Okay (Python 3.5 - 3.9): ``` class Dog(TypedDict, total=False): name: Required[str] owner: Optional[str] # ick; avoid using Optional and Required at same time ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required item opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means an optional item req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means opposite-of-normal-totality item req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * NotRequired -- in negative form; two words * OptionalItem, OptionalKey -- two words * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked Open Issues =========== §"How to Teach This" recommends the use of syntax `T|None` over `Optional[T]` in TypedDict definitions where `Required[...]` is being used to avoid confusion. But in Python versions earlier than 3.10 the `T|None` syntax is not available. Is there an alternative workable recommendation that would be worth making for such Python versions? References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
On 1/31/21 12:25 AM, David Foster wrote:
I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email.
Silence is golden? Can I get a PEP number? Then I can RST-ize it. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
The typical workflow is that you fork the peps repo and edit your file giving it number 9999. When you submit it as a PR, a PEP editor (probably me :-) will assign you a number and then it's a simple commit to change the number in the header and rename the file. This way we don't have to keep a database of assigned PEP numbers -- conflicts are resolved by repeating the process (though I don''t recall this has ever happened). On Thu, Feb 4, 2021 at 6:45 AM David Foster <davidfstr@gmail.com> wrote:
On 1/31/21 12:25 AM, David Foster wrote:
I have drafted an initial PEP for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email.
Silence is golden?
Can I get a PEP number? Then I can RST-ize it.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Thanks for pushing this forward! I am not aware of the practices of writing a PEP, so maybe these other rejected ideas were left out on purpose: - `|Missing` (as magic value) - `TypedDict(optional_as_nullable=...)` (though this only partially accomplishes the target) Could be maybe included under Rejected Ideas if not. (Also, I would suggest to still re-consider using `NotRequired`, too. I personally don't think it's that bad. It's 100% obvious by meaning and harmonizes nicely with `Required`. If in the future something like `+` or `-` would replace the typed form, the wording will became even less important.)
On 2/5/21 1:41 AM, Tuukka Mustonen wrote:
Thanks for pushing this forward!
I am not aware of the practices of writing a PEP, so maybe these other rejected ideas were left out on purpose:
- `|Missing` (as magic value)
Oversight on my part. Will add.
- `TypedDict(optional_as_nullable=...)` (though this only partially accomplishes the target)
Sure I can mention this too.
Could be maybe included under Rejected Ideas if not.
(Also, I would suggest to still re-consider using `NotRequired`, too. I personally don't think it's that bad. It's 100% obvious by meaning and harmonizes nicely with `Required`. If in the future something like `+` or `-` would replace the typed form, the wording will became even less important.)
I'm personally hoping that eventually we'll go toward the `+` or `-` forms if requireness/optionalness marking becomes a common feature. (Although that's a pretty big if, this early in the game.) -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
Hi David, First, I agree with Tukka that the `| Missing` (special vlaue) proposal should be discussed in the "Rejected Ideas" section, as it's one of the first alternatives that come to mind. I would also mention that a similar situation also can occur in regular classes (unlike, say, in Java) which currently cannot be represented with Python typing: class C: def __init__(self, set_it: bool) -> None: if set_it: self.attr = 10 C(set_it=False).attr # type checks with mypy, but crashes While this is bad code IMO, it does occur quite often in the wild (especially in pre-typing code). If there is ever a Python checker that aims for soundness, it would either have to reject this code (like Java etc. do, "possibly uninitialized variable") or it would need to represent it in the type system. And if it is to be represented in the type system, a `Required[..]` wouldn't work well, due to backward compatibility and obviously being the wrong default in terms of boilerplate. Ran
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote:
Hi David,
First, I agree with Tukka that the `| Missing` (special vlaue) proposal should be discussed in the "Rejected Ideas" section, as it's one of the first alternatives that come to mind.
I would also mention that a similar situation also can occur in regular classes (unlike, say, in Java) which currently cannot be represented with Python typing:
class C: def __init__(self, set_it: bool) -> None: if set_it: self.attr = 10
C(set_it=False).attr # type checks with mypy, but crashes
While this is bad code IMO, it does occur quite often in the wild (especially in pre-typing code). If there is ever a Python checker that aims for soundness, it would either have to reject this code (like Java etc. do, "possibly uninitialized variable") or it would need to represent it in the type system. And if it is to be represented in the type system, a `Required[..]` wouldn't work well, due to backward compatibility and obviously being the wrong default in terms of boilerplate.
I believe mypy currently represents the type of `C.attr` in the above example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type Hints"). -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
I don’t think that’s what the star in mypy,means. IIRC it has something to do with whether it is an inferred type. On Sun, Feb 7, 2021 at 22:44 David Foster <davidfstr@gmail.com> wrote:
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote:
Hi David,
First, I agree with Tukka that the `| Missing` (special vlaue) proposal should be discussed in the "Rejected Ideas" section, as it's one of the first alternatives that come to mind.
I would also mention that a similar situation also can occur in regular classes (unlike, say, in Java) which currently cannot be represented with Python typing:
class C: def __init__(self, set_it: bool) -> None: if set_it: self.attr = 10
C(set_it=False).attr # type checks with mypy, but crashes
While this is bad code IMO, it does occur quite often in the wild (especially in pre-typing code). If there is ever a Python checker that aims for soundness, it would either have to reject this code (like Java etc. do, "possibly uninitialized variable") or it would need to represent it in the type system. And if it is to be represented in the type system, a `Required[..]` wouldn't work well, due to backward compatibility and obviously being the wrong default in terms of boilerplate.
I believe mypy currently represents the type of `C.attr` in the above example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type Hints").
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote:
> class C: > def __init__(self, set_it: bool) -> None: > if set_it: > self.attr = 10
On Sun, Feb 7, 2021 at 22:44 David Foster <davidfstr@gmail.com
I believe mypy currently represents the type of `C.attr` in the above example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type
Hints"). On 2/8/21 7:28 AM, Guido van Rossum wrote:
I don’t think that’s what the star in mypy,means. IIRC it has something to do with whether it is an inferred type.
Ah. Then I stand corrected. I've only ever noticed the `SomeType*` form in mypy for this specific scenario, so I made an assumption on its meaning. Best, -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy On 2/8/21 7:28 AM, Guido van Rossum wrote:
I don’t think that’s what the star in mypy,means. IIRC it has something to do with whether it is an inferred type.
On Sun, Feb 7, 2021 at 22:44 David Foster <davidfstr@gmail.com <mailto:davidfstr@gmail.com>> wrote:
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote: > Hi David, > > First, I agree with Tukka that the `| Missing` (special vlaue) proposal should be discussed in the "Rejected Ideas" section, as it's one of the first alternatives that come to mind. > > I would also mention that a similar situation also can occur in regular classes (unlike, say, in Java) which currently cannot be represented with Python typing: > > class C: > def __init__(self, set_it: bool) -> None: > if set_it: > self.attr = 10 > > C(set_it=False).attr # type checks with mypy, but crashes > > While this is bad code IMO, it does occur quite often in the wild (especially in pre-typing code). If there is ever a Python checker that aims for soundness, it would either have to reject this code (like Java etc. do, "possibly uninitialized variable") or it would need to represent it in the type system. And if it is to be represented in the type system, a `Required[..]` wouldn't work well, due to backward compatibility and obviously being the wrong default in terms of boilerplate.
I believe mypy currently represents the type of `C.attr` in the above example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type Hints").
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy _______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig@python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave@python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ <https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: guido@python.org <mailto:guido@python.org>
-- --Guido (mobile)
El jue, 11 feb 2021 a las 7:47, David Foster (<davidfstr@gmail.com>) escribió:
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote:
> class C: > def __init__(self, set_it: bool) -> None: > if set_it: > self.attr = 10
On Sun, Feb 7, 2021 at 22:44 David Foster <davidfstr@gmail.com
I believe mypy currently represents the type of `C.attr` in the
above
example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type
Hints").
On 2/8/21 7:28 AM, Guido van Rossum wrote:
I don’t think that’s what the star in mypy,means. IIRC it has something to do with whether it is an inferred type.
Ah. Then I stand corrected. I've only ever noticed the `SomeType*` form in mypy for this specific scenario, so I made an assumption on its meaning.
I opened https://github.com/python/mypy/issues/10076 to propose removing this mypy feature since as far as I can tell it's only ever been confusing.
Best, -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 2/8/21 7:28 AM, Guido van Rossum wrote:
I don’t think that’s what the star in mypy,means. IIRC it has something to do with whether it is an inferred type.
On Sun, Feb 7, 2021 at 22:44 David Foster <davidfstr@gmail.com <mailto:davidfstr@gmail.com>> wrote:
On 2/5/21 2:16 AM, Ran Benita via Typing-sig wrote: > Hi David, > > First, I agree with Tukka that the `| Missing` (special vlaue) proposal should be discussed in the "Rejected Ideas" section, as it's one of the first alternatives that come to mind. > > I would also mention that a similar situation also can occur in regular classes (unlike, say, in Java) which currently cannot be represented with Python typing: > > class C: > def __init__(self, set_it: bool) -> None: > if set_it: > self.attr = 10 > > C(set_it=False).attr # type checks with mypy, but crashes > > While this is bad code IMO, it does occur quite often in the wild (especially in pre-typing code). If there is ever a Python checker that aims for soundness, it would either have to reject this code (like Java etc. do, "possibly uninitialized variable") or it would need to represent it in the type system. And if it is to be represented in the type system, a `Required[..]` wouldn't work well, due to backward compatibility and obviously being the wrong default in terms of boilerplate.
I believe mypy currently represents the type of `C.attr` in the above example as `int*` (vice `int`) which means "if it's defined then it's an `int`". I don't think this interpretation is standardized by any PEP I've read though. In particular not the original PEP 484 ("Type Hints").
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy _______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig@python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave@python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ <https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: guido@python.org <mailto:guido@python.org>
-- --Guido (mobile)
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Here is a second PEP draft for Required[...], a way to mark individual items of a TypedDict as required and allow defining a TypedDict with mixed required and potentially-missing items. See copy at the bottom of this email. The only changes I made since the last draft are the addition of two subsections within §"Rejected Ideas": 1. § Marking absence of a value with a special constant I see how this proposal could be attractive, so I took some time to offer a thorough rebuttal. 2. § Change Optional to mean "optional item" in certain contexts instead of "nullable" Added for completeness. As my next step I'll plan on RST-izing this PEP and submitting it as a PR to the PEPs repository. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces new syntax `Required[...]` that can be used on individual items of a `total=False` TypedDict to mark them as required. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax to mark *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different function: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` It is an error to use `Required[...]` on an item of a TypedDict whose *totality* is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `Required[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict that contains some keys that are required and some that are potentially-missing, you should define a single TypedDict with `total=False` and mark those items that are required by wrapping the value type with `Required[...]`. For example: ``` class Movie(TypedDict, total=False): # use total=False title: Required[str] # wrap the str type with Required[...] release_year: Required[int] # wrap the int type with Required[...] directors: List[str] # do not mark potentially-missing items writers: List[str] # do not mark potentially-missing items ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over the `Optional[*type*]` syntax for marking such item values, to avoid using both `Required[...]` and `Optional[...]` within the same TypedDict definition: Yes (Python 3.10+): ``` class Dog(TypedDict, total=False): name: Required[str] owner: str|None ``` Avoid (Python 3.10+), Okay (Python 3.5 - 3.9): ``` class Dog(TypedDict, total=False): name: Required[str] owner: Optional[str] # ick; avoid using Optional and Required at same time ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required item opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means an optional item req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means opposite-of-normal-totality item req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript's `undefined` type, perhaps called `Missing`: ``` class MyThing(TypedDict): req1: int opt1: str|Missing req2: float ``` Such a `Missing` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: ``` class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 ``` ``` def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing ``` Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of `...|Missing`, equivalent to `Union[..., Missing]`, doesn't align well with what a union normally means: `Union[...]` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include `Final[...]` and `ClassVar[...]`, which the proposal for `Required[...]` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of `Union[..., Missing]` doesn't align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using `isinstance` checks: ``` class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) ``` However if we were to allow `Union[..., Missing]` you'd either have to eliminate the `Missing` case with `hasattr` for object attributes: ``` class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) ``` or a check against `locals()` for local variables: ``` def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? ``` or a check via other means, such as against `globals()` for global variables: ``` warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? ``` Weird and inconsistent. `Missing` is not really a value at all; it's an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a `Union[..., Missing]`-style syntax would be difficult. [^pyright-no-union] [^pyright-no-union]: https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new `Missing` type-level constant would be very close to introducing a new `Missing` value-level constant at runtime, creating a second null-like runtime value in addition to `None`. Having two different null-like constants in Python (`None` and `Missing`) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants `null` and `undefined`. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Change Optional to mean "optional item" in certain contexts instead of "nullable" --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of `Optional` inside the TypedDict to mean "optional item" rather than its usual meaning of "nullable": ``` class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] ``` or: ``` class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] ``` This would add more confusion for users because it would mean that in *some* contexts the meaning of `Optional[...]` is different than in other contexts. Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * NotRequired -- in negative form; two words * OptionalItem, OptionalKey -- two words * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked Open Issues =========== §"How to Teach This" recommends the use of syntax `T|None` over `Optional[T]` in TypedDict definitions where `Required[...]` is being used to avoid confusion. But in Python versions earlier than 3.10 the `T|None` syntax is not available. Is there an alternative workable recommendation that would be worth making for such Python versions? References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Thanks, David. I'm really in favour of improving the way required keys are managed in TypedDict and glad you're working on it. Some feedback to your latest draft: 1. I understand your rationale for not using Optional; however, I am concerned about the potential for confusion in a world where there is both an Optional[...] and Required[...] type hint. Also, the potential for an ugly (but valid?!) Required[Optional[...]] construct. 2. Has there been any discussion of using of the Annotated type hint? Example: Annotated[..., Required]? The examples for Annotated cite using it for validation. Having a required key in a dict seems like a good fit! If this option has been considered and rejected, I request it be documented as such in the PEP. 3. If we go with Required[...] instead of Annotated[..., ...], would get_type_hints(..., include_extras=False) suppress Required the same way it suppresses Annotated today? If not, this would impose a further burden on code performing dynamic type introspection. 4. When iterating through the TypedDict.__annotations__, will Required[...] be retained there, or would it need to be discovered through __required_keys__, or both? 5. Could we do away (or at least deprecated) totality in TypedDict, and rely on this mechanism? (and/or possibly codify the use of __required_keys__?) 6. The PEP states: "It is an error to use `Required[...]` in any location that is not an item of a TypedDict." I'm curious to know how this would be (easily) enforced. I doubt __class_getitem__ would have the necessary context to enforce it. Paul
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca> wrote:
Thanks, David. I'm really in favour of improving the way required keys are managed in TypedDict and glad you're working on it.
Some feedback to your latest draft:
1. I understand your rationale for not using Optional; however, I am concerned about the potential for confusion in a world where there is both an Optional[...] and Required[...] type hint. Also, the potential for an ugly (but valid?!) Required[Optional[...]] construct.
To reduce confusion we can recommend T|None over Optional[T].
2. Has there been any discussion of using of the Annotated type hint? Example: Annotated[..., Required]? The examples for Annotated cite using it for validation. Having a required key in a dict seems like a good fit! If this option has been considered and rejected, I request it be documented as such in the PEP.
-1. Annotated has a different meaning ("attach third-party data").
3. If we go with Required[...] instead of Annotated[..., ...], would get_type_hints(..., include_extras=False) suppress Required the same way it suppresses Annotated today? If not, this would impose a further burden on code performing dynamic type introspection.
4. When iterating through the TypedDict.__annotations__, will Required[...] be retained there, or would it need to be discovered through __required_keys__, or both?
Together, it seems the best would be to always suppress Required, and only store that info in __required_keys__. Unless I'm missing something subtle (somehow this all sounds like a series of trick questions :-). 5. Could we do away (or at least deprecated) totality in TypedDict, and
rely on this mechanism? (and/or possibly codify the use of __required_keys__?)
Do you mean the __total__ attribute? I guess that's already been superseded by __required_keys__ (and __optional_keys__ -- I don't recall why we need both).
6. The PEP states: "It is an error to use `Required[...]` in any location that is not an item of a TypedDict." I'm curious to know how this would be (easily) enforced. I doubt __class_getitem__ would have the necessary context to enforce it.
That should be enforced by the static type checker. Probably worth being explicit about that in the PEP. Such incorrect uses would just be preserved at runtime. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 2/11/21 9:51 PM, Guido van Rossum wrote:
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca <mailto:pbryan@anode.ca>> wrote: 1. I understand your rationale for not using Optional; however, I am concerned about the potential for confusion in a world where there is both an Optional[...] and Required[...] type hint. Also, the potential for an ugly (but valid?!) Required[Optional[...]] construct.
To reduce confusion we can recommend T|None over Optional[T].
I do in fact make that recommendation for Python 3.10. See §"How to Teach This". But I also have a related open issue in §"Open Issues" that ponders what can be done in Python 3.9 and lower, where the T|None syntax isn't available. Maybe: Yes (Python 3.5 - 3.9): Required[Union[..., None]] Avoid: Required[Optional[...]] -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Thu, Feb 11, 2021 at 22:05 David Foster <davidfstr@gmail.com> wrote:
On 2/11/21 9:51 PM, Guido van Rossum wrote:
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca <mailto:pbryan@anode.ca>> wrote: 1. I understand your rationale for not using Optional; however, I am concerned about the potential for confusion in a world where there is both an Optional[...] and Required[...] type hint. Also, the potential for an ugly (but valid?!) Required[Optional[...]] construct.
To reduce confusion we can recommend T|None over Optional[T].
I do in fact make that recommendation for Python 3.10. See §"How to Teach This".
But I also have a related open issue in §"Open Issues" that ponders what can be done in Python 3.9 and lower, where the T|None syntax isn't available. Maybe:
Yes (Python 3.5 - 3.9): Required[Union[..., None]]
Avoid: Required[Optional[...]]
I think that’s too complex. And note that there’s no reason a type checker couldn’t support T|None in all Python versions that support ‘from __future__ import annotations’. I also expect the case will occur rarely.
-- --Guido (mobile)
On Thu, 2021-02-11 at 21:51 -0800, Guido van Rossum wrote: [snip]
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca> wrote:
4. When iterating through the TypedDict.__annotations__, will Required[...] be retained there, or would it need to be discovered through __required_keys__, or both?
Together, it seems the best would be to always suppress Required, and only store that info in __required_keys__. Unless I'm missing something subtle (somehow this all sounds like a series of trick questions :-).
+1. No trick questions. 😉
5. Could we do away (or at least deprecated) totality in TypedDict, and rely on this mechanism? (and/or possibly codify the use of __required_keys__?)
Do you mean the __total__ attribute? I guess that's already been superseded by __required_keys__ (and __optional_keys__ -- I don't recall why we need both).
Well, yes, __total__ as well as total=... initialization parameter. They must die! Also, I don't think __required_keys__ or __optional_keys__ are documented, at least not in https://docs.python.org/3.10/library/typing.html. Is there any reason we can't codify them in 3.10 docs? Paul
On Thu, Feb 11, 2021 at 10:54 PM Paul Bryan <pbryan@anode.ca> wrote:
On Thu, 2021-02-11 at 21:51 -0800, Guido van Rossum wrote:
[snip]
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca> wrote:
4. When iterating through the TypedDict.__annotations__, will Required[...] be retained there, or would it need to be discovered through __required_keys__, or both?
Together, it seems the best would be to always suppress Required, and only store that info in __required_keys__. Unless I'm missing something subtle (somehow this all sounds like a series of trick questions :-).
+1. No trick questions. 😉
5. Could we do away (or at least deprecated) totality in TypedDict, and rely on this mechanism? (and/or possibly codify the use of __required_keys__?)
Do you mean the __total__ attribute? I guess that's already been superseded by __required_keys__ (and __optional_keys__ -- I don't recall why we need both).
Well, yes, __total__ as well as total=... initialization parameter. They must die!
Explain? The proposal here depends on total=False, and I wouldn't want to flip the default (it would break too much code). I could do without the `__total__` class attribute though.
Also, I don't think __required_keys__ or __optional_keys__ are documented, at least not in https://docs.python.org/3.10/library/typing.html. Is there any reason we can't codify them in 3.10 docs?
Nobody got to it yet? Maybe you'd be willing to submit a small PR for this? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Fri, 2021-02-12 at 14:23 -0800, Guido van Rossum wrote:
On Thu, Feb 11, 2021 at 10:54 PM Paul Bryan <pbryan@anode.ca> wrote:
On Thu, 2021-02-11 at 21:51 -0800, Guido van Rossum wrote:
[snip]
On Thu, Feb 11, 2021 at 9:13 PM Paul Bryan <pbryan@anode.ca> wrote:
4. When iterating through the TypedDict.__annotations__, will Required[...] be retained there, or would it need to be discovered through __required_keys__, or both?
Together, it seems the best would be to always suppress Required, and only store that info in __required_keys__. Unless I'm missing something subtle (somehow this all sounds like a series of trick questions :-).
+1. No trick questions. 😉
5. Could we do away (or at least deprecated) totality in TypedDict, and rely on this mechanism? (and/or possibly codify the use of __required_keys__?)
Do you mean the __total__ attribute? I guess that's already been superseded by __required_keys__ (and __optional_keys__ -- I don't recall why we need both).
Well, yes, __total__ as well as total=... initialization parameter. They must die!
Explain? The proposal here depends on total=False, and I wouldn't want to flip the default (it would break too much code). I could do without the `__total__` class attribute though.
The quarrel I've had with total= is the issue of needing to explicitly create a subclass so as to scope which are required keys and which are not. If total=True is equivalent to wrapping every key specified in this particular TypedDict with Required[...] (with no implications to superclasses or subclasses), and only manifests in __required_keys__, (and not in __total__) then no objection.
Also, I don't think __required_keys__ or __optional_keys__ are documented, at least not in https://docs.python.org/3.10/library/typing.html. Is there any reason we can't codify them in 3.10 docs?
Nobody got to it yet? Maybe you'd be willing to submit a small PR for this?
I would be happy to. Paul
I completely agree with Paul Bryan as I think PEP 593 `Annotated` should be used for this case. Concerning `__required_keys__`, the issue about putting information into type annotations is that they need to be evaluated (they will be stringified in 3.10), but their evaluation could fail because of forward reference. That's an issue (using PEP 593 or not). The `dataclasses` module use regex parsing to solve this issue, but this solution can be debated. I've thought about a solution that slightly overtake the scope of this PEP: typing module could provide an `_eval_annotations` function that would evaluate only the PEP 593 annotations of a type if it's an `Annotated[...]` form (and return `()` otherwise), leaving the annotated type (which could be a forward ref and thus unevaluatable) unevaluated. This would allow retrieving `Required` information (if it uses PEP 593) without problem at the class definition. P.S. One could argue that `ClassVar`, another type modifier, is implemented as a raw type, so it would be coherent to proceed the same way for this PEP, but `ClassVar` was introduced before PEP 593, and in my opinion, `ClassVar` should also evolved to be a PEP 593 annotation, as well as `InitVar`. `dataclasses` module could also benefit of `_eval_annotations` (should it be a public function then?) to avoid regex parsing.
Sounds like you have Annotated on your mind. :-) Anyway, at this point it's not clear that annotations will be stringified in 3.10, PEP 563 notwithstanding -- there's a new proposal under discussion, PEP 649. On Sun, Feb 14, 2021 at 2:22 PM Joseph Perez <joperez@hotmail.fr> wrote:
I completely agree with Paul Bryan as I think PEP 593 `Annotated` should be used for this case.
Concerning `__required_keys__`, the issue about putting information into type annotations is that they need to be evaluated (they will be stringified in 3.10), but their evaluation could fail because of forward reference. That's an issue (using PEP 593 or not). The `dataclasses` module use regex parsing to solve this issue, but this solution can be debated. I've thought about a solution that slightly overtake the scope of this PEP: typing module could provide an `_eval_annotations` function that would evaluate only the PEP 593 annotations of a type if it's an `Annotated[...]` form (and return `()` otherwise), leaving the annotated type (which could be a forward ref and thus unevaluatable) unevaluated. This would allow retrieving `Required` information (if it uses PEP 593) without problem at the class definition.
P.S. One could argue that `ClassVar`, another type modifier, is implemented as a raw type, so it would be coherent to proceed the same way for this PEP, but `ClassVar` was introduced before PEP 593, and in my opinion, `ClassVar` should also evolved to be a PEP 593 annotation, as well as `InitVar`. `dataclasses` module could also benefit of `_eval_annotations` (should it be a public function then?) to avoid regex parsing.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 2/14/21 1:34 PM, Joseph Perez wrote:
I completely agree with Paul Bryan as I think PEP 593 `Annotated` should be used for this case.
The `dataclasses` module use regex parsing to solve this issue, but
Annotated[] appears intended for user code to add its own annotations that do not collide with any existing type hints used by type checkers. So creating a special interpretation of something inside an Annotated[] would be counter to the spirit. Also `Annotated[..., Required]` is unnecessarily longer than `Required[...]`. Why type more? On 2/14/21 1:34 PM, Joseph Perez wrote: this solution can be debated. I've thought about a solution that slightly overtake the scope of this PEP: typing module could provide an `_eval_annotations` function [...] This is getting a bit off-topic, but the `typing.get_type_hints()` function already evaluates ForwardRef() values, although it has some limitations and restrictions on practical usage. Getting something like eval_annotations() to work in more cases is non-trivial, and best put on a new thread. On 2/11/21 8:50 PM, David Foster wrote:
As my next step I'll plan on RST-izing this PEP and submitting it as a PR to the PEPs repository.
In other news, I'm planning to get the PEP submitted via PR this weekend. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Thu, Feb 18, 2021 at 8:24 PM David Foster <davidfstr@gmail.com> wrote:
In other news, I'm planning to get the PEP submitted via PR this weekend.
Sounds like it's going to be a busy weekend. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
A couple of folks in earlier threads were supportive of NotRequired[...] as an alternative (or in addition to) the Required[...] syntax. I've been warming up to that idea. Allowing the use NotRequired[...] would make it possible to succinctly mark a small number of items in an existing TypedDict definition as potentially-missing, which I think may be a more-common case than marking a small number of keys as required. Using NotRequired[...] would also allow folks to avoid using the somewhat cryptic `total=False` syntax and the inverting behavior it implies. Here's an example from one project of mine that makes all "version 1" keys of a JSON structure required but adds potentially-missing keys for all future structure versions: ``` class CodeAssignmentInfo(TypedDict): # v1 id: IntStr title: str subtitle: str assignment_icon_type: AssignmentIconType turned_in_status: TurnedInStatus unlock_status: UnlockStatus # v2 has_teacher_uploaded_video: NotRequired[bool] ``` Notice that I didn't need to use `total=False` and only had to use a single `NotRequired[...]`. By contrast if we only supported `Required[...]`, I'd have to instead write: ``` class CodeAssignmentInfo(TypedDict, total=False): # v1 id: Required[IntStr] title: Required[str] subtitle: Required[str] assignment_icon_type: Required[AssignmentIconType] turned_in_status: Required[TurnedInStatus] unlock_status: Required[UnlockStatus] # v2 has_teacher_uploaded_video: bool ``` Oof. That's a lot of Required[...] wrappers I had to add in order to get just a single potentially-missing item added. So I will pose a few questions directly: 1. How about we support NotRequired[...] in general? 2. How do folks feel about *also* supporting Required[...] for when a total=False TypedDict really is more appropriate? -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 2/20/21 8:15 AM, David Foster wrote:
1. How about we support NotRequired[...] in general?
2. How do folks feel about *also* supporting Required[...] for when a total=False TypedDict really is more appropriate?
As a quick followup comment: If we were to support both NotRequired[...] and Required[...] it would translate well to the proposed eventual shorthand + and - syntax for these wrappers: ``` class MyThing(TypedDict, total=False): req1: +int # + means Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means NotRequired[...] req2: float ``` -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
`+` and `-` already have a meaning in the Python typing world (and some other languages, like Scala), they are used to mark the variance of a type parameter: `+` means covariant and `-` means contravariant (invariant is `~`). ```python from typing import TypeVar print(TypeVar("T"), TypeVar("U", covariant=True), TypeVar("V", contravariant=True)) #> ~T +U -V ``` Yes, they are not used directly in the code, so maybe the meaning of theses operator can be extended (the applications are quite different indeed), but it's maybe too early to plan this kind of shortcuts for this feature. See the time that PEP 604 has taken for `Union` notation shortcut. By the way, I don't know if `TypedDict` is a popular enough feature to bind such common operators to it without taking step back.
I went ahead and revised the PEP for Required[...] to support NotRequired[...] as well. See copy at the bottom of this email. Supporting NotRequired[...] nicely simplifies the "How to Teach This" section and integrates well elsewhere. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: Marking individual TypedDict items as required Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 (Allow writing union types as X | Y) Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021 Abstract ======== [PEP 589] defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with [all potentially-missing keys] however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: `Required[...]` which can be used on individual items of a `total=False` TypedDict to mark them as required, and `NotRequired[...]` which can be used on individual items of a regular (`total=True`) TypedDict to mark them as potentially-missing. [PEP 589]: https://www.python.org/dev/peps/pep-0589/ [all potentially-missing keys]: https://www.python.org/dev/peps/pep-0589/#totality Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for `total=...` and then inherit it from another TypedDict with a different value for `total=...`: ``` class _MovieBase(TypedDict): # implicitly total=True title: str release_year: int class Movie(_MovieBase, total=False): directors: List[str] writers: List[str] ``` Having to declare two different TypedDict types for this purpose is cumbersome. Instead, this PEP defines syntax that allows individual keys to be marked as required in a `total=False` TypedDict, or as potentially-missing in a regular `total=True` TypedDict, which allows specifying a set of required keys and potentially-missing keys all at once in the same TypedDict definition: ``` class Movie(TypedDict): title: str release_year: int directors: NotRequired[List[str]] # mark potentially-missing key writers: NotRequired[List[str]] # mark potentially-missing key ``` or: ``` class Movie(TypedDict, total=False): title: Required[str] # mark required key release_year: Required[int] # mark required key directors: List[str] writers: List[str] ``` Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: ``` interface Movie { title: string; release_year: number; directors?: array<string>; // ? marks potentially-missing keys writers?: array<string>; // ? marks potentially-missing keys } ``` The difficulty is that the best word for marking a potentially-missing key, `Optional[...]`, is already used in Python for a completely different purpose: marking values that could be either of a particular type or `None`. In particular the following does not work: ``` class Movie(TypedDict): title: str release_year: int directors: Optional[List[str]] # means List[str]|None, not potentially-missing! writers: Optional[List[str]] # means List[str]|None, not potentially-missing! ``` Attempting to use any synonym of "optional" to mark potentially-missing keys (like `Missing[...]`) would be too similar to `Optional[...]` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as `Required[...]`. Nevertheless it is common for folks wanting to extend a regular (`total=True`) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the `NotRequired[...]` form for that case: ``` class Movie(TypedDict): # version 1 of a JSON API title: str release_year: int # version 2 of a JSON API, # added as potentially-missing keys for backward compatibility directors: NotRequired[List[str]] writers: NotRequired[List[str]] ``` For other alternative syntaxes considered please see §"Rejected Ideas". Specification ============= The `typing.Required` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key. It may only be used in TypedDict definitions where `total=False`: ``` class Movie(TypedDict, total=False): title: Required[str] release_year: Required[int] directors: List[str] writers: List[str] ``` Additionally the `typing.NotRequired` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key. It may only be used in TypedDict definitions where `total=True`, which is the default totality: ``` class Movie(TypedDict): # implicitly total=True title: str release_year: int directors: NotRequired[List[str]] writers: NotRequired[List[str]] ``` It is an error to use `Required[...]` on an item of a TypedDict whose totality is true: ``` class Movie(TypedDict): # implicitly total=True title: Required[str] # error: Required[...] is only usable when total=False ``` ``` class Movie(TypedDict, total=True): title: Required[str] # error: Required[...] is only usable when total=False ``` It is an error to use `NotRequired[...]` on an item of a TypedDict whose totality is false: ``` class Cell(TypedDict, total=False): value: NotRequired[object] # error: NotRequired[...] is only usable when total=True ``` It is an error to use `Required[...]` or `NotRequired[...]` in any location that is not an item of a TypedDict: ``` # error: Required[...] is only usable on items of TypedDict password: Required[str] # error: Required[...] is only usable on items of TypedDict def login(password: Required[str]) -> None: ... # error: Required[...] is only usable on items of TypedDict def get_signature() -> Required[Signature]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, you should define a single TypedDict and mark those few keys that are potentially-missing by wrapping the value type with `NotRequired[...]`. For example: ``` class Movie(TypedDict): title: str release_year: int directors: NotRequired[List[str]] # mark potentially-missing key writers: NotRequired[List[str]] # mark potentially-missing key ``` If on the other hand most keys are potentially-missing and a few are required, you should instead define a single TypedDict with `total=False` and mark those few keys that are required by wrapping the value type with `Required[...]`. For example: ``` class Node(TypedDict, total=False): value: Required[object] # mark required key label: str weight: float ``` If some items accept `None` in addition to a regular value, it is recommended that the `*type*|None` syntax be preferred over `Optional[*type*]` for marking such item values, to avoid using `Required[...]` or `NotRequired[...]` alongside `Optional[...]` within the same TypedDict definition: Yes: ``` from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] ``` Avoid (unless Python 3.5-3.6): ``` class Dog(TypedDict): name: str owner: NotRequired[Optional[str]] # ick; avoid using both Optional and NotRequired ``` Reference Implementation ======================== The following will be true when [mypy#9867](https://github.com/python/mypy/issues/9867) is implemented: The [mypy] type checker supports `Required` and `NotRequired`. A reference implementation of the runtime component is provided in the [typing_extensions] module. [mypy]: http://www.mypy-lang.org/ [typing_extensions]: https://github.com/python/typing/tree/master/typing_extensions Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- ``` class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` or: ``` class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value ``` These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, "let's just not put funny syntax before the colon." [^funny-syntax] [^funny-syntax]: https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary `+` as shorthand to mark a required key, unary `-` to mark a potentially-missing key, or unary `~` to mark a key with opposite-of-normal totality: ``` class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float ``` Such operators could be implemented on `type` via the `__pos__`, `__neg__` and `__invert__` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. `Required[...]` and `NotRequired[...]`) before introducing shortform syntax. Future PEPs may reconsider introducing these shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript's `undefined` type, perhaps called `Missing`: ``` class MyThing(TypedDict): req1: int opt1: str|Missing req2: float ``` Such a `Missing` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: ``` class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 ``` ``` def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing ``` Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of `...|Missing`, equivalent to `Union[..., Missing]`, doesn't align well with what a union normally means: `Union[...]` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include `Final[...]` and `ClassVar[...]`, which the proposal for `Required[...]` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of `Union[..., Missing]` doesn't align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using `isinstance` checks: ``` class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) ``` However if we were to allow `Union[..., Missing]` you'd either have to eliminate the `Missing` case with `hasattr` for object attributes: ``` class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) ``` or a check against `locals()` for local variables: ``` def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? ``` or a check via other means, such as against `globals()` for global variables: ``` warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? ``` Weird and inconsistent. `Missing` is not really a value at all; it's an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a `Union[..., Missing]`-style syntax would be difficult. [^pyright-no-union] [^pyright-no-union]: https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new `Missing` type-level constant would be very close to introducing a new `Missing` value-level constant at runtime, creating a second null-like runtime value in addition to `None`. Having two different null-like constants in Python (`None` and `Missing`) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants `null` and `undefined`. Replace Optional with Nullable. Repurpose Optional to mean "optional item". --------------------------------------------------------------------------- `Optional[...]` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the `T|None` syntax specified by [PEP 604]. [PEP 604]: https://www.python.org/dev/peps/pep-0604/ Change Optional to mean "optional item" in certain contexts instead of "nullable" --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of `Optional` inside the TypedDict to mean "optional item" rather than its usual meaning of "nullable": ``` class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] ``` or: ``` class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] ``` This would add more confusion for users because it would mean that in *some* contexts the meaning of `Optional[...]` is different than in other contexts. Various synonyms for "potentially-missing item" ----------------------------------------------- * Omittable -- too easy to confuse with optional * OptionalItem, OptionalKey -- two words; too easy to confuse with optional * MayExist, MissingOk -- two words * Droppable -- too similar to Rust's `Drop`, which means something different * Potential -- too vague * Open -- sounds like applies to an entire structure rather then to an item * Excludable * Checked References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Anyway, why not just use field assignment in the declaration of TypedDict? ```python class MyDict(TypedDict): required_key: int optional_key: int = ... ``` It's way easier to implement (you don't have to evaluate annotations or use regex parsing) and IMO easier to read. It's also more consistent with how field requirement is handled in other standard data structures (`dataclass` and `NamedTuple`). I think `Ellipsis`/`...` is a perfect placeholder for this use, but it can be debated. `total` would become quite useless with this proposal (just a shortcut to make all keys optional), but I don't think it would be an issue because again, things are more consistent this way. IMO, `total` could even be deprecated (but kept for backward compatibility, of course). To make it compatible with the assignment-based syntax, a parameter, let's name it `optional_keys`, could be added and used this way: ```python MyDict = TypedDict("MyDict", {"required_key": int, "optional_key": int}, optional_keys=["optional_key"]) ``` Yes, the key name is repeated, but I don't think assignment-based syntax is used a lot, so it should be ok; I might be wrong. By the way, I think a simple placeholder is enough and there is no need for a thing like `dataclasses.field`. It's true that `dataclasses.field` provides a `metadata` parameter that could be useful, but PEP 593 has been released since, and it can be used for that.
Anyway, why not just use field assignment in the declaration of TypedDict?
```python class MyDict(TypedDict): required_key: int optional_key: int = ... ``` It's not clear to me how field assignment would convey "this item is
On 2/20/21 10:40 AM, Joseph Perez wrote: potentially-missing". If anything it says "there's something here but I'm not specifying what".
It's way easier to implement (you don't have to evaluate annotations or use regex parsing) and IMO easier to read.
None of the prior proposals require regex parsing or runtime evaluation of annotations.
It's also more consistent with how field requirement is handled in other standard data structures (`dataclass` and `NamedTuple`).
NamedTuple's assignment syntax is used for specifying *default values*, not potentially-missing items.
I think `Ellipsis`/`...` is a perfect placeholder for this use, but it can be debated.
`total` would become quite useless with this proposal (just a shortcut to make all keys optional), but I don't think it would be an issue because again, things are more consistent this way. IMO, `total` could even be deprecated (but kept for backward compatibility, of course).
To make it compatible with the assignment-based syntax, a parameter, let's name it `optional_keys`, could be added and used this way: ```python MyDict = TypedDict("MyDict", {"required_key": int, "optional_key": int}, optional_keys=["optional_key"]) ``` Yes, the key name is repeated, but I don't think assignment-based syntax is used a lot, so it should be ok; I might be wrong.
The assignment-based syntax was intentionally made less-capable that the class-based syntax to encourage folks to migrate to the class-based syntax. There's no reason to update it. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
None of the prior proposals require regex parsing or runtime evaluation of annotations.
Without regex parsing or runtime evaluation of annotations, how can `TypedDict` metaclass know that a field is required or not? Especially when using `from __future__ import annotations` (aka. PEP 563)? Using `get_type_hints` is not a solution, because it prevents declaring recursive `TypedDict`. `dataclass` decorator has the same issue, and it solves it using regex parsing. IMO runtime evaluation of annotation are better here (in the typing module, private API like `_eval_type` is available), but I don't see an other way of doing this.
NamedTuple's assignment syntax is used for specifying default values, not potentially-missing items.
If you understand `...` in `TypedDict` context as a kind of javascript `undefined`, comparison with `NamedTuple` can be done.
It's not clear to me how field assignment would convey "this item is potentially-missing". If anything it says "there's something here but I'm not specifying what".
Or maybe it can says "the value of this key can be undefined".
The assignment-based syntax was intentionally made less-capable that the class-based syntax to encourage folks to migrate to the class-based syntax. There's no reason to update it.
Could we consider to have a `make_dataclass` function that is outdated compared to `dataclass` decorator. I don't think so, and this can apply here too. Anyway, to continue on the consistency subject, `dataclasses` module use field value to store field metadata. We could consider something like ```python class MyDict(TypedDict): key: int = typed_dict_key(required=False) ``` but I think `...` shortcut is simpler. Joseph Perez
On 2/20/21 12:44 PM, Joseph Perez wrote:
None of the prior proposals require regex parsing or runtime evaluation of annotations.
Without regex parsing or runtime evaluation of annotations, how can `TypedDict` metaclass know that a field is required or not? Especially when using `from __future__ import annotations` (aka. PEP 563)?
`Required[T]` can evaluate to a box of type Required with parameter T, which the TypedDict __new__ can inspect. It can be implemented in a straightforward fashion.
Using `get_type_hints` is not a solution, because it prevents declaring recursive `TypedDict`. `dataclass` decorator has the same issue, and it solves it using regex parsing. IMO runtime evaluation of annotation are better here (in the typing module, private API like `_eval_type` is available), but I don't see an other way of doing this.
NamedTuple's assignment syntax is used for specifying default values, not potentially-missing items.
If you understand `...` in `TypedDict` context as a kind of javascript `undefined`, comparison with `NamedTuple` can be done.
It's not clear to me how field assignment would convey "this item is potentially-missing". If anything it says "there's something here but I'm not specifying what".
Or maybe it can says "the value of this key can be undefined".
I'm not a fan of this syntax. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
Required[T] can evaluate to a box of type Required with parameter T, which the TypedDict __new__ can inspect. It can be implemented in a straightforward fashion.
Again, I think you forget PEP 563; with it, `Required[T]` **will not evaluate** at all. By the way, `ClassVar` and `InitVar` are already boxes like you describe, but `dataclasses` module still has to use regex parsing in case of annotation being stringified.
On 2/21/21 12:04 AM, Joseph Perez wrote:
Required[T] can evaluate to a box of type Required with parameter T, which the TypedDict __new__ can inspect. It can be implemented in a straightforward fashion.
Again, I think you forget PEP 563; with it, `Required[T]` **will not evaluate** at all. By the way, `ClassVar` and `InitVar` are already boxes like you describe, but `dataclasses` module still has to use regex parsing in case of annotation being stringified.
Look, Python 3.10a is already out, which has presumably has some strategy so that TypedDict (and its metaclass) continue to work at runtime. I’ll hook into whatever strategy that is. We’re don’t need to take up the time of typing-sig with this low level discussion of implementation details. Only comments that potentially affect design details or the overall viability of the PEP are relevant here. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
participants (7)
-
David Foster -
Guido van Rossum -
Jelle Zijlstra -
Joseph Perez -
Paul Bryan -
Ran Benita -
Tuukka Mustonen