RFC on PEP 655: Required[] and NotRequired[] for TypedDict
This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as required or not-required. We have implementations for mypy, pyright, and pyanalyze. Consensus on the PEP has been reached in typing-sig. My understanding of the PEP process as described in PEP 1 [2] is that content review should be requested of core developers here in python-dev. So consider this my official request for comments! ^_^ I believe that after review & feedback from python-dev the next step would be to submit this PEP to the Steering Council. However it's not clear to me from [2] where I should actually do that when the time comes. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [1]: https://www.python.org/dev/peps/pep-0655/ [2]: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution
Hi folks, PEP 655 (Required[] and NotRequired[] for TypedDict) is still looking for feedback from core devs. I've copied the latest PEP text at the bottom of this email to make it easier to comment on. Thank you for your time. Best, -- David Foster | Seattle, WA, USA Contributor to Python's type system ### BEGIN PEP 655 ### PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Jan-2021 Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022 Abstract ======== :pep:`589` defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. This PEP introduces two new type qualifiers, ``typing.Required`` and ``typing.NotRequired``, which allow defining a *single* TypedDict with a mix of both required and potentially-missing keys: :: class Movie(TypedDict): title: str year: NotRequired[int] Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[]``) would be too similar to ``Optional[]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[]`` or ``NotRequired[]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[]`` and ``NotRequired[]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] It is an error to use both ``Required[]`` and ``NotRequired[]`` at the same time: :: class Movie(TypedDict): title: str year: NotRequired[Required[int]] # ERROR The :pep:`alternative syntax <589#alternative-syntax>` for TypedDict also supports ``Required[]`` and ``NotRequired[]``: :: Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]}) Interaction with ``total=False`` -------------------------------- Any :pep:`589`-style TypedDict declared with ``total=False`` is equivalent to a TypedDict with an implicit ``total=True`` definition with all of its keys marked as ``NotRequired[]``. Therefore: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int is equivalent to: :: class _MovieBase(TypedDict): title: str class Movie(_MovieBase): year: NotRequired[int] Interaction with ``Annotated[]`` ----------------------------------- ``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``, in any nesting order: :: class Movie(TypedDict): title: str year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok :: class Movie(TypedDict): title: str year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok In particular allowing ``Annotated[]`` to be the outermost annotation for an item allows better interoperability with non-typing uses of annotations, which may always want ``Annotated[]`` as the outermost annotation. [3]_ Runtime behavior ---------------- Interaction with ``get_type_hints()`` ''''''''''''''''''''''''''''''''''''' ``typing.get_type_hints(...)`` applied to a TypedDict will by default strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers, since these qualifiers are expected to be inconvenient for code casually introspecting type annotations. ``typing.get_type_hints(..., include_extras=True)`` however *will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers, for advanced code introspecting type annotations that wishes to preserve *all* annotations in the original source: :: class Movie(TypedDict): title: str year: NotRequired[int] assert get_type_hints(Movie) == \ {'title': str, 'year': int} assert get_type_hints(Movie, include_extras=True) == \ {'title': str, 'year': NotRequired[int]} Interaction with ``get_origin()`` and ``get_args()`` '''''''''''''''''''''''''''''''''''''''''''''''''''' ``typing.get_origin()`` and ``typing.get_args()`` will be updated to recognize ``Required[]`` and ``NotRequired[]``: :: assert get_origin(Required[int]) is Required assert get_args(Required[int]) == (int,) assert get_origin(NotRequired[int]) is NotRequired assert get_args(NotRequired[int]) == (int,) Interaction with ``__required_keys__`` and ``__optional_keys__`` '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' An item marked with ``Required[]`` will always appear in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item marked with ``NotRequired[]`` will always appear in ``__optional_keys__``. :: assert Movie.__required_keys__ == frozenset({'title'}) assert Movie.__optional_keys__ == frozenset({'year'}) Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal (without the ``total`` keyword) and mark those few keys that are potentially-missing with ``NotRequired[]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[]`` or ``NotRequired[]`` alongside ``Optional[]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Okay (required for Python 3.5.3-3.6): :: class Dog(TypedDict): name: str owner: 'NotRequired[str|None]' No: :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Usage in Python <3.11 --------------------- If your code supports Python <3.11 and wishes to use ``Required[]`` or ``NotRequired[]`` then it should use ``typing_extensions.TypedDict`` rather than ``typing.TypedDict`` because the latter will not understand ``(Not)Required[]``. In particular ``__required_keys__`` and ``__optional_keys__`` on the resulting TypedDict type will not be correct: Yes (Python 3.11+ only): :: from typing import NotRequired, TypedDict class Dog(TypedDict): name: str owner: NotRequired[str|None] Yes (Python <3.11 and 3.11+): :: from __future__ import annotations # for Python 3.7-3.9 from typing_extensions import NotRequired, TypedDict # for Python <3.11 with (Not)Required class Dog(TypedDict): name: str owner: NotRequired[str|None] No (Python <3.11 and 3.11+): :: from typing import TypedDict # oops: should import from typing_extensions instead from typing_extensions import NotRequired class Movie(TypedDict): title: str year: NotRequired[int] assert Movie.__required_keys__ == frozenset({'title', 'year'}) # yikes assert Movie.__optional_keys__ == frozenset() # yikes Reference Implementation ======================== The `mypy <http://www.mypy-lang.org/>`__ `0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__, `pyright <https://github.com/Microsoft/pyright>`__ `1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__, and `pyanalyze <https://github.com/quora/pyanalyze>`__ `0.4.0 <https://pyanalyze.readthedocs.io/en/latest/changelog.html#version-0-4-0-november-18-2021>`__ type checkers support ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax causes ``Optional[]`` to take on different meanings depending on where it is positioned, which is inconsistent and confusing. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[]`` and ``NotRequired[]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Note when reconsidering introducing this shortform syntax that ``+``, ``-``, and ``~`` already have existing meanings in the Python typing world: covariant, contravariant, and invariant: :: >>> from typing import TypeVar >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V')) (+T, -U, ~V) Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... .. [3] https://bugs.python.org/issue46491 Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: ### END PEP 655 ###
Hello, I'm sorry for my late reply -- keeping up with all the PEPs is somewhat challenging. The one nitpick I have is that the PEP should make a few things more clear to people outside the typing-sig: - if this PEP really only affects typing.py and external projects/tools, it should say so clearly (so e.g. a parser experts can skip reading the PEP with clear conscience, even though it "introduces two new syntaxes") - in Specification, clarify what "It is an error" means -- is it a Python runtime error, or an error type checkers should raise? Same for "It is valid". To be clear, I don't think this should block the PEP. On Wed, Feb 16, 2022 at 4:58 PM David Foster <davidfstr@gmail.com> wrote:
Hi folks, PEP 655 (Required[] and NotRequired[] for TypedDict) is still looking for feedback from core devs.
I've copied the latest PEP text at the bottom of this email to make it easier to comment on.
Thank you for your time.
Best, -- David Foster | Seattle, WA, USA Contributor to Python's type system
Maybe this shouldn't be called syntax but notation or convention? On Mon, Mar 7, 2022 at 7:00 AM Petr Viktorin <encukou@gmail.com> wrote:
Hello, I'm sorry for my late reply -- keeping up with all the PEPs is somewhat challenging.
The one nitpick I have is that the PEP should make a few things more clear to people outside the typing-sig: - if this PEP really only affects typing.py and external projects/tools, it should say so clearly (so e.g. a parser experts can skip reading the PEP with clear conscience, even though it "introduces two new syntaxes") - in Specification, clarify what "It is an error" means -- is it a Python runtime error, or an error type checkers should raise? Same for "It is valid".
To be clear, I don't think this should block the PEP.
On Wed, Feb 16, 2022 at 4:58 PM David Foster <davidfstr@gmail.com> wrote:
Hi folks, PEP 655 (Required[] and NotRequired[] for TypedDict) is still looking for feedback from core devs.
I've copied the latest PEP text at the bottom of this email to make it easier to comment on.
Thank you for your time.
Best, -- David Foster | Seattle, WA, USA Contributor to Python's type system
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YT23XQSN... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Thanks for the feedback Petr,
- if this PEP really only affects typing.py and external projects/tools, it should say so clearly (so e.g. a parser experts can skip reading the PEP with clear conscience, even though it "introduces two new syntaxes")
Did propose a new paragraph in the Abstract section that should make this clear: "No Python grammar changes are made by this PEP. Correct usage of required and potentially-missing keys of TypedDicts is intended to be enforced only by static type checkers and in particular is not enforced by Python itself at runtime."
- in Specification, clarify what "It is an error" means -- is it a Python runtime error, or an error type checkers should raise? Same for "It is valid".
It is a *logical* error. To be explicit: (1) It's definitely an error that type checkers MUST raise. (2) The runtime implementations of Required[] and NotRequired[] MAY also choose to raise errors at runtime on a best effort basis. In particular the prohibition on nesting Required[] and NotRequired[] inside each other is likely to be enforced at runtime by the __getitem__ functions on the type form implementations, because it is easy to do so. All of the preceding clarifications have been proposed in a PR: https://github.com/python/peps/pull/2388 Best, -- David Foster | Seattle, WA, USA Contributor to Python's type system
This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as required or not-required. We have implementations for mypy, pyright, and pyanalyze.
Consensus on the PEP has been reached in typing-sig.
My understanding of the PEP process as described in PEP 1 [2] is that content review should be requested of core developers here in python-dev. So consider this my official request for comments! ^_^
I believe that after review & feedback from python-dev the next step would be to submit this PEP to the Steering Council. However it's not clear to me from [2] where I should actually do that when the time comes. I think the names `Required` and `NotRequired` are too generic for something which can only be used in one context (`TypedDict`s), and it is not immediately obvious when reading code without context what the difference between `NotRequired` and `Optional` might be. Could they be
On 30/01/2022 05:15, David Foster wrote: put into a pseudo-module like `typing.io`, e.g. `TypedDict.Required`?
On 3/8/22 8:42 AM, Patrick Reader wrote:
I think the names `Required` and `NotRequired` are too generic for something which can only be used in one context (`TypedDict`s), and Could they be put into a pseudo-module like `typing.io`, e.g. `TypedDict.Required`?
It sounds like you are proposing something like: ``` from typing import TypedDict from typing.TypedDict import NotRequired # 👈 class Movie(TypedDict): title: str stars: NotRequired[int] ``` Is that really much different than: ``` from typing import NotRequired, TypedDict # 👈 class Movie(TypedDict): title: str stars: NotRequired[int] ``` Only the imports differ. And now users now have to specially remember that the "typing-related" NotRequired marker needs to imported not from `typing`, as is usual practice, but instead needs to be imported from somewhere else.
it is not immediately obvious when reading code without context what the difference between `NotRequired` and `Optional` might be.
Indeed PEP 655 §"How to Teach This" recommends not using both Optional and NotRequired in the same place, which might also mean the same file, and provides suggestions to avoid using Optional at all. Related: The word on the street is that "T|None" is likely to be a favored replacement for Optional in general going forward, since it's faster to type and doesn't require an extra import (of Optional from typing). Best, -- David Foster | Seattle, WA, USA Contributor to Python's type system
participants (4)
-
David Foster -
Guido van Rossum -
Patrick Reader -
Petr Viktorin