PEP 655: Marking individual TypedDict items as required or potentially-missing
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/ Comments welcome. :) Meanwhile I'll proceed to implement this syntax in mypy. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Avoid (unless Python 3.5-3.6): :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The goal is to be able to make the following statement: The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. The mypy implementation is currently still being worked on. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value or: :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Thanks, David, for working on this. Some questions: Q1. Though the intent is to use in TypedDict, is it necessary to preclude its use elsewhere by making its use outside of TypedDict an error? Q2. Can the PEP explicitly address how Required[...] and NotRequired[...] will manifest in TypedDict state? Namely, its effect on __required_keys__ and __optional_keys__? Q3. How will get_type_hints(...), get_options(...) and get_args(...) behave with TypedDicts (and their __annotations__ values) where keys are explicitly marked Required[...] or NotRequired[...]? Paul On Fri, 2021-02-26 at 19:09 -0800, David Foster wrote:
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially- missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Abstract ========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing.
Motivation ==========
It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True title: str
class Movie(_MovieBase, total=False): year: int
Having to declare two different TypedDict types for this purpose is cumbersome.
Rationale =========
One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript:
::
interface Movie { title: string; year?: number; // ? marks potentially-missing keys }
The difficulty is that the best word for marking a potentially- missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work:
::
class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially- missing!
Attempting to use any synonym of “optional” to mark potentially- missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``.
Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case.
Specification =============
The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False): title: Required[str] year: int
Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int]
It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict.
It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int]
Backwards Compatibility =======================
No backward incompatible changes are made by this PEP.
How to Teach This =================
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``.
To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``.
If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Avoid (unless Python 3.5-3.6):
::
class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]]
Reference Implementation ========================
The goal is to be able to make the following statement:
The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module.
The mypy implementation is currently still being worked on.
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially- missing would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator -------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float
class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float
class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float
Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar.
It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options.
Marking absence of a value with a special constant --------------------------------------------------
We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict): req1: int opt1: str|Missing req2: float
Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined:
::
class MyClass: attr: int|Missing
def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10
::
def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing
Misalignment with how unions apply to values ''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided '''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks:
::
class Packet: data: Union[str, bytes]
def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet: data: Union[str, Missing]
def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global variables:
::
warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!'
if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially.
Difficult to implement ''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_
Introduces a second null-like value into Python '''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------- ------
``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------- ------------
Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str]
This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item” -----------------------------------------------
- Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked
References ==========
.. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD...
.. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
On 2/26/21 7:43 PM, Paul Bryan wrote:
Q1. Though the intent is to use in TypedDict, is it necessary to preclude its use elsewhere by making its use outside of TypedDict an error?
Well, what would it mean to have Required[] somewhere else? For example: def identity(x: Required[int]) -> int: ... A type checker is going to have to apply *some* meaning to Required[] used in a parameter type. Should it be ignored? Feels pretty strongly like an error to me that should be flagged. Is there a useful reason to allow Required[] in other positions that I'm not thinking of?
Q2. Can the PEP explicitly address how Required[...] and NotRequired[...] will manifest in TypedDict state? Namely, its effect on __required_keys__ and __optional_keys__?
Sure. Required[] applied to a key means it will definitely appear in __required_keys__ at runtime. Similarly NotRequired[] applied to a key means it will definitely appear in __optional_keys__. Neither __required_keys__ nor __optional_keys__ are mentioned in PEP 589 (TypedDict) [1] or the "typing" module documentation [2] but I think an earlier post to the list suggested that was an oversight and that these runtime-available keys should be documented (presumably in one of those locations) as well. [1]: https://www.python.org/dev/peps/pep-0589/ [2]: https://docs.python.org/3/library/typing.html#typing.TypedDict I wonder if it would be useful to ban confusing combinations of both Required[] and NotRequired[] at the same time, like: class Movie(TypedDict): year: Required[NotRequired[...]] # um; error?; does Required[] win?
Q3. How will get_type_hints(...), get_options(...) and get_args(...) behave with TypedDicts (and their __annotations__ values) where keys are explicitly marked Required[...] or NotRequired[...]?
Good question. Presumably they would became similarly to how they treat other type qualifiers like Final[] and Annotated[] (whatever that behavior is), but I'll do some further research/thinking here. Thanks for the feedback. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Sat, 2021-02-27 at 09:29 -0800, David Foster wrote:
On 2/26/21 7:43 PM, Paul Bryan wrote:
Q1. Though the intent is to use in TypedDict, is it necessary to preclude its use elsewhere by making its use outside of TypedDict an error?
Well, what would it mean to have Required[] somewhere else? For example:
def identity(x: Required[int]) -> int: ...
A type checker is going to have to apply *some* meaning to Required[] used in a parameter type. Should it be ignored? Feels pretty strongly like an error to me that should be flagged.
Is there a useful reason to allow Required[] in other positions that I'm not thinking of?
My thought is admittedly pretty weak at this point: dynamically defining a as required prior to its use in TypeDict. Is the following valid? TD = TypedDict("TD", {"a": Required[int] if some_condition else NotRequired[int], "b": str}, total=False) If so, why not the following? SomeType = Required[int] if some_condition else NotRequired[int] TD = TypedDict("TD", {"a": SomeType, "b": str}, total=False)
Q2. Can the PEP explicitly address how Required[...] and NotRequired[...] will manifest in TypedDict state? Namely, its effect on __required_keys__ and __optional_keys__?
Sure.
Required[] applied to a key means it will definitely appear in __required_keys__ at runtime. Similarly NotRequired[] applied to a key means it will definitely appear in __optional_keys__.
Neither __required_keys__ nor __optional_keys__ are mentioned in PEP 589 (TypedDict) [1] or the "typing" module documentation [2] but I think an earlier post to the list suggested that was an oversight and that these runtime-available keys should be documented (presumably in one of those locations) as well.
[1]: https://www.python.org/dev/peps/pep-0589/ [2]: https://docs.python.org/3/library/typing.html#typing.TypedDict
I'll create a PR to update [2].
I wonder if it would be useful to ban confusing combinations of both Required[] and NotRequired[] at the same time, like:
class Movie(TypedDict): year: Required[NotRequired[...]] # um; error?; does Required[] win?
The options that make sense to me: 1. Combining them is an error. 2. Outer always wins. I lean toward option 1, as I can't think of a good reason they might wrap each other.
Q3. How will get_type_hints(...), get_options(...) and get_args(...) behave with TypedDicts (and their __annotations__ values) where keys are explicitly marked Required[...] or NotRequired[...]?
Good question. Presumably they would became similarly to how they treat other type qualifiers like Final[] and Annotated[] (whatever that behavior is), but I'll do some further research/thinking here.
Suggestions:
TD = TypedDict("TD", {"a": Required[int]}, total=False) get_type_hints(TD) {'a': typing.Required[int]} a = get_type_hints(TD)["a"] get_origin(a) typing.Required get_args(a) (<class 'int'>,)
Paul
On 2/27/21 2:57 PM, Paul Bryan wrote:
Is there a useful reason to allow Required[] in other positions that I'm not thinking of?
My thought is admittedly pretty weak at this point: dynamically defining a as required prior to its use in TypeDict. Is the following valid?
TD = TypedDict("TD", {"a": Required[int] if some_condition else NotRequired[int], "b": str}, total=False)
It is not valid. Type checkers need to be able to determine the types of keys statically and those expressions are too complex.
Q2. Can the PEP explicitly address how Required[...] and NotRequired[...] will manifest in TypedDict state? Namely, its effect on __required_keys__ and __optional_keys__?
Sure.
Required[] applied to a key means it will definitely appear in __required_keys__ at runtime. Similarly NotRequired[] applied to a key means it will definitely appear in __optional_keys__.
Neither __required_keys__ nor __optional_keys__ are mentioned in PEP 589 (TypedDict) [1] or the "typing" module documentation [2] but I think an earlier post to the list suggested that was an oversight and that these runtime-available keys should be documented (presumably in one of those locations) as well.
[1]: https://www.python.org/dev/peps/pep-0589/ <https://www.python.org/dev/peps/pep-0589/> [2]: https://docs.python.org/3/library/typing.html#typing.TypedDict <https://docs.python.org/3/library/typing.html#typing.TypedDict>
I'll create a PR to update [2].
Cool. I'll probably see it this Thursday or this weekend. Pretty overloaded this week.
I wonder if it would be useful to ban confusing combinations of both Required[] and NotRequired[] at the same time, like:
class Movie(TypedDict): year: Required[NotRequired[...]] # um; error?; does Required[] win?
The options that make sense to me:
1. Combining them is an error. 2. Outer always wins.
I lean toward option 1, as I can't think of a good reason they might wrap each other.
Agreed.
Q3. How will get_type_hints(...), get_options(...) and get_args(...) behave with TypedDicts (and their __annotations__ values) where keys are explicitly marked Required[...] or NotRequired[...]?
Good question. Presumably they would became similarly to how they treat other type qualifiers like Final[] and Annotated[] (whatever that behavior is), but I'll do some further research/thinking here.
Suggestions:
TD = TypedDict("TD", {"a": Required[int]}, total=False)
get_type_hints(TD)
{'a': typing.Required[int]}
a = get_type_hints(TD)["a"]
get_origin(a)
typing.Required
get_args(a)
(<class 'int'>,)
Yes, that's the *preserve* approach. I could also see an argument that the hints be *erased*, leaving only the requireness information in the `__required_keys__` and `__optional_keys__` collections. Erased types are likely to be easier for user code to work with at runtime. Again, I'll do some more research RE the behavior of similar Final[] and Annotated[] here when I have some bandwidth. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Tue, Mar 2, 2021 at 3:06 PM David Foster <davidfstr@gmail.com> wrote:
[...]
Q3. How will get_type_hints(...), get_options(...) and get_args(...) behave with TypedDicts (and their __annotations__ values) where keys are explicitly marked Required[...] or NotRequired[...]?
Good question. Presumably they would became similarly to how they treat other type qualifiers like Final[] and Annotated[] (whatever that behavior is), but I'll do some further research/thinking here.
Suggestions:
TD = TypedDict("TD", {"a": Required[int]}, total=False)
get_type_hints(TD)
{'a': typing.Required[int]}
a = get_type_hints(TD)["a"]
get_origin(a)
typing.Required
get_args(a)
(<class 'int'>,)
Yes, that's the *preserve* approach.
I could also see an argument that the hints be *erased*, leaving only the requireness information in the `__required_keys__` and `__optional_keys__` collections. Erased types are likely to be easier for user code to work with at runtime.
Again, I'll do some more research RE the behavior of similar Final[] and Annotated[] here when I have some bandwidth.
While Final is not allowed in a TypedDict, in contexts where it is allowed it is preserved. I recommend the same for [Not]Required. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
A quick note on some text on the PEP (below): On Sat, 27 Feb 2021 at 03:09, David Foster <davidfstr@gmail.com> wrote:
[snip]
Rejected Ideas
==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
This is factually wrong for the second example; 'Optional[opt1]: str' is perfectly valid Python syntax right now. It will most likely break at runtime with a NameError because it requires opt1 to be defined, but it does not require a grammar change. I guess someone could even argue for Optional["opt1"]: str (which is in some way more accurate given that the key is the string "opt1") which would be implementable at a library level, so perhaps this part of the PEP needs rephrasing. To avoid misunderstanding: this is a comment on the correctness of the PEP text, but I am NOT advocating for this Optional[key]: type syntax, I'm personally happy with the proposal as it is, thanks for writing this :) Best, D.
References to __required_keys__ and __optional_keys__ have now been merged into the 3.10 docs: https://github.com/python/cpython/pull/24668/files As was discussed in another thread, could we deprecate __total__, ideally through this PEP? Perhaps through text like:
The ``total`` parameter, combined with the use of ``Required[...]`` and ``Optional[...]`` manifests in the class ``__required_keys__`` and ``__optional_keys__`` attributes.
Paul On Sat, 2021-02-27 at 17:57 +0000, Daniel Moisset wrote:
A quick note on some text on the PEP (below):
On Sat, 27 Feb 2021 at 03:09, David Foster <davidfstr@gmail.com> wrote:
[snip]
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially- missing would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
This is factually wrong for the second example; 'Optional[opt1]: str' is perfectly valid Python syntax right now. It will most likely break at runtime with a NameError because it requires opt1 to be defined, but it does not require a grammar change. I guess someone could even argue for Optional["opt1"]: str (which is in some way more accurate given that the key is the string "opt1") which would be implementable at a library level, so perhaps this part of the PEP needs rephrasing.
To avoid misunderstanding: this is a comment on the correctness of the PEP text, but I am NOT advocating for this Optional[key]: type syntax, I'm personally happy with the proposal as it is, thanks for writing this :)
Best, D. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
On 2/27/21 9:57 AM, Daniel Moisset wrote:
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null
value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or
potentially-missing
would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
This is factually wrong for the second example; 'Optional[opt1]: str' is perfectly valid Python syntax right now.
Noted. Will update. On 2/28/21 9:26 AM, Paul Bryan wrote:
References to __required_keys__ and __optional_keys__ have now been merged into the 3.10 docs: https://github.com/python/cpython/pull/24668/files <https://github.com/python/cpython/pull/24668/files>
Cool.
As was discussed in another thread, could we deprecate __total__, ideally through this PEP?
I must have missed the other thread. By deprecate do you mean put through some kind of formal deprecation process such that accessing the attribute triggers warnings and later the attribute is removed outright? I am unfamiliar with how Python features are usually deprecated. I could potentially see deprecating `__total__` in this PEP. That flag is definitely misleading when there are keys of mixed totality, either via keys of mixed totality in a single TypedDict type, or through an inheritance chain of mixed totality.
Perhaps through text like:
The ``total`` parameter, combined with the use of ``Required[...]`` and ``Optional[...]`` manifests in the class ``__required_keys__`` and ``__optional_keys__`` attributes.
Best, -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Tue, Mar 2, 2021 at 2:25 PM David Foster <davidfstr@gmail.com> wrote:
[...]
As was discussed in another thread, could we deprecate __total__, ideally through this PEP?
I must have missed the other thread.
By deprecate do you mean put through some kind of formal deprecation process such that accessing the attribute triggers warnings and later the attribute is removed outright? I am unfamiliar with how Python features are usually deprecated.
Sounds about right. I think it takes two full release cycles.
I could potentially see deprecating `__total__` in this PEP. That flag is definitely misleading when there are keys of mixed totality, either via keys of mixed totality in a single TypedDict type, or through an inheritance chain of mixed totality.
It tells us the default totality of the current class. That would still be useful for code that introspects `__annotations__` and wants to interpret any `Required` or `NotRequired` items it finds. I think it's fine to keep it, at least as long as we're not removing `total=` from the class declaration. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
David, the PEP is looking good. I just published a version of pyright (1.1.117) that has support for the latest draft of the PEP. It was pretty straightforward to implement. Here's a [link to the change](https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb...) in case maintainers of other type checkers would like to take a look at the test cases or implementation details. There were a few questions that I ran across in the implementation that were not answered in the latest draft of the PEP. 1. How do `Required` and `NotRequired` interact with `Annotated` (PEP 593)? I assume that an `Annotated` can be contained within a `Required` or `NotRequired`, but the converse is not true. 2. Are `Required` and `NotRequired` supported only in the class form of a TypedDict? Or are they supported in the alternative syntax? For now, I have implemented it such that any use of `Required` and `NotRequired` outside of a TypedDict class definition is considered an error, so it is not allowed in the alternative syntax. -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.
On 3/2/21 1:17 PM, Eric Traut wrote:
David, the PEP is looking good.
I just published a version of pyright (1.1.117) that has support for the latest draft of the PEP. It was pretty straightforward to implement. Here's a [link to the change](https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb...) in case maintainers of other type checkers would like to take a look at the test cases or implementation details.
Fantastic! I'm hoping to get started on the mypy implementation later this weekend, after integrating the previous week's feedback into the PEP draft.
There were a few questions that I ran across in the implementation that were not answered in the latest draft of the PEP.
1. How do `Required` and `NotRequired` interact with `Annotated` (PEP 593)? I assume that an `Annotated` can be contained within a `Required` or `NotRequired`, but the converse is not true.
Agreed. That makes sense to me.
2. Are `Required` and `NotRequired` supported only in the class form of a TypedDict? Or are they supported in the alternative syntax? For now, I have implemented it such that any use of `Required` and `NotRequired` outside of a TypedDict class definition is considered an error, so it is not allowed in the alternative syntax.
I'm leaning toward allowing Required[] and NotRequired[] in the alternative syntax [1] since it already supports specifying totality. [1]: https://www.python.org/dev/peps/pep-0589/#alternative-syntax -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
I'm hoping to get started on the mypy implementation later this weekend, after integrating the previous week's feedback into the PEP draft.
Famous last words. :) I've managed to clear everything off my plate now and so am now actually starting the mypy implementation of Required[] / NotRequired[]. AKA PEP 655. However I am mindful of some upcoming timelines: * Python 3.10a7 (the last alpha) is scheduled for April 5th (in 4 days). I think this may be a last point where a new feature like Required[] can be introduced for Python 3.10. * The Steering Council will need to ultimately review PEP 655 before this feature is merged into CPython/mypy. I understand they have about a month of backlog in reviewing items at this busy time. There is a small amount of feedback that ideally I'd like to integrate before submitting the PEP to the Steering Council but I'm running out of time: * > Are `Required` and `NotRequired` supported only in the class form of a TypedDict? - I'm leaning toward allowing it in the alternative syntax but I really want experience implementing in mypy first before making a decision. If it looks at all challenging to implement, I may lean toward no, since the alternative syntax is effectively deprecated in most new code anyway. * I'm wondering if updating the introspective functions of the "typing" module [1] might involve non-trivial changes that should be discussed in the PEP. I don't have enough experience with this code to discover issues before attempting implementation. Would folks recommend that I submit the current draft of PEP 655 to the Steering Council while I'm still working on the mypy implementation, potentially discovering small issues to integrate into the PEP later, or should I wait until I finish the implementation and thus have very high confidence that all issues have been addressed in the PEP? [1]: https://docs.python.org/3/library/typing.html#introspection-helpers -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy On 3/6/21 9:01 AM, David Foster wrote:
On 3/2/21 1:17 PM, Eric Traut wrote:
David, the PEP is looking good.
I just published a version of pyright (1.1.117) that has support for the latest draft of the PEP. It was pretty straightforward to implement. Here's a [link to the change](https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb...) in case maintainers of other type checkers would like to take a look at the test cases or implementation details.
Fantastic!
I'm hoping to get started on the mypy implementation later this weekend, after integrating the previous week's feedback into the PEP draft.
There were a few questions that I ran across in the implementation that were not answered in the latest draft of the PEP.
1. How do `Required` and `NotRequired` interact with `Annotated` (PEP 593)? I assume that an `Annotated` can be contained within a `Required` or `NotRequired`, but the converse is not true.
Agreed. That makes sense to me.
2. Are `Required` and `NotRequired` supported only in the class form of a TypedDict? Or are they supported in the alternative syntax? For now, I have implemented it such that any use of `Required` and `NotRequired` outside of a TypedDict class definition is considered an error, so it is not allowed in the alternative syntax.
I'm leaning toward allowing Required[] and NotRequired[] in the alternative syntax [1] since it already supports specifying totality.
[1]: https://www.python.org/dev/peps/pep-0589/#alternative-syntax
El jue, 1 abr 2021 a las 8:03, David Foster (<davidfstr@gmail.com>) escribió:
I'm hoping to get started on the mypy implementation later this weekend, after integrating the previous week's feedback into the PEP draft.
Famous last words. :) I've managed to clear everything off my plate now and so am now actually starting the mypy implementation of Required[] / NotRequired[]. AKA PEP 655.
However I am mindful of some upcoming timelines: * Python 3.10a7 (the last alpha) is scheduled for April 5th (in 4 days). I think this may be a last point where a new feature like Required[] can be introduced for Python 3.10. * The Steering Council will need to ultimately review PEP 655 before this feature is merged into CPython/mypy. I understand they have about a month of backlog in reviewing items at this busy time.
There is a small amount of feedback that ideally I'd like to integrate before submitting the PEP to the Steering Council but I'm running out of time: * > Are `Required` and `NotRequired` supported only in the class form of a TypedDict? - I'm leaning toward allowing it in the alternative syntax but I really want experience implementing in mypy first before making a decision. If it looks at all challenging to implement, I may lean toward no, since the alternative syntax is effectively deprecated in most new code anyway. * I'm wondering if updating the introspective functions of the "typing" module [1] might involve non-trivial changes that should be discussed in the PEP. I don't have enough experience with this code to discover issues before attempting implementation.
Would folks recommend that I submit the current draft of PEP 655 to the Steering Council while I'm still working on the mypy implementation, potentially discovering small issues to integrate into the PEP later, or should I wait until I finish the implementation and thus have very high confidence that all issues have been addressed in the PEP?
Does the CPython release cycle matter much for this feature? If people can just get Required/NotRequired out of `typing_extensions` in older Python versions, I don't think it's much of a problem if CPython's typing.py doesn't gain support for the feature until 3.11.
[1]: https://docs.python.org/3/library/typing.html#introspection-helpers
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 3/6/21 9:01 AM, David Foster wrote:
On 3/2/21 1:17 PM, Eric Traut wrote:
David, the PEP is looking good.
I just published a version of pyright (1.1.117) that has support for the latest draft of the PEP. It was pretty straightforward to implement. Here's a [link to the change]( https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb...)
in case maintainers of other type checkers would like to take a look at the test cases or implementation details.
Fantastic!
I'm hoping to get started on the mypy implementation later this weekend, after integrating the previous week's feedback into the PEP draft.
There were a few questions that I ran across in the implementation that were not answered in the latest draft of the PEP.
1. How do `Required` and `NotRequired` interact with `Annotated` (PEP 593)? I assume that an `Annotated` can be contained within a `Required` or `NotRequired`, but the converse is not true.
Agreed. That makes sense to me.
2. Are `Required` and `NotRequired` supported only in the class form of a TypedDict? Or are they supported in the alternative syntax? For now, I have implemented it such that any use of `Required` and `NotRequired` outside of a TypedDict class definition is considered an error, so it is not allowed in the alternative syntax.
I'm leaning toward allowing Required[] and NotRequired[] in the alternative syntax [1] since it already supports specifying totality.
[1]: https://www.python.org/dev/peps/pep-0589/#alternative-syntax
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
On 4/1/21 8:34 AM, Jelle Zijlstra wrote:
Would folks recommend that I submit the current draft of PEP 655 to the Steering Council while I'm still working on the mypy implementation, potentially discovering small issues to integrate into the PEP later, or should I wait until I finish the implementation and thus have very high confidence that all issues have been addressed in the PEP?
Does the CPython release cycle matter much for this feature? If people can just get Required/NotRequired out of `typing_extensions` in older Python versions, I don't think it's much of a problem if CPython's typing.py doesn't gain support for the feature until 3.11.
Good point. I'll retarget my implementation toward "typing_extensions" rather than "typing" itself under the assumption that I'll probably miss the CPython release window. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
I just wanted to show my support and say that I think this is a great idea. It would help with several internal type annotations for FastAPI, improving the developer experience (for final users). And it would help a lot in some other places like the type annotations for the ASGI spec.
On 5/10/21 4:34 AM, Sebastián Ramírez wrote:
I just wanted to show my support and say that I think this is a great idea.
Thanks Sebastián. Quick status update on the progress: * Required[] support for mypy is drafted and pending review. * Required[] support for typing_extensions is drafted is going through at least one round of major revisions. * Some minor PEP revisions based on implementation experience are pending draft. Best, -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
After an extended hiatus I have a new draft of PEP 655 implementing the Required[] and NotRequired[] syntax for marking individual keys of a TypedDict. Please see the latest text at the bottom of this email. It is my goal to get this PEP ready for submission to the steering council in time for implementation in Python 3.11, with feature freeze scheduled for 2022-05-06. [0] The following notable changes have been integrated since the last draft posted 26-Feb-2021: - Clarifications: * Required & NotRequired at the same time is an error: > class Movie(TypedDict): year: NotRequired[Required[int]] # ERROR * (Not)Required can be used in the alternative syntax for TypedDict: > Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]}) # ok - New sections: + Interaction with ``Annotated[...]`` * (Not)Required can be used with Annotated[], in ANY nesting order 🤔[1] + Interaction with ``get_type_hints()`` * Strips (Not)Required by default, unless include_extras=True 🤔 + Interaction with ``get_origin()`` and ``get_args()`` * (Not)Required is recognized by these functions + Interaction with ``__required_keys__`` and ``__optional_keys__`` * (Not)Required implies that the marked key is definitely in one of these key-sets - News: * Both mypy and pyright have working implementations of (Not)Required! 🎉 🤔 = potentially controversial Regarding allowing mixing (Not)Required[] with Annotated[] in ANY nesting order, I am somewhat conflicted: It would be easier (at least in mypy and perhaps in pyright [1]) to require that (Not)Required[] be on the *outside* of Annotated[] and never on the inside. But from a user's point of view, that restriction feels somewhat arbitrary to me. Comments? [0]: https://www.python.org/dev/peps/pep-0664/ [1]: https://mail.python.org/archives/list/typing-sig@python.org/message/S36LHF2T... -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Jan-2021 Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[]``) would be too similar to ``Optional[]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[]`` or ``NotRequired[]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[]`` and ``NotRequired[]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] It is an error to use both ``Required[]`` and ``NotRequired[]`` at the same time: :: class Movie(TypedDict): title: str year: NotRequired[Required[int]] # ERROR The `alternative syntax <https://www.python.org/dev/peps/pep-0589/#alternative-syntax>`__ for TypedDict also supports ``Required[]`` and ``NotRequired[]``: :: Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]}) Interaction with ``Annotated[]`` ----------------------------------- ``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``, in any nesting order: :: class Movie(TypedDict): title: str year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok :: class Movie(TypedDict): title: str year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok Interaction with ``get_type_hints()`` ------------------------------------- ``typing.get_type_hints(...)`` applied to a TypedDict will by default strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers, since these qualifiers are expected to be inconvenient for code casually introspecting type annotations. ``typing.get_type_hints(..., include_extras=True)`` however *will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers, for advanced code introspecting type annotations that wishes to preserve *all* annotations in the original source: :: class Movie(TypedDict): title: str year: NotRequired[int] assert get_type_hints(Movie) == \ {'title': str, 'year': int} assert get_type_hints(Movie, include_extras=True) == \ {'title': str, 'year': NotRequired[int]} Interaction with ``get_origin()`` and ``get_args()`` ---------------------------------------------------- ``typing.get_origin()`` and ``typing.get_args()`` will be updated to recognize ``Required[]`` and ``NotRequired[]``: :: assert get_origin(Required[int]) is Required assert get_args(Required[int]) == (int,) assert get_origin(NotRequired[int]) is NotRequired assert get_args(NotRequired[int]) == (int,) Interaction with ``__required_keys__`` and ``__optional_keys__`` ---------------------------------------------------------------- An item marked with ``Required[]`` will always appear in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item marked with ``NotRequired[]`` will always appear in ``__optional_keys__``. :: assert Movie.__required_keys__ == frozenset({'title'}) assert Movie.__optional_keys__ == frozenset({'year'}) Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[]`` or ``NotRequired[]`` alongside ``Optional[]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Okay (required for Python 3.5.3-3.6): :: class Dog(TypedDict): name: str owner: 'NotRequired[str|None]' No: :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The `mypy <http://www.mypy-lang.org/>`__ `0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__ and `pyright <https://github.com/Microsoft/pyright>`__ `1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__ type checkers support ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax causes ``Optional[]`` to take on different meanings depending on where it is positioned, which is inconsistent and confusing. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[]`` and ``NotRequired[]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Note when reconsidering introducing this shortform syntax that ``+``, ``-``, and ``~`` already have existing meanings in the Python typing world: covariant, contravariant, and invariant: :: >>> from typing import TypeVar >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V')) (+T, -U, ~V) Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< On 2/26/21 7:09 PM, David Foster wrote:
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
El lun, 17 ene 2022 a las 15:20, David Foster (<davidfstr@gmail.com>) escribió:
After an extended hiatus I have a new draft of PEP 655 implementing the Required[] and NotRequired[] syntax for marking individual keys of a TypedDict. Please see the latest text at the bottom of this email.
It is my goal to get this PEP ready for submission to the steering council in time for implementation in Python 3.11, with feature freeze scheduled for 2022-05-06. [0]
The following notable changes have been integrated since the last draft posted 26-Feb-2021:
- Clarifications: * Required & NotRequired at the same time is an error: > class Movie(TypedDict): year: NotRequired[Required[int]] # ERROR
* (Not)Required can be used in the alternative syntax for TypedDict: > Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]}) # ok
- New sections: + Interaction with ``Annotated[...]`` * (Not)Required can be used with Annotated[], in ANY nesting order 🤔[1] + Interaction with ``get_type_hints()`` * Strips (Not)Required by default, unless include_extras=True 🤔 + Interaction with ``get_origin()`` and ``get_args()`` * (Not)Required is recognized by these functions + Interaction with ``__required_keys__`` and ``__optional_keys__`` * (Not)Required implies that the marked key is definitely in one of these key-sets
- News: * Both mypy and pyright have working implementations of (Not)Required! 🎉
And pyanalyze too :)
🤔 = potentially controversial
Regarding allowing mixing (Not)Required[] with Annotated[] in ANY nesting order, I am somewhat conflicted: It would be easier (at least in mypy and perhaps in pyright [1]) to require that (Not)Required[] be on the *outside* of Annotated[] and never on the inside. But from a user's point of view, that restriction feels somewhat arbitrary to me. Comments?
Both mypy and pyright currently disallow `Annotated[ClassVar[...], ...]`, which feels closely analogous to me, so I think we should also disallow `Annotated[Required[...], ...]`.
[0]: https://www.python.org/dev/peps/pep-0664/ [1]:
https://mail.python.org/archives/list/typing-sig@python.org/message/S36LHF2T...
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Jan-2021 Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022
Abstract ========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[]`` which can be used on individual items to mark them as potentially-missing.
Motivation ==========
It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True title: str
class Movie(_MovieBase, total=False): year: int
Having to declare two different TypedDict types for this purpose is cumbersome.
Rationale =========
One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript:
::
interface Movie { title: string; year?: number; // ? marks potentially-missing keys }
The difficulty is that the best word for marking a potentially-missing key, ``Optional[]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work:
::
class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[]``) would be too similar to ``Optional[]`` and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[]``.
Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[]`` form for that case.
Specification =============
The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False): title: Required[str] year: int
Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int]
It is an error to use ``Required[]`` or ``NotRequired[]`` in any location that is not an item of a TypedDict.
It is valid to use ``Required[]`` and ``NotRequired[]`` even for items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int]
It is an error to use both ``Required[]`` and ``NotRequired[]`` at the same time:
::
class Movie(TypedDict): title: str year: NotRequired[Required[int]] # ERROR
The `alternative syntax <https://www.python.org/dev/peps/pep-0589/#alternative-syntax>`__ for TypedDict also supports ``Required[]`` and ``NotRequired[]``:
::
Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})
Interaction with ``Annotated[]`` -----------------------------------
``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``, in any nesting order:
::
class Movie(TypedDict): title: str year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok
::
class Movie(TypedDict): title: str year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok
Interaction with ``get_type_hints()`` -------------------------------------
``typing.get_type_hints(...)`` applied to a TypedDict will by default strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers, since these qualifiers are expected to be inconvenient for code casually introspecting type annotations.
``typing.get_type_hints(..., include_extras=True)`` however *will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers, for advanced code introspecting type annotations that wishes to preserve *all* annotations in the original source:
::
class Movie(TypedDict): title: str year: NotRequired[int]
assert get_type_hints(Movie) == \ {'title': str, 'year': int} assert get_type_hints(Movie, include_extras=True) == \ {'title': str, 'year': NotRequired[int]}
Interaction with ``get_origin()`` and ``get_args()`` ----------------------------------------------------
``typing.get_origin()`` and ``typing.get_args()`` will be updated to recognize ``Required[]`` and ``NotRequired[]``:
::
assert get_origin(Required[int]) is Required assert get_args(Required[int]) == (int,)
assert get_origin(NotRequired[int]) is NotRequired assert get_args(NotRequired[int]) == (int,)
Interaction with ``__required_keys__`` and ``__optional_keys__`` ----------------------------------------------------------------
An item marked with ``Required[]`` will always appear in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.
::
assert Movie.__required_keys__ == frozenset({'title'}) assert Movie.__optional_keys__ == frozenset({'year'})
Backwards Compatibility =======================
No backward incompatible changes are made by this PEP.
How to Teach This =================
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[]``.
To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[]``.
If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[]`` or ``NotRequired[]`` alongside ``Optional[]`` within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Okay (required for Python 3.5.3-3.6):
::
class Dog(TypedDict): name: str owner: 'NotRequired[str|None]'
No:
::
class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]]
Reference Implementation ========================
The `mypy <http://www.mypy-lang.org/>`__ `0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__ and `pyright <https://github.com/Microsoft/pyright>`__ `1.1.117 < https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb...
`__ type checkers support ``Required`` and ``NotRequired``.
A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module.
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value
This syntax would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes.
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value
This syntax causes ``Optional[]`` to take on different meanings depending on where it is positioned, which is inconsistent and confusing.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator -------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[] opt1: str req2: +float
class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[] req2: float
class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float
Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar.
It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[]`` and ``NotRequired[]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options.
Note when reconsidering introducing this shortform syntax that ``+``, ``-``, and ``~`` already have existing meanings in the Python typing world: covariant, contravariant, and invariant:
::
>>> from typing import TypeVar >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V')) (+T, -U, ~V)
Marking absence of a value with a special constant --------------------------------------------------
We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict): req1: int opt1: str|Missing req2: float
Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined:
::
class MyClass: attr: int|Missing
def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10
::
def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing
Misalignment with how unions apply to values ''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided '''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks:
::
class Packet: data: Union[str, bytes]
def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet: data: Union[str, Missing]
def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global variables:
::
warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!'
if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially.
Difficult to implement ''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_
Introduces a second null-like value into Python '''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”. ---------------------------------------------------------------------------
``Optional[]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of “nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str]
This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[]`` is different than in other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item” -----------------------------------------------
- Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked
References ==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD...
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
On 2/26/21 7:09 PM, David Foster wrote:
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
While I agree with the sentiment and the comments below, I think we should nevertheless always allow `Annotated` on the outside, even in the case of `ClassVar` and `Final`. In https://bugs.python.org/msg411067 a user is complaining about the inability to place `Annotated` on the outside of `ClassVar`, causing difficulty. I've seen several complaints on the mailing list that there's too much tension between typing users of annotation space and non-typing users of annotation space. If we want to continue to claim typing annotations aren't going to interfere with people we owe it to them to take issues with our blessed typing-friendly annotation format (`Annotated`) seriously. In that vein, it seems to me the cost of typing being a good citizen to other users of annotations should require us to always accept `Annotated` on the outside of annotations to avoid forcing the burden of being able to parse typing on all other users of annotations. If someone wishes to annotate a specific inner type with `Annotated`, then they are already buying into being able to parse typing anyway. There seems to be a tendency to view `Annotated` annotations as metadata on a *type* to justify where it should go, but I worry we're being too prescriptive about what an annotation can or should be and by doing so we risk further alienating other users of annotation space who just want a simple way to introspect metadata they wish to add.
On 1/21/22 3:43 PM, Gregory Beauregard via Typing-sig wrote:
While I agree with the sentiment and the comments below, I think we should nevertheless always allow `Annotated` on the outside, even in the case of `ClassVar` and `Final`. In https://bugs.python.org/msg411067 a user is complaining about the inability to place `Annotated` on the outside of `ClassVar`, causing difficulty.
I've seen several complaints on the mailing list that there's too much tension between typing users of annotation space and non-typing users of annotation space. If we want to continue to claim typing annotations aren't going to interfere with people we owe it to them to take issues with our blessed typing-friendly annotation format (`Annotated`) seriously.
In that vein, it seems to me the cost of typing being a good citizen to other users of annotations should require us to always accept `Annotated` on the outside of annotations to avoid forcing the burden of being able to parse typing on all other users of annotations. If someone wishes to annotate a specific inner type with `Annotated`, then they are already buying into being able to parse typing anyway.
On 1/21/22 5:50 PM, Mehdi2277 wrote:
If you add [the requirement that mixing Annotated and Required must force Required on the outside and disallow Annotated on the outside], I'll need to add logic that works around this for runtime annotation using library. [...] I would expect any other library using Annotated for it's purpose of examining other metadata will need similar edge case logic to handle typeddicts properly.
Annotated I ideally would view as purely Identity to any typeform. [...]
I am receptive to this line of argument and was thinking there might be users of Annotated[T, ...] that wanted it to always be possible to put Annotated[] as the outermost item. Medhi's example confirms this suspicion. Therefore: (1) I'm looking at leaving in my current language in PEP 655 that allows (Not)Required[] and Annotated[] to be nested in *any* order. In particular Annotated[] is allowed to wrap (Not)Required[]. - I plan to add some explanatory language as to *why* I'm allowing nesting in any order. (2) I think that ClassVar[] and Final[] should be altered to also allow Annotated[] to be mixed in with any nesting order, but I'll looking at leaving that work to a future PEP, since it feels unrelated to the TypedDict-specific alterations that PEP 655 is making. - So I'll mention this inconsistency in PEP 655 (but not attempt to amend it immediately). -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
This inconsistency should be resolved now. It turned out the `ClassVar` and `Final` behavior wasn't specified in a PEP or test at all so I opened an issue at https://bugs.python.org/issue46491 Since there was agreement, it was merged in 3.11 and backported to 3.10 and 3.9. I've also backported the change to `typing_extensions` and opened issues to change type checker behavior or verified it already works in mypy, pyright, pyre, and pytype. Thanks everyone for help. I've opened an issue for the corresponding change in `dataclasses` at https://bugs.python.org/issue46511 which is a tad more involved and currently seeking feedback on the typing approach to take. https://github.com/python/mypy/issues/12061 https://github.com/microsoft/pyright/commit/8ce7fa6e0ad883fb17407add955273bb... https://github.com/facebook/pyre-check/issues/577 https://github.com/google/pytype/issues/1110
LGTM. Thanks for working on this. Comments inline: On Mon, 2022-01-17 at 15:20 -0800, David Foster wrote: [snip]
Regarding allowing mixing (Not)Required[] with Annotated[] in ANY nesting order, I am somewhat conflicted: It would be easier (at least in mypy and perhaps in pyright [1]) to require that (Not)Required[] be on the *outside* of Annotated[] and never on the inside. But from a user's point of view, that restriction feels somewhat arbitrary to me. Comments?
I already have to navigate this with Annotated[Optional[...]] and Optional[Annotated[...]]. I've definitely seen cases where each is valid, so currently allow for both. [snip]
- Omittable – too easy to confuse with optional
The train is probably too far out of the station, but... I'd actually be pretty comfortable with just `Omittable`, which would only be usable if `total=True`. I think its meaning is clear, and given Optional should be eventually deprecated by PEP 604, think such confusion could be overcome. [snip]
I agree with Jelle that `Annotated[Required[...], ...]` should be disallowed. The `Required` part is associated with the field that it's annotating, so it logically belongs on the outside, as in `Required[Annotated[..., ...]]`. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft
+1. Good point. On Mon, 2022-01-17 at 23:54 +0000, Eric Traut wrote:
I agree with Jelle that `Annotated[Required[...], ...]` should be disallowed. The `Required` part is associated with the field that it's annotating, so it logically belongs on the outside, as in `Required[Annotated[..., ...]]`.
-Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
Following up on Gregory's good overview on my bpo comment, If you add this rule, I'll need to add logic that works around this for runtime annotation using library. The primary motivation of Annotated[T, x, y, ...] is for x to be used for non-type checkers to attach arbitrary metadata they use as needed. I currently use the metadata to define parser information + docstrings on top of the type T as a way to have type safe json/yaml for more complex python objects. For my usage anywhere var: T is legal, var: Annotated[T, ...] should be legal and each edge case requires adding an edge condition to my parsing rules to work around it. Annotated I ideally would view as purely Identity to any typeform. Not just python class types, but other special type form like final/classvar/others I'm unaware of. The only type checker expectation I have is the first argument is a valid type annotation and other arguments are syntactically valid python objects. I would expect any other library using Annotated for it's purpose of examining other metadata will need similar edge case logic to handle typeddicts properly.
Recent feedback has been integrated. Please see latest PEP text at the bottom of this email. I think this draft is likely stable enough to submit to the Steering Council. Major changes since last revision: * Integrate feedback RE how typing.TypedDict will (not) support typing_extensions.Required in certain Python versions. In particular update the How to Teach This section * Explain why `Annotated[Required[...], ...]` is allowed. See the full diff at: https://github.com/python/peps/pull/2287 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Jan-2021 Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022 Abstract ======== :pep:`589` defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[]``) would be too similar to ``Optional[]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[]`` or ``NotRequired[]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[]`` and ``NotRequired[]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] It is an error to use both ``Required[]`` and ``NotRequired[]`` at the same time: :: class Movie(TypedDict): title: str year: NotRequired[Required[int]] # ERROR The :pep:`alternative syntax <589#alternative-syntax>` for TypedDict also supports ``Required[]`` and ``NotRequired[]``: :: Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]}) Interaction with ``Annotated[]`` ----------------------------------- ``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``, in any nesting order: :: class Movie(TypedDict): title: str year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok :: class Movie(TypedDict): title: str year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok In particular allowing ``Annotated[]`` to be the outermost annotation for an item allows better interoperability with non-typing uses of annotations, which may always want ``Annotated[]`` as the outermost annotation. [3]_ Interaction with ``get_type_hints()`` ------------------------------------- ``typing.get_type_hints(...)`` applied to a TypedDict will by default strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers, since these qualifiers are expected to be inconvenient for code casually introspecting type annotations. ``typing.get_type_hints(..., include_extras=True)`` however *will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers, for advanced code introspecting type annotations that wishes to preserve *all* annotations in the original source: :: class Movie(TypedDict): title: str year: NotRequired[int] assert get_type_hints(Movie) == \ {'title': str, 'year': int} assert get_type_hints(Movie, include_extras=True) == \ {'title': str, 'year': NotRequired[int]} Interaction with ``get_origin()`` and ``get_args()`` ---------------------------------------------------- ``typing.get_origin()`` and ``typing.get_args()`` will be updated to recognize ``Required[]`` and ``NotRequired[]``: :: assert get_origin(Required[int]) is Required assert get_args(Required[int]) == (int,) assert get_origin(NotRequired[int]) is NotRequired assert get_args(NotRequired[int]) == (int,) Interaction with ``__required_keys__`` and ``__optional_keys__`` ---------------------------------------------------------------- An item marked with ``Required[]`` will always appear in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item marked with ``NotRequired[]`` will always appear in ``__optional_keys__``. :: assert Movie.__required_keys__ == frozenset({'title'}) assert Movie.__optional_keys__ == frozenset({'year'}) Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[]`` or ``NotRequired[]`` alongside ``Optional[]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Okay (required for Python 3.5.3-3.6): :: class Dog(TypedDict): name: str owner: 'NotRequired[str|None]' No: :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Usage in Python <3.11 --------------------- If your code supports Python <3.11 and wishes to use ``Required[]`` or ``NotRequired[]`` then it should use ``typing_extensions.TypedDict`` rather than ``typing.TypedDict`` because the latter will not understand ``(Not)Required[]``. In particular ``__required_keys__`` and ``__optional_keys__`` on the resulting TypedDict type will not be correct: Yes (Python 3.11+ only): :: from typing import NotRequired, TypedDict class Dog(TypedDict): name: str owner: NotRequired[str|None] Yes (Python <3.11 and 3.11+): :: from __future__ import annotations # for Python 3.7-3.9 from typing_extensions import NotRequired, TypedDict # for Python <3.11 with (Not)Required class Dog(TypedDict): name: str owner: NotRequired[str|None] No (Python <3.11 and 3.11+): :: from typing import TypedDict # oops: should import from typing_extensions instead from typing_extensions import NotRequired class Movie(TypedDict): title: str year: NotRequired[int] assert Movie.__required_keys__ == frozenset({'title', 'year'}) # yikes assert Movie.__optional_keys__ == frozenset() # yikes Reference Implementation ======================== The `mypy <http://www.mypy-lang.org/>`__ `0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__, `pyright <https://github.com/Microsoft/pyright>`__ `1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__, and `pyanalyze <https://github.com/quora/pyanalyze>`__ `0.4.0 <https://pyanalyze.readthedocs.io/en/latest/changelog.html#version-0-4-0-november-18-2021>`__ type checkers support ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value This syntax causes ``Optional[]`` to take on different meanings depending on where it is positioned, which is inconsistent and confusing. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[]`` and ``NotRequired[]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Note when reconsidering introducing this shortform syntax that ``+``, ``-``, and ``~`` already have existing meanings in the Python typing world: covariant, contravariant, and invariant: :: >>> from typing import TypeVar >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V')) (+T, -U, ~V) Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD... .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6... .. [3] https://bugs.python.org/issue46491 Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
To define a TypedDict where most keys are required and some are
Hey David, Thanks for the useful PEP! Users will find it easier to use a single TypedDict with Required/NotRequired than to use TypedDict inheritance. I've rarely seen anyone use the latter. # Type Compatibility with PEP 589 TypedDicts Could you mention how type compatibility checks work for a PEP-655-style TypedDict against a PEP-589-style TypedDict (and vice versa)? For example, is a single-class Movie with a Required `name` and NotRequired `year` compatible with the inherited `Movie` in your Motivation example? I'm guessing the answer is "obviously yes", but it's probably worth explicitly laying out in the PEP. We could do this by saying that the following two TypedDicts are compatible with each other: ``` # old style class MovieBase(TypedDict): name: str class Movie1(MovieBase, total=False): year: int # new style class Movie2(TypedDict): name: Required[str] year: NotRequired[int] def expect_movie1(movie: Movie1) -> None: ... movie2: Movie2 expect_movie1(movie2) # and vice versa ``` Similarly, is there any difference in runtime or type-checking behavior between a TypedDict with a redundant `name: Required[str]` and one with `name: str`? I'm guessing not but might be worth specifying. Otherwise, there might be questions like, do they generate the same constructor for the Movie class? Once we specify that they are equivalent for type-checking purposes, type checker implementers (such as me on Pyre) can just desugar the latter to the former without any issues. # Other comments A couple of quick readability comments: **Motivation section**: You've shown the "Before" example. Could you show the "After" example (i.e., the same code using your PEP's feature)? That would make it easy for people who just want to get the gist of the PEP without diving into the Specification. Would also be great if you can also demonstrate both Required and NotRequired, but this is... not required :) **Specification**: Could you nest the three sections for "Interaction with get_type_hints/get_origin/__required_keys" under a "Runtime Behavior" section? That way, users who care just about typing can skip that section. Is it valid to use `Required` in an inherited TypedDict? **How to Teach This section**: potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with NotRequired[]. Could you clarify that "define a single TypedDict as normal" means "without a `total` keyword"? Not obvious otherwise. Best, On Fri, Jan 28, 2022 at 8:07 AM David Foster <davidfstr@gmail.com> wrote:
Recent feedback has been integrated. Please see latest PEP text at the bottom of this email. I think this draft is likely stable enough to submit to the Steering Council.
Major changes since last revision:
* Integrate feedback RE how typing.TypedDict will (not) support typing_extensions.Required in certain Python versions. In particular update the How to Teach This section
* Explain why `Annotated[Required[...], ...]` is allowed.
See the full diff at: https://github.com/python/peps/pull/2287
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Jan-2021 Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022
Abstract ========
:pep:`589` defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[]`` which can be used on individual items to mark them as potentially-missing.
Motivation ==========
It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True title: str
class Movie(_MovieBase, total=False): year: int
Having to declare two different TypedDict types for this purpose is cumbersome.
Rationale =========
One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript:
::
interface Movie { title: string; year?: number; // ? marks potentially-missing keys }
The difficulty is that the best word for marking a potentially-missing key, ``Optional[]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work:
::
class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[]``) would be too similar to ``Optional[]`` and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[]``.
Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[]`` form for that case.
Specification =============
The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False): title: Required[str] year: int
Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int]
It is an error to use ``Required[]`` or ``NotRequired[]`` in any location that is not an item of a TypedDict.
It is valid to use ``Required[]`` and ``NotRequired[]`` even for items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int]
It is an error to use both ``Required[]`` and ``NotRequired[]`` at the same time:
::
class Movie(TypedDict): title: str year: NotRequired[Required[int]] # ERROR
The :pep:`alternative syntax <589#alternative-syntax>` for TypedDict also supports ``Required[]`` and ``NotRequired[]``:
::
Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})
Interaction with ``Annotated[]`` -----------------------------------
``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``, in any nesting order:
::
class Movie(TypedDict): title: str year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok
::
class Movie(TypedDict): title: str year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok
In particular allowing ``Annotated[]`` to be the outermost annotation for an item allows better interoperability with non-typing uses of annotations, which may always want ``Annotated[]`` as the outermost annotation. [3]_
Interaction with ``get_type_hints()`` -------------------------------------
``typing.get_type_hints(...)`` applied to a TypedDict will by default strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers, since these qualifiers are expected to be inconvenient for code casually introspecting type annotations.
``typing.get_type_hints(..., include_extras=True)`` however *will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers, for advanced code introspecting type annotations that wishes to preserve *all* annotations in the original source:
::
class Movie(TypedDict): title: str year: NotRequired[int]
assert get_type_hints(Movie) == \ {'title': str, 'year': int} assert get_type_hints(Movie, include_extras=True) == \ {'title': str, 'year': NotRequired[int]}
Interaction with ``get_origin()`` and ``get_args()`` ----------------------------------------------------
``typing.get_origin()`` and ``typing.get_args()`` will be updated to recognize ``Required[]`` and ``NotRequired[]``:
::
assert get_origin(Required[int]) is Required assert get_args(Required[int]) == (int,)
assert get_origin(NotRequired[int]) is NotRequired assert get_args(NotRequired[int]) == (int,)
Interaction with ``__required_keys__`` and ``__optional_keys__`` ----------------------------------------------------------------
An item marked with ``Required[]`` will always appear in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.
::
assert Movie.__required_keys__ == frozenset({'title'}) assert Movie.__optional_keys__ == frozenset({'year'})
Backwards Compatibility =======================
No backward incompatible changes are made by this PEP.
How to Teach This =================
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[]``.
To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[]``.
If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[]`` or ``NotRequired[]`` alongside ``Optional[]`` within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Okay (required for Python 3.5.3-3.6):
::
class Dog(TypedDict): name: str owner: 'NotRequired[str|None]'
No:
::
class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]]
Usage in Python <3.11 ---------------------
If your code supports Python <3.11 and wishes to use ``Required[]`` or ``NotRequired[]`` then it should use ``typing_extensions.TypedDict`` rather than ``typing.TypedDict`` because the latter will not understand ``(Not)Required[]``. In particular ``__required_keys__`` and ``__optional_keys__`` on the resulting TypedDict type will not be correct:
Yes (Python 3.11+ only):
::
from typing import NotRequired, TypedDict
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Yes (Python <3.11 and 3.11+):
::
from __future__ import annotations # for Python 3.7-3.9
from typing_extensions import NotRequired, TypedDict # for Python <3.11 with (Not)Required
class Dog(TypedDict): name: str owner: NotRequired[str|None]
No (Python <3.11 and 3.11+):
::
from typing import TypedDict # oops: should import from typing_extensions instead from typing_extensions import NotRequired
class Movie(TypedDict): title: str year: NotRequired[int]
assert Movie.__required_keys__ == frozenset({'title', 'year'}) # yikes assert Movie.__optional_keys__ == frozenset() # yikes
Reference Implementation ========================
The `mypy <http://www.mypy-lang.org/>`__ `0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html
`__, `pyright <https://github.com/Microsoft/pyright>`__ `1.1.117 < https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cb... `__, and `pyanalyze <https://github.com/quora/pyanalyze>`__ `0.4.0 < https://pyanalyze.readthedocs.io/en/latest/changelog.html#version-0-4-0-nove... `__ type checkers support ``Required`` and ``NotRequired``.
A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module.
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value
This syntax would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes.
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value
This syntax causes ``Optional[]`` to take on different meanings depending on where it is positioned, which is inconsistent and confusing.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator -------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[] opt1: str req2: +float
class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[] req2: float
class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float
Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar.
It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[]`` and ``NotRequired[]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options.
Note when reconsidering introducing this shortform syntax that ``+``, ``-``, and ``~`` already have existing meanings in the Python typing world: covariant, contravariant, and invariant:
::
>>> from typing import TypeVar >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V')) (+T, -U, ~V)
Marking absence of a value with a special constant --------------------------------------------------
We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict): req1: int opt1: str|Missing req2: float
Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined:
::
class MyClass: attr: int|Missing
def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10
::
def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing
Misalignment with how unions apply to values ''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided '''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks:
::
class Packet: data: Union[str, bytes]
def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet: data: Union[str, Missing]
def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global variables:
::
warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!'
if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially.
Difficult to implement ''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_
Introduces a second null-like value into Python '''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”. ---------------------------------------------------------------------------
``Optional[]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`.
Change Optional to mean “optional item” in certain contexts instead of “nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str]
This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[]`` is different than in other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item” -----------------------------------------------
- Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked
References ==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD...
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6...
.. [3] https://bugs.python.org/issue46491
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
-- S Pradeep Kumar
On 2/1/22 3:07 PM, S Pradeep Kumar wrote:
# Type Compatibility with PEP 589 TypedDicts
Could you mention how type compatibility checks work for a PEP-655-style TypedDict against a PEP-589-style TypedDict (and vice versa)? For example, is a single-class Movie with a Required `name` and NotRequired `year` compatible with the inherited `Movie` in your Motivation example?
I'm guessing the answer is "obviously yes", but it's probably worth explicitly laying out in the PEP. [...]
I agree that being explicit RE the interaction with total=False would be valuable. I've added a new "Interaction with total=False" section.
Similarly, is there any difference in runtime or type-checking behavior between a TypedDict with a redundant `name: Required[str]` and one with `name: str`? I'm guessing not but might be worth specifying. Otherwise, there might be questions like, do they generate the same constructor for the Movie class?
There is no difference. I think this is probably clear enough just based on the word "redundant".
Once we specify that they are equivalent for type-checking purposes, type checker implementers (such as me on Pyre) can just desugar the latter to the former without any issues.
Aye, that should be possible to do. However I suspect typecheckers may be able to provide more targeted error messages if they provide some degree of special recognition for the Required[] or NotRequired[] syntax.
# Other comments
A couple of quick readability comments:
**Motivation section**: You've shown the "Before" example. Could you show the "After" example (i.e., the same code using your PEP's feature)? That would make it easy for people who just want to get the gist of the PEP without diving into the Specification.
Agreed. Done.
Would also be great if you can also demonstrate both Required and NotRequired, but this is... not required :)
I'll leave that detail for the Specification section.
**Specification**: Could you nest the three sections for "Interaction with get_type_hints/get_origin/__required_keys" under a "Runtime Behavior" section? That way, users who care just about typing can skip that section.
I had thought about doing that earlier. I'll take your mention as a vote to actually do it. Done.
Is it valid to use `Required` in an inherited TypedDict?
Yes; it is not disallowed. Hopefully the equivalence related to a total=False TypedDict should make this clear implicitly.
**How to Teach This section**:
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with NotRequired[].
Could you clarify that "define a single TypedDict as normal" means "without a `total` keyword"? Not obvious otherwise.
Done. All of the preceding feedback is pending integration to the PEP in the following PR: -> https://github.com/python/peps/pull/2323 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
I just thought of an additional motivation for this PEP—up to you whether to include it. The functional-style TypedDict syntax (`X = TypedDict("X", {"a": int})`) was added to support pre-PEP 526 users, but it's also useful if one of your keys isn't a valid Python identifier. For example, we use it in typeshed for a tkinter TypedDict that uses "in" as a key. You can use total=True/False with this syntax, but you can't use inheritance, so under the current syntax, there's no way to create a TypedDict with non-identifier keys that mixes required and non-required keys. I can't imagine that's a very common use case, but PEP 655 makes it possible. El vie, 26 feb 2021 a las 19:09, David Foster (<davidfstr@gmail.com>) escribió:
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Abstract ========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing.
Motivation ==========
It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True title: str
class Movie(_MovieBase, total=False): year: int
Having to declare two different TypedDict types for this purpose is cumbersome.
Rationale =========
One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript:
::
interface Movie { title: string; year?: number; // ? marks potentially-missing keys }
The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work:
::
class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``.
Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case.
Specification =============
The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False): title: Required[str] year: int
Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int]
It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict.
It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int]
Backwards Compatibility =======================
No backward incompatible changes are made by this PEP.
How to Teach This =================
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``.
To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``.
If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Avoid (unless Python 3.5-3.6):
::
class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]]
Reference Implementation ========================
The goal is to be able to make the following statement:
The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module.
The mypy implementation is currently still being worked on.
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator -------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float
class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float
class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float
Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar.
It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options.
Marking absence of a value with a special constant --------------------------------------------------
We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict): req1: int opt1: str|Missing req2: float
Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined:
::
class MyClass: attr: int|Missing
def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10
::
def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing
Misalignment with how unions apply to values ''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided '''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks:
::
class Packet: data: Union[str, bytes]
def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet: data: Union[str, Missing]
def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global variables:
::
warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!'
if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially.
Difficult to implement ''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_
Introduces a second null-like value into Python '''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”. ---------------------------------------------------------------------------
``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of “nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str]
This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item” -----------------------------------------------
- Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked
References ==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD...
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
I've used this feature before. I've only needed it once so far, but I'd expect if you have heavy usage of json that hyphens can be an issue. Sometimes I work with keys with hyphens in key name and that forces functional style.
Update: PEP 655 was submitted to python-dev on Jan 29 (1 month ago) [1] and to Discourse/PEPs on Feb 20 (3 days ago) [2] for core dev review, here on this email thread in typing-sig. I have since then received 2 items of feedback, one of which is still pending integration. [3] [4] **Please send me any further feedback by the end of this weekend, Sun Feb 27.** After that point I plan to submit the PEP to the Steering Council for final review as per PEP 1 [5]. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [1]: https://mail.python.org/archives/list/python-dev@python.org/message/C7QCZ5J2... [2]: https://discuss.python.org/t/pep-655-required-and-notrequired-for-typeddict/... [3]: https://github.com/python/peps/pull/2323 [4]: https://github.com/python/peps/pull/2359 [5]: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution
participants (10)
-
Daniel Moisset -
David Foster -
Eric Traut -
Gregory Beauregard -
Guido van Rossum -
Jelle Zijlstra -
Mehdi2277 -
Paul Bryan -
S Pradeep Kumar -
Sebastián Ramírez