I just thought of an additional motivation for this PEP—up to you whether to include it. The functional-style TypedDict syntax (`X = TypedDict("X", {"a": int})`) was added to support pre-PEP 526 users, but it's also useful if one of your keys isn't a valid Python identifier. For example, we use it in typeshed for a tkinter TypedDict that uses "in" as a key. You can use total=True/False with this syntax, but you can't use inheritance, so under the current syntax, there's no way to create a TypedDict with non-identifier keys that mixes required and non-required keys. I can't imagine that's a very common use case, but PEP 655 makes it possible. El vie, 26 feb 2021 a las 19:09, David Foster (<davidfstr@gmail.com>) escribió:
The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Abstract ========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing.
Motivation ==========
It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True title: str
class Movie(_MovieBase, total=False): year: int
Having to declare two different TypedDict types for this purpose is cumbersome.
Rationale =========
One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript:
::
interface Movie { title: string; year?: number; // ? marks potentially-missing keys }
The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work:
::
class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``.
Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case.
Specification =============
The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False): title: Required[str] year: int
Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int]
It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict.
It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int]
Backwards Compatibility =======================
No backward incompatible changes are made by this PEP.
How to Teach This =================
To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``.
To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``.
If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict): name: str owner: NotRequired[str|None]
Avoid (unless Python 3.5-3.6):
::
class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]]
Reference Implementation ========================
The goal is to be able to make the following statement:
The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module.
The mypy implementation is currently still being worked on.
Rejected Ideas ==============
Special syntax around the *key* of a TypedDict item ---------------------------------------------------
::
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator -------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float
class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float
class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float
Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar.
It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options.
Marking absence of a value with a special constant --------------------------------------------------
We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict): req1: int opt1: str|Missing req2: float
Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined:
::
class MyClass: attr: int|Missing
def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10
::
def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing
Misalignment with how unions apply to values ''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided '''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks:
::
class Packet: data: Union[str, bytes]
def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet: data: Union[str, Missing]
def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global variables:
::
warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!'
if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially.
Difficult to implement ''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_
Introduces a second null-like value into Python '''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”. ---------------------------------------------------------------------------
``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of “nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str]
This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item” -----------------------------------------------
- Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked
References ==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWD...
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com