The new syntax Required[...] and NotRequired[...] for marking individual
keys of a TypedDict has now been formally proposed as a draft PEP.
Please see the latest text at the bottom of this email, or online at:
https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
--
David Foster | Seattle, WA, USA
Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655
Title: Marking individual TypedDict items as required or potentially-missing
Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: typing-sig at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 604
Created: 30-Jan-2021
Python-Version: 3.10
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Abstract
========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
for declaring a TypedDict with all required keys and syntax for defining
a TypedDict with `all potentially-missing
keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it
does not provide any syntax to declare some keys as required and others
as potentially-missing. This PEP introduces two new syntaxes:
``Required[...]`` which can be used on individual items of a
TypedDict to mark them as required, and
``NotRequired[...]`` which can be used on individual items
to mark them as potentially-missing.
Motivation
==========
It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for ``total`` and then inherit it from another TypedDict with a
different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True
title: str
class Movie(_MovieBase, total=False):
year: int
Having to declare two different TypedDict types for this purpose is
cumbersome.
Rationale
=========
One might think it unusual to propose syntax that prioritizes marking
*required* keys rather than syntax for *potentially-missing* keys, as is
customary in other languages like TypeScript:
::
interface Movie {
title: string;
year?: number; // ? marks potentially-missing keys
}
The difficulty is that the best word for marking a potentially-missing
key, ``Optional[...]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:
::
class Movie(TypedDict):
...
year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing
keys (like ``Missing[...]``) would be too similar to ``Optional[...]``
and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as ``Required[...]``.
Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
``NotRequired[...]`` form for that case.
Specification
=============
The ``typing.Required`` type qualifier is used to indicate that a
variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False):
title: Required[str]
year: int
Additionally the ``typing.NotRequired`` type qualifier is used to
indicate that a variable declared in a TypedDict definition is a
potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True
title: str
year: NotRequired[int]
It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any
location that is not an item of a TypedDict.
It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for
items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict):
title: Required[str] # redundant
year: NotRequired[int]
Backwards Compatibility
=======================
No backward incompatible changes are made by this PEP.
How to Teach This
=================
To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal
and mark those few keys that are potentially-missing with
``NotRequired[...]``.
To define a TypedDict where most keys are potentially-missing and a few are
required, define a ``total=False`` TypedDict
and mark those few keys that are required with ``Required[...]``.
If some items accept ``None`` in addition to a regular value, it is
recommended that the ``TYPE|None`` syntax be preferred over
``Optional[TYPE]`` for marking such item values, to avoid using
``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]``
within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict):
name: str
owner: NotRequired[str|None]
Avoid (unless Python 3.5-3.6):
::
class Dog(TypedDict):
name: str
# ick; avoid using both Optional and NotRequired
owner: NotRequired[Optional[str]]
Reference Implementation
========================
The goal is to be able to make the following statement:
The `mypy <http://www.mypy-lang.org/>`__ type checker supports
``Required`` and ``NotRequired``. A reference implementation of the
runtime component is provided in the
`typing_extensions
<https://github.com/python/typing/tree/master/typing_extensions>`__
module.
The mypy implementation is currently still being worked on.
Rejected Ideas
==============
Special syntax around the *key* of a TypedDict item
---------------------------------------------------
::
class MyThing(TypedDict):
opt1?: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict):
Optional[opt1]: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator
-------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary
``-`` to mark a potentially-missing key, or unary ``~`` to mark a key
with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False):
req1: +int # + means a required key, or Required[...]
opt1: str
req2: +float
class MyThing(TypedDict):
req1: int
opt1: -str # - means a potentially-missing key, or
NotRequired[...]
req2: float
class MyThing(TypedDict):
req1: int
opt1: ~str # ~ means a opposite-of-normal-totality key
req2: float
Such operators could be implemented on ``type`` via the ``__pos__``,
``__neg__`` and ``__invert__`` special methods without modifying the
grammar.
It was decided that it would be prudent to introduce longform syntax
(i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing
any shortform syntax. Future PEPs may reconsider introducing this
or other shortform syntax options.
Marking absence of a value with a special constant
--------------------------------------------------
We could introduce a new type-level constant which signals the absence
of a value when used as a union member, similar to JavaScript’s
``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict):
req1: int
opt1: str|Missing
req2: float
Such a ``Missing`` constant could also be used for other scenarios such
as the type of a variable which is only conditionally defined:
::
class MyClass:
attr: int|Missing
def __init__(self, set_attr: bool) -> None:
if set_attr:
self.attr = 10
::
def foo(set_attr: bool) -> None:
if set_attr:
attr = 10
reveal_type(attr) # int|Missing
Misalignment with how unions apply to values
''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to
``Union[..., Missing]``, doesn’t align well with what a union normally
means: ``Union[...]`` always describes the type of a *value* that is
present. By contrast missingness or non-totality is a property of a
*variable* instead. Current precedent for marking properties of a
variable include ``Final[...]`` and ``ClassVar[...]``, which the
proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided
'''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the
usual ways that union values are broken down: Normally you can eliminate
components of a union type using ``isinstance`` checks:
::
class Packet:
data: Union[str, bytes]
def send_data(packet: Packet) -> None:
if isinstance(packet.data, str):
reveal_type(packet.data) # str
packet_bytes = packet.data.encode('utf-8')
else:
reveal_type(packet.data) # bytes
packet_bytes = packet.data
socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to
eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet:
data: Union[str, Missing]
def send_data(packet: Packet) -> None:
if hasattr(packet, 'data'):
reveal_type(packet.data) # str
packet_bytes = packet.data.encode('utf-8')
else:
reveal_type(packet.data) # Missing? error?
packet_bytes = b''
socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None:
packet_bytes: Union[str, Missing]
if packet_data is not None:
packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals():
reveal_type(packet_bytes) # bytes
socket.send(packet_bytes)
else:
reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global
variables:
::
warning: Union[str, Missing]
import sys
if sys.version_info < (3, 6):
warning = 'Your version of Python is unsupported!'
if 'warning' in globals():
reveal_type(warning) # str
print(warning)
else:
reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s
an absence of definition and such an absence should be treated
specially.
Difficult to implement
''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that
implementing a ``Union[..., Missing]``-style syntax would be
difficult. [2]_
Introduces a second null-like value into Python
'''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to
introducing a new ``Missing`` value-level constant at runtime, creating
a second null-like runtime value in addition to ``None``. Having two
different null-like constants in Python (``None`` and ``Missing``) would
be confusing. Many newcomers to JavaScript already have difficulty
distinguishing between its analogous constants ``null`` and
``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
---------------------------------------------------------------------------
``Optional[...]`` is too ubiquitous to deprecate. Although use of it
*may* fade over time in favor of the ``T|None`` syntax specified by `PEP
604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of
“nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter
the interpretation of ``Optional`` inside the TypedDict to mean
“optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True):
req1: int
opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False):
req1: int
opt1: Optional[str]
This would add more confusion for users because it would mean that in
*some* contexts the meaning of ``Optional[...]`` is different than in
other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item”
-----------------------------------------------
- Omittable – too easy to confuse with optional
- OptionalItem, OptionalKey – two words; too easy to confuse with
optional
- MayExist, MissingOk – two words
- Droppable – too similar to Rust’s ``Drop``, which means something
different
- Potential – too vague
- Open – sounds like applies to an entire structure rather then to an
item
- Excludable
- Checked
References
==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW…
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG…
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Since the introduction of PEP 647 (User-defined Type Guards), we've received a steady stream of input from users saying that they don't like the limitation that type narrowing is applied only in the positive case and is not applied in the negative case.
In general, type narrowing is not safe to perform if a user-defined type guard returns False, so I've stood by the original decision not to provide type narrowing in the negative case, but there are cases where such narrowing is safe and desirable.
Here's a recent thread where this was discussed in some detail: https://github.com/python/typing/issues/996#issuecomment-1002716830
In this discussion, Ilya Kamen proposed a solution. The proposal is to extend the existing `TypeGuard` to support an optional second type argument. If present, the second argument indicates the narrowed type in the negative type narrowing situation.
Here's a simple (admittedly contrived) example:
```python
def is_str(val: str | int) -> TypeGuard[str, int]:
return isinstance(val, str)
def func(val: str | int):
if is_str(val):
reveal_type(val) # str
else:
reveal_type(val) # int
```
I've implemented this proposal in the latest published version of pyright (1.1.202) so folks can experiment with it and see if they like it. If there's general consensus that it's the right approach, I can file an amendment for the existing PEP 647 to include this functionality. It was trivial to add to pyright, so I'm optimistic that it would likewise be easy to add to the other type checkers.
Another common request for PEP 647 is the desire for an "assert" form of TypeGuard — a way to indicate that a function performs runtime validation of a type, raising an exception if the type is incorrect. It occurred to me that this two-argument form of `TypeGuard` could also be used to handle this use case. The second argument would be specified as a `NoReturn`. I've provisionally implemented this in pyright as well (although it's not in the currently-published version — you'll need to wait until 1.1.203 is published).
Here's what this would look like:
```python
def validate_sequence(val: Sequence[_T | None]) -> TypeGuard[Sequence[_T], NoReturn]:
"""Verifies that the sequence contains no None values"""
if len([x for x in val if x is not None]) > 0:
raise Exception()
return True
def func(val: Sequence[int | None]):
validate_sequence(val)
reveal_type(val) # Sequence[int]
```
I'm interested in input on these proposals.
-Eric
--
Eric Traut
Contributor to Pyright & Pylance
Microsoft
Updating the Typing Meetup date to Wednesday as there were some conflicts.
Let me know if there are any major time conflicts.
Date:
*Wednesday, November 03, 2021.*Time: 10 am San Francisco time (UTC-7) / 5
pm London time (UTC)
Duration: 1 hour
Potential discussion topics:
+ Typing **kwargs using a TypedDict (Franek Magiera?, Pradeep) [1]
+ Remaining questions about Callable syntax (Steven)
+ Non-required Protocol fields (Sebastian Rittau?) [2]
+ Type support for sealed classes (Adrian Freund?) [3]
+ Safe, typed JSON validation
+ Type system documentation (Shannon?)
If you're working on these or any other topics and would like to start a
discussion at the meetup, reach out to me.
If you have interesting topics you'd like to see discussed, share them too.
Once we have topics, we can always start a discussion even if there's no
proposer yet. We will probably have time for 3-4 topics in this session.
On Fri, Oct 15, 2021 at 12:55 PM S Pradeep Kumar <gohanpra(a)gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> We plan to hold a virtual Typing Meetup to discuss key typing-related
>>>> issues in real-time. The idea is that we can make progress on important or
>>>> controversial discussions, get early feedback on in-progress PEPs, and
>>>> gauge interest in new proposals.
>>>>
>>>> A virtual meetup will be more accessible to people in other timezones
>>>> than Bay Area in-person meetups. It should also help consolidate
>>>> discussions that are often spread out across mailing lists or Github repos.
>>>> The tentative plan is to have a monthly meetup, but we can change that as
>>>> needed.
>>>>
>>>> Date: Monday, November 01, 2021.
>>>> Time: 10 am San Francisco time (UTC-7) / 5 pm London time (UTC)
>>>> Duration: 1 hour
>>>>
>>>> Time and date are flexible. I'll share the zoom link and agenda the
>>>> week before. I'll also record the meetup and share afterwards.
>>>>
>>>> If you're interested in presenting a new proposal or your work on some
>>>> existing proposal, please respond here. We can shoot for 15 minutes per
>>>> presentation. I know there are quite a few interesting threads on
>>>> typing-sig that haven't seen much activity in a while :)
>>>> --
>>>> S Pradeep Kumar
>>>> _______________________________________________
>>>> Typing-sig mailing list -- typing-sig(a)python.org
>>>> To unsubscribe send an email to typing-sig-leave(a)python.org
>>>> https://mail.python.org/mailman3/lists/typing-sig.python.org/
>>>> Member address: guido(a)python.org
>>>
>>>
Links:
[1] Typing **kwargs using TypedDict:
https://mail.python.org/archives/list/typing-sig@python.org/thread/IUCXT2IQ…
[2] Non-required Protocol fields:
https://mail.python.org/archives/list/typing-sig@python.org/thread/P6FCGRJU…
[3] Sealed classes:
https://mail.python.org/archives/list/typing-sig@python.org/thread/QSCT2N4R…
--
S Pradeep Kumar
Hi,
I'd like to write a PEP for a new typing construct: NotType which, would be equivalent to any type but x. There has been an issue open on the typing repo for quite some time about this (https://github.com/python/typing/issues/801). After the recent-ish proposal for an IntersectionType, this would be a nice-to-have feature as it would complete the set theory operations for typed Python. I'd also like to propose implementing type.__invert__ to make this work, so it has built-in syntax like intersection and union as this is the closest thing to a not operator.
Usecases:
Something that takes any non-empty string
```py
def fn(x: str & ~Literal[""]): ...
```
This also has nice integration with PEP 675 where the type of a Literal str is important. If you’d want to call this from any str you’d need a type guard.
```py
def guard(x: str) -> TypeGuard[str & ~Literal[“”]]:
return isinstance(x, str) and x != “” # or maybe bool(x) although this is probably much harder to implement
```
Something that takes any value but None
```py
def fn(x: ~Literal[None]): ...
```
Something that requires not an integer
```py
def not_an_int(x: Any) -> TypeGuard[~int]: ...
```
This also allows for narrowing in the negative case which is another requested feature (https://github.com/python/typing/issues/926)
```py
@overload
def is_none(value: None) -> Literal[True]: ...
@overload
def is_none(value: ~Literal[None]) -> Literal[False]: ...
def is_none(value: Any) -> bool: return value is None
def func(value: Optional[str]):
if is_none(value): reveal_type(value) # None
else: reveal_type(value) # str
```
Currently this isn’t the case
```py
def is_none(value: Any) -> TypeGuard[None]: return value is None
def func(value: Optional[str]):
if is_none(value): reveal_type(value) # None
else: reveal_type(value) # str | None
```
Overloading operators more correctly, more easily
```py
@overload
def __eq__(self, other: ~Self) -> Literal[False]: ...
@overload
def __eq__(self, other: Self) -> bool: ...
def __eq__(self, other: object) -> bool: return isinstance(other, self.__class__) and ...
```
Something fun
```py
reveal_type(~~str) # str
```
A few questions:
Can you see any issues here?
Do you have ideas for use cases I haven't thought of?
Does anybody want to be a co-author on this?
The following is an overview of all issues and pull requests in the
typing repository on GitHub with the label 'topic: feature'
that were opened or updated last week, excluding closed issues.
---------------------------------------------------
The following issues and pull requests were updated last week:
#256 Possible to distinguish between Sequence[str]/Iterable[str] and str?
opened by @jtatum
https://github.com/python/typing/issues/256
#548 Higher-Kinded TypeVars
opened by @tek
https://github.com/python/typing/issues/548
#985 Make `builtins.property` generic
opened by @sobolevn
https://github.com/python/typing/issues/985
---------------------------------------------------
All issues and pull requests with the label 'topic: feature'
can be viewed under the following URL:
https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22
(Hoping I figured out how to reply in the thread now... )
Yes, sorry for the cryptic message but it looks like you got it right. The
example isn't impressive but just like Steven is saying the value comes in
larger applications. In a web application for instance only the parsing
layer will have to deal with the runtime validation of the type. The
business logic and database layers can use static type checking to make
sure only the narrowed value is allowed. So it's kind of like a NewType
plus an "instantiation predicate" (I'm failing to find the correct term,
nothing is actually instantiated).
I thought it's relevant here since I imagine a NotType would also have to
rely a lot on runtime narrowing. If a downstream API takes Sequence &
Not[str] and we have a Sequence available, we'd have to narrow it with
something like assert not isinstance(..., str).
Pradeep, Guido and I are leaning toward removing the `async` part of PEP 677.
There are several reasons:
- If we leave `async` in, then `async (int) -> bool` and `(int) -> Awaitable[bool]` are the same
- This violates that there should only be one obvious way to do things
- It also might lead to a mismatch between written types and the errors returned by type
checkers, leading to unnecessary confusion
- It looks like < 5% of callable types return Awaitable anyway
- It has explicitly been brought up in python-dev as unnecessary complexity, and we largely
agree with this criticism.
I wanted to float the idea of cutting `async` here first since so many folks from this thread
contributed to the proposal in PEP 677. If there's no strong opposition, I'll probably remove
it after the holidays.
Cheers,
- Steven
(Hoping I figured out how to reply in the thread now... )
Yes, sorry for the cryptic message but it looks like you got it right. The
example isn't impressive but just like Steven is saying the value comes in
larger applications. In a web application for instance only the parsing
layer will have to deal with the runtime validation of the type. The
business logic and database layers can use static type checking to make
sure only the narrowed value is allowed. So it's kind of like a NewType
plus an "instantiation predicate" (I'm failing to find the correct term,
nothing is actually instantiated).
I thought it's relevant here since I imagine a NotType would also have to
rely a lot on runtime narrowing. If a downstream API takes Sequence &
Not[str] and we have a Sequence available, we'd have to narrow it with
something like assert not isinstance(..., str).
