The new syntax Required[...] and NotRequired[...] for marking individual
keys of a TypedDict has now been formally proposed as a draft PEP.
Please see the latest text at the bottom of this email, or online at:
https://www.python.org/dev/peps/pep-0655/
Comments welcome. :)
Meanwhile I'll proceed to implement this syntax in mypy.
--
David Foster | Seattle, WA, USA
Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 655
Title: Marking individual TypedDict items as required or potentially-missing
Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: typing-sig at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 604
Created: 30-Jan-2021
Python-Version: 3.10
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Abstract
========
`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
for declaring a TypedDict with all required keys and syntax for defining
a TypedDict with `all potentially-missing
keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it
does not provide any syntax to declare some keys as required and others
as potentially-missing. This PEP introduces two new syntaxes:
``Required[...]`` which can be used on individual items of a
TypedDict to mark them as required, and
``NotRequired[...]`` which can be used on individual items
to mark them as potentially-missing.
Motivation
==========
It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for ``total`` and then inherit it from another TypedDict with a
different value for ``total``:
::
class _MovieBase(TypedDict): # implicitly total=True
title: str
class Movie(_MovieBase, total=False):
year: int
Having to declare two different TypedDict types for this purpose is
cumbersome.
Rationale
=========
One might think it unusual to propose syntax that prioritizes marking
*required* keys rather than syntax for *potentially-missing* keys, as is
customary in other languages like TypeScript:
::
interface Movie {
title: string;
year?: number; // ? marks potentially-missing keys
}
The difficulty is that the best word for marking a potentially-missing
key, ``Optional[...]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:
::
class Movie(TypedDict):
...
year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing
keys (like ``Missing[...]``) would be too similar to ``Optional[...]``
and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as ``Required[...]``.
Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
``NotRequired[...]`` form for that case.
Specification
=============
The ``typing.Required`` type qualifier is used to indicate that a
variable declared in a TypedDict definition is a required key:
::
class Movie(TypedDict, total=False):
title: Required[str]
year: int
Additionally the ``typing.NotRequired`` type qualifier is used to
indicate that a variable declared in a TypedDict definition is a
potentially-missing key:
::
class Movie(TypedDict): # implicitly total=True
title: str
year: NotRequired[int]
It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any
location that is not an item of a TypedDict.
It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for
items where it is redundant, to enable additional explicitness if desired:
::
class Movie(TypedDict):
title: Required[str] # redundant
year: NotRequired[int]
Backwards Compatibility
=======================
No backward incompatible changes are made by this PEP.
How to Teach This
=================
To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal
and mark those few keys that are potentially-missing with
``NotRequired[...]``.
To define a TypedDict where most keys are potentially-missing and a few are
required, define a ``total=False`` TypedDict
and mark those few keys that are required with ``Required[...]``.
If some items accept ``None`` in addition to a regular value, it is
recommended that the ``TYPE|None`` syntax be preferred over
``Optional[TYPE]`` for marking such item values, to avoid using
``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]``
within the same TypedDict definition:
Yes:
::
from __future__ import annotations # for Python 3.7-3.9
class Dog(TypedDict):
name: str
owner: NotRequired[str|None]
Avoid (unless Python 3.5-3.6):
::
class Dog(TypedDict):
name: str
# ick; avoid using both Optional and NotRequired
owner: NotRequired[Optional[str]]
Reference Implementation
========================
The goal is to be able to make the following statement:
The `mypy <http://www.mypy-lang.org/>`__ type checker supports
``Required`` and ``NotRequired``. A reference implementation of the
runtime component is provided in the
`typing_extensions
<https://github.com/python/typing/tree/master/typing_extensions>`__
module.
The mypy implementation is currently still being worked on.
Rejected Ideas
==============
Special syntax around the *key* of a TypedDict item
---------------------------------------------------
::
class MyThing(TypedDict):
opt1?: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
or:
::
class MyThing(TypedDict):
Optional[opt1]: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
These syntaxes would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.
Also, “let’s just not put funny syntax before the colon.” [1]_
Marking required or potentially-missing keys with an operator
-------------------------------------------------------------
We could use unary ``+`` as shorthand to mark a required key, unary
``-`` to mark a potentially-missing key, or unary ``~`` to mark a key
with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False):
req1: +int # + means a required key, or Required[...]
opt1: str
req2: +float
class MyThing(TypedDict):
req1: int
opt1: -str # - means a potentially-missing key, or
NotRequired[...]
req2: float
class MyThing(TypedDict):
req1: int
opt1: ~str # ~ means a opposite-of-normal-totality key
req2: float
Such operators could be implemented on ``type`` via the ``__pos__``,
``__neg__`` and ``__invert__`` special methods without modifying the
grammar.
It was decided that it would be prudent to introduce longform syntax
(i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing
any shortform syntax. Future PEPs may reconsider introducing this
or other shortform syntax options.
Marking absence of a value with a special constant
--------------------------------------------------
We could introduce a new type-level constant which signals the absence
of a value when used as a union member, similar to JavaScript’s
``undefined`` type, perhaps called ``Missing``:
::
class MyThing(TypedDict):
req1: int
opt1: str|Missing
req2: float
Such a ``Missing`` constant could also be used for other scenarios such
as the type of a variable which is only conditionally defined:
::
class MyClass:
attr: int|Missing
def __init__(self, set_attr: bool) -> None:
if set_attr:
self.attr = 10
::
def foo(set_attr: bool) -> None:
if set_attr:
attr = 10
reveal_type(attr) # int|Missing
Misalignment with how unions apply to values
''''''''''''''''''''''''''''''''''''''''''''
However this use of ``...|Missing``, equivalent to
``Union[..., Missing]``, doesn’t align well with what a union normally
means: ``Union[...]`` always describes the type of a *value* that is
present. By contrast missingness or non-totality is a property of a
*variable* instead. Current precedent for marking properties of a
variable include ``Final[...]`` and ``ClassVar[...]``, which the
proposal for ``Required[...]`` is aligned with.
Misalignment with how unions are subdivided
'''''''''''''''''''''''''''''''''''''''''''
Furthermore the use of ``Union[..., Missing]`` doesn’t align with the
usual ways that union values are broken down: Normally you can eliminate
components of a union type using ``isinstance`` checks:
::
class Packet:
data: Union[str, bytes]
def send_data(packet: Packet) -> None:
if isinstance(packet.data, str):
reveal_type(packet.data) # str
packet_bytes = packet.data.encode('utf-8')
else:
reveal_type(packet.data) # bytes
packet_bytes = packet.data
socket.send(packet_bytes)
However if we were to allow ``Union[..., Missing]`` you’d either have to
eliminate the ``Missing`` case with ``hasattr`` for object attributes:
::
class Packet:
data: Union[str, Missing]
def send_data(packet: Packet) -> None:
if hasattr(packet, 'data'):
reveal_type(packet.data) # str
packet_bytes = packet.data.encode('utf-8')
else:
reveal_type(packet.data) # Missing? error?
packet_bytes = b''
socket.send(packet_bytes)
or a check against ``locals()`` for local variables:
::
def send_data(packet_data: Optional[str]) -> None:
packet_bytes: Union[str, Missing]
if packet_data is not None:
packet_bytes = packet.data.encode('utf-8')
if 'packet_bytes' in locals():
reveal_type(packet_bytes) # bytes
socket.send(packet_bytes)
else:
reveal_type(packet_bytes) # Missing? error?
or a check via other means, such as against ``globals()`` for global
variables:
::
warning: Union[str, Missing]
import sys
if sys.version_info < (3, 6):
warning = 'Your version of Python is unsupported!'
if 'warning' in globals():
reveal_type(warning) # str
print(warning)
else:
reveal_type(warning) # Missing? error?
Weird and inconsistent. ``Missing`` is not really a value at all; it’s
an absence of definition and such an absence should be treated
specially.
Difficult to implement
''''''''''''''''''''''
Eric Traut from the Pyright type checker team has stated that
implementing a ``Union[..., Missing]``-style syntax would be
difficult. [2]_
Introduces a second null-like value into Python
'''''''''''''''''''''''''''''''''''''''''''''''
Defining a new ``Missing`` type-level constant would be very close to
introducing a new ``Missing`` value-level constant at runtime, creating
a second null-like runtime value in addition to ``None``. Having two
different null-like constants in Python (``None`` and ``Missing``) would
be confusing. Many newcomers to JavaScript already have difficulty
distinguishing between its analogous constants ``null`` and
``undefined``.
Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
---------------------------------------------------------------------------
``Optional[...]`` is too ubiquitous to deprecate. Although use of it
*may* fade over time in favor of the ``T|None`` syntax specified by `PEP
604 <https://www.python.org/dev/peps/pep-0604/>`__.
Change Optional to mean “optional item” in certain contexts instead of
“nullable”
---------------------------------------------------------------------------------
Consider the use of a special flag on a TypedDict definition to alter
the interpretation of ``Optional`` inside the TypedDict to mean
“optional item” rather than its usual meaning of “nullable”:
::
class MyThing(TypedDict, optional_as_missing=True):
req1: int
opt1: Optional[str]
or:
::
class MyThing(TypedDict, optional_as_nullable=False):
req1: int
opt1: Optional[str]
This would add more confusion for users because it would mean that in
*some* contexts the meaning of ``Optional[...]`` is different than in
other contexts, and it would be easy to overlook the flag.
Various synonyms for “potentially-missing item”
-----------------------------------------------
- Omittable – too easy to confuse with optional
- OptionalItem, OptionalKey – two words; too easy to confuse with
optional
- MayExist, MissingOk – two words
- Droppable – too similar to Rust’s ``Drop``, which means something
different
- Potential – too vague
- Open – sounds like applies to an entire structure rather then to an
item
- Excludable
- Checked
References
==========
.. [1]
https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW…
.. [2]
https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG…
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I would like to draft a PEP for a typing.Self class which would perform similarly to rust's Self keyword in function annotations.
Examples
```python
from typing import TypeVar
F = TypeVar("F", bound="Foo")
class Foo:
def parse(self: F, data: bytes) -> F:
...
return self
```
vs
```python
from typing import Self
class Foo:
def parse(self, data: bytes) -> Self:
...
return self
```
Another example of where this could be useful is classmethods:
```python
class Foo:
@classmethod
def bar(cls: type[F], *args: Any, **kwargs: Any) -> F:
return cls()
```
vs
```python
class Foo:
@classmethod
def bar(cls, *args: Any, **kwargs: Any) -> Self:
return cls()
```
Implementation wise it would be a copy of NoReturn, i.e. not subscriptable, a _SpecialForm and should only be valid as a return type.
I haven't seen any discussion of the idea on the mailing list, so I wanted to run it by everyone.
Thanks,
James.
Hello all,
User-friendly Callable syntax is something we all want to see. However,
there has been no real movement on the discussion in the last one month.
People have had differing opinions based on design principles, stats, and
personal experience. This has led to half a dozen proposals with some
subtle and not-so-subtle differences [1]. And Guido pointed out that we
want to settle on a full proposal instead of going for a partial PEP now
and settling on the remaining details in a later PEP.
How about we meet up over a zoom call and discuss some of the key concerns?
Things proceed noticeably faster when people from the typing community are
able to get in the same (virtual) room (I saw that with PEP 646 and the
typing summit). We don't have to reach a consensus in the first meeting
itself. It's more a way to break the deadlock and make progress.
Date: 10 am San Francisco time (6 pm UK time), Monday, September 13, 2021.
Date is flexible.
Duration: 1 hour.
If there is enough interest, I can set up the video call and share the link
and agenda. If you're interested in presenting pros and cons for your
favorite proposal, reply to this mail.
[1]: Document containing the various proposals and use cases:
https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB…
--
S Pradeep Kumar
Hi,
I’m in a position, where I’d like to be able to pass a reference to a
Protocol Class, however, I can’t find a way to describe that variable with
the current type system, as Type[ProtocolClass] means a class implementing
said protocol, where I want the protocol itself, while ProtocolClass means
an instance of a class implementing the protocol.
So I haven’t figured out if PEP 544 specifies a type that describes the
Protocol class itself, rather than just its compatible implementations.
I tried to file this as a mypy bug
<https://github.com/python/mypy/issues/10988>, but no response yet, and
alas, figure it may even be overlooked by the PEP in question.
Anyway, I’d very much appreciate a few more eyes and input on this matter.
Thank you,
Cheers
This is a cross post from
https://discuss.python.org/t/how-to-pass-a-reference-to-a-protocol-class/10…
Reported MyPy Issue: https://github.com/python/mypy/issues/10988
Is anything like mypy's `NamedArg` coming to the Python standard
library? Meanwhile, if I want to include named arguments in a Callable
type, is the right approach to use something like `Callable[..., int]`?
This was previously briefly discussed in
https://github.com/python/typing/issues/601.
Occasionally it's useful if it was possible to mark protocol fields as
optional. While the existence of such a field is not guaranteed, if it
exists, it follows the described protocol. This is especially useful for
I/O protocols, where implementations often check for the existence of a
particular method, before using it. One prominent example is the WSGI
spec, where the iterable returned by an application can optionally have
a close() method.
My suggestion is to add an @optional decorator to typing to be used like
this:
from typing import Protocol, optional
class MyProto(Protocol):
def write(self, __b: bytes) -> Any: ...
@optional
def close(self) -> Any: ...
On the Python side, this is fairly trivial to implement: @optional would
just return the passed in function, possible adding an attribute for
runtime inspection:
def optional(f: _F) -> _F:
f.__optional__ = True
return f
Full support for type checkers is more complicated, as hasattr() support
would be required to check implementations. A quick & dirty way for type
checkers to support the status quo is to ignore @optional fields.
- Sebastian
Typing is hard. And constantly evolving. I've seen questions about
correct typing in many different forums and bug trackers, and often the
same questions get repeatedly asked. This is why we set up a central
forum to help users with typing at
https://github.com/python/typing/discussions. Currently, it's a bit bare
bones, but we have two categories for general Q&A and for asking for
reviews. If users have questions about typing, please feel free to
direct them there or to Gitter (https://gitter.im/python/typing). If you
maintain a typing-related package or typing-related documentation, we
would be happy if you could add a link to the discussions forum to your
documentation. Contributors to the forum and suggestions on how to
improve it are most welcome.
That forum is intended as a user help forum. Discussions about new
features, improvements, and implementation should continue to go to this
list or to the appropriate issue trackers.
- Sebastian
Hello typing-sig,
I wanted to suggest adding support for sealed classes to the python type
system.
What are sealed classes?
Sealed classes are classes of which all subclasses are known statically.
This is usually accomplished by requiring all subclasses to be created
in the same file/module as the parent class.
What languages support sealed classes?
Sealed classes are supported by C#, Kotlin and starting with Java 15
even by Java.
What can sealed classes be used for?
Sealed classes are useful for checking the exhaustiveness of if/else
towers and (python 3.10) match statements for example when working with
abstract syntax trees.
Take a look at this example:
```
class Node(ABC): ...
class Value(Node): ...
class BinOp(Node): ...
class UnOp(Node): ...
n: Node
match n:
case Value(v): ...
case BinOp(n1, op, n2): ...
case UnOp(op, n1): ...
# Can we statically check if all cases are handled?
```
Sealed classes in python:
My suggestion would be to add a new class "Sealed" to typing and
typing_extensions that can be inherited from:
class Node(Sealed): ...
Type checkers should make sure that no subclasses of Node are created in
modules other than the module Node was created in and should treat
sealed classes as abstract classes. Additionally they can keep track of
all the subclasses of Node do implement exhaustiveness checking.
The runtime implementation of sealed could also prevent non-typechecked
code from creating additional subclasses:
```
class Sealed(ABC):
def __init_subclass__(cls):
for c in cls.__mro__:
if Sealed in c.__bases__ and c.__module__ != cls.__module__:
raise RuntimeError('Sealed class can\'t be subclassed
from another module. Sealed in module "{}".'.format(c.__module__))
```
Final example:
a.py:
```
from typing import Sealed
class Base(Sealed):
pass
class A(Base):
pass
class B(A):
pass
```
b.py:
```
from a import Base, A
class C(Base): # raises an exception
pass
class D(A): # also raises an exception
pass
```
Adrian Freund
Hi,
Context: during the typing meeting of this week Brandon presented the idea of introducing a Compose type operator, which takes a variadic of callables, checks that they compose, and returns the type of the composed function.
To follow-up on the discussion about Compose:
We were wondering what can be achieved nowadays without custom operators, just relying on overload. Let's think about the example of a compose function, we could do the following right?
from typing import Callable as Call
# Overloads
def compose(f1: Call[A,B]) -> Call[A,B]
def compose(f1: Call[A,B], f2: Call[B,C]) -> Call[A,C]
def compose(f1: Call[A,B], f2: Call[B,C], f3: Call[C,D]) -> Call[A,D]
...
What is less obvious is how would we type a class like torch.nn.Sequential, which would look like this with the Compose operator:
class Sequential(Generic[A,*Ts]):
def __init__(self, fs: *Ts)
def __call__(self, in: A) -> Compose[*Ts]
Here we can't overload the class definition, so what can be done? We had a couple of ideas but they would not really work for a variable number of arguments because we can't overload. An example of those approaches would be:
class Sequential(Generic[A,B,C]):
def __init__(self, f1: Callable[A,B], f2: Callable[B,C]): ...
def __call__(self, in: A) -> C: ....
Best,
Alfonso.
Hi all,
Out next tensor typing meeting will be *Monday the 16th of August at 6pm UK
time/10am San Francisco time*. Our tentative agenda is:
1. PEP 646 progress (basically that grammar changes are done and we're
just waiting for review on updates to the PEP itself)
2. Discussion of feedback we got from TensorFlow/PyTorch core devs
Give me a shout if anyone's interested in talking about something else!
Cheers,
Matthew
