Mailman 3 March 2021 - Typing-sig

PEP 655: Marking individual TypedDict items as required or potentially-missing

by David Foster

The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/ Comments welcome. :) Meanwhile I'll proceed to implement this syntax in mypy. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Avoid (unless Python 3.5-3.6): :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The goal is to be able to make the following statement: The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. The mypy implementation is currently still being worked on. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value or: :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

4 months

7
16
0 0

How to ensure a generic function only accepts args of the same type

by Teddy Sudol

Hi typing-sig, This question has come up a few times from pytype's users: How do you write a generic function that only takes arguments that are the same subclass of a particular base class? That is, if you have a function like: def do_stuff(x, y): ... If you want it to take two args of the same type, but that type is otherwise not constrained, use a regular TypeVar: T = TypeVar("T") def do_stuff(x: T, y: T): ... If you want do_stuff to only accept args of certain types, you can limit the TypeVar to those types: T = TypeVar("T", int, str) def do_stuff(x: T, y: T): ... do_stuff(<int>, <int>) or do_stuff(<str>, <str>) are the only valid calls. This works if the constraining types are known ahead of time. If you want do_stuff to only accept args of a particular base type, use bound: T = TypeVar("T", bound=Base) def do_stuff(x: T, y: T): ... If A and B are subclasses of Base, then do_stuff(A(), A()), do_stuff(A(), B()) and do_stuff(A(), Base()) are all legal. As far as I can tell, there is no way to express that do_stuff(A(), A()) and do_stuff(B(), B()) are legal while do_stuff(A(), B()) and do_stuff(A(), Base()) are illegal, for all subclasses of Base. Am I missing something? (Pytype does accept one way to do it, though this is unintended: T = TypeVar("T", Base, Base) def do_stuff(x: T, y: T): ... mypy treats this the same as the bound=Base case, but pytype will complain about do_stuff(A(), B()). PEP 484 doesn't have anything to say about this case, as far as I know.) -- Teddy

5 months, 1 week

3
2
0 0

PEP-589 TypedDict inheritance rules and typing.Literal / PEP-586

by Greg Werbin

Note: this was originally posted on the Discourse forum (https://discuss.python.org/t/7721), but it was recommended that I post here instead. In PEP-589: “Inheritance”, it is stated: > Changing a field type of a parent TypedDict class in a subclass is not allowed. However, this restriction seems perhaps too restrictive in the presence of typing.Literal and PEP-586. Consider a situation like the following, which might describe some polymorphic data returned from a web API: from typing import Literal, TypedDict # Assume these all have several other fields class BaseDatum(TypedDict): id: str datumType: str class Thing(BaseDatum): datumType: Literal["thing"] class Widget(BaseDatum): datumType: Literal["widget"] This seems like a reasonable thing that someone might want to do, especially in a context where you can’t use something like Marshmallow for runtime validation & deserialization (don’t want performance overhead, restricted from using 3rd-party dependencies, etc). However, in the above example, MyPy emits: > tmp1.py:10: error: Overwriting TypedDict field "datumType" while extending > tmp1.py:13: error: Overwriting TypedDict field "datumType" while extending > Found 2 errors in 1 file (checked 1 source file) Typically, Literal["widget"] is considered a subclass of str, as in: from typing import Literal x: Literal["hello"] = "hello" y: str = x It appears that the MyPy devs faithfully honored the restriction on changing type that is stated in PEP-589, but I think in this specific case the restriction is excessive. Should there be a revision to the TypedDict specification to allow this particular case? For that matter, why shouldn’t TypedDict members be covariant in general? Or is there a good reason why the restriction is what it is?

6 months

2
2
1 0

isinstance(instance, protocol) with protocol members without default values (PEP 544)

by Paul Dest

I hope to have been directed to the right place for my question from https://bugs.python.org/issue43512 I'm completely new to the Python dev process. Therefore, I ask for your indulgence if my approach is not in line with the process. I would like to collect opinions (especially from authors of PEP 544) if the CPython behavior described below should be regarded as a violation of PEP 544 or at least as a behavior that should be improved. The section "Subtyping relationships with other types" of PEP 544 states: "A concrete type X is a subtype of protocol P if and only if X implements all protocol members of P with compatible types. In other words, subtyping with respect to a protocol is always structural." In contrast to that, the behavior of CPython 3.9.2 for protocols with protocol members without default values is as follows: ``` from typing import Protocol class P(Protocol): protocol_member: str # no default value, but still a protocol member class X(P): # inherits P but does NOT implement protocol_member, since P did not provide a default value pass assert isinstance(X(), P) # violates the PEP 544 requirement cited above X().protocol_member # raises: AttributeError: 'X' object has no attribute 'protocol_member' ``` In this regard, it was argued that "At runtime, protocol classes will be simple ABCs." (PEP 544) But unfortunately, this is currently not the case. Actually, there is an extra metaclass for protocols, solely to provide an __instancecheck__. https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1096 ``` class _ProtocolMeta(ABCMeta): # This metaclass is really unfortunate and exists only because of # the lack of __instancehook__. def __instancecheck__(cls, instance): # We need this method for situations where attributes are # assigned in __init__. if ((not getattr(cls, '_is_protocol', False) or _is_callable_members_only(cls)) and issubclass(instance.__class__, cls)): return True if cls._is_protocol: if all(hasattr(instance, attr) and # All *methods* can be blocked by setting them to None. (not callable(getattr(cls, attr, None)) or getattr(instance, attr) is not None) for attr in _get_protocol_attrs(cls)): return True return super().__instancecheck__(instance) ``` I am inclined to assess the behavior described above as a violation of PEP 544 and an incomplete implementation of `_ProtocolMeta`.

6 months

2
3
0 0

isinstance(instance, protocol) with protocol members without default values (PEP 544)

by Paul Dest

I hope to have been directed to the right place for my question from https://bugs.python.org/issue43512 I'm completely new to the Python dev process. Therefore, I ask for your indulgence if my approach is not in line with the process. I would like to collect opinions (especially from authors of PEP 544) if the CPython behavior described below should be regarded as a violation of PEP 544 or at least as a behavior that should be improved. The section "Subtyping relationships with other types" of PEP 544 states: "A concrete type X is a subtype of protocol P if and only if X implements all protocol members of P with compatible types. In other words, subtyping with respect to a protocol is always structural." In contrast to that, the behavior of CPython 3.9.2 for protocols with protocol members without default values is as follows: ``` from typing import Protocol class P(Protocol): protocol_member: str # no default value, but still a protocol member class X(P): # inherits P but does NOT implement protocol_member, since P did not provide a default value pass assert isinstance(X(), P) # violates the PEP 544 requirement cited above X().protocol_member # raises: AttributeError: 'X' object has no attribute 'protocol_member' ``` In this regard, it was argued that "At runtime, protocol classes will be simple ABCs." (PEP 544) But unfortunately, this is currently not the case. Actually, there is an extra metaclass for protocols, solely to provide an __instancecheck__. https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1096 ``` class _ProtocolMeta(ABCMeta): # This metaclass is really unfortunate and exists only because of # the lack of __instancehook__. def __instancecheck__(cls, instance): # We need this method for situations where attributes are # assigned in __init__. if ((not getattr(cls, '_is_protocol', False) or _is_callable_members_only(cls)) and issubclass(instance.__class__, cls)): return True if cls._is_protocol: if all(hasattr(instance, attr) and # All *methods* can be blocked by setting them to None. (not callable(getattr(cls, attr, None)) or getattr(instance, attr) is not None) for attr in _get_protocol_attrs(cls)): return True return super().__instancecheck__(instance) ``` I am inclined to assess the behavior described above as a violation of PEP 544 and an incomplete implementation of `_ProtocolMeta`.

6 months

1
0
0 0

PEP 647 (type guards) -- final call for comments

by Guido van Rossum

I think we have reached consensus on PEP 647 in typing-sig. We have implementations for mypy and pyright, not sure about the rest. This PEP does not affect CPython directly except for the addition of one special item (TypeGuard) to typing.py -- it would be nice to get that in the 3.10 stdlib. I'm CC'ing python-dev here to see if there are any further comments; if not, we can initiate the approval process by creating an issue at https://github.com/python/steering-council. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

6 months

7
22
0 0

Change of list archive behavior (and moderators wanted)

by Guido van Rossum

I just received an email from the Mailman 3 admin for mail.python.org about rendering of archived messages. For some time, HyperKitty has been rendering messages in the archive > using a subset of markdown to produce a rich text rendering of the > message. We have just made several improvements to the rendering. > These are described at > https://docs.mailman3.org/projects/hyperkitty/en/latest/rendering.html > > You will note that the markdown rendering is now optional and is > controlled by the Archive Rendering Mode setting at > > https://mail.python.org/mailman3/lists/typing-sig@python.org/settings/archi… > and defaults to plain text rendering. If you wish to use markdown > rendering for this list, you will need to change this setting. > > Note that the setting has no effect on the stored messages themselves. > It only affects how they are rendered in HyperKitty, so you can switch > back and forth without a problem. > I've verified that this affects the rendering of all messages already archived. I've checked that this setting is off for the typing-sig right now, and despite the improvements I would suggest that we keep it off -- I personally don't like having to type backticks just to avoid dunders being rendered as bold, or indentation/whitespace in code samples getting lost. But if people overwhelmingly prefer the markup-aware rendering in the archives please respond to this message and maybe we can change it. (And yes, in GitHub comments I am fine with markdown, the difference there is that (a) you can preview how it renders, and (b) you can edit your message to fix markup after the fact.) Also, if anyone wants to volunteer to be an additional list moderator, please contact me off-list. The list attracts a fair amount of spam that needs to be dealt with by the moderator, and as a result we also need to moderate all new legit members to the list until their first message. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

6 months

2
1
0 0

Typing persistent data structure

by Caleb Donovick

I am trying to type check a persistent (immutable) data structure. However I am stumbling on restrictions on using covariant types as arguments. Is there a standard work around for this? Simplified Example: class Node(Generic[T_co]): def __init__(self, val: T_co, succ: Optional[Node[T_co]]) -> None: self.val: Final[T_co] = val self.succ: Final[Optional[Node[T_co]]] = succ class List(Sequence[T_co]): def __init__(self, *, _head: Optional[Node[T_co]] = None) -> None: self.head: Final[Optional[Node[T_co]]] = _head # The problem method def push_front(self, val: T_co) -> List[T_co]: new_node = Node(val, self.head) return List(_head=new_node) def __iter__(self) -> Iterator[T_co]: ptr = self.head while ptr is not None: yield ptr.val ptr = ptr.succ ...

6 months

2
1
0 0

Typing Summit 2021 Interest Form

by Shannon Zhu

Hi everyone! Alfonso and I have been working on organizing a Typing Summit in partnership with PyCon, for developers and users of Python static type checkers. If you are interested in attending, please fill out this signup form<https://docs.google.com/forms/d/e/1FAIpQLSct42iXWscqznadBYSSsJUWm0wLd9IFKS0…>. Because the summit is co-located with PyCon, it will take place on the 12th or 13th of May. Our tentative schedule is Wednesday, May 12th from 11AM-3PM EST (8AM-12PM PST, 5PM-9PM CET) to accommodate as many time zones as possible. As in previous years, we hope to make the summit as discussion-oriented as possible. To facilitate this, we encourage presenters to prepare an agenda for their topic, including an introductory presentation and any materials that discussion participants should check in advance. Although different topics may require different discussion formats, the goal is to make sure that participants are up to date with the state of the topic and prepared to provide constructive ideas after the presentation. Finally, we are sending this form to understand what topics you want to see covered, whether you want or are willing to present the topic, or just want to participate in its discussion. No worries in case that multiple people want to present the same topic, or if you are interested in discussing a topic that nobody has proposed presenting: soon after we will share the form results so that we can discuss how to get the topics properly covered. Thanks! We are looking forward to seeing everyone at the summit! Alfonso & Shannon

6 months

2
2
1 0

Next tensor typing meeting

by Matthew Rahtz

Hi everyone, Our next tensor typing meeting will be happening this coming *Monday the 15th of March*, at 6pm London time, 10am San Francisco time. As always, we'll be meeting at meet.google.com/fft-dzjq-ksu. You can take a look at what we've talked about in our past meetings here <https://docs.google.com/document/d/1oaG0V2ZE5BRDjd9N-Tr1N0IKGwZQcraIlZ0N8ay…> . The agenda this time is a short one: - Summary of where we're at with PEP 646 - What the next steps are once the PEP is done If anyone would like to present or talk about anything else, give me a shout. Hope to see you there! Matthew

6 months

4
9
0 0

2021

2020

2019

2018

Typing-sig March 2021