Mailman 3 August 2021 - Typing-sig

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon June 19, 2023

June 19, 2023

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

17 35

Runtime Access to Type Parameters
by Saul Shanabrook June 7, 2023

June 7, 2023

I would like to propose being able to access, at runtime, the type parameters in classmethods and in constructors. I wrote up a draft PEP on this to capture the thinking here: https://github.com/saulshanabrook/peps/blob/master/pep-9999.md It seems easier to add them to classmethods, so even that addition alone would be very welcome! It could be added to Guido's existing draft PR for pep 585, by making a small change in the getattr function (https://github.com/python/cpython/pull/18239/files#diff-a34ae826f869897e56e…), to first do a `__getattribute__` and check if the result is a descriptor (has a `__get__` attribute), and if so passing the generic class itself in as the class, instead of the origin class. I wrote up a little example of that in the PEP. To make it available in the constructor, I proposed adding a global function `get_parameters` to `typing` which returns the parameters in the constructor. I also provide a little mock implementation of that. Note that work is only for custom classes which inherit from `typing.Generic` not for any built in collections.

5 6

Explicit Type Aliases
by Shannon Zhu Aug. 17, 2022

Aug. 17, 2022

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

PEP 655: Marking individual TypedDict items as required or potentially-missing
by David Foster Feb. 23, 2022

Feb. 23, 2022

The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/ Comments welcome. :) Meanwhile I'll proceed to implement this syntax in mypy. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Avoid (unless Python 3.5-3.6): :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The goal is to be able to make the following statement: The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. The mypy implementation is currently still being worked on. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value or: :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

10 31

Thoughts on a typing.Self?
by gobot1234yt＠gmail.com Nov. 2, 2021

Nov. 2, 2021

I would like to draft a PEP for a typing.Self class which would perform similarly to rust's Self keyword in function annotations. Examples ```python from typing import TypeVar F = TypeVar("F", bound="Foo") class Foo: def parse(self: F, data: bytes) -> F: ... return self ``` vs ```python from typing import Self class Foo: def parse(self, data: bytes) -> Self: ... return self ``` Another example of where this could be useful is classmethods: ```python class Foo: @classmethod def bar(cls: type[F], *args: Any, **kwargs: Any) -> F: return cls() ``` vs ```python class Foo: @classmethod def bar(cls, *args: Any, **kwargs: Any) -> Self: return cls() ``` Implementation wise it would be a copy of NoReturn, i.e. not subscriptable, a _SpecialForm and should only be valid as a return type. I haven't seen any discussion of the idea on the mailing list, so I wanted to run it by everyone. Thanks, James.

12 27

Meetup to discuss Callable Syntax
by S Pradeep Kumar Oct. 8, 2021

Oct. 8, 2021

Hello all, User-friendly Callable syntax is something we all want to see. However, there has been no real movement on the discussion in the last one month. People have had differing opinions based on design principles, stats, and personal experience. This has led to half a dozen proposals with some subtle and not-so-subtle differences [1]. And Guido pointed out that we want to settle on a full proposal instead of going for a partial PEP now and settling on the remaining details in a later PEP. How about we meet up over a zoom call and discuss some of the key concerns? Things proceed noticeably faster when people from the typing community are able to get in the same (virtual) room (I saw that with PEP 646 and the typing summit). We don't have to reach a consensus in the first meeting itself. It's more a way to break the deadlock and make progress. Date: 10 am San Francisco time (6 pm UK time), Monday, September 13, 2021. Date is flexible. Duration: 1 hour. If there is enough interest, I can set up the video call and share the link and agenda. If you're interested in presenting pros and cons for your favorite proposal, reply to this mail. [1]: Document containing the various proposals and use cases: https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB… -- S Pradeep Kumar

9 33

How to pass a reference to a protocol class
by Andreas Stenius Aug. 27, 2021

Aug. 27, 2021

Hi, I’m in a position, where I’d like to be able to pass a reference to a Protocol Class, however, I can’t find a way to describe that variable with the current type system, as Type[ProtocolClass] means a class implementing said protocol, where I want the protocol itself, while ProtocolClass means an instance of a class implementing the protocol. So I haven’t figured out if PEP 544 specifies a type that describes the Protocol class itself, rather than just its compatible implementations. I tried to file this as a mypy bug <https://github.com/python/mypy/issues/10988>, but no response yet, and alas, figure it may even be overlooked by the PEP in question. Anyway, I’d very much appreciate a few more eyes and input on this matter. Thank you, Cheers This is a cross post from https://discuss.python.org/t/how-to-pass-a-reference-to-a-protocol-class/10… Reported MyPy Issue: https://github.com/python/mypy/issues/10988

1 1

NamedArg?
by Paul Bryan Aug. 20, 2021

Aug. 20, 2021

Is anything like mypy's `NamedArg` coming to the Python standard library? Meanwhile, if I want to include named arguments in a Callable type, is the right approach to use something like `Callable[..., int]`?

2 2

Optional protocol fields
by Sebastian Rittau Aug. 19, 2021

Aug. 19, 2021

This was previously briefly discussed in https://github.com/python/typing/issues/601. Occasionally it's useful if it was possible to mark protocol fields as optional. While the existence of such a field is not guaranteed, if it exists, it follows the described protocol. This is especially useful for I/O protocols, where implementations often check for the existence of a particular method, before using it. One prominent example is the WSGI spec, where the iterable returned by an application can optionally have a close() method. My suggestion is to add an @optional decorator to typing to be used like this: from typing import Protocol, optional class MyProto(Protocol): def write(self, __b: bytes) -> Any: ... @optional def close(self) -> Any: ... On the Python side, this is fairly trivial to implement: @optional would just return the passed in function, possible adding an attribute for runtime inspection: def optional(f: _F) -> _F: f.__optional__ = True return f Full support for type checkers is more complicated, as hasattr() support would be required to check implementations. A quick & dirty way for type checkers to support the status quo is to ignore @optional fields. - Sebastian

2 2

Discussion forum for typing Q&A and review requests
by Sebastian Rittau Aug. 19, 2021

Aug. 19, 2021

Typing is hard. And constantly evolving. I've seen questions about correct typing in many different forums and bug trackers, and often the same questions get repeatedly asked. This is why we set up a central forum to help users with typing at https://github.com/python/typing/discussions. Currently, it's a bit bare bones, but we have two categories for general Q&A and for asking for reviews. If users have questions about typing, please feel free to direct them there or to Gitter (https://gitter.im/python/typing). If you maintain a typing-related package or typing-related documentation, we would be happy if you could add a link to the discussions forum to your documentation. Contributors to the forum and suggestions on how to improve it are most welcome. That forum is intended as a user help forum. Discussions about new features, improvements, and implementation should continue to go to this list or to the appropriate issue trackers. - Sebastian

1 0

Support for sealed classes
by Adrian Freund Aug. 19, 2021

Aug. 19, 2021

Hello typing-sig, I wanted to suggest adding support for sealed classes to the python type system. What are sealed classes? Sealed classes are classes of which all subclasses are known statically. This is usually accomplished by requiring all subclasses to be created in the same file/module as the parent class. What languages support sealed classes? Sealed classes are supported by C#, Kotlin and starting with Java 15 even by Java. What can sealed classes be used for? Sealed classes are useful for checking the exhaustiveness of if/else towers and (python 3.10) match statements for example when working with abstract syntax trees. Take a look at this example: ``` class Node(ABC): ... class Value(Node): ... class BinOp(Node): ... class UnOp(Node): ... n: Node match n: case Value(v): ... case BinOp(n1, op, n2): ... case UnOp(op, n1): ... # Can we statically check if all cases are handled? ``` Sealed classes in python: My suggestion would be to add a new class "Sealed" to typing and typing_extensions that can be inherited from: class Node(Sealed): ... Type checkers should make sure that no subclasses of Node are created in modules other than the module Node was created in and should treat sealed classes as abstract classes. Additionally they can keep track of all the subclasses of Node do implement exhaustiveness checking. The runtime implementation of sealed could also prevent non-typechecked code from creating additional subclasses: ``` class Sealed(ABC): def __init_subclass__(cls): for c in cls.__mro__: if Sealed in c.__bases__ and c.__module__ != cls.__module__: raise RuntimeError('Sealed class can\'t be subclassed from another module. Sealed in module "{}".'.format(c.__module__)) ``` Final example: a.py: ``` from typing import Sealed class Base(Sealed): pass class A(Base): pass class B(A): pass ``` b.py: ``` from a import Base, A class C(Base): # raises an exception pass class D(A): # also raises an exception pass ``` Adrian Freund

5 7

Discussion on Compose type operator
by Alfonso L. Castaño Aug. 18, 2021

Aug. 18, 2021

Hi, Context: during the typing meeting of this week Brandon presented the idea of introducing a Compose type operator, which takes a variadic of callables, checks that they compose, and returns the type of the composed function. To follow-up on the discussion about Compose: We were wondering what can be achieved nowadays without custom operators, just relying on overload. Let's think about the example of a compose function, we could do the following right? from typing import Callable as Call # Overloads def compose(f1: Call[A,B]) -> Call[A,B] def compose(f1: Call[A,B], f2: Call[B,C]) -> Call[A,C] def compose(f1: Call[A,B], f2: Call[B,C], f3: Call[C,D]) -> Call[A,D] ... What is less obvious is how would we type a class like torch.nn.Sequential, which would look like this with the Compose operator: class Sequential(Generic[A,*Ts]): def __init__(self, fs: *Ts) def __call__(self, in: A) -> Compose[*Ts] Here we can't overload the class definition, so what can be done? We had a couple of ideas but they would not really work for a variable number of arguments because we can't overload. An example of those approaches would be: class Sequential(Generic[A,B,C]): def __init__(self, f1: Callable[A,B], f2: Callable[B,C]): ... def __call__(self, in: A) -> C: .... Best, Alfonso.

4 5

Next tensor typing meeting: 16th August
by Matthew Rahtz Aug. 17, 2021

Aug. 17, 2021

Hi all, Out next tensor typing meeting will be *Monday the 16th of August at 6pm UK time/10am San Francisco time*. Our tentative agenda is: 1. PEP 646 progress (basically that grammar changes are done and we're just waiting for review on updates to the PEP itself) 2. Discussion of feedback we got from TensorFlow/PyTorch core devs Give me a shout if anyone's interested in talking about something else! Cheers, Matthew

1 2

Compact syntax for Callable: `(Arg1, Arg2) -> Ret`
by Steven Troxler Aug. 15, 2021

Aug. 15, 2021

Hi typing-sig, I’d like to start a discussion about moving forward with improved syntax for typing.Callable based on discussion in the Typing Summit: * Allow `(ArgType0, ArgType1) -> ReturnType` * In place of `Callable[[ArgType0, ArgType1], ReturnType]` A lot of folks also want a way to express things like named, optional, and variadic args, but this is more complicated and has harder tradeoffs. I'll start a separate thread next week to discuss that, let's aim to keep this thread focused on the syntactic sugar described above. If we think this is a good idea, I'm happy to work on drafting a PEP. ## Background The existing Callable type is useful for simple functions with required arguments passed by position, but there’s consensus that it can be hard to keep track of brackets, especially as the types get complex. For example, even in very simple functional-style code with higher-order functions the brackets can be cumbersome: ``` from typing import Callable def accepts_callback(callback: Callable[[str, int], str]) -> None: print(callback("answer", 42) def f(s: str, x: int) -> str: return s + ": " + str(x) accepts_callback(f) accepts_callback(lambda s, _: s) ``` In agreement with a poll from the Typing Summit, I’d like to propose allowing * `(Argtype1, Argtype2) -> ReturnType` to be used in place of * `Callable[[Argtype1, Argtype2], ReturnType]` This allows us to rewrite the example above as: ``` def accepts_callback(callback: (str, int) -> str) -> None: print(callback("answer", 42) def f(s: str, x: int) -> str: return s + ": " + str(x) accepts_callback(f) accepts_callback(lambda s, _: s) ``` Advantages of arrow syntax include: * eliminate the need to import Callable * easier-to-skim types * they are shorter, and one of the problems with Callables is just verbosity * the parentheses and arrow have more visual structure than existing bracket-based Callable syntax * a syntax that should feel familiar: * it mimics the syntax for ordinary functions and stubs * it is similar to several other languages e.g. * [typescript](https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax) * [flow](https://flow.org/en/docs/types/functions/#toc-function-types) * [kotlin](https://kotlinlang.org/docs/lambdas.html#function-types) ## Open questions about syntax rules Would we require parentheses around a callable return type as above, or allow it to be bare (which causes two ->s to be visually at the same level of the code, even though one is nested in the AST)? Consider this function (annotated using a conservative syntax where we wrap the return type in paranetheses): ``` def identity( f: (str, int, int) -> int, ) -> ((str, int, int) -> int): return f ``` In this case, would we want to * require parentheses around a callable return type as above, or * allow it to be bare (which causes two ->s to look a little bit as if they are at the same code level) Here’s what the example above would look like if we allow bare arrow syntax in the return type: ``` def identity( f: (str, int, int) -> int, ) -> (str, int, int) -> int: return f ``` Another option, if we feel like the outer parentheses are necessary would be to make the parameter grouping optional in this case, for example: ``` def identity( f: (str, int, int) -> int, ) -> (str, int, int -> int): return f ``` ## Alternatives considered We ran a poll at the typing summit comparing three options for nicer Callable syntax: * This proposal `(ArgType1, ArgType2, ArgType3) -> ReturnType` * 72% of participants upvoted this * A typescript-like syntax `(a: ArgType1, b: ArgType2) => ReturnType` * 62% of participants upvoted this * Disadvantages are * a new symbol `=>` * argument names are nuisance parameters for positional args (which is all the existing Callable supports) * `[ArgType1, ArgType2, ArgType3] -> ReturnType` * 8% of participants upvoted this * The main disadvantage is that brackets don’t seem as intuitive - Steven

10 55

Re: Arbitrary Literal Strings
by Anton Agestam Aug. 13, 2021

Aug. 13, 2021

Some alternative name suggestions for AnyLiteral: - Static[str] - Safe[str] - Untainted[str] IMO it would be preferable for the name to be more focused on the property of "untaintedness", than how the value is defined.

1 0

Arbitrary Literal Strings
by S Pradeep Kumar Aug. 11, 2021

Aug. 11, 2021

Literal Types ([PEP 586](https://www.python.org/dev/peps/pep-0586/)) allow us to type a specific literal string like `x: Literal[“foo”] = “foo”`. This is useful when we know exactly which string or set of strings we want to accept. However, I’ve run into use cases where we'd want to accept *any* literal string such as “foo”, “bar”, etc. For example, we might have a custom format string function. For security reasons, we would want the typechecker to enforce that the format string is a literal, not an arbitrary string. Otherwise, an attacker could read or write arbitrary data by changing the format string (the so-called “format string attack” [1]): ``` def my_format_string(s: str, *args: FormatArgument) -> str: … my_format_string(“hello: %A”, a) # OK my_format_string(user_controlled_string, a) # BAD ``` Likewise, if we have a custom shell execution command like `my_execute`, we might want to enforce that the command name is a literal, not an arbitrary string. Otherwise, an attacker might be able to insert arbitrary shell code in the string and execute it: ``` def my_execute(command: str, *args: str) -> None: ... my_execute("ls", file1, file2) # OK command = input() my_execute(command, file1, file2) # BAD ``` There is no way to specify the above in the current type system. # Proposal We can allow `Literal[str]`, which would represent *any* literal string: ``` from typing import Literal def my_format_string(s: Literal[str], *args: FormatArgument) -> str: … my_format_string(“hello: %A: %B”, a, b) # OK because it is a literal string. my_format_string(user_controlled_string, sensitive_data) # Type error: Expected Literal[str], got str. ``` The same goes for the shell command function: ``` def my_execute(command: Literal[str], *args: CommandArgument) -> None: … my_execute(“ls”, files) # OK my_execute(arbitrary_string, files) # Type error: Expected Literal[str], got str. ``` Other usage will work as expected: ``` from typing import Literal, TypeVar # Type variable that accepts only literal strings. TLiteral = TypeVar("TLiteral", bound=Literal[str]) def identity(s: TLiteral) -> TLiteral: ... y = identity("hello") reveal_type(y) # Literal[“hello”] s: Literal[str] y2 = identity(s) reveal_type(y2) # Literal[str] literal_string: Literal[str] s: str = literal_string # OK literal_string: Literal[str] = s # Type error literal_string: Literal[str] = “hello” # OK x = “hello” literal_string: Literal[str] = x # OK ``` ## Backward compatibility **Backward compatibility**: `Literal[str]` is acceptable at runtime, so this doesn’t require any changes to Python itself. **Reference Implementation**: This was quite easy to implement and is available in Pyre v0.9.3. **Rejected alternatives**: `T = TypeVar(“T”, bound=Literal[Any])` isn’t something allowed in PEP 586 and would anyway be too broad. It would also allow literal bools to be passed in when we want only literal strings. ## Other uses for Literal[str] Other places where it might be useful to statically enforce literal strings for safety and readability: ``` # struct struct.unpack("<I", self.read(n)) # datetime datetime.now().strftime('%B %d, %Y - %X') # builtins open(path, encoding='utf-8') my_string.encode('latin-1') # PyTorch self.register_buffer("weight", torch.zeros(a, b)) # argparse parser.add_argument("--my-flag", action="store_true") # argparse ``` The same idea would apply to `Literal[int]` and `Literal[bool]`, but I don’t have compelling use cases for them yet. I suspect `Literal[int]` will be useful for Tensor types in the future. Others might have run into use cases in the wild. Thoughts? Opinions? [1]: [ https://owasp.org/www-community/attacks/Format_string_attack](https://owasp… -- S Pradeep Kumar

11 29

Add Expand special form to allow typing **kwargs with a TypedDict
by framagie＠gmail.com Aug. 11, 2021

Aug. 11, 2021

Hello, the ability to type **kwargs using TypedDict is a popular request (https://github.com/python/mypy/issues/4441). The proposal to use the form **kwargs: **TypedDictType is problematic because it causes a syntax error. Introducing a new Expand special form would enable adding this feature quite easily. There is a preliminary PR to support this feature in mypy - https://github.com/python/mypy/pull/10576 What are your opinions on this?

4 3

Type for arbitrary JSON data.
by Artemis Aug. 10, 2021

Aug. 10, 2021

Very often while writing web-related scripts, I have a function that either accepts or returns some arbitrary JSON data. Sometimes I know that this will be an object, so I annotate it as `dict[str, Any]` - which isn't quite accurate, since the values can't be `Any`. Sometimes I'm feeling lazy and annotate it as `dict[str, Any]` even when it may be any other JSON type. Very occasionally do I go to the effort of making something more accurate, maybe JsonData = Union[dict[str, 'JsonData'], list['JsonData'], int, float, bool, None] I think adding something like this called `Json`, `JsonData`, `JsonEncodable` or similar to `typing` or `json` would be very helpful for people working with web APIs. Perhaps a `JsonObject`/`JsonDict` aliased to `dict[JsonData]` and a `JsonArray`/`JsonList` aliased to `list[JsonData]` (like in Java) would also be helpful.

7 7

Support for type checking fixed width integers
by Arun Sharma Aug. 7, 2021

Aug. 7, 2021

I'm aware that the python3 type system primarily supports arbitrary precision integers and there is no support for type checking fixed width integers. However, when python is used as an approachable first language for data scientists, scientists interested in numerical analysis and possibly other generic use cases, one common technique is to map python's type system to that of a lower level underlying system. I'm primarily coming at it from the third use case in the form of py2many, a transpiler for converting python to many statically typed C-like languages. I suspect pytorch, tensorflow or anything that maps python to vectorized hardware efficiently has to deal with the precision of integers. Canonical use case: ``` #!/usr/bin/env python3 from ctypes import c_int8 as i8 def test(): a: i8 = 10 b: i8 = 20 c = a + b reveal_type(c) ``` Also would love to have type checkers infer ranges and throw errors for cases like: c: i8 = 300 Doing so would significantly benefit the use case of generating secure C++/Rust code from python source: https://github.com/adsharma/py2many/issues/233 Eric Traut suggested I post to the mailing list. More context here: https://github.com/microsoft/pyright/issues/1872 -Arun

2 4

Interpreting PEP 586
by Paul Bryan Aug. 4, 2021

Aug. 4, 2021

This question is based on my fundamental misunderstanding of Literal that was pointed-out in the "Arbitrary Literal Strings" thread. As a mental exercise, I'm imagining `open` having a mode that is `Literal["r", "w", "a", ...]` per PEP 586. If one then wants to pass a non-literal string value of "rb" (supposing it was dynamically generated), to the newly Literalized `mode` param, would the only way to accomplish this be to compare and supplying literals in if-elif statements, maps of values to their literal selves, (or structural pattern matching when available)? Is there a better way of coercing a string value to be a Literal? A casual reading of the PEP results in an ambiguous understanding of this point as well. It states: > Literal types indicate that a variable has a specific and concrete > value. For example, if we define some variable foo to have type > Literal[3], we are declaring that foo must be exactly equal to 3 and > no other value. It says "exactly equal to", and makes no mention of how the value was evaluated. Arguably, `foo: Literal[3] = 1 + 2` assigns a specific and concrete value of 3 to foo. foo will be exactly equal to 3 and no other value.

1 0

Generic TypedDict: Details of generic support for TypedDict
by Samodya Abey Aug. 4, 2021

Aug. 4, 2021

Hello, I have drafted an initial PEP detailing adding support for generic TypedDicts. Please see the copy at the bottom of this email or the following repository where I tracked the work: https://github.com/sransara/py-generic-typeddict/ I would love to here your comments / guidance on the writing as responses to this email on typing-sig or in the repository. Some places that definitely need improvement: - Wording of the motivation - Getting TypeVar parameter order in alternative syntax without OrderedDict for older versions I have a few implementations available (also available as sub modules in the tracking repo: https://github.com/sransara/py-generic-typeddict/): - [-] Generic `TypedDict` in `typing.py` - [x] [Patch](https://github.com/python/cpython/compare/3.9...sransara:py-generic-… typing.py for versions >= 3.9 - [ ] Integrate to test suite - [-] Generic `TypedDict` runtime in `typing_extensions` - [x] [Patch](https://github.com/python/typing/compare/master...sransara:py-generi… typing_extensions.py for versions < 3.9.2 - [ ] Integrate to test suite - [-] Generic `TypedDict` in one of the type checkers ([pyright](https://github.com/microsoft/pyright/)) - [x] [Patch](https://github.com/microsoft/pyright/compare/main...sransara:generic… to Pyright ([PR](https://github.com/microsoft/pyright/pull/1390)). - [ ] Rebase the PR on to a current version - [ ] Support for declaring with TypedDict constructor notation - [ ] Adding proper test cases to the automated test suite For further background: - Originating PR: https://github.com/microsoft/pyright/pull/1390 - Origin typing sig thread with suggestions to move forward: https://mail.python.org/archives/list/typing-sig@python.org/thread/GKKSWMON… - Another request in mypy: https://github.com/python/mypy/issues/3863 Thank you, Samodya Abeysiriwardane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 9999 Title: Details of generic support for TypedDict Author: Samodya Abeysiriwardane <hi at sransara.com> Status: Draft Type: Standard Track Content-Type: text/x-rst Created: 19-Jul-2021 Python-Version: 3.11 Post-History: Abstract ======== PEP 589 [#PEP-589]_ introduces a type constructor ``TypedDict`` to support the use case where a dictionary object has a specific set of string keys, each with a value of a specific type. But it does not specify the details of supporting generic TypedDicts, where the type of a value can be parameterized. This PEP specifies the details of a type parameterized TypedDicts as an extension of PEP 589 [#PEP-589]_. Motivation ========== TypedDict is a great choice to give a precise type for pure data containers. Almost all container data types of the standard library support generics. TypedDicts should also support generics. Following is few example usecases for having generic TypedDicts. Using type annotations to convey relationships between keys. class Page(TypedDict, Generic[T]): uid: Hash[T] desc: str item: T An example where the function on container type does not concern with the concrete type of the contents. def get_items(*pages: Page[T]) -> List[T]: return [page["item"] for page in pages] Another usage example in annotating polymorphic JSON responses. class Item(TypedDict): type: str class ValueItem(Item): value: int T = TypeVar('T') class ItemPager(TypedDict, Generic[T]): next_page: Optional[str] items: List[T] def get_value_items() -> List[ValueItem]: value_items: List[ValueItem] = [] url = "https://example.com/value_items" while url: response = requests.get(url) pager: ItemPager[ValueItem] = response.json() url = pager['next_page'] value_items += pager['items'] return value_items Specification ============= This PEP is an extension of PEP 589 [#PEP-589]_, hence details specified here for TypedDicts will be in addition to what is specified in PEP 589 [#PEP-589]_. Class based syntax ------------------ Class based syntax for TypedDicts will follow the same specification as user defined generic types in PEP 484 [#PEP-484]_ and class based syntax specification detailed in PEP 589 [#PEP-589]_. from typing import TypedDict T = TypeVar("T") class Pager(TypedDict, Generic[T]): next_page: str items: List[T] ``Generic[T]`` as a base class defines that the class ``Pager`` takes a single type parameter ``T``. ``Pager[int]`` is a TypedDict that has a key ``items`` with value type ``List[int]``. Alternative syntax ------------------ The following alternative syntax is semantically equivalent to the previous class based syntax example. Order of Generic subscript TypeVars will be in the order of their appearance. For Python versions before 3.7 where dict item order is not guaranteed, an ordered dictionary needs to be used with typevars. from typing import TypedDict T = TypeVar("T") Pager = TypedDict("Pager", { 'next_page': str, 'items': List[T], }) Inheritance ----------- A TypedDict can only inherit from another TypedDict or TypedDict with Generic. This is in contrast to PEP 589 [#PEP-589]_, which specifies that a TypedDict can only inherit from a TypedDict. Subclassing a generic TypedDict without specifying type parameters assumes ``Any`` for each position. All other details of TypedDict inheritance from PEP 589 [#PEP-589]_ applies. Using generic TypedDict types ----------------------------- Here is an example of how the type ``Pager`` can be used. pager: Pager[str] = { 'next_page': 'https://example.com/value_items?p=2', 'items': ['item 1', 'item2',] } If the type parameter is not specified, type checker can follow the same behavior as how it treats other generic instances without a type parameter. Type consistency ---------------- Same type consistency rules from PEP 589 [#PEP-589]_ applies. Since TypedDict objects are mutable, value types should behave invariantly. Therefore at declaration a type checker should only accept invariant TypeVars. from typing import TypedDict T = TypeVar("T", covariant=True) class Page(TypedDict, Generic[T]): # Covariant T should not be allowed item: T def set_item(p: Page[Animal]): p["item"] = Animal() page: Page[Dog] = Page(item=Dog()) set_item(page) page["item"].bark() # Run time error References ========== .. [#PEP-589] PEP 589, TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys, Lehtosalo (https://www.python.org/dev/peps/pep-0589/) .. [#PEP-484] PEP 484, Type Hints, Rossum, Lehtosalo, Langa (https://www.python.org/dev/peps/pep-0484/) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

2 2

Type bounds on PEP 646
by Alfonso L. Castaño Aug. 2, 2021

Aug. 2, 2021

Hi, I have been thinking about the implications of not supporting bounds in PEP 646. Although when I mentioned it the last time that I checked the PEP I did not give it much importance, after having thought more about it and heard about more use cases I wonder if leaving the bound for another PEP would not limit too much the value of the PEP. The main motivation of the PEP is to use variadics to model the shape of a tensor, either by specifying the exact size of the dimension or by specifying their name. However, if for example a library like NumPy or PyTorch would want to take advantage of this feature, they would need to restrict the bound of the variadic. For example, if they go for typing exact dimensions they would like to accept Tensor[L[20],L[40]] but not Tensor[L[20], str, ClassFoo]. (L = Literal) Similarly, if they prefer to go for the naming route (Tensor[Width, Height]) but they don't want arbitrary types in there. Therefore, I was wondering if leaving it for a future PEP is the best approach. The initial reason was to keep the PEP simple but I think that the additional complexity could come from supporting Variance (which can be left out), but bounds would not introduce any new behavior, they would just replicate the logic of the bounds in TypeVar. Any thoughts? Alfonso.

4 12