Mailman 3 April 2021 - Typing-sig

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 20 Jun '23

20 Jun '23

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

17 35

Runtime Access to Type Parameters
by Saul Shanabrook 07 Jun '23

07 Jun '23

I would like to propose being able to access, at runtime, the type parameters in classmethods and in constructors. I wrote up a draft PEP on this to capture the thinking here: https://github.com/saulshanabrook/peps/blob/master/pep-9999.md It seems easier to add them to classmethods, so even that addition alone would be very welcome! It could be added to Guido's existing draft PR for pep 585, by making a small change in the getattr function (https://github.com/python/cpython/pull/18239/files#diff-a34ae826f869897e56e…), to first do a `__getattribute__` and check if the result is a descriptor (has a `__get__` attribute), and if so passing the generic class itself in as the class, instead of the origin class. I wrote up a little example of that in the PEP. To make it available in the constructor, I proposed adding a global function `get_parameters` to `typing` which returns the parameters in the constructor. I also provide a little mock implementation of that. Note that work is only for custom classes which inherit from `typing.Generic` not for any built in collections.

5 6

Explicit Type Aliases
by Shannon Zhu 17 Aug '22

17 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

PEP 655: Marking individual TypedDict items as required or potentially-missing
by David Foster 23 Feb '22

23 Feb '22

The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/ Comments welcome. :) Meanwhile I'll proceed to implement this syntax in mypy. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Avoid (unless Python 3.5-3.6): :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The goal is to be able to make the following statement: The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. The mypy implementation is currently still being worked on. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value or: :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

10 31

Towards statically checked ORM projections
by Tin Tvrtković 18 May '21

18 May '21

Dear typing-sig, I've noticed recently that there is practically no support for type-safe ORM/ODM projections in the broader Python ecosystem. A little context: ORMs (object-relational mapper) and ODM (object-document mapper/mapping) are tools or libraries for interacting with databases. SQLAlchemy and the Django ORM are probably the most famous examples. Basically, you model a database table as a class, and rows in the table are instances of that class. ODMs are a little simpler than ORMs, they practically only map rows/documents to instances of a class, ORMs are more complex. Writing a type-safe ODM nowadays is super simple, and there are loads out there. The idea is this: ``` @dataclass class User: id: int username: str user = await fetch(User, id=1) ``` This can be made to work with the proper type annotations without much fuss. Now, the issue is doing projections in a type-safe manner. A projection basically means loading only a subset of the fields, usually for performance. Let's say that we're only interested in the user username (imagine there are 30 other fields in the class that we don't care about). That would look kinda like: ``` from dataclasses import fields user_projection: tuple[str] = await fetch_projection(User, id=1, fields(User)[1]) ``` This can't really be very type-safe since Mypy treats `fields(User)[1]` as `dataclasses.Field*[Any]`. Now, if Mypy treated it as `dataclasses.Field*[str]`, I assume that would be a different story, and the function could be annotated to return a 1-tuple of `str`. As mentioned, I've looked at a bunch of ORM/ODM libraries (and written a few internally) and as far as I can tell this use-case is very much unsupported as of yet. (I would appreciate counter-examples, obviously!) This made me sad so I came to the list to see if there's anything to be done. To be super honest I'm more interested in getting attrs support for this, but attrs and dataclasses are so similar I figured if someone did the work in the dataclass plugin, the logic could also be ported over to the attrs plugin. attrs also has a nicer API for actually getting the fields, so the equivalent attrs example would be: ``` from attrs import fields as f user_projection = await fetch_projection(User, id=1, f(User).username) ``` Wouldn't it be super cool if Mypy (or other type checkers) could check this statically?

6 11

Supporting dataclass-like semantics
by Eric Traut 15 May '21

15 May '21

PEP 557 introduced the dataclass to the Python stdlib. Several popular libraries (including attrs, pydantic, and various libraries that support database ORMs such as django and edgedb) have behaviors that are similar to dataclass, but these behaviors cannot be described using standard type annotations. To work around this limitation, mypy custom plugins have been developed for many of these libraries, but these plugins don't work with other type checkers, linters or language servers. They are also costly to maintain for library authors, and they require that Python developers know about the existence of these plugins and download and configure them within their environment. I've been exploring ways to extend support for dataclass semantics to other other libraries in a standardized manner. I've posted a [proposal in the form of a proto-PEP](https://github.com/microsoft/pyright/blob/master/specs/dataclass…. The latest versions of pylance and pyright implement this proposal as a proof of concept. It was relatively easy to implement (on the order of a couple hundred lines of code), and I assume that it would be similarly easy for other type checkers to add this support. I've created a [github discussion and feedback forum](https://github.com/microsoft/pyright/discussions/1782), and I welcome feedback on the proposal. I've already been in contact with the maintainers of attrs and pydantic. They are generally supportive, and I've incorporated their initial input into the latest draft of the proposal. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.

9 19

Type assignment statement (and extending the annotation syntax)
by Jukka Lehtosalo 30 Apr '21

30 Apr '21

Many people seem to agree that Python typing still has some rough edges. Various PEPs have fixed warts (or proposed fixes), including PEPs 585, 563, and 649. Still, several issues remain: 1. Type annotation syntax is not always elegant (consider callable types). There's no good way to add custom type syntax without "polluting" non-typing use cases. 2. String literal escaping is ugly, and it's still needed from time to time. 3. Runtime introspection of types feels like a second-class citizen, even though it's pretty popular (pydantic, etc.). Instead of trying to tackle these issues piecemeal, I've tried to come up with ideas to solve all of the above issues. They don't form a complete proposal, and I haven't worked out many details yet, but I'm hoping that this could be a discussion starter, and perhaps it can help us eventually design something that is workable. Here I explain my first idea -- type assignment. Right now, we can't easily add dedicated syntax for callable types, for example, without either making this syntax available everywhere, or making it *un*available in type aliases and other common use cases, which don't use the annotation syntax. Supporting new syntax in annotations seems manageable: def doit(f: () => int) -> None: ... # Could be supported However, what about type aliases: A = () => int # Hmm? If this works, nothing stops people from writing weird code like this: d = {'x': (1 + y) => foo(1, 2)} I'd rather have dedicated typing syntax reserved for type-related uses, mainly to avoid confusing Python users who don't use typing. If typing syntax is only available via a small number of special syntactic forms, it's easier to teach users to ignore it, if they don't care about it. I'm proposing a new construct called "type assignment" where the RHS is lazily evaluated, and in the future it might support typing-only syntax. There would be two variants. First, we can define a type alias like this: type A = list[int] If we'd use the new syntax for callable types, we could write an alias like this: type C = (int) => str # No problem However, the following would be a syntax error, since it's not in a "type context": C = (int) => str # Error, as it probably should be Since the right hand side of type assignment contains types, we can avoid evaluating them during module import time, using either PEP 563 semantics (stringification) or something resembling PEP 649 (lazy evaluation). This means that we won't need to use explicit string literal escaping: type A = B # OK! class B: ... If the RHS is a function call, we'd call the function at runtime, but arguments would be (partially) stringified/lazy and they can contain type syntax (I'm glossing over some details): type N = NewType(int) type T = TypeVar(bound=(int) => str) # Type-specific syntax is ok type D = typing.TypedDict({"x": C}) # C can be a forward reference! class C: ... Note that the results here are usable in runtime contexts, since we call the function: x = N(2) # Ok The general syntax would be "type <id> = <name>(...)". We'd pass the stringified name as an implicit first argument for a cleaner syntax. We can write the earlier examples today like this: T = TypeVar("T", bound=Callable[[int], str]) N = NewType("N", int) D = typing.TypedDict("D", {"x": "C"}) # Need escaping here class C: ... The new syntax uses 'type', which is widely used as a name. The new meaning would only apply in this assignment context, and all existing uses are still fine. 'type' wouldn't be a keyword. To support the function call syntax lazily, we want to be able to distinguish between types and non-types in arguments. First, any literals would be eagerly calculated and not escaped: type T = f(1, "y", kw=False) # evaluated as T = f(1, "y", kw=False) Name expressions are lazy, to allow forward references: type T = f(C, kw=D) # T = f("C", kw="D") [if using PEP 563] If using delayed evaluation instead of stringification, we can create some "lazy type reference" objects instead of strings (this is kind of hand-wavy). Similarly, tuples, lists and dictionaries would be eager (but items can be lazy): type T = f(C, [("x", list[C])]) # T = f("C", [("x", "list[C]")]) This would resemble PEP 563 stringification, but it needs to be a little more flexible. The type statement syntax is general enough to support various existing use cases, including NewType, TypeVar, TypedDict, NamedTuple, and others, and it's also extensible to new forms. I'll write separate posts about my other ideas I mentioned earlier. Jukka

7 16

PyCon 2021 Typing Summit
by Shannon Zhu 26 Apr '21

26 Apr '21

Hello everyone, PyCon 2021 is coming up fast, and so is the first Typing Summit in over a year. We’re really excited and hope you are too! Thanks to the hard work of everyone who volunteered to present a topic for discussion, Alfonso and I are now able to share an agenda for the summit: Typing Summit Schedule Thursday, May 13th from 11AM-3:30PM EST, 8AM-12:30PM PST, 5PM-9:30PM CET All times below listed in PST. 8:00 AM - 8:15 AM: Greetings, Starting Up 8:15 AM- 8:45 AM: Type Syntax Simplifications 8:45 AM - 9:10 AM: Validating JSON with TypedDict, trycast, and TypeForm 9:10 AM - 9:40 AM: Type Variables for All 9:40 AM - 9:50 AM: Break 9:50 AM - 10:15 AM: Static Python: Types in Bytecode Compilation & Runtime 10:15 AM - 10:35 AM: PyType’s Approach to Large-Scale Type Checking 10:35 AM - 11:00 AM: Incremental Check in Pyre 11:00 AM - 11:10 AM: Break 11:10 AM - 11:35 AM: Scaling Typeshed to 1000 Packages 11:35 AM - 12:05 AM: Catching Tensor Shape Errors Using the Type Checker 12:05 AM - 12:30 PM: Type Arithmetic 12:30 PM - End All of the items on the agenda have time for both a presentation and an active discussion built in, so everyone will have a chance to ask questions and actively engage with the topic. If any of this sounds interesting to you, please be sure you are RSVP’d<https://docs.google.com/forms/d/e/1FAIpQLSct42iXWscqznadBYSSsJUWm0wLd9IFKS0…> to attend the summit*! Because we are not requiring a PyCon registration in order to attend this event, we can only share details around how to log in and participate with RSVP’d email addresses close to the event date. *Feel free to ignore all fields in the form except name and email. You can also find the RSVP link and agenda information published on the PyCon 2021 website, under Summits and Events<https://us.pycon.org/2021/summits/typing/>. Please feel free to reach out to me or Alfonso if you have any questions. Looking forward to seeing everyone virtually soon!

1 0

PEP 646: It Probably Doesn't Do What Everyone Wants It To Do
by Cecil Curry 13 Apr '21

13 Apr '21

PEP 646: It Probably Doesn't Do What Everyone Wants It To Do Hi, all! Warmest greetings on this drizzly Spring evening from the squalid Canadian wetlands, where even the mosquitoes think it's really too cold. I'm Cecil Curry (@leycec), the plucky author of beartype (https://github.com/beartype/beartype) – the pure-Python O(1) runtime type checker that pretends to do alot while actually doing not too much at all. Thus O(1). "You gets what you pays for," right? My wife and I also co-authored a several million-line pure-Python multiphysics biology simulator (https://gitlab.com/betse/betse) heavily leveraging the standard scientific stack (e.g., NumPy, SciPy, Matplotlib, NetworkX). Those were good years. I still had hair back then, so they had to have been good. I was kindly redirected here by Ryan Soklaski (@rsokl), the brilliant @beartype user I never knew I had and head instructor of the CogWorks course at the MIT Beaver Works Summer Institute. Ryan accurately predicted that PEP 646 specifically and the vigorous discussion here surrounding tensor typing in general would be right up my lakeside wheelhouse. Boy, is it! We're all over this sort of deep structural science-oriented QA. In fact, @beartype's userbase (https://github.com/beartype/beartype/stargazers) is predominated by machine learning, computer vision, and data scientists. We're poised on the awkward precipice between typing and tensors, because what everyone (including my wife and I) always wanted from @beartype was runtime type checking of scientific data structures. PEP 646 promises all this... and more! But does it deliver? Does @leycec ever regain his hair? Do Canadian brown bears groom themselves on spindly trees on our front yard? The answers to all these questions and more follow shortly. First, I'd like to profusely thank all the PEP 646 authors – Mark Mendoza, Matthew Rahtz, Pradeep Kumar Srinivasan, Vincent Siles, and Guido van Rossum – for their profuse work and dedication to the good cause. Second, I'd like to profusely apologize a priori for any untoward argumentation or firework shows I am about to generate. That's not the intention. Really! PEP 646 is currently at the steering committee stage. Nobody wants to see that derailed, because hard work deserves its just reward and GitHub karma. All that said... let's do this. The real-world motivation of PEP 646 is to convince the various authors of various popular numerical frameworks to adopt, embrace, and extend PEP 646. This is a noble effort. But can this effort actually be realized? The PEP 646 API assumes a Python-centric worldview. Specifically, PEP 646 assumes that numerical frameworks will subclass their data structures from "typing.Generic", a magical pure-Python class. Those assumptions derive from the first example in PEP 646: from typing import TypeVar, TypeVarTuple DType = TypeVar('DType') Shape = TypeVarTuple('Shape') class Array(Generic[DType, *Shape]): def __abs__(self) -> Array[DType, *Shape]: ... def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ... But this assumption may not necessarily hold. Most popular numerical frameworks worth supporting (including numerical frameworks explicitly referenced in PEP 646, which means NumPy and TensorFlow) are implemented with at least two layers: a low-level mostly Python-agnostic layer implemented in some combination of C, C++, Fortran, or Rust referred to as the "core API" and one or more high-level layers providing bindings to high-level languages. Python is only one of many high-level languages supported by these frameworks. This includes popular numerical frameworks one might erroneously assume from their names to be Python-specific like PyTorch. Despite the name, PyTorch is also implemented as a low-level Python-agnostic C++ layer (https://pytorch.org/tutorials/advanced/cpp_frontend.html). The same goes for: * The entirety of TensorFlow's core C++ API (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framew…). * *Most* but not all of NumPy's core C API (https://numpy.org/doc/stable/reference/c-api/coremath.html), which NumPy developers are desperately refactoring to be Python-agnostic. I believe the core "numpy.core._multiarray_umath.ndarray" type is still Python-specific, but that's likely to change sometime this decade. (Don't quote me on that. Please. My GitHub karma can only take so many body blows.) Is this a problem? It might very well be. Of course, it should be admitted at this crucial juncture that my idea of a fun time includes scrambling over tick-infested shrubland in an inhospitable wasteland for eight hours yesterday, which only goes to show that you can't trust anyone who voluntarily lives in a poorly insulated cottage plagued with mosquitoes. </ahem> "typing.Generic" is magical, because it defines the magical PEP 487 __init_subclass__() dunder method. Despite banging my balding head against CPython documentation and my ketchup-stained keyboard for several hours, I'm kinda unconvinced that a C[++]-based class can even subclass one or more magical or non-magical pure-Python classes. I've never seen anyone do either. Then again, I've never seen a circumpolar double rainbow on Christmas Eve composed of flying winged bugbears. Who's to say that can't happen? I'd sure like to see that someday. So let's assume for the sake of debate that's feasible – that C[++]-based classes can in fact subclass one or more magical pure-Python classes. Under that generous assumption, it's still unlikely that numerical frameworks that were either Python-agnostic from the get-go (like TensorFlow) or are rapidly refactoring their core APIs away from Python (like NumPy) would revert those fundamental decisions just to appease our small but vocal cadre of Python type-checking aficionados. (That's us: the good guys in the distinct minority.) Now I *know* what you're thinking. "Ah-ha! You poor sad Canadian hillbilly. All we need to do is strenuously encourage the authors of these numerical frameworks to define entirely new Python-specific classes in their Python bindings that comply with PEP 646 by subclassing "typing.Generic" in the exact way we want them to! It's so blindingly simple! If only you still had luxurious hair, @leycec. Then you might have thought of that. Alas. Here we are." In the dark ruminations of your monitor-lit devcave, you may even be dangerously considering unspeakable horrors like adding a new type-checking shim like this into the perfidious bowels of the NumPy codebase: from numpy.core._multiarray_umath import ndarray from typing import TypeVar, TypeVarTuple DType = TypeVar('DType') Shape = TypeVarTuple('Shape') class NDArray(ndarray, Generic[DType, *Shape]): def __abs__(self) -> Array[DType, *Shape]: return super().__abs__() def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: return super().__add__(other) Right? Something unmentionable like that, right? Of course, we can all admit to ourselves that's horrible, because that imposes one additional stack frame per method call. So, nobody will do that. Now I *know* what you're thinking. "Ah-ha! You poor sad Canadian person of indeterminate mental stability. All we need to do is define .pyi-formatted stub files instead expressing the same type semantics! If only your limbs weren't painfully arthritic, @leycec. Then you might have thought of that. Alas. Here we are." Right? Something inexpressible like that, right? Of course, we can all admit to ourselves that's horrible too, because .pyi-formatted stub files aren't sanely accessible at runtime. And we are back where we started in a cursed reenactment of "Blair Witch Project: Tensor Typing Edition." You're now thinking of either bolting type hints onto C[++]-based callables or wholly reimplementing "typing.Generic" in C, aren't you? I knew it! You are! And at this point, everyone should be thinking: "This doesn't quite seem right anymore." The core issue here is that PEP 646 requires someone who is not us to change their workflow for us. It's not necessarily clear that that will ever happen. Sure, it might. I'm an optimistic man, which means I'm not a betting man. Even if that utopic Xanadu were to become manifest, PEP 646 still seems problematic. My PEP 484 isn't strong, but it's unclear how you would: * Type tensor methods returning new tensors satisfying different constraints (especially of differing shapes). * Type tensor constraints in a forward-compatible manner. Tensor constraints declared by PEP 646 are crazy fragile. They lock any numerical framework that complies into a specific preordained ordering of tensor constraints. A tensor typed as "class Array(Generic[DType, *Shape])" can *never* be typed as anything else for all time, which is a pretty long time. Want to add another constraint like, say, PyTorch's newly introduced "layout" tensor attribute that fundamentally changes how one uses passed or returned tensors? (https://pytorch.org/docs/stable/tensor_attributes.html#torch.torch.layout) **Sorry.** No one can ever do that, because the original specification of "class Array(Generic[DType, *Shape])" prematurely locked everyone into checking only those two constraints. Surely we have sane alternatives? We do. Many! Oodles! Bucket-loads! And other colourful idioms for "lots." They all share one attribute in common: they don't require someone who is not us to change their fundamental workflow for us. The PEP 593 "typing.Annotated" class offers an illustrative example: import numpy as np from tensorflow.types.core import Tensor from typing import Annotated PixellatedRetroImage = Annotated[Tensor, np.float32, [640, 480]] def upscale_image(image: PixellatedRetroImage) -> PixellatedRetroImage: ... That has the benefit of working now under Python >= 3.9 without necessitating that external authors, open-source communities, or for-profit corporations do unlikely (and perhaps infeasible) stuff for us. We'll be pursuing a similar solution to that in @beartype, because it works now and satisfies everyone's downstream concerns. Of course, "typing" wouldn't want to do something like that. "typing" would want to do something even more robust, extensible, and absurdly awesome like: import numpy as np from tensorflow.types.core import Tensor from typing import Structure PixellatedRetroImage = Structure[ Tensor, {dtype=np.float32, shape=[640, 480]}] def upscale_image(image: PixellatedRetroImage) -> PixellatedRetroImage: ... Even better, right? To avoid conflicts with user-defined "typing.Annotated" type hints that already exist in the wild, we define a new "typing.Structure" class suitable for type-hinting data structure constraints. We then define two constraints in an extensible manner supporting forward-compatible changes, improvements, and modifications to third-party APIs. Of course, that's probably not quite "typing" enough. Right? Not quite enough square braces above and more than a little hostile to static analysis, which is fair enough. So let's delve one step deeper into the cavernous labyrinth of awesomeness that solves all tensor typing concerns: import numpy as np from tensorflow.types.core import Tensor from typing import Structure, Dtype, Shape PixellatedRetroImage = Structure[ Tensor, Dtype[np.float32], Shape[640, 480]] def upscale_image(image: PixellatedRetroImage) -> PixellatedRetroImage: ... Yup. It doesn't get more "typing" than that. You've got square braces all over the place; you've got well-named constraints generically applicable to the gamut of popular frameworks; you've got order-invariant extensibility, because it no longer matters what order users declare "Dtype[...]" and "Shape[...]" constraints in (or whether users even declare those constraints); you've got built-in support for both static and runtime analysis. And you've got all that without burdening third-party vendors with a Python-specific C[++]-unfriendly API they're unlikely to ever adopt. In short, you've got everything you could probably, possibly, or likely ever want. ...which means it'll never happen. PEP 646 is all but a fait accompli at this eleventh hour. That's fine, I guess. But that's also kinda *not* fine, you know? We can and should do substantially better. I reckon we can. In closing, thanks for all the Pythonic goodness over the past several decades. Everyone's amazing! Regardless of where and how Python type-checking intersects with scientific computing in the decades to come, I'm indefatigably proud of everything everyone's accomplished here. Keep on rocking the type-safe world.

5 4

How to ensure a generic function only accepts args of the same type
by Teddy Sudol 12 Apr '21

12 Apr '21

Hi typing-sig, This question has come up a few times from pytype's users: How do you write a generic function that only takes arguments that are the same subclass of a particular base class? That is, if you have a function like: def do_stuff(x, y): ... If you want it to take two args of the same type, but that type is otherwise not constrained, use a regular TypeVar: T = TypeVar("T") def do_stuff(x: T, y: T): ... If you want do_stuff to only accept args of certain types, you can limit the TypeVar to those types: T = TypeVar("T", int, str) def do_stuff(x: T, y: T): ... do_stuff(<int>, <int>) or do_stuff(<str>, <str>) are the only valid calls. This works if the constraining types are known ahead of time. If you want do_stuff to only accept args of a particular base type, use bound: T = TypeVar("T", bound=Base) def do_stuff(x: T, y: T): ... If A and B are subclasses of Base, then do_stuff(A(), A()), do_stuff(A(), B()) and do_stuff(A(), Base()) are all legal. As far as I can tell, there is no way to express that do_stuff(A(), A()) and do_stuff(B(), B()) are legal while do_stuff(A(), B()) and do_stuff(A(), Base()) are illegal, for all subclasses of Base. Am I missing something? (Pytype does accept one way to do it, though this is unintended: T = TypeVar("T", Base, Base) def do_stuff(x: T, y: T): ... mypy treats this the same as the bound=Base case, but pytype will complain about do_stuff(A(), B()). PEP 484 doesn't have anything to say about this case, as far as I know.) -- Teddy

3 2