Mailman 3 December 2022 - Typing-sig

Re: Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Adrian Garcia Badaracco Dec. 23, 2022

Dec. 23, 2022

I really want this feature for the scenario where you are getting data from some API/library that doesn’t fully describe the payload but you know one or two keys that you use. For example, boto3 returns a bunch of untyped dicts but I know that I want payload[“Key”] and that it is a string. I see that one of the major issues that were brought up was describing the type of the “extra” keys (even if that is Any). Could we do this by allowing classes to subclass both TypedDict and Dict? from typing import Dict, TypedDict class Coefficients(Dict[str, float], TypedDict, allow_extra=True): x1: int x2: int f1: Coefficients = {"x1": 1, "x2": 2, "x3": 3.3} # ok f2: Coefficients = {"x1": "bad", "x2": 2, "x3": 3.3} # bad, x1 has wrong type f3: Coefficients = {"x1": 1, "x2": 2, "x3": "bad"} # bad, x3 has wrong type I’m no type theorist but I think this transmits the idea that this is a dict with str keys and float values where some specific keys are narrowed to int (or it could be Dict[str, Any] with some keys of known types). I’m not sure what the right thing to do is w.r.t. mutability, but I suppose this could be restricted to be Mapping instead of Dict if we wanted to only allow this pattern for immutable objects?

1 1

Create an `AbstractType`, similar to `Type` in the typing module
by miguelguthridge＠gmail.com Dec. 23, 2022

Dec. 23, 2022

I've also posted this on the discuss.python.org forum, but as suggested in PEP 1, I'm also posting here to get additional ideas/feedback/criticism. Currently, using a `type[Something]` annotation requires the given type to be a concrete class in Mypy and Pyright, which is often useful, since it means that the class can be safely instantiated within the code using the annotated object. However there are cases where a class does not need to be instantiated in order to operate on it. Here is a (somewhat contrived) example: ``` from abc import abstractmethod class Abstract: @abstractmethod def foo(self): pass class Concrete(Abstract): def foo(self): print("bar") def my_special_isinstance(o: object, t: type[Abstract]) -> bool: return isinstance(o, t) my_special_isinstance(Concrete(), Abstract) # Only concrete class can be given where "Type[Abstract]" is expected [type-abstract] ``` Note that in reality, my code that reproduces this would be far more complex (rather than just being a wrapper around isinstance). Some other examples can be found in this issue . It’s clear that unless type safety is reduced for other cases, this case will never satisfy mypy, and as such I believe the best solution is to implement a separate type annotation for this case, tentatively named `AbstractType` (although `Interface` was also suggested). This could come with a few benefits: * Functions using it as an annotation could be prevented from attempting to instantiate the abstract class * The type annotation could give users of libraries that utilise it a bit more information on how the function behaves (ie knowing the class isn’t instantiated, or knowing not to instantiate a class returned by it) * It would be possible to make the above example type-safe with relatively trivial edits (changing `type[Abstract]` to `AbstractType[Abstract]`) As far as I can tell, this is the best compromise to allow for code such as the above to be type-safe without reducing type safety for other code (thereby ensuring backward compatibility). What does everyone think? I'd love to hear other ideas, or criticism and feedback on this one. If the idea seems reasonable, I'm also happy to start drafting a PEP for it!

8 14

Opened and changed typing issues week 2022-W50
by Typing Bot Dec. 19, 2022

Dec. 19, 2022

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were opened last week: #1311 Suggestion: Allow free type variables in type variable bounds that permeate beyond opened by @schuelermine https://github.com/python/typing/issues/1311 --------------------------------------------------- The following issues and pull requests were updated last week: #307 Allow specifying a default for omitted type parameters opened by @refi64 https://github.com/python/typing/issues/307 #548 Higher-Kinded TypeVars opened by @tek https://github.com/python/typing/issues/548 #912 allow `...` in place of generic parameters opened by @DetachHead https://github.com/python/typing/issues/912 #1216 TypeVarTuple Transformations Before Unpack opened by @llchan https://github.com/python/typing/issues/1216 #1250 Generic specialization? opened by @vnmabus https://github.com/python/typing/issues/1250 #1304 range typing opened by @Godwhitelight https://github.com/python/typing/issues/1304 #1309 Annotations for Type factories opened by @vors https://github.com/python/typing/issues/1309 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Re: Type-only imports
by Raphael Krupinski Dec. 18, 2022

Dec. 18, 2022

Is the current pattern for handling optional dependencies not sufficient? ``` try: import numpy as np MyType: TypeAlias = list[int] | np.ndarray[np.int32] except: MyType: TypeAlias = list[int] def mean(x: MyType) -> float: if isinstance(x, list): return sum(x) / len(x) else: return float(x.sum() / x.shape[0]) ``` -- Pozdrawiam Rafał Krupiński

2 2

Type-only imports
by Sebastian Rittau Dec. 17, 2022

Dec. 17, 2022

Since this comes up from time to time: What are everybody's thoughts on easier type-only imports? For example, instead of writing: ``` from typing import TYPE_CHECKING: if TYPE_CHECKING: from foo import Bar ``` Use something like this: ``` from foo type import Bar ``` This has a much more concise and more obvious syntax and could be useful for 1. Performance, since those imports can just be ignored at runtime. 2. Cases, where a type is only available in stubs and not at runtime. - Sebastian

6 5

A proposal for shape-safe array programming in Python with a Range type
by Thomas Kehrenberg Dec. 16, 2022

Dec. 16, 2022

Shape mismatch in arrays is a problem with libraries like numpy and pytorch because you can only reliably discover them at runtime. PEP 646 was a step towards solving this problem, but it left the actual "shape arithmetic" for future work: https://peps.python.org/pep-0646/#shape-arithmetic Recently, Google published a new programming language, dex, which has a new approach to the shape problem, that I find quite elegant. And I think their approach could be adapted to the Python type system without much problem. I think three new types would be needed to make it work. All of them need special treatment from type checkers. The first one is `Range[]`. We will use this type to represent the index of an array. It's a generic type, but it may only be specialized with non-negative `int` literals: `Range[3]`, `Range[10]`, etc. But no `Range[-2]` or `Range[n]` where `n` is a variable. (Maybe constants could also be allowed.) The point is that the type checker has to know the number passed to `Range[]`, otherwise the whole approach doesn't work. It is possible to create `Range[]` objects with ``` index: Range[3] = range(3) ``` The numbers must be the same. The other two types we need are `Chain` and `Product`, modeled after `itertools.chain` and `itertools.product`. Both need to be variadically generic with an arbitrary number of `Iterable` types: `Chain[*Iterables]`, `Product[*Iterables]`. Now to the special treatment that type checkers have to give these types. Type checkers must consider to following two types equal: ``` Chain[Range[3], Range[4]] == Range[7] ``` In general: ``` Chain[Range[<n>], Range[<m>]] == Range[<n+m>] ``` I wrote `<n>` here to make clear that these are meta variables. At no point should actual variables appear in `Range[]` (except, potentially, statically-known constants, as mentioned). The intuition behind the notation is that `itertools.chain(range(3), range(4))` has 7 elements. We also have a special behavior for `Product`: ``` Product[Range[<n>], Range[<m>]] == Range[<n*m>] ``` So, for example ``` Product[Range[3], Range[4]] == Range[12] ``` With this setup, we can now define shape-safe container types and functions. First, a one-dimensional array: ``` DType = TypeVar("DType") Index = TypeVar("Index", bound=Iterable[int]) class Vector(Generic[DType, Index]): ... ``` So, `Vector[float, Range[3]]` is a floating-point array of length 3. We can, for example, write a concatenation function for this: ``` I = TypeVar("I", bound=Iterable[int]) J = TypeVar("J", bound=Iterable[int]) def concat( x: Vector[float, I], y: Vector[float, J] ) -> Vector[float, Chain[I, J]]: ... ``` Thus: ``` a: Vector[float, Range[3]] = Vector([0.0, 1.0, 3.5], range(3)) b: Vector[float, Range[6]] = concat(a, a) ``` because of the special interaction of `Range[]` and `Chain`. The `concat` function defined this way is generic in the shape of the vector, but we can still statically determine the shape when we pass in a concrete vector. Note specifically, that we didn't need to mention `Range` at all in the function signature. In theory, the vectors can have other index types, as long as they're chainable. In this example, I explicitly passed the index `range(3)` to the constructor of `Vector`, to make sure the type checker can infer the right index type. Alternatively, type checkers could recognize that the given list has 3 elements and infer that the index type must be `Range[3]`. We can also define a two-dimensional array: ``` class Matrix(Generic[DType, I, J]): ... ``` Such that `Matrix[float, 3, 4]` corresponds to a 3x4 matrix. We can then write a flatten function like this: ``` def flatten_matrix( x: Matrix[float, I, J] ) -> Vector[float, Product[I, J]]: ... ``` Instead of defining a separate type for each array dimension, we can of course also define one for arbitrary dimensions, thanks to PEP 646: ``` Indexes = TypeVarTuple("Indexes") class Tensor(Generic[DType, *Indexes]): ... ``` Then ``` def flatten( x: Tensor[float, *Indexes] ) -> Tensor[float, Product[*Indexes]]: ... ``` More examples: ``` def duplicate_elements( x: Vector[int, I] ) -> Vector[int, Product[Range[2], I]]: ... def remove_2_entries( x: Vector[float, Chain[I, Range[2]]] ) -> Vector[float, I]: ... # if the result would have negative length, it should be a type error def split_in_2( x: Vector[float, Product[I, Range[2]]] ) -> tuple[Vector[float, I], Vector[float, I]]: ... # if the input length can’t be divided by 2, it should be a type error # we can also do this for arbitrary `n` # but we have to pass in an index instead of an `int` def split( x: Vector[float, Product[I, J]], n: J ) -> list[Vector[float, I]]: ... split(Vector([1., 2., 0. 4.], range(4)), n=range(2)) ``` Note that if a shape is not known at "compile-time", then the type checker can never compute the concrete shapes, but it can still catch mistakes! For example, in machine learning, it's common that the batch size is a commandline parameter. Then we can write ``` BatchIndex = NewType("BatchIndex", Iterable) def process_mnist( x: Tensor[float, BatchIndex, Range[28], Range[28], Range[3]] ) -> float: ... # somewhere: batch_index = BatchIndex(range(int(argv[1]))) ``` And the type checker will still notice if the first index is not `BatchIndex`. And `Chain` and `Product` will still work on it. And that's the core of the proposal. The following are thoughts about some of the details. One thing that seems natural is to identify `Chain` and `Product` with `itertools.chain` and `itertools.product` from the standard library. However, they don't quite have the right behavior when applied to `range()`: `itertools.product(range(2), range(4))` is *not* equal to `range(8)` and `itertools.chain(range(2), range(4))` is not equal to `range(6)`. Maybe one could introduce specialized versions of these, which are special-cased for `range()`. Then the following would work as expected: ``` v = Vector([0, -1, 2, 8, -2, 11], range(6)) for i in Product(range(2), range(3)): print(v[i]) ``` Speaking of iterating over a `Range[]` object, the elements of `Range[<n>]` should remember that they're limited to [0...n-1]: ``` for i in range(10): index: RangedIndex[10] = i ``` This would allow for out-of-bounds checking when accessing arrays. Finally, third-party libraries should probably be able to create their own `Range[]` types. For example, in numpy ``` a: Range[10] = np.arange(10) ``` Kind regards, Thomas

3 3

Opened and changed typing issues week 2022-W49
by Typing Bot Dec. 12, 2022

Dec. 12, 2022

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were opened last week: #1304 range typing opened by @Godwhitelight https://github.com/python/typing/issues/1304 #1308 More Protocols opened by @Eclips4 https://github.com/python/typing/issues/1308 #1309 Annotations for Type factories opened by @vors https://github.com/python/typing/issues/1309 --------------------------------------------------- The following issues and pull requests were updated last week: #307 Allow specifying a default for omitted type parameters opened by @refi64 https://github.com/python/typing/issues/307 #548 Higher-Kinded TypeVars opened by @tek https://github.com/python/typing/issues/548 #1273 Type lookup table for generics opened by @jonathanslenders https://github.com/python/typing/issues/1273 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Variance for protocols is awkward!
by Kevin Millikin Dec. 9, 2022

Dec. 9, 2022

I've been playing with generic Protocols a lot lately, and I'm finding TypeVar variance to be really bad from a usability standpoint. PEP-544 required them to be exactly correct ( https://peps.python.org/pep-0544/#overriding-inferred-variance-of-protocol-… ). In practice this means that if you add a method to a generic protocol class, you might have to rename the type parameters (because variance is essentially encoded in the name). And the same if you remove a method, or split a Protocol class into simpler sub-protocols. I had to have invariant, covariant, and contravariant versions of my `StateT` type parameter and I constantly had to think about which one to use. When I'm using structural subtyping, I *never* want to think about variance :) Could we change it? I found the discussion in the PEP, linked above, unconvincing. Perhaps I'm not understanding this section of the PEP. 1. It doesn't break transitivity of subtyping. The example has: ```python T = TypeVar('T') class P(Protocol[T]): # Protocol is declared as invariant. def meth(self) -> T: ... class C: def meth(self) -> float: ... class D(C): def meth(self) -> int: ... ``` And the PEP just asserts that D is not a structural subtype of P[float]. But I think it is! The PEP doesn't actually say "structural subtype", but subtyping of a protocol is always structural, so it's implied. The sentence from the PEP should be read as: "Now we have that D is a nominal subtype of C, and C is a structural subtype of P[float]. But D is not (?) a structural subtype of P[float] since D is a structural subtype of P[int], and P[int] is not a nominal subtype of P[float] because P is invariant." Isn't this reasoning wrong? I would say that programmers might just find it weird that P[int] is a structural subtype of P[float] but not a nominal subtype. But that's OK, there are lots of things that are structural subtypes of P[float] and not nominal subtypes. That's the whole point of structural subtyping. The PEP continues with: "There is a possibility to “cure” this by looking for protocol implementations in MROs but this will be too complex in a general case, and this “cure” requires abandoning simple idea of purely structural subtyping for protocols." I would say that the actual cure is to use (not abandon!) the simple idea of purely structural subtyping. If you want to know if D is a subtype of P[float], you would just check it directly (structurally). Nothing to do with MRO (well, you need to look in D's MRO to find the signatures of its methods, obviously). 2. Requires type inference. I'm not sure what this means. No type inference is necessary. You don't have to guess a supertype P[X] of D and check if it's a nominal subtype of P[float]. What might be weird is if you were trying to figure out how to instantiate a generic function taking P[T] and you wanted to find the type T that gives the least P[T] (wrt. subtyping). But here, I would just say that you don't want to do that wrt. nominal subtyping because P is a protocol. So it would be wrt. structural subtyping. 3. Prevents detailed error messages. Not sure what this means. Maybe there was a specific example? 4. Explicit is better than implicit. I would claim that what the PEP has is *implicit* in the sense that variance is determined by the uses of the type parameter. The only thing explicit is that I have to explicitly think about it. It's not explicit in the sense that I have any control over it. ==== Am I the only one that finds this awkward? Can we just relax the requirements and allow invariant type parameters that happen to be only used co- or contra-variantly? (It's still an error to use a covariant type parameter in a contravariant position and vice versa). -- [image: logo] Kevin Millikin Software Engineer kmillikin(a)deepmind.com If you received this email in error, please do not forward it to anyone else (it may contain confidential or privileged information), erase all copies and attachments, and let me know that it has gone to the wrong person.

2 2

Could we add frozen_default to "PEP 681 – Data Class Transforms"?
by Neil Girdhar Dec. 6, 2022

Dec. 6, 2022

The parameters provide various defaults: https://peps.python.org/pep-0681/#dataclass-transform-parameters However, frozen dataclasses are very popular when working with Jax (flax.struct.dataclass, and tjax.dataclass). Could we add `frozen_default` analogous to the other defaulted parameters?

6 7

Opened and changed typing issues week 2022-W48
by Typing Bot Dec. 5, 2022

Dec. 5, 2022

No issues or pull requests with the label 'topic: feature' were opened or updated last week in the typing repository on GitHub. All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0