I have read the proposed PEP about variadic generics (PEP 646) and I like it enough that I want to sponsor it and want to help getting it over the finish line (we have to get the Steering Council to understand enough of it that they'll delegate approval to me :-).
A good review starts by briefly summarizing the proposal being reviewed, so here's the proposal in my own words.
**Motivation A:** We want to create generic types that take an arbitrary number of type parameters, like Tuple. For example, Tensors where each dimension is a "type". There is a demonstration of this without variadics, but it requires defining types `Tensor1[T1]`, `Tensor2[T1, T2]`, etc.:
https://github.com/deepmind/tensor_annotations. We want just `Tensor[T1]`, `Tensor[T1, T2]`, etc., for any number of parameters.
**Motivation B:** The type of functions like map() and zip() cannot be expressed using the existing type system. The simplest example would be the type of
```
def foo(*args):
return args
a = foo(42, "abc") # Should have type Tuple[int, str]
```
**Proposal:** Introduce a new kind of type variable that can be instantiated with an arbitrary number of types, some new syntax, and a new type operator:
```
Ts = TypeVarTuple("Ts") # NEW
T = TypeVar("T")
def f(*args: *Ts) -> Tuple[*Ts]: ...
class C(Generic[*Ts]): ...
Callable[[*Ts], T]
Tuple[*Ts]
Map[SomeType, Ts] # SomeType is a generic of one parameter
```
In most cases the form `*Ts` may be preceded and/or followed by any number of non-variadic types, e.g., `Tuple[int, int, *Ts, str]`. In cases where it's unambiguous, multiple variadic type variables are also allowed, e.g., `Tuple[*Ts1, *Ts2]`. For older Python versions, `Expand[Ts]` would mean the same as `*Ts`.
So now let me go on with my (generally favorable) review. (I left many detailed editorial comments in the Google Doc -- I will not repeat those here.)
I like the proposal a lot, and I am glad that we now have (apparently) a working prototype in Pyre. This has been on our wish list since at least 2016 -- much early discussion happened in
https://github.com/python/typing/issues/193 and at various meetings at PyCon and at the Bay Area Typing Meetups (links in the PEP). The proposed syntax has cycled through endless variations, and I am fine with the current proposal, even though it is still slightly clunky. There's also
https://github.com/python/typing/issues/513, which is specifically about array types.
There are probably other motivating applications that the PEP doesn't mention, for example certain decorator types (I doubt that all of these are taken care by PEP 612, ParamSpec).
I wonder why the proposal left out `Union[*Ts]`. This would seem useful, e.g. to type this function:
```
def f(*args):
return random.choice(args)
```
which could be typed naturally as follows:
```
def f(*args: *Ts) -> Union[*Ts]:
return
random.choice(args)
```
I'm not sure that `Tensor[T1, T2, ...]` is the be-all and end-all of tensor types (e.g. where would you put the data type of numpy arrays?) but maybe that can be handled by just adding one non-variadic type variable (a complete example would be nice though). There are also proposals for integer generics, which deserve their own PEP (presumably aiming at Python 3.11).
Eric Traut proposed an extension that would allow defining variadic subtypes of Sequence which behave similar to Tuple (where `Tuple[int, int]` is a subtype of `Tuple[int, ...]` which is a subtype of `Sequence[int]`), but I'm not sure we would need that a lot -- we could always add that later.
The introduction of a prefix `*` operator requires new syntax in a few cases. While `Callable[[*Ts], T]` is already valid (the parser interprets this as sequence unpacking), `Tuple[*Ts]` is not, and neither is `def f(*a: *Ts)`. For `Tuple[*Ts]` we can piggy-back on PEP 637 (keyword indexing, which adds this as well), but for the `def` example we'll need to add something new specifically for this PEP. I think that's fine -- we can give it runtime semantics that turns `*Ts` into `(*Ts,)`, which is similar to the other places: at runtime it iterates over the argument, producing a tuple. In all cases we need to support `Expand[Ts]` as well for backwards compatibility with Python 3.9 and before.
The `Map[]` operator is, as I said, fairly clunky. In the past various other syntaxes have been proposed. In particular, @sixolet
proposed a syntax that would allow defining `zip()` as follows:
def zip(*args: Iterable[Ts]) -> Iterator[Tuple[Ts, ...]]: ...```
Compare this to what it would look using the current proposal:
```
def zip(*args: *Map[Iterable, Ts]) -> Iterator[Ts]): ...
# Note that Iterator[Ts] is the same as Iterator[Tuple[*Ts]]
```
Sixolet's syntax made the iteration over the elements of Ts implicit, which is slightly shorter, and doesn't require "higher-order type functions" (is there an official name for that?), but also slightly more cryptic, and created yet another use for the ellipsis: `Tuple[Ts, ...]` is not quite analogous to `Tuple[T, ...]`, since the latter is *homogeneous* while the former is still heterogeneous. The new notation uses an explicit `Map[]` operator, which is similar to the choice we made in PEP 612 for `Concatenate[]`. (Speaking of this choice, we could drop the `*` prefix and rely purely on `Expand[]`, but that feels unnecessarily verbose, and we'll get most of the needed syntax for free with PEP 637, assuming it's accepted.)
All in all my recommendation for this PEP is: clean up the text based on the GDoc feedback, add `Union[*Ts]`, and submit to the Steering Council.
--