I agree that "variadic" is a term that casual Python coders may be unfamiliar with, but it's a pretty standard term used in other languages, as opposed to "covariant" and "contravariant", which I had never encountered prior to Python. I also don't think variadic type variables will be used by a typical Python coder. It's a pretty advanced feature. Most Python coders don't use simple (non-variadic) type variables today.
Based on your experiments in Pyright so far, how difficult would introducing the new grammar be?
Introducing the grammar change to allow the star operator within a subscript is easy, just a few dozen lines of new code. The difficult part is with all the error cases this introduces. The star operator is allowed only in the case of variadic type variables. All other uses of a star operator within a subscript are not allowed. A few of these cases can be detected and reported by the parser (e.g. when used in conjunction with slice expressions), but most require semantic information to detect, so the checks will need to be added in many places within the type checker — and presumably the runtime as well. When a new construct introduces many ways to produce new error conditions, my natural instinct is to look for a way to eliminate the possibility of those errors rather than trying to enumerate and plug each of them individually.
The star operator will also require changes beyond the parser and type checker. It will also require updates to completion suggestion and signature help logic.
This is all doable, but it adds significant work across many code bases and will result in many more bugs as we work out all of the kinks and edge cases. I'm not convinced that the readability benefits justify the added complexity. I think naming conventions could work fine here. After all, we've adopted naming conventions to designate covariant and contravariant type variables, and that seems to work fine.
I'm continuing to work on the implementation in pyright (currently on a private branch). Of course, none of this is set in stone — I'm just trying to inform the discussion. Once I get a critical mass of functionality working, I'll merge the changes and give you a chance to play with them in Pyright. I find that it helps to be able to write real code with real tooling when playing with new language constructs.
Here's what I have implemented so far: * Support for "variadic=True" in TypeVar constructor. * Support for a variadic TypeVar used at the end of a generic class declaration * Support for subscripts within type expressions that contain an arbitrary number of type arguments and matching of those type arguments to type parameters when the last type parameter is a variadic * Support for "()" (empty tuple) notation when used with variadic TypeVar * Support for "*args: Ts" matching * Support for zero-length matching
What I haven't done yet: * Reporting error for bound, variance, or constraints used in conjunction with variadic TypeVar * Reporting errors for situations where a variadic TypeVar is used in cases where it shouldn't be * Reporting errors for situations where a variadic TypeVar is not used in cases where it is needed * Detecting and reporting errors for variadic TypeVar when it's not at the end of a list of TypeVars in a generic class declaration * Detecting and reporting errors for multiple variadic TypeVars appearing in a generic class declaration * Support for Union[Ts] * Support for Tuple[Ts] * Support for Concatenate[x, y, Ts] * Variadics in generic type aliases * Support for open-ended (arbitrary-length) variadics * Tests for all of the above
I've run across a few additional questions:
1. PEP 484 indicates that if a type argument is omitted from a generic type, that type argument is assumed to be `Any`. What is the assumption with a variadic TypeVar? Should it default to `()` (empty tuple)? If we support open-ended tuples, then we could also opt for `(Any, ...)`.
2. What is the type of `def foo(*args: Ts) -> Union[Ts]` if foo is called with no arguments? In other words, what is the type of `Union[*()]`? Is it `Any`? Is this considered an error?
3. When the constraint solver is solving for a variadic type variable, does it need to solve for the individual elements of the tuple independently? Consider, for example, `def foo(a: Tuple[Ts], b: Tuple[Ts]) -> Tuple[Ts]`. Now, let's consider the expression `foo((3, "hi"), ("hi", 5.6))`? Would this be an error? Or would you expect that the constraint solver produce an answer of `Tuple[int | str, str | float]` (or `Tuple[object, object]`)? It's much easier to implement if we can treat this as an error, but I don't know if that satisfies the use cases you have in mind.
4. Along the lines of the previous question, consider the expression `foo((3, "hi"), ("hi", ))`. In this case, the lengths of the tuples don't match. If we don't support open-ended variadics, this needs to be an error. If we support open-ended variadics, we have the option of solving this as `Tuple[int | str, ...]` (or `Tuple[object, ...]`). Once again, it's easiest if we don't allow this and treat it as an error.
-- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.