Thanks for the review and thoughtful questions, Eric. We had been curious about opinions from Mypy and Pyright in terms of implementation complexity.
I understand the motivation behind adding Map to this spec, but I don't
think it's a very strong motivation. I would _strongly_ advocate for its removal. I think it unnecessarily complicates an already complex PEP. It effectively adds higher-kinded type support but in a limited and half-thought-out manner. To support Map, type checkers will need to do most of the work required to support higher-kinded types.
That's a lot of work with relatively little benefit. I'd rather save this
functionality for a future PEP that introduces higher-kinded types in a proper and holistic manner.
You're right, the PEP special-cases `Map` instead of properly supporting higher-order higher-kinded types.
The rest of the PEP's features are useful for Tensor functions, whereas `Map` is useful mainly for variadic functions like `map`, `zip`, and `asyncio.gather`. These use cases seem quite orthogonal.
The section "Concatenating Multiple Type Variable Tuples" clarifies some
of my questions above but provides an answer I was hoping not to hear. Supporting multiple TypeVariableTuples adds _significant_ complexity, and in most use cases I can think of will result in ambiguities (and therefore errors). The spec provides some examples of where these ambiguities can be resolved, but my preference is to disallow the use of multiple type variable tuples in all cases. This is another case of where the added complexity doesn't seem to be merited given the limited usage.
I agree this does add significant complexity. The main usage is in Tensor functions, which require such prefix removal. Concatenation of multiple variadics will become important going forward with type arithmetic. For example, Alfonso had collected quite a few functions where, in the future, we would need to match, strip, or add a prefix. For example, `def mean(t: Tensor[*Ts, T, *Rs], axis: Length[Ts]) -> Tensor[*Ts, *Rs]`: https://docs.google.com/document/d/1IlByrIjZPPxTa_1ZWHLZDaQrxdGrAbL71plzmXi0... .
(I'll address your individual questions separately after we've settled the below.)
Overall, I agree that the PEP is pretty complex to implement. However, features like concatenation of variadics are crucial for typing common tensor functions (which is the underlying motivation behind the PEP).
Would it be better to split this into smaller PEPs?
1. Variadic tuples with no `Map` and no concatenation of variadics 2. Concatenation of variadic tuples 3. `Map` and other higher-kinded types
This would let us incrementally evaluate the cost-benefit ratio. We wouldn't have to close the door on key features because they are complex to implement in one go. And the subsequent features are backward-compatible, so that shouldn't be a problem.
(1) will cover basic Tensor dimension-checking. (2) will be useful for more advanced Tensor functions that require removing a prefix or a particular dimension. (3) will be useful mainly for variadic functions that transform `*args`.
Guido, Matthew: thoughts?
On Thu, Jan 21, 2021 at 10:09 PM Eric Traut firstname.lastname@example.org wrote:
I started to implement parts of the draft PEP 646 within pyright this evening. I figured this exercise might be helpful in informing our discussion and teasing out additional questions and issues. Here's what I've uncovered so far.
*Grammar Changes* The grammar will need to change to support star expressions (i.e. unpack operator) within type annotations and in subscripts. If unpack operators are permitted within a subscript, how will that be handled at runtime? For example, in the expression `x[A, *B]`, what value will be passed to the `__getitem__` method for instance `x`? Will the runtime effectively replace `*B` with `typing.Unpack[B]`?
Will star expressions be allowed in slice expressions? I presume no.
*Zero-length Variadics* It's legal to pass no arguments to a `*args` parameter. For example:
def foo(*args: Any): ... foo() # This is fine
*Unknown-length Variadics* Am I correct in assuming that it's not OK to pass zero arguments to a `*args` parameter that has a variadic TypeVar annotation?
def foo(*args: *T): ... foo() I presume this is an error?
Also, it's generally OK to pass an arbitrary-length list of arguments to an `*args` parameter using an unpack operator.
def foo(*args: Any): ... def bar(x: Tuple[int, ...]): foo(*x) # This is allowed
I presume that it should be an error pass an arbitrary-length list of arguments to an `*args` parameter if it has a variadic TypeVar annotation?
def foo(*args: *T): ... def bar(x: Tuple[int, ...], y: Iterable[int], z: Tuple[int]): foo(*x) # I presume this is an error? foo(*y) # I presume this is an error? foo(*z) # This is allowed
If my assumption is correct that this should be flagged as an error by a type checker, will it also be a runtime error? I'm guessing the answer is no, there's no way to distinguish such an error at runtime. If my assumption is incorrect and this permitted, does the variadic type variable effectively become "open-ended" (i.e. the dimensionality of the variadic becomes unknown)? If so, how does it work with concatenation? I think it's better to make this an error.
- Other Observations *
This won't be an easy PEP to implement in type checkers, even with the simplifications I've recommended. It's going to be a heavy lift, which means it's going to be a long time before all type checkers support it. Compared to other recent type-related PEPs like 604, 612, 613, and 647, this PEP will require significantly more work to implement and get everything right. That could significantly delay the time before it can be used in typeshed and other public stubs. This bolsters my conviction that we should embrace simplifications where possible.
After this exercise, I'm even more convinced that we should support only unpacked usage ("*T") and not support packed ("T") for variadic type variables — and that we should use the existing TypeVar rather than introducing TypeVarTuple. In the rare cases where a packed version is desired, it can be expressed as `Tuple[*T]`. For example:
T = TypeVar("T") def func(*args: *T) -> Tuple[*T]: ...
By requiring `*T` everywhere for a variadic type variable, we will reduce confusion for users and simplify the spec and implementation.
-- Eric Traut Contributor to pyright and pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- email@example.com To unsubscribe send an email to firstname.lastname@example.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: email@example.com