Variadic generics PEP draft
Hi all, Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work. https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXsE... (Attached is also an HTML render of the RST for easier reading.) This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you). Thanks! Matthew
Has there been any thought given to a way to "collapse" a ListVariadic into a single type? This would be useful, for example, in the declaration of the built-in `tuple` class. It accepts a variadic TypeVar, but its base class `Sequence` does not. The type argument for `Sequence` needs to be the union of the types in the tuple's ListVariadic. The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this. -Eric -- Eric Traut Contributor to pyright & pylance Microsoft Corp.
Greg - sorry for the slow reply. I agree the ListVariadic is a bit jargony. I'm currently leaning towards TypeVar(variadic=True), pending some discussion in the doc about whether this might cause problems for us down the line if we want to add more features to TypeVar. Eric - to make sure I understand what you're saying, are you thinking of something like the following? Ts = ListVariadic('Ts') TsUnion = Union[Ts] class Tuple(Sequence[UnionTs]): ... I'm wondering whether there are any use-cases for this apart from making definitions of builtins cleaner - did you have any in mind? On Mon, 19 Oct 2020 at 18:42, Eric Traut <eric@traut.com> wrote:
Has there been any thought given to a way to "collapse" a ListVariadic into a single type? This would be useful, for example, in the declaration of the built-in `tuple` class. It accepts a variadic TypeVar, but its base class `Sequence` does not. The type argument for `Sequence` needs to be the union of the types in the tuple's ListVariadic.
The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this.
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Another late reply. :-) On Thu, Oct 22, 2020 at 2:17 AM Matthew Rahtz via Typing-sig < typing-sig@python.org> wrote:
Greg - sorry for the slow reply. I agree the ListVariadic is a bit jargony. I'm currently leaning towards TypeVar(variadic=True), pending some discussion in the doc about whether this might cause problems for us down the line if we want to add more features to TypeVar.
Eric - to make sure I understand what you're saying, are you thinking of something like the following?
Ts = ListVariadic('Ts') TsUnion = Union[Ts]
class Tuple(Sequence[UnionTs]): ...
I'm wondering whether there are any use-cases for this apart from making definitions of builtins cleaner - did you have any in mind?
Looks like this dropped off Eric's radar, but what I thought he meant was more the following (using the PEP-585-compliant `tuple` instead of `Tuple`). The class `tuple` is currently defined like this: ``` class tuple(Sequence[T]): ... ``` but that really only corresponds to `tuple[T, ...]`. Other forms of tuples must be special-cased in the type checker, since they cannot be expressed in the type system directly. Nevertheless the above definition is important, because suppose we have this: ``` def first(a: Sequence[T]) -> T: return a[0] def foo(a: Tuple[int, str]): b = first(a) # What is the type of b? ``` Here I'd hope that b has type `Union[int, str]` and not `object` or `Any` (mypy currently says `object`). With variadics we may have the chance of fixing this, removing (most) support for it from the type checker. We could be writing something like ``` class tuple(Sequence[Union[*Ts]], Generic[*Ts]): ... ``` ...and here I am actually not sure how to proceed. Maybe Eric has an idea? How would one spell the constructor? --Guido
On Mon, 19 Oct 2020 at 18:42, Eric Traut <eric@traut.com> wrote:
Has there been any thought given to a way to "collapse" a ListVariadic into a single type? This would be useful, for example, in the declaration of the built-in `tuple` class. It accepts a variadic TypeVar, but its base class `Sequence` does not. The type argument for `Sequence` needs to be the union of the types in the tuple's ListVariadic.
The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this.
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
In your example above, you asked "What is the type of b?". I agree that it should be `Union[int, str]`. That's consistent with pyright's current behavior. Here are a few thoughts about how we could generalize variadic type variables to work with tuple. We could extend variadic (list) type variables to accept an optional parameter called `arbitrary_len` that indicates whether variadic type variable supports homogenous, arbitrary-length forms. By default, variadic type variables wouldn't allow this. We'd need to make it illegal to use type variables that support arbitrary lengths with an unpack operator or `Expand` because the result would be undefined. For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types. 1. For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T. 2. For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to [this discussion](https://github.com/microsoft/pyright/issues/1249). This would allow us to define the `tuple` class and its constructor as follows: ```python _T = TypeVar("_T") _Ts = TypeVar("_Ts", list=True, arbitrary_len=True) class tuple(Sequence[_Ts]): @overload def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T: ... @overload def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T: ... ``` Thoughts? -- Eric Traut Contributor to pyright/pylance
Let me check whether I understand: 1. tuple currently inherits from Sequence[T], where T is a single type. 2. That's a problem, considering that tuple is variadic. To deal with types like Tuple[int, str], type checkers have to special-case the conversion from the e.g. two types specified to the type of the Sequence (e.g. with a Union). 3. If we introduced variadic type variables, we could get rid of that special-casing - either because the conversion from multiple types to a single type is specified explicitly (as in Sequence[Union[*Ts]]), or because the conversion is an explicit part of the specification (as in Sequence[Ts]). If this is right, my preference would be for making the conversion from a list of types to a single type explicit, as in Union[*Ts]. I think that preference is mostly based on the intuition that it's usually better to be explicit, but I'd be curious to hear arguments the other way. --- Eric - to make sure we're on the same page about variadic type variables in general - I was a little confused about this: We could extend variadic (list) type variables to accept an optional
parameter called `arbitrary_len` that indicates whether a variadic type variable supports homogenous, arbitrary-length forms. *By default, variadic type variables wouldn't allow this.* *We'd need to make it illegal to use type variables that support arbitrary lengths with an unpack operator or `Expand` because the result would be undefined.*
I was thinking that variadic type variables *would* be arbitrary-length (and heterogeneous) by default. Is the point that unless we enforce some restrictions here, we can't properly define how to handle the conversion to a single type? Wouldn't the following work, even without those restrictions? Ts = TypeVar('Ts', list=True) # Arbitrary-length, homogeneous or heterogeneous class tuple(Generic[*Ts], Sequence[Union[*Ts]]): ... x: tuple[int] # Ts is bound to [int]; x is a Sequence[int] y: tuple[int, str] # Ts is bound to [int, str]; y is a Sequence[Union[int, str]] z: tuple[int, ...] # Ts is bound to [int, ...]; z is a Sequence[int] Hmm, maybe this is equivalent to your proposal - except here, the "work" is done by the Union. Oh, I think I see what you're saying. So variadic type variables would be arbitrary-length in that they can be bound to a fixed-size type list of any size, but they're *not* arbitrary length in that they *can't* be bound to an arbitrary-sized type list [int, ...]. Is that right? OK, so it seems like the crux of the matter is whether [int, ...] is a valid thing for a variadic type variable to be bound to. Because in order to define tuple using a variadic type variable, [int, ...] *would* have to be a valid thing for a variadic type variable to be bound to. Alright, I'll pause here to check whether I've understood so far :) On Fri, 11 Dec 2020 at 00:24, Eric Traut <eric@traut.com> wrote:
In your example above, you asked "What is the type of b?". I agree that it should be `Union[int, str]`. That's consistent with pyright's current behavior.
Here are a few thoughts about how we could generalize variadic type variables to work with tuple.
We could extend variadic (list) type variables to accept an optional parameter called `arbitrary_len` that indicates whether variadic type variable supports homogenous, arbitrary-length forms. By default, variadic type variables wouldn't allow this. We'd need to make it illegal to use type variables that support arbitrary lengths with an unpack operator or `Expand` because the result would be undefined.
For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types. 1. For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T. 2. For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to [this discussion]( https://github.com/microsoft/pyright/issues/1249).
This would allow us to define the `tuple` class and its constructor as follows: ```python _T = TypeVar("_T") _Ts = TypeVar("_Ts", list=True, arbitrary_len=True)
class tuple(Sequence[_Ts]): @overload def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T: ...
@overload def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T: ... ```
Thoughts?
-- Eric Traut Contributor to pyright/pylance _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Let me look at this from the other end. Suppose we have a variadic class Tensor and some tensor types: ``` Ts = TypeTupleVar("Ts") class Tensor(Generic[*Ts]): ... class Width: pass class Height: pass class Dim: pass A = Tensor[Width, Height] B = Tensor[Height, Width] C = Tensor[Dim, Width, Height] ``` This Tensor class is not a subclass of Sequence and has no homogeneous "fallback" type. A and B are incompatible, and the join of A and C is either their union or object. There is no concept of Tensor[T, ...] in this case. If you wanted to use e.g. Tensor[Dim, ...] for some universal dimension, you could change the Tensor class to this: ``` class Tensor(Sequence[Union[*Ts]], Generic[*Ts]): ... def f(seq: Sequence[T]) -> T: ... t1: Tensor[Dim, Dim] i1 = f(t1) t2: Tensor[Width, Height] i2 = f(t2) ``` In the call to f(t1), t1 is converted to its fallback class Sequence[Union[Dim, Dim]], i.e. Sequence[Dim], and the type of i1 is Dim. In the call to f(t2), we end up with i2 having type Union[Width, Height], or perhaps object (depending on how the checker implements join()). Note that in the first example, there is no way to even express the type of Tensor[T, ...], whereas in the second example, it is expressed as Sequence[T]. From all this I also realize that there is one thing that remains special about tuples: the notation Tuple[T, ...] is not supported by any other type. So the specialness that Eric wants to convey using arbitrary_len=True is exactly that. IOW we could write this: ``` t3: Tensor[Dim, ...] i3 = f(t3) ``` Here t3's type is converted from Tensor[Dim, ...] to Sequence[Dim] in the call, and the type of i3 is Dim. Formulating the rule for the specification for arbitrary_len=True is relatively simple: if a class C inherits from Generic[*Ts] and Ts has arbitrary_len=True, then the class can be parameterized as C[T, ...], and if arbitrary_len=False that notation is not available. However, what is the fallback type? Why would Sequence be special? With the Sequence[Union[*Ts]] notation it is a bit murkier to write the rule for when C[T, ...] is allowed (maybe it's just "whenever there's another base class that uses *Ts" ?), but the fallback type is precisely specified in the class definition., and we could have some other generic type (e.g. LinkedList[Union[*Ts]] or possibly Tuple[*Ts]). I like that aspect. OT bikeshed: having typed it half a dozen times no, I can testify that arbitrary_len is a very awkward thing to type. And why not arbitrary_length? :-) On Fri, Dec 11, 2020 at 11:45 AM Matthew Rahtz via Typing-sig < typing-sig@python.org> wrote:
Let me check whether I understand:
1. tuple currently inherits from Sequence[T], where T is a single type. 2. That's a problem, considering that tuple is variadic. To deal with types like Tuple[int, str], type checkers have to special-case the conversion from the e.g. two types specified to the type of the Sequence (e.g. with a Union). 3. If we introduced variadic type variables, we could get rid of that special-casing - either because the conversion from multiple types to a single type is specified explicitly (as in Sequence[Union[*Ts]]), or because the conversion is an explicit part of the specification (as in Sequence[Ts]).
If this is right, my preference would be for making the conversion from a list of types to a single type explicit, as in Union[*Ts]. I think that preference is mostly based on the intuition that it's usually better to be explicit, but I'd be curious to hear arguments the other way.
---
Eric - to make sure we're on the same page about variadic type variables in general - I was a little confused about this:
We could extend variadic (list) type variables to accept an optional
parameter called `arbitrary_len` that indicates whether a variadic type variable supports homogenous, arbitrary-length forms. *By default, variadic type variables wouldn't allow this.* *We'd need to make it illegal to use type variables that support arbitrary lengths with an unpack operator or `Expand` because the result would be undefined.*
I was thinking that variadic type variables *would* be arbitrary-length (and heterogeneous) by default. Is the point that unless we enforce some restrictions here, we can't properly define how to handle the conversion to a single type?
Wouldn't the following work, even without those restrictions?
Ts = TypeVar('Ts', list=True) # Arbitrary-length, homogeneous or heterogeneous
class tuple(Generic[*Ts], Sequence[Union[*Ts]]): ...
x: tuple[int] # Ts is bound to [int]; x is a Sequence[int] y: tuple[int, str] # Ts is bound to [int, str]; y is a Sequence[Union[int, str]] z: tuple[int, ...] # Ts is bound to [int, ...]; z is a Sequence[int]
Hmm, maybe this is equivalent to your proposal - except here, the "work" is done by the Union.
Oh, I think I see what you're saying. So variadic type variables would be arbitrary-length in that they can be bound to a fixed-size type list of any size, but they're *not* arbitrary length in that they *can't* be bound to an arbitrary-sized type list [int, ...]. Is that right?
OK, so it seems like the crux of the matter is whether [int, ...] is a valid thing for a variadic type variable to be bound to. Because in order to define tuple using a variadic type variable, [int, ...] *would* have to be a valid thing for a variadic type variable to be bound to.
Alright, I'll pause here to check whether I've understood so far :)
On Fri, 11 Dec 2020 at 00:24, Eric Traut <eric@traut.com> wrote:
In your example above, you asked "What is the type of b?". I agree that it should be `Union[int, str]`. That's consistent with pyright's current behavior.
Here are a few thoughts about how we could generalize variadic type variables to work with tuple.
We could extend variadic (list) type variables to accept an optional parameter called `arbitrary_len` that indicates whether variadic type variable supports homogenous, arbitrary-length forms. By default, variadic type variables wouldn't allow this. We'd need to make it illegal to use type variables that support arbitrary lengths with an unpack operator or `Expand` because the result would be undefined.
For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types. 1. For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T. 2. For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to [this discussion]( https://github.com/microsoft/pyright/issues/1249).
This would allow us to define the `tuple` class and its constructor as follows: ```python _T = TypeVar("_T") _Ts = TypeVar("_Ts", list=True, arbitrary_len=True)
class tuple(Sequence[_Ts]): @overload def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T: ...
@overload def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T: ... ```
Thoughts?
-- Eric Traut Contributor to pyright/pylance _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length". It occurs to me that there's one other attribute that makes `tuple` special. It accepts zero-length lists (i.e. `tuple[()]`). Would the proposed variadic TypeVar mechanism also support this? For example, would `Tensor[()]` be legal? Perhaps `tuple` is just too much of an oddity for us to model using the proposed variadic TypeVar mechanism. I figured it was at least worth exploring the idea, so thanks for the discussion. -Eric
It occurs to me that there's one other attribute that makes `tuple` special. It accepts zero-length lists (i.e. `tuple[()]`). Would the proposed variadic TypeVar mechanism also support this? For example, would `Tensor[()]` be legal?
Yes, zero-length Tuples are valid as arguments for Ts. That is the case, for example, when we have def foo(*args: *Ts) -> Ts: ... and we call it with no arguments: foo(). Ts would then resolve to Tuple[()]. Likewise, Tensor[()] is also legal. -- Pradeep ________________________________ From: Eric Traut <eric@traut.com> Sent: Monday, December 14, 2020 9:38 AM To: typing-sig@python.org <typing-sig@python.org> Subject: [Typing-sig] Re: Variadic generics PEP draft Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length". It occurs to me that there's one other attribute that makes `tuple` special. It accepts zero-length lists (i.e. `tuple[()]`). Would the proposed variadic TypeVar mechanism also support this? For example, would `Tensor[()]` be legal? Perhaps `tuple` is just too much of an oddity for us to model using the proposed variadic TypeVar mechanism. I figured it was at least worth exploring the idea, so thanks for the discussion. -Eric _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
Thanks for the comments, Guido! I'll be slowly working through them over the next few days. For the `Tuple` discussion: overall, it sounds like this would be complicated enough to sort out that it would be best left for a future PEP. I don't think there's anything in the PEP as it stands that would prevent us tacking on either of these two mechanisms later - as long as we make the constructor for variadic type variables different to `TypeVar`, so there's no problem potentially adding extra arguments to the constructor later on. On Mon, 14 Dec 2020 at 19:38, Pradeep Kumar Srinivasan via Typing-sig < typing-sig@python.org> wrote:
It occurs to me that there's one other attribute that makes `tuple` special. It accepts zero-length lists (i.e. `tuple[()]`). Would the proposed variadic TypeVar mechanism also support this? For example, would `Tensor[()]` be legal?
Yes, zero-length Tuples are valid as arguments for Ts. That is the case, for example, when we have def foo(*args: *Ts) -> Ts: ... and we call it with no arguments: foo(). Ts would then resolve to Tuple[()]. Likewise, Tensor[()] is also legal.
-- Pradeep
------------------------------ *From:* Eric Traut <eric@traut.com> *Sent:* Monday, December 14, 2020 9:38 AM *To:* typing-sig@python.org <typing-sig@python.org> *Subject:* [Typing-sig] Re: Variadic generics PEP draft
Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length".
It occurs to me that there's one other attribute that makes `tuple` special. It accepts zero-length lists (i.e. `tuple[()]`). Would the proposed variadic TypeVar mechanism also support this? For example, would `Tensor[()]` be legal?
Perhaps `tuple` is just too much of an oddity for us to model using the proposed variadic TypeVar mechanism. I figured it was at least worth exploring the idea, so thanks for the discussion.
-Eric _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
FWIW, I've finally read the draft and left a large number of comments. That said, I am not quite done digesting all aspects of the proposal, but I am finally serious about making time to do so. Stay tuned. On Wed, Oct 7, 2020 at 5:48 AM Matthew Rahtz via Python-ideas < python-ideas@python.org> wrote:
Hi all,
Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work.
https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXsE... (Attached is also an HTML render of the RST for easier reading.)
This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you).
Thanks! Matthew _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SQVTQY... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (4)
-
Eric Traut -
Guido van Rossum -
Matthew Rahtz -
Pradeep Kumar Srinivasan