Hi all,
Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work.
https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXsE... (Attached is also an HTML render of the RST for easier reading.)
This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you).
Thanks! Matthew
Has there been any thought given to a way to "collapse" a ListVariadic into a single
type? This would be useful, for example, in the declaration of the built-in
tuple class. It accepts a variadic TypeVar, but its base class
Sequence does not. The type argument for Sequence needs to be
the union of the types in the tuple's ListVariadic.
The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this.
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft Corp.
Greg - sorry for the slow reply. I agree the ListVariadic is a bit jargony. I'm currently leaning towards TypeVar(variadic=True), pending some discussion in the doc about whether this might cause problems for us down the line if we want to add more features to TypeVar.
Eric - to make sure I understand what you're saying, are you thinking of something like the following?
Ts = ListVariadic('Ts') TsUnion = Union[Ts]
class Tuple(Sequence[UnionTs]): ...
I'm wondering whether there are any use-cases for this apart from making definitions of builtins cleaner - did you have any in mind?
On Mon, 19 Oct 2020 at 18:42, Eric Traut eric@traut.com wrote:
Has there been any thought given to a way to
"collapse" a ListVariadic
into a single type? This would be useful, for example, in the declaration
of the built-in tuple class. It accepts a variadic TypeVar, but its base
class Sequence does not. The type argument for Sequence needs to
be the
union of the types in the tuple's ListVariadic.
The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this.
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft Corp.
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Another late reply. :-)
On Thu, Oct 22, 2020 at 2:17 AM Matthew Rahtz via Typing-sig typing-sig@python.org wrote:
Greg - sorry for the slow reply. I agree the ListVariadic is a bit jargony. I'm currently leaning towards TypeVar(variadic=True), pending some discussion in the doc about whether this might cause problems for us down the line if we want to add more features to TypeVar.
Eric - to make sure I understand what you're saying, are you thinking of something like the following?
Ts = ListVariadic('Ts') TsUnion = Union[Ts]
class Tuple(Sequence[UnionTs]): ...
I'm wondering whether there are any use-cases for this apart from making definitions of builtins cleaner - did you have any in mind?
Looks like this dropped off Eric's radar, but what I thought he meant was
more the following (using the PEP-585-compliant tuple instead of
Tuple).
The class tuple is currently defined like this:
class tuple(Sequence[T]):
...
but that really only corresponds to tuple[T, ...]. Other forms of tuples
must be special-cased in the type checker, since they cannot be expressed
in the type system directly. Nevertheless the above definition is
important, because suppose we have this:
def first(a: Sequence[T]) -> T:
return a[0]
def foo(a: Tuple[int, str]):
b = first(a)
# What is the type of b?
Here I'd hope that b has type Union[int, str] and not object
or Any
(mypy currently says object).
With variadics we may have the chance of fixing this, removing (most) support for it from the type checker. We could be writing something like
class tuple(Sequence[Union[*Ts]], Generic[*Ts]):
...
...and here I am actually not sure how to proceed. Maybe Eric has an idea? How would one spell the constructor?
--Guido
On Mon, 19 Oct 2020 at 18:42, Eric Traut eric@traut.com wrote:
Has there been any thought given to a way to
"collapse" a ListVariadic
into a single type? This would be useful, for example, in the declaration
of the built-in tuple class. It accepts a variadic TypeVar, but its base
class Sequence does not. The type argument for Sequence needs to
be the
union of the types in the tuple's ListVariadic.
The reason I ask is that the pyright code contains a bunch of special-case handling for tuples because they are currently the only type that allows variadic TypeVars. It would be great to replace all of this special-case logic with generalized support for variadic generics. The current proposal comes close to enabling this.
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft Corp.
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him **(why is my pronoun here?) http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
In your example above, you asked "What is the type of b?". I agree that it should be
Union[int, str]. That's consistent with pyright's current behavior.
Here are a few thoughts about how we could generalize variadic type variables to work with tuple.
We could extend variadic (list) type variables to accept an optional parameter called
arbitrary_len that indicates whether variadic type variable supports
homogenous, arbitrary-length forms. By default, variadic type variables wouldn't allow
this. We'd need to make it illegal to use type variables that support arbitrary lengths
with an unpack operator or Expand because the result would be undefined.
For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types.
- For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T.
- For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to this discussion.
This would allow us to define the tuple class and its constructor as
follows:
_T = TypeVar("_T")
_Ts = TypeVar("_Ts", list=True, arbitrary_len=True)
class tuple(Sequence[_Ts]):
@overload
def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T:
...
@overload
def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T:
...
Thoughts?
-- Eric Traut Contributor to pyright/pylance
Let me check whether I understand:
- tuple currently inherits from Sequence[T], where T is a single type.
- That's a problem, considering that tuple is variadic. To deal with types like Tuple[int, str], type checkers have to special-case the conversion from the e.g. two types specified to the type of the Sequence (e.g. with a Union).
- If we introduced variadic type variables, we could get rid of that special-casing - either because the conversion from multiple types to a single type is specified explicitly (as in Sequence[Union[*Ts]]), or because the conversion is an explicit part of the specification (as in Sequence[Ts]).
If this is right, my preference would be for making the conversion from a list of types to a single type explicit, as in Union[*Ts]. I think that preference is mostly based on the intuition that it's usually better to be explicit, but I'd be curious to hear arguments the other way.
Eric - to make sure we're on the same page about variadic type variables in general - I was a little confused about this:
We could extend variadic (list) type variables to accept an optional
parameter called arbitrary_len that
indicates whether a variadic type
variable supports homogenous, arbitrary-length forms. By default,
variadic type variables wouldn't allow this. We'd need to make it
illegal to use type variables that support arbitrary lengths with an unpack
operator or Expand because the result would be undefined.
I was thinking that variadic type variables would be arbitrary-length (and heterogeneous) by default. Is the point that unless we enforce some restrictions here, we can't properly define how to handle the conversion to a single type?
Wouldn't the following work, even without those restrictions?
Ts = TypeVar('Ts', list=True) # Arbitrary-length, homogeneous or heterogeneous
class tuple(Generic[Ts], Sequence[Union[Ts]]): ...
x: tuple[int] # Ts is bound to [int]; x is a Sequence[int] y: tuple[int, str] # Ts is bound to [int, str]; y is a Sequence[Union[int, str]] z: tuple[int, ...] # Ts is bound to [int, ...]; z is a Sequence[int]
Hmm, maybe this is equivalent to your proposal - except here, the "work" is done by the Union.
Oh, I think I see what you're saying. So variadic type variables would be arbitrary-length in that they can be bound to a fixed-size type list of any size, but they're not arbitrary length in that they can't be bound to an arbitrary-sized type list [int, ...]. Is that right?
OK, so it seems like the crux of the matter is whether [int, ...] is a valid thing for a variadic type variable to be bound to. Because in order to define tuple using a variadic type variable, [int, ...] would have to be a valid thing for a variadic type variable to be bound to.
Alright, I'll pause here to check whether I've understood so far :)
On Fri, 11 Dec 2020 at 00:24, Eric Traut eric@traut.com wrote:
In your example above, you asked "What is the type of
b?". I agree that it
should be Union[int, str]. That's consistent with pyright's current
behavior.
Here are a few thoughts about how we could generalize variadic type variables to work with tuple.
We could extend variadic (list) type variables to accept an optional
parameter called arbitrary_len that indicates whether variadic type
variable supports homogenous, arbitrary-length forms. By default, variadic
type variables wouldn't allow this. We'd need to make it illegal to use
type variables that support arbitrary lengths with an unpack operator or
Expand because the result would be undefined.
For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types.
- For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T.
- For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to this discussion.
This would allow us to define the tuple class and its constructor as
follows:
_T = TypeVar("_T")
_Ts = TypeVar("_Ts", list=True, arbitrary_len=True)
class tuple(Sequence[_Ts]):
@overload
def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T:
...
@overload
def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T:
...
Thoughts?
-- Eric Traut Contributor to pyright/pylance
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Let me look at this from the other end. Suppose we have a variadic class Tensor and some tensor types:
Ts = TypeTupleVar("Ts")
class Tensor(Generic[*Ts]): ...
class Width: pass
class Height: pass
class Dim: pass
A = Tensor[Width, Height]
B = Tensor[Height, Width]
C = Tensor[Dim, Width, Height]
This Tensor class is not a subclass of Sequence and has no homogeneous "fallback" type. A and B are incompatible, and the join of A and C is either their union or object. There is no concept of Tensor[T, ...] in this case.
If you wanted to use e.g. Tensor[Dim, ...] for some universal dimension, you could change the Tensor class to this:
class Tensor(Sequence[Union[*Ts]], Generic[*Ts]): ...
def f(seq: Sequence[T]) -> T: ...
t1: Tensor[Dim, Dim]
i1 = f(t1)
t2: Tensor[Width, Height]
i2 = f(t2)
In the call to f(t1), t1 is converted to its fallback class Sequence[Union[Dim, Dim]], i.e. Sequence[Dim], and the type of i1 is Dim. In the call to f(t2), we end up with i2 having type Union[Width, Height], or perhaps object (depending on how the checker implements join()).
Note that in the first example, there is no way to even express the type of Tensor[T, ...], whereas in the second example, it is expressed as Sequence[T].
From all this I also realize that there is one thing that remains special about tuples: the notation Tuple[T, ...] is not supported by any other type. So the specialness that Eric wants to convey using arbitrary_len=True is exactly that. IOW we could write this:
t3: Tensor[Dim, ...]
i3 = f(t3)
Here t3's type is converted from Tensor[Dim, ...] to Sequence[Dim] in the call, and the type of i3 is Dim.
Formulating the rule for the specification for arbitrary_len=True is relatively simple: if a class C inherits from Generic[*Ts] and Ts has arbitrary_len=True, then the class can be parameterized as C[T, ...], and if arbitrary_len=False that notation is not available. However, what is the fallback type? Why would Sequence be special?
With the Sequence[Union[Ts]] notation it is a bit murkier to write the rule for when C[T, ...] is allowed (maybe it's just "whenever there's another base class that uses Ts" ?), but the fallback type is precisely specified in the class definition., and we could have some other generic type (e.g. LinkedList[Union[Ts]] or possibly Tuple[Ts]). I like that aspect.
OT bikeshed: having typed it half a dozen times no, I can testify that arbitrary_len is a very awkward thing to type. And why not arbitrary_length? :-)
On Fri, Dec 11, 2020 at 11:45 AM Matthew Rahtz via Typing-sig typing-sig@python.org wrote:
Let me check whether I understand:
- tuple currently inherits from Sequence[T], where T is a single type.
- That's a problem, considering that tuple is variadic. To deal with types like Tuple[int, str], type checkers have to special-case the conversion from the e.g. two types specified to the type of the Sequence (e.g. with a Union).
- If we introduced variadic type variables, we could get rid of that special-casing - either because the conversion from multiple types to a single type is specified explicitly (as in Sequence[Union[*Ts]]), or because the conversion is an explicit part of the specification (as in Sequence[Ts]).
If this is right, my preference would be for making the conversion from a list of types to a single type explicit, as in Union[*Ts]. I think that preference is mostly based on the intuition that it's usually better to be explicit, but I'd be curious to hear arguments the other way.
Eric - to make sure we're on the same page about variadic type variables in general - I was a little confused about this:
We could extend variadic (list) type variables to accept an optional
parameter called arbitrary_len that
indicates whether a variadic type
variable supports homogenous, arbitrary-length forms. By default,
variadic type variables wouldn't allow this. We'd need to make it
illegal to use type variables that support arbitrary lengths with an unpack
operator or Expand because the result would be undefined.
I was thinking that variadic type variables would be arbitrary-length (and heterogeneous) by default. Is the point that unless we enforce some restrictions here, we can't properly define how to handle the conversion to a single type?
Wouldn't the following work, even without those restrictions?
Ts = TypeVar('Ts', list=True) # Arbitrary-length, homogeneous or heterogeneous
class tuple(Generic[Ts], Sequence[Union[Ts]]): ...
x: tuple[int] # Ts is bound to [int]; x is a Sequence[int] y: tuple[int, str] # Ts is bound to [int, str]; y is a Sequence[Union[int, str]] z: tuple[int, ...] # Ts is bound to [int, ...]; z is a Sequence[int]
Hmm, maybe this is equivalent to your proposal - except here, the "work" is done by the Union.
Oh, I think I see what you're saying. So variadic type variables would be arbitrary-length in that they can be bound to a fixed-size type list of any size, but they're not arbitrary length in that they can't be bound to an arbitrary-sized type list [int, ...]. Is that right?
OK, so it seems like the crux of the matter is whether [int, ...] is a valid thing for a variadic type variable to be bound to. Because in order to define tuple using a variadic type variable, [int, ...] would have to be a valid thing for a variadic type variable to be bound to.
Alright, I'll pause here to check whether I've understood so far :)
On Fri, 11 Dec 2020 at 00:24, Eric Traut eric@traut.com wrote:
In your example above, you asked "What is the
type of b?". I agree that
it should be Union[int, str]. That's consistent with pyright's current
behavior.
Here are a few thoughts about how we could generalize variadic type variables to work with tuple.
We could extend variadic (list) type variables to accept an optional
parameter called arbitrary_len that indicates whether variadic type
variable supports homogenous, arbitrary-length forms. By default, variadic
type variables wouldn't allow this. We'd need to make it illegal to use
type variables that support arbitrary lengths with an unpack operator or
Expand because the result would be undefined.
For type variables that support arbitrary length lists, we could define some simple rules for how they "collapse" to traditional (non-list) types.
- For a variadic TypeVar that is bound to a homogenous, arbitrary-length list, the type list [T, ...] collapses to a type of T.
- For a variadic TypeVar that is bound to a heterogenous, fixed-length list, the type list collapses to a union of the individual subtypes with literals stripped. For example, [Literal['a'], int] collapses to Union[str, int]. For an explanation of why it's important to strip literals in this case, refer to this discussion.
This would allow us to define the tuple class and its constructor as
follows:
_T = TypeVar("_T")
_Ts = TypeVar("_Ts", list=True, arbitrary_len=True)
class tuple(Sequence[_Ts]):
@overload
def __new__(cls: Type[_T], iterable: Tuple[_Ts] = ...) -> _T:
...
@overload
def __new__(cls: Type[_T], iterable: Iterable[_Ts] = ...) -> _T:
...
Thoughts?
-- Eric Traut Contributor to pyright/pylance
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him **(why is my pronoun here?) http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length".
It occurs to me that there's one other attribute that makes tuple special.
It accepts zero-length lists (i.e. tuple[()]). Would the proposed variadic
TypeVar mechanism also support this? For example, would Tensor[()] be
legal?
Perhaps tuple is just too much of an oddity for us to model using the
proposed variadic TypeVar mechanism. I figured it was at least worth exploring the idea,
so thanks for the discussion.
-Eric
It occurs to me that there's one other attribute that
makes tuple special. It accepts zero-length lists (i.e.
tuple[()]). Would the proposed variadic TypeVar mechanism also support this?
For example, would Tensor[()] be legal?
Yes, zero-length Tuples are valid as arguments for Ts. That is the case, for example, when we have def foo(args: Ts) -> Ts: ... and we call it with no arguments: foo(). Ts would then resolve to Tuple[()]. Likewise, Tensor[()] is also legal.
-- Pradeep
From: Eric Traut eric@traut.com Sent: Monday, December 14, 2020 9:38 AM To: typing-sig@python.org typing-sig@python.org Subject: [Typing-sig] Re: Variadic generics PEP draft
Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length".
It occurs to me that there's one other attribute that makes tuple special.
It accepts zero-length lists (i.e. tuple[()]). Would the proposed variadic
TypeVar mechanism also support this? For example, would Tensor[()] be
legal?
Perhaps tuple is just too much of an oddity for us to model using the
proposed variadic TypeVar mechanism. I figured it was at least worth exploring the idea,
so thanks for the discussion.
-Eric
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
Thanks for the comments, Guido! I'll be slowly working through them over the next few days.
For the Tuple discussion: overall, it sounds like this would be
complicated enough to sort out that it would be best left for a future PEP.
I don't think there's anything in the PEP as it stands that would prevent
us tacking on either of these two mechanisms later - as long as we make the
constructor for variadic type variables different to TypeVar, so there's
no problem potentially adding extra arguments to the constructor later on.
On Mon, 14 Dec 2020 at 19:38, Pradeep Kumar Srinivasan via Typing-sig typing-sig@python.org wrote:
It occurs to
me that there's one other attribute that makes tuple
special. It accepts zero-length lists (i.e. tuple[()]). Would the
proposed variadic TypeVar mechanism also support this? For example, would
Tensor[()] be legal?
Yes, zero-length Tuples are valid as arguments for Ts. That is the case, for example, when we have def foo(args: Ts) -> Ts: ... and we call it with no arguments: foo(). Ts would then resolve to Tuple[()]. Likewise, Tensor[()] is also legal.
-- Pradeep
From: Eric Traut eric@traut.com Sent: Monday, December 14, 2020 9:38 AM To: typing-sig@python.org typing-sig@python.org Subject: [Typing-sig] Re: Variadic generics PEP draft
Perhaps my choice of the term "arbitrary length" was confusing here. Alternatives might be "unknown length" or "unspecified length".
It occurs to me that there's one other attribute that makes tuple
special. It accepts zero-length lists (i.e. tuple[()]). Would the
proposed variadic TypeVar mechanism also support this? For example, would
Tensor[()] be legal?
Perhaps tuple is just too much of an oddity for us to model using the
proposed variadic TypeVar mechanism. I figured it was at least worth
exploring the idea, so thanks for the discussion.
-Eric
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
FWIW, I've finally read the draft and left a large number of comments. That said, I am not quite done digesting all aspects of the proposal, but I am finally serious about making time to do so. Stay tuned.
On Wed, Oct 7, 2020 at 5:48 AM Matthew Rahtz via Python-ideas python-ideas@python.org wrote:
Hi all,
Something that's on many of our wishlists is support for variadic generics. Here's a first draft of a PEP detailing how they might work.
https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXsE... (Attached is also an HTML render of the RST for easier reading.)
This is an early draft so there are likely many things we're still missing. Please do take a look and give us your feedback (in the doc directly or by email, whichever is easiest for you).
Thanks! Matthew
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SQVTQY... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him **(why is my pronoun here?) http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...
-
Eric Traut -
Guido van Rossum -
Matthew Rahtz -
Pradeep Kumar Srinivasan