Defaults for TypeVar-likes
I've been toying with the idea of writting a PEP for this long requested feature (https://github.com/python/typing/issues/307). I've got a preliminary draft of it with the motivation and a specification for the feature. In short features like ```py T = TypeVar("T", default=int) # This means that if no type is specified T = int @dataclass class Box(Generic[T]): value: T | None = None reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` would be supported as would ```py YieldT = TypeVar("YieldT") SendT = TypeVar("SendT", default=None) ReturnT = TypeVar("ReturnT", default=None) class Generator(Generic[YieldT, SendT, ReturnT]): ... Generator[int] == Generator[int, None] == Generator[int, None, None] ``` Feedback and any questions on this would be very appreciated. https://gist.github.com/Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7 Thanks, James H-B.
Thanks for looking at this! This comes up in typeshed every now and then and is something that could be quite nice for evolving generically typed APIs. I have a preference for having the defaults live close to the class. TypeVar scoping and binding is kind of confusing as is. Could you spell out what the backward compatibility concerns with `class Box(Generic[T], T=int)` are? It sucks that we don't have PEP 637 for this. Maybe we consider doing something more dramatic like `class Box(Generic(S, T, U=int))`? It might be worth spelling out the rules for generic subclasses of generic classes that both have defaulted TypeVars are. It'd also be good to write up some more real-world-code places where this would be useful. To what extent would we still want this if we had a commonly used `GenIter = Generator[T, None, None]` generic type alias in typing.py? On Fri, 4 Mar 2022 at 14:19, James H-B <gobot1234yt@gmail.com> wrote:
I've been toying with the idea of writting a PEP for this long requested feature (https://github.com/python/typing/issues/307).
I've got a preliminary draft of it with the motivation and a specification for the feature. In short features like ```py T = TypeVar("T", default=int) # This means that if no type is specified T = int
@dataclass class Box(Generic[T]): value: T | None = None
reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` would be supported as would ```py YieldT = TypeVar("YieldT") SendT = TypeVar("SendT", default=None) ReturnT = TypeVar("ReturnT", default=None)
class Generator(Generic[YieldT, SendT, ReturnT]): ...
Generator[int] == Generator[int, None] == Generator[int, None, None] ```
Feedback and any questions on this would be very appreciated. https://gist.github.com/Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7
Thanks, James H-B. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
I would be happy to make tensors in most numeric libraries default to float. Most tensors I come across in numpy/tensorflow are float/int tensors and given int < float making default be float would capture most usages. There are occasionally times where other tensors like string tensor are used, but I think it's reasonable to expect those to specify type variable. Similarly for tensorflow most layers take 1 tensor as input and return 1 tensor as output. But layer is free to take much more complex types so I still use Layer(Generic[InputT, OutputT]). In practice how I've handled this so far is not with aliases, but with subclasses. So I have, class LayerGeneric(Generic[InputT, OutputT]): ... class Layer(LayerGeneric[Tensor, Tensor]): ... Since alias does not work for isinstance/issubclass checks, but a subclass does. Those are the 2 biggest examples I've encountered where I would probably use this.
Hi, This is a great idea, thanks for taking the initiative. In django-stubs there is a type (QuerySet [1]) where we would like the second typevar parameter to default to the type of the first typevar parameter. Most users don’t care about providing the second type parameter, but it’s important for annotating the types correctly. Could this PEP be modified such that there would be a way to express this? E.g. by setting the default to the other TypeVar? Currently, as a workaround, we define the original class as a private _QuerySet class and then make an alias to the original: QuerySet = _QuerySet[_T, _T] This works, but causes other problems. Originally, instead of the alias trick, a mypy plugin would handle defaulting the second typevar parameter. [1] https://github.com/typeddjango/django-stubs/blob/eb6991b008e7afd2b191d19ca4c...
On 5 Mar 2022, at 00.41, Mehdi2277 <med2277@gmail.com> wrote:
I would be happy to make tensors in most numeric libraries default to float. Most tensors I come across in numpy/tensorflow are float/int tensors and given int < float making default be float would capture most usages. There are occasionally times where other tensors like string tensor are used, but I think it's reasonable to expect those to specify type variable.
Similarly for tensorflow most layers take 1 tensor as input and return 1 tensor as output. But layer is free to take much more complex types so I still use Layer(Generic[InputT, OutputT]). In practice how I've handled this so far is not with aliases, but with subclasses. So I have,
class LayerGeneric(Generic[InputT, OutputT]): ...
class Layer(LayerGeneric[Tensor, Tensor]): ...
Since alias does not work for isinstance/issubclass checks, but a subclass does.
Those are the 2 biggest examples I've encountered where I would probably use this. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: syastrov@gmail.com
Maybe I'm just misinformed but I can't see any reason to add support for this, _QuerySet should probably only have one type parameter if you can use that work around. Ontop of that, It would probably be very hard to support this, are there any other languages that you know of that have this feature?
I've just realised Rust actually supports this, I might mention this, I'll have to think about how hard it is to implement. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=9c7043e365f44f97f42a36e1cd62655d
I’m currently skiing and just following along between runs so I can’t offer links or a concrete example, but Hacklang/HHVM also supports defaults for generics based on other generics on a type. Jon
On Mar 5, 2022, at 12:22, James H-B <gobot1234yt@gmail.com> wrote:
I've just realised Rust actually supports this, I might mention this, I'll have to think about how hard it is to implement. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=9c7043e365f44f97f42a36e1cd62655d _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jon@jonjanzen.com
Thanks for writing this proposal. I do think this is a valuable addition to the type system, and I’m generally supportive of the idea as long as it remains simple and we can avoid feature creep. Here are some thoughts… *Default ordering* I'm not convinced that we need to enforce ordering here. In effect, every type variable already has an implicit default value of `Any`, so it is unnecessary for type checkers or the runtime to detect cases where a TypeVar with no explicit default comes after a TypeVar with an explicit default. This is just added complexity that limits flexibility of the feature. I recommend deleting this entire section. I noticed that there's a section under "rejected alternatives" that tries to justify the default ordering constraint. I don't buy the justification here because, as I mentioned above, there's really no such thing as a TypeVar without a default. There's just "explicit defaults" and "implicit defaults". That's a very different situation from parameters in a call signature. *Constrained TypeVars* You talk about the interaction between bound TypeVars and default, but you don’t specify the interaction between constrained TypeVars and default. For constrained TypeVars, I think the default needs to be the same type as one of the constraints. It would be an error if it were a subtype. ```python T1 = TypeVar("T1", int, str, default=int) # OK T2 = TypeVar("T2", int, str, default=float) # Error ``` *Use of generics in default* The PEP doesn't explicitly disallow the use of generics in the default type. I think this is implied, but it would be better if it was explicitly called out as invalid. ```python T1 = TypeVar("T1", default=T) # Error T2 = TypeVar("T2", default=list[T]) # Error ``` *Function defaults* I don't understand why a function parameter annotated with a TypeVar with a default must have a default argument value associated with it. Why are the two concepts (a default type argument type and a default argument value for a parameter) related? If this is required, what about more complicated parameter type annotations that involve a TypeVar with a default type argument, such as `T | None` or `List[T]`? What about cases where the TypeVar appears for multiple input parameters? ```python DefaultIntT = TypeVar("DefaultIntT", default=int) def bar(a: DefaultInt = 0, b: DefaultInt = 0) -> DefaultInt: ... bar(3.14) ``` Unless there's a strong justification for tying these two concepts together, I would recommend deleting this entire section and removing this constraint. I think it unnecessarily complicates the design and takes us down a rabbit hole of additional concerns that would need to be discussed and addressed in the PEP. Let's avoid that if possible. *TypeVarTuple* Can more complex forms of unpacked tuples be used in the default value? I think the answer is probably yes, but it would be good to include an example. ```python Ts = TypeVarTuple("Ts", default=(int, *tuple[float, ...], str)) ``` *TypeVarTuple "subscription"* This subsection could use some additional text to explain the problem and the proposed solution. I'm trying to infer the intent from the code sample. I think the intent here is to avoid an ambiguity when a TypeVarTuple is used with a "suffix" that includes a TypeVar with a default. There are two ways to avoid that ambiguity. One is to disallow the use of a TypeVar with a default in a suffix. The other way is to simply document how it will be interpreted in that case. Your sample here seems to imply that it works in a prefix position by requiring that a type argument be provided explicitly. Maybe that's the same rule that should apply for suffixes. I don't have a strong opinion on this one, but I prefer consistency between prefixes and suffixes unless there's a good reason for them to be inconsistent. *ParamSpec Defaults* Is `...` a legal default for ParamSpec? It probably should be, since that's the implicit default. I presume that Concatenate cannot be used in a ParamSpec default. It might be good to call that out explicitly. *Missing type argument detection and reporting* Pyright has a diagnostic rule called reportMissingTypeArgument that emits a diagnostic when a generic class is used in an annotation without providing the expected type arguments. I don't know if mypy, pyre or pytype have a similar feature. The introduction of default type argument values will necessarily affect that logic. I don't know if this is worth mentioning in the PEP. It's something I'll need to consider if/when I implement this feature in pyright. *Specifying a default of Any* I wonder if the PEP should express an opinion about an explicit default of `Any`. The implicit default is already `Any`, but there's an important distinction between the two, at least for pyright which distinguishes between explicit `Any` and implicit `Any`, which it refers to as `Unknown`. I would like to encourage people (especially library authors) _not_ to use an explicit `Any` default, because that will potentially mask type errors. *Alternative syntax* Shantanu suggested an alternative syntax that brings the default closer to the code. I'd recommend against doing that in this PEP. I think we all recognize that the current syntax for TypeVars in Python is problematic and a fix is needed, but let's not do it piecemeal. That's bound to create problems. I like that this proposal fits in nicely with the existing TypeVar infrastructure and syntax. If we deviate from that, getting the PEP accepted will be more difficult, and we're likely to complicate our efforts in the future to improve the overall syntax for TypeVars. *Specifying defaults of one TypeVar in terms of another TypeVar* @syastrov asked whether the default type argument for one TypeVar could depend on the type argument provided for a different TypeVar. I understand the use case, but I'd strongly push back on this requirement. Many programming languages have support for default type arguments, but I've never seen one that supports defaults that depend on other type argument values — and there's a good reason for this. It would greatly complicate the design. Perhaps there are other ways to solve the problem with django stubs, but this doesn't strike me as a viable solution. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft
*Default ordering*
I strongly disagree with changing this for a few reasons: - I think keeping this similar to function signatures is the best way to do this as it’s much easier to teach. You can just imagine it expanding the class e.g.: ```py class Foo(Generic[T, DefaultIntT]): ... ``` Would expand to (assuming the args would be unpacked) ```py class Foo: def __class_getitem__(cls, T, DefaultIntT = int) -> GenericAlias: ... ``` - Rust will fail to compile code like where a non-default follows a default https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=89533c099f99d1d721c3ea5b44461728, you might think this is part of Rust not allowing unknowns at runtime (it's actually because type parameters are positional) but this is also something that typescript doesn’t allow https://www.typescriptlang.org/play?#code/MYGwhgzhAEBiD28A8AVaBeaA7ArgWwCMBT... - This would allow something like dict[str] == dict[str, Any] which I think is problematic as this doesn’t currently work and I don’t think it should. The point about the default being Any vs Unknown is a good one but, I’m not sure how this would work at runtime.
*Constrained TypeVars*
Good idea done.
*Use of generics in default*
As I mentioned, this is something that Rust supports and TypeScript also supports (which I didn’t even realise when writing my reply) this so maybe it’s not that bad to implement? *Function defaults*
If this is required, what about more complicated parameter type annotations that involve a TypeVar with a default type argument, such as `T | None` or `List[T]`?
The signature default should have to be compatible with the annotation default: `T | None` => (_: T | None = T) or (_: T | None = None) `List[T]` => (_: List[T] = []) This is however, something that TypeScript and Rust don’t enforce (though Rust doesn’t have default parameters). I might reconsider this but I’d be interested to hear your additional concerns over this maybe that would sway me.
What about cases where the TypeVar appears for multiple input parameters?
Maybe it should just require one of the arguments to have a default. I don’t see a use case for this if there isn’t a default signature argument present, maybe I’m missing something though. *TypeVarTuple*
Can more complex forms of unpacked tuples be used in the default value? I think the answer is probably yes, but it would be good to include an example.
Good idea, done.
*TypeVarTuple "subscription"*
I think I just forgot the first Foo here, my bad Foo == Foo[()] == Foo[int] == Foo[int, *tuple[Any, ...]]
*ParamSpec Defaults*
Good idea, done.
*Specifying a default of Any*
Originally I wanted to add a typing.Unknown special form as the default for default but for fear of an already broad scope creep I chose not too, but, yeah I completely agree.
*Missing type argument detection and reporting*
I feel like this is something that type checkers can chose how they want to handle themselves, this also a continuation of the previous point.
I have a preference for having the defaults live close to the class. TypeVar scoping and binding is kind of confusing as is.
Completely agree on this point, it's really unfortunate that 637 was rejected. I've updated the draft to add a note about how some metaclasses/supertypes might already consume a parameter with the same name as the TypeVar.
It might be worth spelling out the rules for generic subclasses of generic classes that both have defaulted TypeVars are.
Good idea, done.
It'd also be good to write up some more real-world-code places where this would be useful.
I've added the both the projects from Mehdi2277 and also discord.py which is where the Context example is from.
To what extent would we still want this if we had a commonly used `GenIter = Generator[T, None, None]` generic type alias in typing.py?
Whilst Generator is a compelling use case for this it's certainly not the only place this comes up. Adding type aliases for all of these cases would be annoying and potentially error prone. As others have pointed out it means creating intermediary classes which makes code more confusing to read. Not only that there are also cases that currently can't be supported without defaults like Context where there would have to be two different exported classes for it to work (e.g. BotContext and Context[BotT]) where users could be confused as when they should use one over the other.
Do we have any common examples of Paramspec/TypeVarTuple defaults being desirable? I understand symmetry with TypeVar, but I'm unsure it's worth complexity unless it happens that implementing it for typevar gives it for free somehow. I think we'll have 90% of value of this pep from normal typevar defaults and we can do a follow up pep if people find needs for defaults for paramspec/typevartuple. If you need to ponder hard for good examples then I think it's worth leaving as follow up work My preference is that Paramspec/TypeVarTuple defaults are undefined behavior to make pep's scope smaller and still have most of the value. If a type checker needs to have more logic to handle those two it can just skip them.
Am 05.03.22 um 00:33 schrieb Shantanu Jain:
I have a preference for having the defaults live close to the class.
Ideally, generics would be initialized at the point where they are used, like in all other languages I'm familiar with. But I don't think we can do this is in a readable manner without new syntax, and that is unlikely to happen soon, considering the SC's stance on typing features. For the moment I think defining the default as part of the TypeVar declaration is the best option, so as not to split the definition of the type var across multiple places. - Sebastian
On Mon, Mar 7, 2022 at 3:11 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Ideally, generics would be initialized at the point where they are used, like in all other languages I'm familiar with. But I don't think we can do this is in a readable manner without new syntax, and that is unlikely to happen soon, considering the SC's stance on typing features.
I wouldn't be quite so pessimistic. The SC rejected PEP 677 at least in part because, being a type annotation, it has to be an expression, but it's a form of expression that's not useful in other contexts. If we're considering syntax to replace TypeVar(), like def foo<T>(arg1: T, arg2: T) -> list[T]: ... class C<T, S>: def __init__(self, arg1: T, arg2: S): self.attr1 = arg1 self.attr2 = attr2 ... that would not be general expression syntax. (It would be good to poll on python-dev what people would think of such a proposal though.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Am 07.03.22 um 17:46 schrieb Guido van Rossum:
On Mon, Mar 7, 2022 at 3:11 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Ideally, generics would be initialized at the point where they are used, like in all other languages I'm familiar with. But I don't think we can do this is in a readable manner without new syntax, and that is unlikely to happen soon, considering the SC's stance on typing features.
I wouldn't be quite so pessimistic. The SC rejected PEP 677 at least in part because, being a type annotation, it has to be an expression, but it's a form of expression that's not useful in other contexts.
If we're considering syntax to replace TypeVar(), like
def foo<T>(arg1: T, arg2: T) -> list[T]: ...
class C<T, S>: def __init__(self, arg1: T, arg2: S): self.attr1 = arg1 self.attr2 = attr2 ...
that would not be general expression syntax. (It would be good to poll on python-dev what people would think of such a proposal though.)
Not python-dev and not really relevant for this discussion, but I would really like to see "standard" generic syntax, as in your example, in Python. - Sebastian
I've been away for a bit but I'm working on this again re-reviews would be appreciated again.
For those who want more discussion on this, as the initial PR (https://peps.python.org/pep-0696) has been merged, so, I'm open to more questions. Now the main question is, should TypeVar defaults be able to default to other TypeVars? It is something that both Rust and TypeScript support although it requires a few changes to PEP 695. Anyone have anything else they wish to discuss?
Friendly ping here, I think if no one has any objections I'm going to leave this section (https://peps.python.org/pep-0696/#using-another-typevarlike-as-the-default) in.
I see the benefits of allowing other TypeVarLike as the default, but I suspect there are complexities and edge-case behaviors that are not contemplated or described in the PEP. A few comments about this section as it's currently written: * It says "they have to be of the same type". By "type", I presume that you mean TypeVar vs ParamSpec vs TypeVarTuple, right? The word "type" is a bit ambiguous here. * Is it allowed for the second type variable to be bound to some type that is parameterized by the first type variable? For example, T1 = TypeVar("T1"), T2 = TypeVar("T2", default=list[T1])? Or is this disallowed? * Can a default refer to a TypeVar scoped to an outer class or function? If not, why? * It says "must be used before in the signature of the class". This wording probably needs to be tightened up somewhat because the order in which TypeVars appear is not always the order in which they parameterize the class. The order can be explicitly overridden through the use of a Generic or Protocol. And if PEP 695 passes, it will always be explicit. * The section mentions only classes. Is it intended to work also with generic functions and type aliases? * There are two ways to specialize a generic type: explicitly (using a subscript expression) and through the use of a call expression, which uses the constraint solver to "solve" the type arguments. This section focuses on explicit specialization. Have you considered the impact on specialization through a call expression? Consider, for example, if T2 has a default type of T1. If the constraint solver doesn't produce a solution for T2, it presumably "adopts" the default type which is solved value of T1. What if the converse occurs (i.e. T1 has no solution but T2 does)? I'd feel more confident if someone were to prototype this feature in an existing type checker and work out all of the above edge cases and discover if there are any others. Is such an effort under way? -Eric
- Agreed the word type here is ambiguous but yes I did mean the same type as the TypeVarLike (TypeVar defaults have to be TypeVars, etc.). - Yes, `T1 = TypeVar("T1"), T2 = TypeVar("T2", default=list[T1])` should work. - Yes, TypeVar scoping should allow for that. - For this case, I was just concerned about having to insert placeholder nodes that would cause deadlocks finding TypeVar names. After looking into mypy it seems fine there, so I'd assume type checkers are fine with any order. - Yeah exactly right if T1 has a solution but T2 doesn't, the constraint solver should use the type of T1 as T2, I don't think the converse makes sense but I'm happy to be proven wrong on this. I'm currently working on the mypy implementation of this feature (https://github.com/Gobot1234/mypy/tree/TypeVar-defaults) and have most of the TypeVar features working on my local version of the branch I'm just working on squashing some bugs. I'll send a PR to python/peps to fix the wording issues as soon as I can. Thanks!
Cross posting https://github.com/python/typing/issues/1274 for more visibility any feedback on the topic of classes generic over a ParamSpec and the recommended way to parameterise them. TLDR; should the default for a ParamSpec be a tuple or a list and why?
participants (8)
-
Eric Traut -
Guido van Rossum -
James H-B -
Jon Janzen -
Mehdi2277 -
Sebastian Rittau -
Shantanu Jain -
syastrov＠gmail.com