I was thinking about Joren Hammudoglu's proposed pep adding `+T`/`-T` syntax for TypeVar variance and had a crazy idea: what if we had syntactic support for other kinds of TypeVar customization too? For bounds we could overload the <= operator: from typing import TypeVar, SupportsAbs T = TypeVar("T") def largest_in_absolute_value(*xs: T <= SupportsAbs[float]) -> T: return max(xs, key=abs) And for value restrictions we could use `in`: def concat(x: T in (str, bytes), y: T) -> T: return x + y (Examples taken from https://mypy.readthedocs.io/en/stable/generics.html) The first use of the TypeVar in the function definition would have the bound or constraints, and other uses would then follow the constraint set in the first use. Using in or <= on the same TypeVar more than once in a function definition is an error. The nice thing about this syntax is that it puts all the information about the function definition in one place. You no longer have to create a named (and usually awkwardly named) TypeVar for each possible bound. The syntax is reminiscent of that used in languages like Scala and TypeScript, although in those you would write something like `def largest_in_absolute_value<T <= SupportsAbs[float]>(*xs: T) -> T:`, which would require new syntax in Python. I'm curious if other people think this would be a useful enhancement to the language.
Personally, I don't find `<=` to be very readable for bounds. Would the `is` operator be better? Or perhaps the `in` operator with a single-element tuple? If there is going to be a push to add new syntax for TypeVar customizations, I'd rather see us first focus on addressing the a much bigger usability issue with type variables in Python today. Many Python users are confused by type variables because the allocation of a TypeVar is divorced from its use, and the scoping rules for a TypeVar are not obvious and difficult to understand even after reading PEP 484 and the mypy docs. The source of confusion doesn't exist in other language because type variables are defined in context where they are used, making their scopes obvious. Fixing this issue is way more important, IMO, than making it slightly easier to specify the variance of a TypeVar. I wouldn't want any of these proposed variance syntax changes to get in the way of fixing the larger issue. I remember this broader issue being raised in the typing-sig previously, but I don't recall if anyone proposed a viable solution. -Eric -- Eric Traut Contributor to pyright & pylance Microsoft
I second the point on confusion around TypeVar declarations and scoping rules. I would very much like Python to have dedicated syntax for type parameters. However, given the recent discussion on python-dev and the position of the Steering Council on typing-only syntax, I'm not sure this is realistic... On Mon, Oct 25, 2021 at 5:12 AM Eric Traut <eric@traut.com> wrote:
I think that's unclear. In the rejection of PEP637 they stated: "The Steering Council is not particularly convinced it is of significant benefit to the static type checking language, but even if it were, at this point we’re reluctant to add general Python syntax that only (or mostly) benefits the static typing language."
Yeah, I'd brought up better TypeVar syntax at the Typing Summit [1]. At this point, it's probably best to wait and see what the Steering Council decides about the syntax changes for PEP 646 and for the Callable syntax PEP before proposing new TypeVar syntax. [1]: Slides (starting from slide 18): https://drive.google.com/file/d/1x-qoDVY_OvLpIV1EwT7m3vm4HrgubHPG/view?usp=s... On Mon, Oct 25, 2021 at 6:00 AM <henbruas@gmail.com> wrote:
-- S Pradeep Kumar
I really like the proposed syntax! Not only does it read better, but it also makes it impossible to create a TypeVar with nonsensical combinations of features, e.g. bound= with value restrictions. Do you think we could go one step further and free users from the need to declare type vars? As with the +T/-T proposal, my concern is that unless we put together a deprecation timeline for the existing syntax, we would end up in a situation where users have to learn both variants and tooling developers have to do redundant work to support them. On Sun, Oct 24, 2021 at 2:33 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El dom, 24 oct 2021 a las 15:02, Sergei Lebedev (<sergei.a.lebedev@gmail.com>) escribió:
That's more difficult because it requires a language-level change so you don't just get a NameError. For example, we could introduce new syntax so that `def concat<T in (str, bytes)>(...)` is valid. There was a talk proposing that at a typing meetup (I think by Pradeep), but it's much harder to introduce new syntax than to just change the behavior of typing.py.
I'd be OK with deprecating the current behavior once the last version that doesn't have it is EOL. For this proposal (assuming it is implemented in 3.11), that could mean we deprecate bound= after 3.10 is dead. Hopefully users will just naturally gravitate to the new syntax, just like 3.10+ code will naturally use `X | Y` instead of `Union[X, Y]`.
I like it! I remember seeing the hypothetical `<:` operator used for describing subclass relations in the Python documentation and other languages, and `<=` looks pretty similar. But since `a <= b` it often implies `a < b or a == b`, you'd also expect `<` and `==` to be valid here. It also raises the question of whether `*xs: SupportsAbs[float] >= T` is valid, since that `>=` could also imply a lower type bound, which is a thing in e.g. Scala. So what about making it look like the subclass constructor syntax instead: `*xs: T(SupportsAbs[float])` ? For value restrictions, the `in` operator can be problematic at runtime, since it always returns a boolean. So perhaps something like `x: T(str) | T(bytes)` could be used for this instead?
Like Eric already mentioned, I think that if we are talking about making type variables more ergonomic, what would proved the biggest benefit is improving the way how type variables are declared. I don't have a strong opinion about the proposed syntax though.
This proposal got me thinking about TypeVar(..., A, B) vs TypeVar(..., bound=Union[A, B]). From what I can tell, the only difference is that the first one accepts an instance of class C(A, B), whereas the second one does not. This appears to be analogous to OR vs XOR. See https://mypy-play.net/?mypy=latest&python=3.10&gist=7f25463261d4775b37fcbdba04796e41 With this in mind, this syntax could unify typing constraints/restrictions and upper bounds by e.g. writing TypeVar('T', A, B) as T <= A | B , and TypeVar('T', bound=Union[A, B]) as T <= A ^ B. However, the latter is a bit confusing, since Union[A, B] can be written with an A | B. But mypy does not allow it for TypeVar(..., bound=A | B), which makes sense if you consider that `class C(A | B)` and `class C(A, B)` are not the same (neither is `class C(Union[A, B])`, but that's besides the point).
That's not actually the key difference between the two forms of TypeVar. The version with bound= can substitute any subclass of the given bound, while the version with two or more positional type arguments can substitute only *exactly* those types. Example: ``` from typing import TypeVar class C(str): ... AnyStr = TypeVar("AnyStr", str, bytes) def f(arg: AnyStr) -> AnyStr: ... reveal_type(f(C())) # Line 7 T = TypeVar("T", bound=str) def g(arg: T) -> T: ... reveal_type(g(C())) # Line 11 ``` Output: ``` main.py:7: note: Revealed type is "builtins.str*" main.py:11: note: Revealed type is "__main__.C*" ``` On Sat, Oct 30, 2021 at 5:29 PM Joren Hammudoglu <jhammudoglu@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Thanks for clearing that up. But this is a pretty confusing thing to me. For instance, consider this example ``` from typing import TypeVar class A: ... class B: ... class C(A): ... T = TypeVar("T", A, B) def f(x: T) -> T: return x x = C() reveal_type(x) reveal_type(f(x)) ``` Output: ``` main.py:13: note: Revealed type is "__main__.C" main.py:14: note: Revealed type is "__main__.A*" ``` Up to now, I always assumed that a type parameter simply binds to the type of its value. However, in this example, that would imply that `T != T`. This assumption is one that I think most people have, and as far as I know, this is the only exception to it.
Joren, yes this is an inconsistency. You're not alone in finding it confusing. IMO, the wording in PEP 484 doesn't make this behavior very clear. It's definitely not intuitive. However, the current behavior is well established at this point, and many type stubs and typed code bases rely on the behavior. I don't think it would be feasible to change it or remove it from the type system. I think the original justification for adding this behavior was to support use cases like this one: ``` from typing import TypeVar, Union _T1 = TypeVar("_T1", bytes, str) def add_space_constrained(val: _T1) -> _T1: if isinstance(val, str): return val + " " # No type error else: return val + b" " # No type error _T2 = TypeVar("_T2", bound=Union[bytes, str]) def add_one_bound(val: _T2) -> _T2: if isinstance(val, str): return val + " " # Type error else: return val + b" " # Type error ``` Eric Traut Contributor to Pylance & Pyright Microsoft
Personally, I don't find `<=` to be very readable for bounds. Would the `is` operator be better? Or perhaps the `in` operator with a single-element tuple? If there is going to be a push to add new syntax for TypeVar customizations, I'd rather see us first focus on addressing the a much bigger usability issue with type variables in Python today. Many Python users are confused by type variables because the allocation of a TypeVar is divorced from its use, and the scoping rules for a TypeVar are not obvious and difficult to understand even after reading PEP 484 and the mypy docs. The source of confusion doesn't exist in other language because type variables are defined in context where they are used, making their scopes obvious. Fixing this issue is way more important, IMO, than making it slightly easier to specify the variance of a TypeVar. I wouldn't want any of these proposed variance syntax changes to get in the way of fixing the larger issue. I remember this broader issue being raised in the typing-sig previously, but I don't recall if anyone proposed a viable solution. -Eric -- Eric Traut Contributor to pyright & pylance Microsoft
I second the point on confusion around TypeVar declarations and scoping rules. I would very much like Python to have dedicated syntax for type parameters. However, given the recent discussion on python-dev and the position of the Steering Council on typing-only syntax, I'm not sure this is realistic... On Mon, Oct 25, 2021 at 5:12 AM Eric Traut <eric@traut.com> wrote:
I think that's unclear. In the rejection of PEP637 they stated: "The Steering Council is not particularly convinced it is of significant benefit to the static type checking language, but even if it were, at this point we’re reluctant to add general Python syntax that only (or mostly) benefits the static typing language."
Yeah, I'd brought up better TypeVar syntax at the Typing Summit [1]. At this point, it's probably best to wait and see what the Steering Council decides about the syntax changes for PEP 646 and for the Callable syntax PEP before proposing new TypeVar syntax. [1]: Slides (starting from slide 18): https://drive.google.com/file/d/1x-qoDVY_OvLpIV1EwT7m3vm4HrgubHPG/view?usp=s... On Mon, Oct 25, 2021 at 6:00 AM <henbruas@gmail.com> wrote:
-- S Pradeep Kumar
I really like the proposed syntax! Not only does it read better, but it also makes it impossible to create a TypeVar with nonsensical combinations of features, e.g. bound= with value restrictions. Do you think we could go one step further and free users from the need to declare type vars? As with the +T/-T proposal, my concern is that unless we put together a deprecation timeline for the existing syntax, we would end up in a situation where users have to learn both variants and tooling developers have to do redundant work to support them. On Sun, Oct 24, 2021 at 2:33 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El dom, 24 oct 2021 a las 15:02, Sergei Lebedev (<sergei.a.lebedev@gmail.com>) escribió:
That's more difficult because it requires a language-level change so you don't just get a NameError. For example, we could introduce new syntax so that `def concat<T in (str, bytes)>(...)` is valid. There was a talk proposing that at a typing meetup (I think by Pradeep), but it's much harder to introduce new syntax than to just change the behavior of typing.py.
I'd be OK with deprecating the current behavior once the last version that doesn't have it is EOL. For this proposal (assuming it is implemented in 3.11), that could mean we deprecate bound= after 3.10 is dead. Hopefully users will just naturally gravitate to the new syntax, just like 3.10+ code will naturally use `X | Y` instead of `Union[X, Y]`.
I like it! I remember seeing the hypothetical `<:` operator used for describing subclass relations in the Python documentation and other languages, and `<=` looks pretty similar. But since `a <= b` it often implies `a < b or a == b`, you'd also expect `<` and `==` to be valid here. It also raises the question of whether `*xs: SupportsAbs[float] >= T` is valid, since that `>=` could also imply a lower type bound, which is a thing in e.g. Scala. So what about making it look like the subclass constructor syntax instead: `*xs: T(SupportsAbs[float])` ? For value restrictions, the `in` operator can be problematic at runtime, since it always returns a boolean. So perhaps something like `x: T(str) | T(bytes)` could be used for this instead?
Like Eric already mentioned, I think that if we are talking about making type variables more ergonomic, what would proved the biggest benefit is improving the way how type variables are declared. I don't have a strong opinion about the proposed syntax though.
This proposal got me thinking about TypeVar(..., A, B) vs TypeVar(..., bound=Union[A, B]). From what I can tell, the only difference is that the first one accepts an instance of class C(A, B), whereas the second one does not. This appears to be analogous to OR vs XOR. See https://mypy-play.net/?mypy=latest&python=3.10&gist=7f25463261d4775b37fcbdba04796e41 With this in mind, this syntax could unify typing constraints/restrictions and upper bounds by e.g. writing TypeVar('T', A, B) as T <= A | B , and TypeVar('T', bound=Union[A, B]) as T <= A ^ B. However, the latter is a bit confusing, since Union[A, B] can be written with an A | B. But mypy does not allow it for TypeVar(..., bound=A | B), which makes sense if you consider that `class C(A | B)` and `class C(A, B)` are not the same (neither is `class C(Union[A, B])`, but that's besides the point).
That's not actually the key difference between the two forms of TypeVar. The version with bound= can substitute any subclass of the given bound, while the version with two or more positional type arguments can substitute only *exactly* those types. Example: ``` from typing import TypeVar class C(str): ... AnyStr = TypeVar("AnyStr", str, bytes) def f(arg: AnyStr) -> AnyStr: ... reveal_type(f(C())) # Line 7 T = TypeVar("T", bound=str) def g(arg: T) -> T: ... reveal_type(g(C())) # Line 11 ``` Output: ``` main.py:7: note: Revealed type is "builtins.str*" main.py:11: note: Revealed type is "__main__.C*" ``` On Sat, Oct 30, 2021 at 5:29 PM Joren Hammudoglu <jhammudoglu@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Thanks for clearing that up. But this is a pretty confusing thing to me. For instance, consider this example ``` from typing import TypeVar class A: ... class B: ... class C(A): ... T = TypeVar("T", A, B) def f(x: T) -> T: return x x = C() reveal_type(x) reveal_type(f(x)) ``` Output: ``` main.py:13: note: Revealed type is "__main__.C" main.py:14: note: Revealed type is "__main__.A*" ``` Up to now, I always assumed that a type parameter simply binds to the type of its value. However, in this example, that would imply that `T != T`. This assumption is one that I think most people have, and as far as I know, this is the only exception to it.
Joren, yes this is an inconsistency. You're not alone in finding it confusing. IMO, the wording in PEP 484 doesn't make this behavior very clear. It's definitely not intuitive. However, the current behavior is well established at this point, and many type stubs and typed code bases rely on the behavior. I don't think it would be feasible to change it or remove it from the type system. I think the original justification for adding this behavior was to support use cases like this one: ``` from typing import TypeVar, Union _T1 = TypeVar("_T1", bytes, str) def add_space_constrained(val: _T1) -> _T1: if isinstance(val, str): return val + " " # No type error else: return val + b" " # No type error _T2 = TypeVar("_T2", bound=Union[bytes, str]) def add_one_bound(val: _T2) -> _T2: if isinstance(val, str): return val + " " # Type error else: return val + b" " # Type error ``` Eric Traut Contributor to Pylance & Pyright Microsoft
participants (9)
-
Alfonso L. Castaño -
Eric Traut -
Guido van Rossum -
henbruas＠gmail.com -
Jelle Zijlstra -
Joren Hammudoglu -
S Pradeep Kumar -
Sebastian Rittau -
Sergei Lebedev