PEP 613 and type aliases in nested scope
Hello everybody :-) Linking https://www.python.org/dev/peps/pep-0613/#scope-restrictions PEP 613 appears to disallow explicit type aliases in nested scopes:
With explicit aliases, [...] the inner assignment can either be a valid local variable or a clear error [...]
It also maybe seem to not allow for the possibility of type aliases in nested scope at all:
[...] because type aliases cannot be defined inside a nested scope
If this was the intention of the PEP, this restriction feels unnecessary to me. Type aliases in nested scope are a valid use case. mypy previously never considered assignments in nested scopes to be type aliases, and we've had multiple issues concerning this. Eric mentioned that as a result of this wording, Pyright errors for explicit type aliases in nested scope, but allows implicit type aliases. I propose we loosen the wording in PEP 613. If type checkers don't wish to add support for type aliases in nested scope that seems fine, but I see no reason to prohibit this via PEP. The following code seems very clear to me in its intention (as a result of PEP 613!) and type checkers should be free to follow that intention: ``` import typing T = typing.NewType('T', int) class C: V: typing.TypeAlias = T x: C.V = C.V(5) ``` Conversely, if this proves controversial and we decide we do actually want to prohibit type aliases in nested scopes, we should probably amend PEP 484 to mention this. Thanks to Nipunn for bringing this up!
Thanks for linking the question and starting the discussion here! My understanding is that in PEP 484, type aliases were global variables that were assigned to types. Any locals assigned to a type would be treated as a value rather than part of the type hierarchy. The reason behind this is to ensure we have a consistent class hierarchy when performing type checking across a codebase. If more precise scoping is allowed, can type aliases be defined inline in a function (like the counter-examples in PEP 613)? Do we care if it’s used before it’s referenced, and if so, does that break assumptions if it’s in scope when it’s used as an annotation, but out of scope when that annotation needs to be compared to something else? I’d argue that we don’t ever want local variables to be allowable as type aliases to avoid scoping issues changing the type hierarchy at different points. That said, if the proposal instead is to allow type aliases strictly as toplevel class attributes, that seems doable because class attributes are essentially treated like qualified globally accessible values. However, we will have to explicitly prohibit type aliases that capture any scoped generics to avoid a situation where constraints need to be solved globally. For example, class Foo(Generic[T]): A: TypeAlias = T This a huge can of worms, and we lose the simplicity of the assertion (see PEP 484) that “type aliases can be as complex as any type annotation” if we allow aliases nested inside classes. Is it worth it? Finally, a slightly unrelated part of the discussion: * If type checkers don't wish to add support for type aliases in nested scope that seems fine, but I see no reason to prohibit this via PEP. The following code seems very clear to me in its intention (as a result of PEP 613!) and type checkers should be free to follow that intention: I’d make the case that the typing PEP should describe a consistent typing experience for the language and that type checkers should hold themselves to follow the system described by the PEP as closely as they can. The idea that type checkers should be free to do as they wish and PEPs should be more vague to allow for different handling I think makes for a much messier Python typing environment. IMO, we should take into account the opinions of type checker willingness to implement, but in the end still aim to compromise on a consistent decision across the board. Anyway, I am curious what others think on whether nested scope type aliases should be allowed and if so, what set of restrictions we need to put on it. Shannon From: Shantanu Jain <hauntsaninja@gmail.com> Date: Sunday, October 31, 2021 at 12:41 AM To: None via Typing-sig <typing-sig@python.org> Subject: [Typing-sig] PEP 613 and type aliases in nested scope Hello everybody :-) Linking https://www.python.org/dev/peps/pep-0613/#scope-restrictions<https://www.python.org/dev/peps/pep-0613/#scope-restrictions> PEP 613 appears to disallow explicit type aliases in nested scopes:
With explicit aliases, [...] the inner assignment can either be a valid local variable or a clear error [...]
It also maybe seem to not allow for the possibility of type aliases in nested scope at all:
[...] because type aliases cannot be defined inside a nested scope
If this was the intention of the PEP, this restriction feels unnecessary to me. Type aliases in nested scope are a valid use case. mypy previously never considered assignments in nested scopes to be type aliases, and we've had multiple issues concerning this. Eric mentioned that as a result of this wording, Pyright errors for explicit type aliases in nested scope, but allows implicit type aliases. I propose we loosen the wording in PEP 613. If type checkers don't wish to add support for type aliases in nested scope that seems fine, but I see no reason to prohibit this via PEP. The following code seems very clear to me in its intention (as a result of PEP 613!) and type checkers should be free to follow that intention: ``` import typing T = typing.NewType('T', int) class C: V: typing.TypeAlias = T x: C.V = C.V(5) ``` Conversely, if this proves controversial and we decide we do actually want to prohibit type aliases in nested scopes, we should probably amend PEP 484 to mention this. Thanks to Nipunn for bringing this up!
My understanding is that in PEP 484, type aliases were global variables that were assigned to types.
I don't see anywhere in PEP 484 that limits type aliases to the global (module) level. Pyright currently allows type aliases to be defined in classes and functions as well. Mypy apparently does as well. This type checks without error in both mypy and pyright: ```python def func1(val: int) -> None: Alias = list[int] x: Alias = [val] class Class1: Alias = list[int] def method1(self) -> Alias: return [2] ``` For generic type aliases defined within a nested scope, mypy does emit an error if the type variable is bound in that scope. ("Can't use bound type variable to define generic alias".) Pyright doesn't flag this an error but should. I'll file a bug to track this. ```python T = TypeVar("T") def func2(val: T): Alias = list[T] x: Alias = [val] class Class2(Generic[T]): Alias = list[T] def method1(self, x: T) -> Alias: return [x] ```
I’d make the case that the typing PEP should describe a consistent typing experience for the language
I also prefer that we define the desired behavior in the PEPs and strive for consistent behavior across type checkers. That would seem to be in the best interest of the Python community. I think there are several defensible positions to take, but let me propose the following: * Both implied type aliases (introduced in PEP 484) and explicit type aliases (introduced in PEP 613) are allowed within module, class and function scopes. * Type aliases that use bound type variables (those defined within a class or function scope) are forbidden and should generate an error. This would require us to 1) revise PEP 484 for clarity, 2) revise PEP 613 to loosen the requirements. I don't think it would require any changes to mypy. It would require a small bug fix in pyright, but I plan to make that change regardless. I haven't tested the behavior of the other type checkers (pyre, pytest, PyCharm), so I'm not sure what changes would be required in those cases. Thoughts? -- Eric Traut Contributor to Pyright & Pylance Microsoft
Hi friends. Thanks for the discussion! As it may be helpful, here is a motivating use case for nested scope aliases: backward compatible name changes to types. For example, in a library we might have ``` class A: class B_new: pass B_old: TypeAlias = B_new def get_B() -> B_new: # impl ``` Here, we renamed the inner class from `B_old` to `B_new` but would like to keep some backward compatibility for some time between old name and new name for customers of this library who may have preexisting code: ``` b: B_old = A().get_B() ```
I think there are several defensible positions to take, but let me propose the following: * Both implied type aliases (introduced in PEP 484) and explicit type aliases (introduced in PEP 613) are allowed within module, class and function scopes. * Type aliases that use bound type variables (those defined within a class or function scope) are forbidden and should generate an error. This would require us to 1) revise PEP 484 for clarity, 2) revise PEP 613 to loosen the requirements. I don't think it would require any changes to mypy. It would require a small bug fix in pyright, but I plan to make that change regardless. I haven't tested the behavior of the other type checkers (pyre, pytest, PyCharm), so I'm not sure what changes would be required in those cases. Thoughts?
This makes sense to me. There is one further place of ambiguity that should be clarified in the PEP ``` X = int class A: Y = int ``` Should X and Y be inferred as a type alias on `int` or `Type[int]`? Mypy has a whole section of documentation about this ambiguity - https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-a... - though it does not describe the explicit `TypeAlias` syntax or the behavior heuristic inside nested scope. Currently in mypy, `X` is inferred as a type alias and `A.Y` is inferred as a class variable of type `Type[int]`. Mypy chooses to treat `A.Y` as a variable (intentionally) as a heuristic inside nested scope. See this comment from a couple years back https://mail.python.org/archives/list/typing-sig@python.org/message/QNJEVUIH... In pyright, both `X` and `A.Y` are inferred as type aliases. I think without an explicit type of either `Y: TypeAlias = int` or `Y: Type[int] = int`, we should have a consistent experience specified by PEP613 on what `Y = int` should look like. Some options - Inferred as a TypeAlias - Inferred as a class variable - Specify that the type checker should raise an error here - asking the user to be explicit IMO - inferring as TypeAlias will be more useful in more cases. It also has the benefit of having consistent behavior across scopes (least surprise), and more compatible with PEP 484's initial introduction of type aliases, meaning more compatible with existing code.
FWIW, pyright currently uses the following heuristics to distinguish an implicit (inferred) type alias definitions from a regular variable assignment. A type alias is inferred if all of the following conditions apply: 1. There is no type annotation provided 2. There is only one assignment to the symbol 3. The expression on the RHS of the assignment does not contain any syntactic form that would be considered illegal for a type annotation (call expressions, lambdas, comprehensions, etc.) 4. The type evaluation of the RHS evaluates to an instantiable type or a union of instantiable types The current heuristics don't require that the symbol is in the global (module-level) scope. Points #1 and #2 generally distinguish class variables from class-scoped type aliases in the vast majority of cases. Even if these heuristics fail and incorrectly identify a class variable as a type alias, it tends to work fine (no false positive errors). I don't recall ever receiving a bug report or a question about this, so it seems to work well. I do agree that it would be beneficial to standardize this behavior across type checkers. I know that some pyright users rely on our current behavior, so there could be some pain if we were to change it, but (depending on what we conclude here) there's likely a straightforward workaround we can offer the affected users. -- Eric Traut Contributor to Pyright & Pylance Microsoft
As is often the case, I agree with Eric and find myself with little to add beyond a +1 to the position Eric suggested :-) mypy's current heuristics are similar to pyright's, except that simple assignments in nested scope are considered variables, not aliases (as Nipunn mentioned, relevant code <https://github.com/python/mypy/blob/63c414abae02c9afaebe9d0d183aeb911de1807c...>). With PEP 613, I personally don't feel particularly strongly about that additional heuristic... Although maybe there is an appeal to be made to Chesterton's Fence, given that Ivan and Jukka went out of their way to include that heuristic, even though the pre-PEP 613 world left users no recourse to get mypy to treat those as type aliases. I'll check how much code gets flagged by mypy_primer if I remove it. Both of you are right that consistency is better for users. I did a bad job of expressing the core of the point I wanted to make, which was something like: if the intent of the user is clear (as the case of `def f(): X: TypeAlias = int` feels to me), I tend to default to the position that type checkers should follow that intent and it shouldn't be prohibited without a discussion like the one we're having now :-) If we find we're in agreement / this thread dies off, I think I have the following action items: 1) Make PRs to clarify PEP 484 and loosen requirements in PEP 613 2) See if changing mypy's heuristic regarding nested scopes to match pyright causes regressions for mypy users 3) Update the mypy documentation page Nipunn linked On Wed, 3 Nov 2021 at 11:56, Eric Traut <eric@traut.com> wrote:
FWIW, pyright currently uses the following heuristics to distinguish an implicit (inferred) type alias definitions from a regular variable assignment.
A type alias is inferred if all of the following conditions apply: 1. There is no type annotation provided 2. There is only one assignment to the symbol 3. The expression on the RHS of the assignment does not contain any syntactic form that would be considered illegal for a type annotation (call expressions, lambdas, comprehensions, etc.) 4. The type evaluation of the RHS evaluates to an instantiable type or a union of instantiable types
The current heuristics don't require that the symbol is in the global (module-level) scope.
Points #1 and #2 generally distinguish class variables from class-scoped type aliases in the vast majority of cases. Even if these heuristics fail and incorrectly identify a class variable as a type alias, it tends to work fine (no false positive errors). I don't recall ever receiving a bug report or a question about this, so it seems to work well.
I do agree that it would be beneficial to standardize this behavior across type checkers.
I know that some pyright users rely on our current behavior, so there could be some pain if we were to change it, but (depending on what we conclude here) there's likely a straightforward workaround we can offer the affected users.
--
Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
Pyre hasn’t supported non-global aliases in the past, but I’m OK with consistent behavior for aliases nested in classes as with global aliases (ie., an explicit `TypeAlias` annotation is an option if clarity is needed, or you’re using quoted annotations, etc., but backwards-compatible inference following the rules Eric laid out still works). I agree that we should throw type errors if any bound type variable is referenced in a type alias. My only remaining question is – why support type aliases inside function scopes? I don’t think there is meaningful value-add, and having to deal with a different type hierarchy based on scoping sounds like unnecessary complexity to me. Similarly with type aliases defined as instance variables, or anything beyond toplevel globals or class variables, which are essentially qualified globals. From: Shantanu Jain <hauntsaninja@gmail.com> Date: Thursday, November 4, 2021 at 1:20 AM To: Eric Traut <eric@traut.com> Cc: None via Typing-sig <typing-sig@python.org> Subject: [Typing-sig] Re: PEP 613 and type aliases in nested scope As is often the case, I agree with Eric and find myself with little to add beyond a +1 to the position Eric suggested :-) mypy's current heuristics are similar to pyright's, except that simple assignments in nested scope are considered variables, not aliases (as Nipunn mentioned, relevant code<https://github.com/python/mypy/blob/63c414abae02c9afaebe9d0d183aeb911de1807c...>). With PEP 613, I personally don't feel particularly strongly about that additional heuristic... Although maybe there is an appeal to be made to Chesterton's Fence, given that Ivan and Jukka went out of their way to include that heuristic, even though the pre-PEP 613 world left users no recourse to get mypy to treat those as type aliases. I'll check how much code gets flagged by mypy_primer if I remove it. Both of you are right that consistency is better for users. I did a bad job of expressing the core of the point I wanted to make, which was something like: if the intent of the user is clear (as the case of `def f(): X: TypeAlias = int` feels to me), I tend to default to the position that type checkers should follow that intent and it shouldn't be prohibited without a discussion like the one we're having now :-) If we find we're in agreement / this thread dies off, I think I have the following action items: 1) Make PRs to clarify PEP 484 and loosen requirements in PEP 613 2) See if changing mypy's heuristic regarding nested scopes to match pyright causes regressions for mypy users 3) Update the mypy documentation page Nipunn linked On Wed, 3 Nov 2021 at 11:56, Eric Traut <eric@traut.com<mailto:eric@traut.com>> wrote: FWIW, pyright currently uses the following heuristics to distinguish an implicit (inferred) type alias definitions from a regular variable assignment. A type alias is inferred if all of the following conditions apply: 1. There is no type annotation provided 2. There is only one assignment to the symbol 3. The expression on the RHS of the assignment does not contain any syntactic form that would be considered illegal for a type annotation (call expressions, lambdas, comprehensions, etc.) 4. The type evaluation of the RHS evaluates to an instantiable type or a union of instantiable types The current heuristics don't require that the symbol is in the global (module-level) scope. Points #1 and #2 generally distinguish class variables from class-scoped type aliases in the vast majority of cases. Even if these heuristics fail and incorrectly identify a class variable as a type alias, it tends to work fine (no false positive errors). I don't recall ever receiving a bug report or a question about this, so it seems to work well. I do agree that it would be beneficial to standardize this behavior across type checkers. I know that some pyright users rely on our current behavior, so there could be some pain if we were to change it, but (depending on what we conclude here) there's likely a straightforward workaround we can offer the affected users. -- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org<mailto:typing-sig@python.org> To unsubscribe send an email to typing-sig-leave@python.org<mailto:typing-sig-leave@python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/<https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: hauntsaninja@gmail.com<mailto:hauntsaninja@gmail.com>
Shannon, I'm fine with disallowing type aliases defined within a function scope. I struggle to come up with a compelling use case for this. It sounds like we have consensus that type alias definitions (both implicit and explicit) should be allowed within class scopes. Unless someone disagrees, I think we can move forward with that plan. Shantanu, thanks for volunteering to drive the next steps. I've filed these three issues for myself in the pyright issue tracker: https://github.com/microsoft/pyright/issues/2541 (Generate error when a type alias within an inner scope references a TypeVar bound to the scope) https://github.com/microsoft/pyright/issues/2542 (Disallow type aliases within function scope) https://github.com/microsoft/pyright/issues/2543 (Allow explicit type aliases within class scope) -- Eric Traut Contributor to Pyright & Pylance Microsoft
I don’t know about aliases specifically, but there are definitely people out there who like to define and use classes in function scope. It is commonly used in test suites, which often define similar structures, where it would be tedious to have to define everything at the global level and pick unique names for things in slightly different test scenarios. I would imagine this could include aliases. Is there a technical reason to exclude them? Python in general has a philosophy that allows you to combine structures arbitrarily. We allow imports in functions. Why not type aliases? —Guido On Fri, Nov 5, 2021 at 14:51 Eric Traut <eric@traut.com> wrote:
Shannon, I'm fine with disallowing type aliases defined within a function scope. I struggle to come up with a compelling use case for this.
It sounds like we have consensus that type alias definitions (both implicit and explicit) should be allowed within class scopes. Unless someone disagrees, I think we can move forward with that plan.
Shantanu, thanks for volunteering to drive the next steps.
I've filed these three issues for myself in the pyright issue tracker:
https://github.com/microsoft/pyright/issues/2541 (Generate error when a type alias within an inner scope references a TypeVar bound to the scope) https://github.com/microsoft/pyright/issues/2542 (Disallow type aliases within function scope) https://github.com/microsoft/pyright/issues/2543 (Allow explicit type aliases within class scope)
-- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
I took a stab at a PR updating PEP 613, which gave me some thoughts.
My only remaining question is – why support type aliases inside function scopes? I don’t think there is meaningful value-add, and having to deal with a different type hierarchy based on scoping sounds like unnecessary complexity to me. Similarly with type aliases defined as instance variables, or anything beyond toplevel globals or class variables, which are essentially qualified globals.
Is there a technical reason to exclude them? Python in general has a philosophy that allows you to combine structures arbitrarily. We allow imports in functions. Why not type aliases?
In terms of spec complexity, I believe having different behavior for TypeAlias within functions vs classes vs global scope adds complexity to the spec (eg see scope-restrictions section of https://github.com/python/peps/pull/2154). It may be worth adding such spec-complexity in order to mitigate technical complexity - but I think such cases are rare. Usually spec simplicity is worth working for. If we allowed type aliases in functions, I think we could leave out the entire scope-restrictions section. I'm not familiar with the technical challenges of implementing type hierarchies - so I'll certainly defer to others here who work on type checkers. Per guido's point, this would be in line with python's general philosophy, but I'm also sympathetic to the point that type checking in python might want to be stricter than python's general philosophy. Is there some way to get the appropriate strictness while maintaining spec simplicity?
Thanks Nipunn! I concur... Everywhere that typed Python differs from untyped Python incurs a cost for the user, so I feel we should be approaching this from the default of allowing it. I'm not sure I see how supporting type aliases defined in functions is more complex than supporting classes that are locally defined in function scope — or does Pyre not support locally defined classes? The line between classes and functions is blurry. For example, disallowing type aliases in functions would break code like: ``` def foo(x): bar = map return bar(lambda y: y+1, x) ``` because map is actually a class (and is typed as such in typeshed). ``` def foo(x): bar = map return bar(x) ``` On Tue, 16 Nov 2021 at 18:15, Nipunn Koorapati <nipunn1313@gmail.com> wrote:
I took a stab at a PR updating PEP 613, which gave me some thoughts.
My only remaining question is – why support type aliases inside function scopes? I don’t think there is meaningful value-add, and having to deal with a different type hierarchy based on scoping sounds like unnecessary complexity to me. Similarly with type aliases defined as instance variables, or anything beyond toplevel globals or class variables, which are essentially qualified globals.
Is there a technical reason to exclude them? Python in general has a philosophy that allows you to combine structures arbitrarily. We allow imports in functions. Why not type aliases?
In terms of spec complexity, I believe having different behavior for TypeAlias within functions vs classes vs global scope adds complexity to the spec (eg see scope-restrictions section of https://github.com/python/peps/pull/2154).
It may be worth adding such spec-complexity in order to mitigate technical complexity - but I think such cases are rare. Usually spec simplicity is worth working for. If we allowed type aliases in functions, I think we could leave out the entire scope-restrictions section. I'm not familiar with the technical challenges of implementing type hierarchies - so I'll certainly defer to others here who work on type checkers.
Per guido's point, this would be in line with python's general philosophy, but I'm also sympathetic to the point that type checking in python might want to be stricter than python's general philosophy. Is there some way to get the appropriate strictness while maintaining spec simplicity? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
Interesting example. I don't think disallowing type aliases in functions would break that code. However, it limits the ability to use type aliases to give new names to types (eg for renaming a class defined within a function). Take a look at this: from typing_extensions import TypeAlias def foo() -> None: class Hello: pass foo = Hello bar: TypeAlias = Hello foo1 = foo() foo2: foo = foo() bar1 = bar() bar2: bar = bar() Today, foo gets inferred to either a TypeAlias or Type[Hello] - depending on the typechecker's inference heuristic. bar's legality is also up for debate - again depending on typechecker. I think it's a pretty reasonable use case to consider wanting to rename `Hello` in a backward compatible manner, even within function scope - given python's willingness to allow classes defined within functions. Supporting a type alias in this scenario would be least surprising to me. With the heuristics that Eric Traut outlined, both `foo` and `bar` would be inferred as a TypeAlias matching my expectation. Eric's heuristics try to infer a TypeAlias whenever possible and fallback to a Type[_] - a behavior I think would be nice to standardize in the PEP613/PEP484. Shannon mentioned Type Aliases defined as instance variables - which seems non-useful, since you couldn't use it in a type context. self.baz: TypeAlias = Hello baz1: self.baz = self.baz() # self probably shouldn't resolve in type context I propose clarifying in PEP484 that the LHS must be an identifier in order to be a type alias - so that self.baz = Hello is not a type alias (and self.baz: TypeAlias = Hello is invalid) In favor of both - allowing explicit type alias at any scope (in PEP 613) - standardizing the implicit type-var vs alias inference heuristic to the one Eric described (in PEP 484) Would be interested in hearing Shannon's thoughts about function scope type alias implementation difficulties. --Nipunn
The line between classes and functions is blurry. For example, disallowing type aliases in functions would break code like: ``` def foo(x): bar = map return bar(lambda y: y+1, x) ``` because map is actually a class (and is typed as such in typeshed).
``` def foo(x): bar = map return bar(x) ```
On Tue, Nov 16, 2021 at 10:46 PM Shantanu Jain <hauntsaninja@gmail.com> wrote:
The line between classes and functions is blurry. For example, disallowing type aliases in functions would break code like: ``` def foo(x): bar = map return bar(lambda y: y+1, x) ``` because map is actually a class (and is typed as such in typeshed).
Note that this should pass a type checker regardless of what we choose. Either `bar = map` is an alias definition or it is just a variable assignment. It shouldn't be flagged as an invalid alias definition. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (5)
-
Eric Traut -
Guido van Rossum -
Nipunn Koorapati -
Shannon Zhu -
Shantanu Jain