PEP 563: get_type_hints should use closure when available
The discussion around PEP 649 got me thinking about what I believe is the largest downside to PEP 563: the inability to evaluate annotations created with closures. While this is in general unavoidable, if the type is ever referenced in an annotated function (including as an annotation) it should be resolvable via `__closure__`. For example: ``` from __future__ import annotations import typing def gen(T): def f(x: T): y: T = ... return f f = gen(int) nonlocal_vars = { var : cell.cell_contents for var, cell in zip(f.__code__.co_freevars, f.__closure__) } assert typing.get_type_hints(f, localns=nonlocal_vars) == {'x': int} ``` I would just open a PR to have `get_type_hints` attempt to resolve closure variables by default. However, this would require an update to PEP 563 and I don't know what the protocol is there. - Caleb Donovick
On 2/2/2021 8:37 PM, Caleb Donovick wrote:
Once a PEP is accepted and implemented, the PEP document itself is frozen and not updated. The Python language and docs and the CPython implementation are still subject to what is hopefully further progress. You are free to open an thread on python-ideas list or an issue on bpo, optionally with a PR, or both. It would be good to reference the PEP and note whether this idea appears there or not. Further discussion of the idea itself would depend on whether is was a rejected alternative, a deferred addition, or just mentioned. A speculative PR for an idea not yet accepted in principle may be rejected but may help gain acceptance. PS: I don't use type hints yet and have not much opinion about about your idea (it naively seems plausible), so the above is a generic answer. -- Terry Jan Reedy
Sadly this won't work, because the annotations are "evaluated" (or not) in the surrounding scope, not in the function's scope. Example: ```
Here we want to know what the 'C' refers to in g's annotations. But g's
closures are no help (unless g's body were to also reference C, which is
not likely in typical code).
--Guido
On Tue, Feb 2, 2021 at 6:10 PM Caleb Donovick <donovick@cs.stanford.edu>
wrote:
> The discussion around PEP 649 got me thinking about what I believe is the
> largest downside to PEP 563: the inability to evaluate annotations created
> with closures. While this is in general unavoidable, if the type is ever
> referenced in an annotated function (including as an annotation) it should
> be resolvable via `__closure__`.
>
> For example:
> ```
> from __future__ import annotations
> import typing
> def gen(T):
> def f(x: T):
> y: T = ...
> return f
>
> f = gen(int)
> nonlocal_vars = {
> var : cell.cell_contents
> for var, cell in zip(f.__code__.co_freevars, f.__closure__)
> }
> assert typing.get_type_hints(f, localns=nonlocal_vars) == {'x': int}
> ```
>
> I would just open a PR to have `get_type_hints` attempt to resolve closure
> variables by default. However, this would require an update to PEP 563 and
> I don't know what the protocol is there.
>
> - Caleb Donovick
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/L2XH2EIFCEPT5DQRPRFR6WYEM6M2LM2V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I understand the limitation. I was proposing changing https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1431-L1433 to use the `nonlocal_vars` instead of `globalns` for the default localns (or possibly a chainmap of the two). I agree that it may only work on a small fraction of code but the performance cost seems pretty minimal especially for a function like `get_type_hints`, so why not? But I trust judgement, if you think that creating an apparent inconsistency around when `get_type_hints` will work and when it won't is not worth being able to resolve a few extra cases it probably isn't. - Caleb Donovick On Wed, Feb 3, 2021 at 8:55 AM Guido van Rossum <guido@python.org> wrote:
I think this would just perpetuate misunderstanding about how things work. You're proposing to make it so that changing the body of the function (in particular, removing `y: T`) would affect the interpretation of the type annotation. That's just wrong. On Wed, Feb 3, 2021 at 12:47 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 2/2/2021 8:37 PM, Caleb Donovick wrote:
Once a PEP is accepted and implemented, the PEP document itself is frozen and not updated. The Python language and docs and the CPython implementation are still subject to what is hopefully further progress. You are free to open an thread on python-ideas list or an issue on bpo, optionally with a PR, or both. It would be good to reference the PEP and note whether this idea appears there or not. Further discussion of the idea itself would depend on whether is was a rejected alternative, a deferred addition, or just mentioned. A speculative PR for an idea not yet accepted in principle may be rejected but may help gain acceptance. PS: I don't use type hints yet and have not much opinion about about your idea (it naively seems plausible), so the above is a generic answer. -- Terry Jan Reedy
Sadly this won't work, because the annotations are "evaluated" (or not) in the surrounding scope, not in the function's scope. Example: ```
Here we want to know what the 'C' refers to in g's annotations. But g's
closures are no help (unless g's body were to also reference C, which is
not likely in typical code).
--Guido
On Tue, Feb 2, 2021 at 6:10 PM Caleb Donovick <donovick@cs.stanford.edu>
wrote:
> The discussion around PEP 649 got me thinking about what I believe is the
> largest downside to PEP 563: the inability to evaluate annotations created
> with closures. While this is in general unavoidable, if the type is ever
> referenced in an annotated function (including as an annotation) it should
> be resolvable via `__closure__`.
>
> For example:
> ```
> from __future__ import annotations
> import typing
> def gen(T):
> def f(x: T):
> y: T = ...
> return f
>
> f = gen(int)
> nonlocal_vars = {
> var : cell.cell_contents
> for var, cell in zip(f.__code__.co_freevars, f.__closure__)
> }
> assert typing.get_type_hints(f, localns=nonlocal_vars) == {'x': int}
> ```
>
> I would just open a PR to have `get_type_hints` attempt to resolve closure
> variables by default. However, this would require an update to PEP 563 and
> I don't know what the protocol is there.
>
> - Caleb Donovick
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/L2XH2EIFCEPT5DQRPRFR6WYEM6M2LM2V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I understand the limitation. I was proposing changing https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1431-L1433 to use the `nonlocal_vars` instead of `globalns` for the default localns (or possibly a chainmap of the two). I agree that it may only work on a small fraction of code but the performance cost seems pretty minimal especially for a function like `get_type_hints`, so why not? But I trust judgement, if you think that creating an apparent inconsistency around when `get_type_hints` will work and when it won't is not worth being able to resolve a few extra cases it probably isn't. - Caleb Donovick On Wed, Feb 3, 2021 at 8:55 AM Guido van Rossum <guido@python.org> wrote:
I think this would just perpetuate misunderstanding about how things work. You're proposing to make it so that changing the body of the function (in particular, removing `y: T`) would affect the interpretation of the type annotation. That's just wrong. On Wed, Feb 3, 2021 at 12:47 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (3)
-
Caleb Donovick -
Guido van Rossum -
Terry Reedy