__future__ annotations loses closure scope
It appears that when from future import __annotations__, a type hint annotation derived from a closure loses scope. Simplistic example: Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... import typing foo = make_a_class(str)() typing.get_type_hints(foo.put_data) {'data': <class 'str'>}
If I add a single import to the top, it breaks: Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
from __future__ import annotations # added this line def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... import typing foo = make_a_class(str)() typing.get_type_hints(foo.put_data) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints value = _eval_type(value, globalns, localns) File "/usr/lib/python3.9/typing.py", line 254, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/usr/lib/python3.9/typing.py", line 493, in _evaluate eval(self.__forward_code__, globalns, localns), File "<string>", line 1, in <module> NameError: name 'data_type' is not defined
I don't see how I can supply the closure scope as localns to get_type_hints. Any suggestions? Is constructing a (dynamically-type- annotated) class in a function like this an anti-pattern?
Yeah, static type checkers won't like it regardless. On Tue, Dec 8, 2020 at 6:39 PM Paul Bryan <pbryan@anode.ca> wrote:
It appears that when from future import __annotations__, a type hint annotation derived from a closure loses scope.
Simplistic example:
Python 3.9.0 (default, Oct 7 2020, 23:09:01)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
def make_a_class(data_type):
... class Foo:
... def put_data(self, data: data_type):
... self.data = data
... return Foo
...
import typing
foo = make_a_class(str)()
typing.get_type_hints(foo.put_data)
{'data': <class 'str'>}
If I add a single import to the top, it breaks:
Python 3.9.0 (default, Oct 7 2020, 23:09:01)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
*>>> from __future__ import annotations # added this line*
def make_a_class(data_type):
... class Foo:
... def put_data(self, data: data_type):
... self.data = data
... return Foo
...
import typing
foo = make_a_class(str)()
typing.get_type_hints(foo.put_data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints
value = _eval_type(value, globalns, localns)
File "/usr/lib/python3.9/typing.py", line 254, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "/usr/lib/python3.9/typing.py", line 493, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
NameError: name 'data_type' is not defined
I don't see how I can supply the closure scope as localns to get_type_hints. Any suggestions? Is constructing a (dynamically-type-annotated) class in a function like this an anti-pattern?
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5RK6VXF2... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Let's try an example that static type checkers should have no problem with: Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
from __future__ import annotations
def make_a_class(): ... class A: ... def get_b(self) -> B: ... return B() ... class B: ... def get_a(self) -> A: ... return A() ... return A ... A = make_a_class() a = A()
import typing typing.get_type_hints(a.get_b) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints value = _eval_type(value, globalns, localns) File "/usr/lib/python3.9/typing.py", line 254, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/usr/lib/python3.9/typing.py", line 493, in _evaluate eval(self.__forward_code__, globalns, localns), File "<string>", line 1, in <module> NameError: name 'B' is not defined
On Tue, 2020-12-08 at 18:48 -0800, Guido van Rossum wrote:
Yeah, static type checkers won't like it regardless.
On Tue, Dec 8, 2020 at 6:39 PM Paul Bryan <pbryan@anode.ca> wrote:
It appears that when from future import __annotations__, a type hint annotation derived from a closure loses scope.
Simplistic example:
Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... import typing foo = make_a_class(str)() typing.get_type_hints(foo.put_data) {'data': <class 'str'>}
If I add a single import to the top, it breaks:
Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
from __future__ import annotations # added this line def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... import typing foo = make_a_class(str)() typing.get_type_hints(foo.put_data) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints value = _eval_type(value, globalns, localns) File "/usr/lib/python3.9/typing.py", line 254, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/usr/lib/python3.9/typing.py", line 493, in _evaluate eval(self.__forward_code__, globalns, localns), File "<string>", line 1, in <module> NameError: name 'data_type' is not defined
I don't see how I can supply the closure scope as localns to get_type_hints. Any suggestions? Is constructing a (dynamically- type-annotated) class in a function like this an anti-pattern?
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/5RK6VXF2...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NRH4HBD3... Code of Conduct: http://python.org/psf/codeofconduct/
What is the utility of a type annotation when the thing it refers to cannot exist? Deferred annotation lookups are intended to be something that analysis time can make sense of but can always have no useful meaning at runtime. No nesting required: ``` from __future__ import annotations Class X: ... def foo(hi: X): ... del X ``` Now try analyzing foo at runtime... I assume "Boom" with that NameError again? (*On a phone, can't try it now)* I believe this isn't a problem get_type_hints() can ever solve. Code that does this isn't what I'd call "reasonably" structured for use with type hints. If anything, type checkers should try to warn about it? -gps On Tue, Dec 8, 2020, 7:03 PM Paul Bryan <pbryan@anode.ca> wrote:
Let's try an example that static type checkers should have no problem with:
Python 3.9.0 (default, Oct 7 2020, 23:09:01)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
from __future__ import annotations
def make_a_class():
... class A:
... def get_b(self) -> B:
... return B()
... class B:
... def get_a(self) -> A:
... return A()
... return A
...
A = make_a_class()
a = A()
import typing
typing.get_type_hints(a.get_b)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints
value = _eval_type(value, globalns, localns)
File "/usr/lib/python3.9/typing.py", line 254, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "/usr/lib/python3.9/typing.py", line 493, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
NameError: name 'B' is not defined
On Tue, 2020-12-08 at 18:48 -0800, Guido van Rossum wrote:
Yeah, static type checkers won't like it regardless.
On Tue, Dec 8, 2020 at 6:39 PM Paul Bryan <pbryan@anode.ca> wrote:
It appears that when from future import __annotations__, a type hint annotation derived from a closure loses scope.
Simplistic example:
Python 3.9.0 (default, Oct 7 2020, 23:09:01)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
def make_a_class(data_type):
... class Foo:
... def put_data(self, data: data_type):
... self.data = data
... return Foo
...
import typing
foo = make_a_class(str)()
typing.get_type_hints(foo.put_data)
{'data': <class 'str'>}
If I add a single import to the top, it breaks:
Python 3.9.0 (default, Oct 7 2020, 23:09:01)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
*>>> from __future__ import annotations # added this line*
def make_a_class(data_type):
... class Foo:
... def put_data(self, data: data_type):
... self.data = data
... return Foo
...
import typing
foo = make_a_class(str)()
typing.get_type_hints(foo.put_data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints
value = _eval_type(value, globalns, localns)
File "/usr/lib/python3.9/typing.py", line 254, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "/usr/lib/python3.9/typing.py", line 493, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
NameError: name 'data_type' is not defined
I don't see how I can supply the closure scope as localns to get_type_hints. Any suggestions? Is constructing a (dynamically-type-annotated) class in a function like this an anti-pattern?
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5RK6VXF2... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NRH4HBD3... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6P5GFROA... Code of Conduct: http://python.org/psf/codeofconduct/
Yep, your example, *boom*. My use case is to for type annotations to resolve type encoders, decoders and validators at runtime. Without __future__, type annotations specified in closure scope are correctly attached to class variables, function parameters and return types. Because they're in scope at the time they're evaluated. __future__ annotations breaks because the hint is not evaluated until get_type_hints is called, which is too late; the scope is lost. Instead of just storing an annotation as a string, how about storing an object containing the string, plus the local scope? Then get_type_hint could be made to successfully resolve it. On Tue, 2020-12-08 at 20:44 -0800, Gregory P. Smith wrote:
What is the utility of a type annotation when the thing it refers to cannot exist?
Deferred annotation lookups are intended to be something that analysis time can make sense of but can always have no useful meaning at runtime.
No nesting required:
``` from __future__ import annotations Class X: ...
def foo(hi: X): ...
del X ```
Now try analyzing foo at runtime... I assume "Boom" with that NameError again? (On a phone, can't try it now)
I believe this isn't a problem get_type_hints() can ever solve.
Code that does this isn't what I'd call "reasonably" structured for use with type hints.
If anything, type checkers should try to warn about it?
-gps
On Tue, Dec 8, 2020, 7:03 PM Paul Bryan <pbryan@anode.ca> wrote:
Let's try an example that static type checkers should have no problem with:
Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
from __future__ import annotations
def make_a_class(): ... class A: ... def get_b(self) -> B: ... return B() ... class B: ... def get_a(self) -> A: ... return A() ... return A ... A = make_a_class() a = A()
import typing typing.get_type_hints(a.get_b) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints value = _eval_type(value, globalns, localns) File "/usr/lib/python3.9/typing.py", line 254, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/usr/lib/python3.9/typing.py", line 493, in _evaluate eval(self.__forward_code__, globalns, localns), File "<string>", line 1, in <module> NameError: name 'B' is not defined
On Tue, 2020-12-08 at 18:48 -0800, Guido van Rossum wrote:
Yeah, static type checkers won't like it regardless.
On Tue, Dec 8, 2020 at 6:39 PM Paul Bryan <pbryan@anode.ca> wrote:
It appears that when from future import __annotations__, a type hint annotation derived from a closure loses scope.
Simplistic example:
Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
> def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... > import typing > foo = make_a_class(str)() > typing.get_type_hints(foo.put_data) {'data': <class 'str'>} >
If I add a single import to the top, it breaks:
Python 3.9.0 (default, Oct 7 2020, 23:09:01) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
> from __future__ import annotations # added this line > def make_a_class(data_type): ... class Foo: ... def put_data(self, data: data_type): ... self.data = data ... return Foo ... > import typing > foo = make_a_class(str)() > typing.get_type_hints(foo.put_data) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.9/typing.py", line 1386, in get_type_hints value = _eval_type(value, globalns, localns) File "/usr/lib/python3.9/typing.py", line 254, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/usr/lib/python3.9/typing.py", line 493, in _evaluate eval(self.__forward_code__, globalns, localns), File "<string>", line 1, in <module> NameError: name 'data_type' is not defined >
I don't see how I can supply the closure scope as localns to get_type_hints. Any suggestions? Is constructing a (dynamically-type-annotated) class in a function like this an anti-pattern?
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/5RK6VXF2...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/NRH4HBD3...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6P5GFROA...
Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Dec 8, 2020 at 8:58 PM Paul Bryan <pbryan@anode.ca> wrote:
My use case is to for type annotations to resolve type encoders, decoders and validators at runtime.
What is the reason you can't insist that the annotations be globals or at least accessible from there (e.g. class attributes)? If the reason is that they're dynamically computed, maybe you can dynamically compute the annotation (effectively overruling the "future" semantics)? `__annotations__` is a mutable attribute so there's nothing to stop you from writing e.g. def make_foo(symbol): def foo(arg: symbol): ... foo.__annotations__['arg'] = symbol # Or you could do e.g. this in a decorator return foo
Without __future__, type annotations specified in closure scope are correctly attached to class variables, function parameters and return types. Because they're in scope at the time they're evaluated.
__future__ annotations breaks because the hint is not evaluated until get_type_hints is called, which is too late; the scope is lost.
Instead of just storing an annotation as a string, how about storing an object containing the string, plus the local scope? Then get_type_hint could be made to successfully resolve it.
That would be a problem because it would keep everything in the local scope alive forever (since perhaps the annotation is never used). There may be a reason why my suggestion won't work either. If that's so, please show a realistic example that captures the essence of your problem, so we won't keep going back and forth attacking each other's toy examples. Also please understand that this behavior is prescribed by an accepted PEP and has been around since 3.7, so the bar to change this is very high. (It's not the default behavior until 3.10 though, so theoretically there might be a little bit more wiggle room for that. So far you haven't shown sufficient evidence to consider that though.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Rereading PEP 563, I'm clearly railing against it. I believe I can work with __annotations__ as you suggest assuming the contract going forward is for get_type_hints to return annotations verbatim if they're not strings or ForwardRefs (the current implementation). Thank you and Gregory for your time. On Tue, 2020-12-08 at 21:49 -0800, Guido van Rossum wrote:
On Tue, Dec 8, 2020 at 8:58 PM Paul Bryan <pbryan@anode.ca> wrote:
My use case is to for type annotations to resolve type encoders, decoders and validators at runtime.
What is the reason you can't insist that the annotations be globals or at least accessible from there (e.g. class attributes)? If the reason is that they're dynamically computed, maybe you can dynamically compute the annotation (effectively overruling the "future" semantics)? `__annotations__` is a mutable attribute so there's nothing to stop you from writing e.g.
def make_foo(symbol): def foo(arg: symbol): ... foo.__annotations__['arg'] = symbol # Or you could do e.g. this in a decorator return foo
Without __future__, type annotations specified in closure scope are correctly attached to class variables, function parameters and return types. Because they're in scope at the time they're evaluated.
__future__ annotations breaks because the hint is not evaluated until get_type_hints is called, which is too late; the scope is lost.
Instead of just storing an annotation as a string, how about storing an object containing the string, plus the local scope? Then get_type_hint could be made to successfully resolve it.
That would be a problem because it would keep everything in the local scope alive forever (since perhaps the annotation is never used).
There may be a reason why my suggestion won't work either. If that's so, please show a realistic example that captures the essence of your problem, so we won't keep going back and forth attacking each other's toy examples.
Also please understand that this behavior is prescribed by an accepted PEP and has been around since 3.7, so the bar to change this is very high. (It's not the default behavior until 3.10 though, so theoretically there might be a little bit more wiggle room for that. So far you haven't shown sufficient evidence to consider that though.)
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DMHM5BTB... Code of Conduct: http://python.org/psf/codeofconduct/
Great. If you’d like to add something to the docs for get_type_hints() about this behavior please submit a PR — I don’t see a reason this would suddenly change but it’s better to make sure. On Tue, Dec 8, 2020 at 22:29 Paul Bryan <pbryan@anode.ca> wrote:
Rereading PEP 563, I'm clearly railing against it. I believe I can work with __annotations__ as you suggest assuming the contract going forward is for get_type_hints to return annotations verbatim if they're not strings or ForwardRefs (the current implementation).
Thank you and Gregory for your time.
On Tue, 2020-12-08 at 21:49 -0800, Guido van Rossum wrote:
On Tue, Dec 8, 2020 at 8:58 PM Paul Bryan <pbryan@anode.ca> wrote:
My use case is to for type annotations to resolve type encoders, decoders and validators at runtime.
What is the reason you can't insist that the annotations be globals or at least accessible from there (e.g. class attributes)? If the reason is that they're dynamically computed, maybe you can dynamically compute the annotation (effectively overruling the "future" semantics)? `__annotations__` is a mutable attribute so there's nothing to stop you from writing e.g.
def make_foo(symbol): def foo(arg: symbol): ... foo.__annotations__['arg'] = symbol # Or you could do e.g. this in a decorator return foo
Without __future__, type annotations specified in closure scope are correctly attached to class variables, function parameters and return types. Because they're in scope at the time they're evaluated.
__future__ annotations breaks because the hint is not evaluated until get_type_hints is called, which is too late; the scope is lost.
Instead of just storing an annotation as a string, how about storing an object containing the string, plus the local scope? Then get_type_hint could be made to successfully resolve it.
That would be a problem because it would keep everything in the local scope alive forever (since perhaps the annotation is never used).
There may be a reason why my suggestion won't work either. If that's so, please show a realistic example that captures the essence of your problem, so we won't keep going back and forth attacking each other's toy examples.
Also please understand that this behavior is prescribed by an accepted PEP and has been around since 3.7, so the bar to change this is very high. (It's not the default behavior until 3.10 though, so theoretically there might be a little bit more wiggle room for that. So far you haven't shown sufficient evidence to consider that though.)
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DMHM5BTB...
Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
participants (3)
-
Gregory P. Smith -
Guido van Rossum -
Paul Bryan