Postponed annotations break inspection of dataclasses
The new postponed annotations have an unexpected interaction with
dataclasses. Namely, you cannot get the type hints of any of the data
classes methods.
For example, I have some code that inspects the type parameters of a
class's `__init__` method. (The real use case is to provide a default
serializer for the class, but that is not important here.)
```
from dataclasses import dataclass
from typing import get_type_hints
class Foo:
pass
@dataclass
class Bar:
foo: Foo
print(get_type_hints(Bar.__init__))
```
In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
This is a good catch -- thanks for bringing it up. I'm adding Eric Smith
(author of dataclasses) and Ivan Levkivskyi (co-author of typing) as well
as Łukasz Langa (author of PEP 563) to the thread to see if they have
further insights.
Personally I don't think it's feasible to change PEP 563 to use lambdas (if
it were even advisable, which would be a long discussion), but I do think
we might be able to make small improvements to the dataclasses and/or
typing modules to make sure your use case works.
Probably a bugs.python.org issue is a better place to dive into the details
than python-dev.
Thanks again,
--Guido (top-poster in chief)
On Sat, Sep 22, 2018 at 8:32 AM David Hagen
The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods.
For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)
``` from dataclasses import dataclass from typing import get_type_hints
class Foo: pass
@dataclass class Bar: foo: Foo
print(get_type_hints(Bar.__init__)) ```
In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
, 'return': }`. However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:
``` NameError: name 'Foo' is not defined ```
I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.
I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 9/22/2018 12:41 PM, Guido van Rossum wrote:
This is a good catch -- thanks for bringing it up. I'm adding Eric Smith (author of dataclasses) and Ivan Levkivskyi (co-author of typing) as well as Łukasz Langa (author of PEP 563) to the thread to see if they have further insights.
I don't see Ivan and Łukasz cc'd, so I'm adding them here.
Personally I don't think it's feasible to change PEP 563 to use lambdas (if it were even advisable, which would be a long discussion), but I do think we might be able to make small improvements to the dataclasses and/or typing modules to make sure your use case works.
Probably a bugs.python.org http://bugs.python.org issue is a better place to dive into the details than python-dev.
Agreed that opening a bug would be good. And then I'll ruin that suggestion by answering here, too: I think this problem is endemic to get_type_hints(). I've never understood how you're supposed to use the globals and locals arguments to it, but this works: print(get_type_hints(Bar.__init__, globals())) as does: print(get_type_hints(Bar.__init__, Bar.__module__)) But that seems like you'd have to know a lot about how a class were declared in order to call get_type_hints on it. I'm not sure __module__ is always correct (but again, I haven't really thought about it). The docs for get_type_hints() says: "In addition, forward references encoded as string literals are handled by evaluating them in globals and locals namespaces." Every once in a while someone will bring up the idea of delayed evaluation, and the answer is always "use a lambda". If we ever wanted to do something more with delayed evaluation, this is a good use case for it. Eric
Thanks again,
--Guido (top-poster in chief)
On Sat, Sep 22, 2018 at 8:32 AM David Hagen
mailto:david@drhagen.com> wrote: The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods.
For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)
``` from dataclasses import dataclass from typing import get_type_hints
class Foo: pass
@dataclass class Bar: foo: Foo
print(get_type_hints(Bar.__init__)) ```
In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
, 'return': }`. However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:
``` NameError: name 'Foo' is not defined ```
I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.
I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once. _______________________________________________ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40tru...
On Sat, Sep 22, 2018 at 11:29 AM Eric V. Smith
I think this problem is endemic to get_type_hints(). I've never understood how you're supposed to use the globals and locals arguments to it, but this works:
print(get_type_hints(Bar.__init__, globals()))
as does:
print(get_type_hints(Bar.__init__, Bar.__module__))
But that seems like you'd have to know a lot about how a class were declared in order to call get_type_hints on it. I'm not sure __module__ is always correct (but again, I haven't really thought about it).
Still, I wonder if there's a tweak possible of the globals and locals used when exec()'ing the function definitions in dataclasses.py, so that get_type_hints() gets the right globals for this use case. It's really tough to be at the intersection of three PEPs... -- --Guido van Rossum (python.org/~guido)
On Sat, Sep 22, 2018 at 3:11 PM Guido van Rossum
Still, I wonder if there's a tweak possible of the globals and locals used when exec()'ing the function definitions in dataclasses.py, so that get_type_hints() gets the right globals for this use case.
It's really tough to be at the intersection of three PEPs...
If it's possible to fix exec() to accept any Mapping (not just dicts), then we can create a proxy mapping for "Dataclass.__init__.__module__" module and everything would work as expected. Here's a very hack-ish fix we can use in meanwhile (even in 3.7.1?): https://gist.github.com/1st1/37fdd3cc84cd65b9af3471b935b722df Yury
On Sat, Sep 22, 2018 at 3:11 PM Guido van Rossum
Still, I wonder if there's a tweak possible of the globals and locals used when exec()'ing the function definitions in dataclasses.py, so that get_type_hints() gets the right globals for this use case.
On Sat, Sep 22, 2018 at 4:38 PM Yury Selivanov
If it's possible to fix exec() to accept any Mapping (not just dicts), then we can create a proxy mapping for "Dataclass.__init__.__module__" module and everything would work as expected.
Another possible solution is that `__annotations__` are *permitted* to be lambdas, but not created as such by default, and `get_type_hints` is aware of this. Operations that are known to break type hints (such as when data classes copy the type hints from the class body to the `__init__` method) could covert the type hint from a string to a lambda before moving it. It does not even have to be a lambda; an instance of some `TypeHint` class, which stores the string and pointers to the original `globals` and `locals` (or just the original object whose `globals` and `locals` are relevant), but that's essentially what a lambda is anyway.
Probably a bugs.python.org issue is a better place to dive into the
On Sat, Sep 22, 2018 at 12:41 PM Guido van Rossum
On Sep 22, 2018, at 1:38 PM, Yury Selivanov
wrote: On Sat, Sep 22, 2018 at 3:11 PM Guido van Rossum
wrote: [..] Still, I wonder if there's a tweak possible of the globals and locals used when exec()'ing the function definitions in dataclasses.py, so that get_type_hints() gets the right globals for this use case.
It's really tough to be at the intersection of three PEPs...
If it's possible to fix exec() to accept any Mapping (not just dicts), then we can create a proxy mapping for "Dataclass.__init__.__module__" module and everything would work as expected
FWIW, the locals() dict for exec() already accepts any mapping (not just dicts): >>> class M: def __getitem__(self, key): return key.upper() def __setitem__(self, key, value): print(f'{key!r}: {value!r}') >>> exec('a=b', globals(), M()) 'a': 'B' Raymond
Do we have a b.p.o. issue about this? If no, then I would recommend to open
one, so that we will not loose track of this.
--
Ivan
On Sat, 22 Sep 2018 at 16:32, David Hagen
The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods.
For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)
``` from dataclasses import dataclass from typing import get_type_hints
class Foo: pass
@dataclass class Bar: foo: Foo
print(get_type_hints(Bar.__init__)) ```
In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
, 'return': }`. However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:
``` NameError: name 'Foo' is not defined ```
I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.
I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
Yes, it’s https://bugs.python.org/issue34776 -- Eric
On Sep 27, 2018, at 12:05 PM, Ivan Levkivskyi
wrote: Do we have a b.p.o. issue about this? If no, then I would recommend to open one, so that we will not loose track of this.
-- Ivan
On Sat, 22 Sep 2018 at 16:32, David Hagen
wrote: The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods. For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)
``` from dataclasses import dataclass from typing import get_type_hints
class Foo: pass
@dataclass class Bar: foo: Foo
print(get_type_hints(Bar.__init__)) ```
In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
, 'return': }`. However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:
``` NameError: name 'Foo' is not defined ```
I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.
I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40tru...
participants (6)
-
David Hagen
-
Eric V. Smith
-
Guido van Rossum
-
Ivan Levkivskyi
-
Raymond Hettinger
-
Yury Selivanov