PEP 649: Deferred Evaluation Of Annotations
I'd like to revive the discussion of PEP 649 [https://www.python.org/dev/peps/pep-0649/] that started shortly before development of 3.10 features was closed. This is Larry's PEP to defer evaluation of __annotations__ until it's actually accessed. During the discussion the decision was made to back out the change that made "from __future__ import annotations" (PEP 563 [https://www.python.org/dev/peps/pep-0563/]) the default behavior for 3.10. My understanding is that PEP 649 is currently in front of the SC. But do we need to have any additional discussion here? My recollection is that we backed out the PEP 563 change because we didn't feel we had enough time to come to a good decision one way or the other before 3.10. Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses. Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in. -- Eric
Some prior discussions: "PEP 563 in light of PEP 649": https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6C... "In support of PEP 649": https://mail.python.org/archives/list/python-dev@python.org/message/7VMJWFGH... "PEP 563 and 649: The Great Compromise": https://mail.python.org/archives/list/python-dev@python.org/message/WUZGTGE4... I'm sure there are other threads I missed. -- Eric On 8/9/2021 11:27 AM, Eric V. Smith wrote:
I'd like to revive the discussion of PEP 649 [https://www.python.org/dev/peps/pep-0649/] that started shortly before development of 3.10 features was closed. This is Larry's PEP to defer evaluation of __annotations__ until it's actually accessed. During the discussion the decision was made to back out the change that made "from __future__ import annotations" (PEP 563 [https://www.python.org/dev/peps/pep-0563/]) the default behavior for 3.10.
My understanding is that PEP 649 is currently in front of the SC. But do we need to have any additional discussion here? My recollection is that we backed out the PEP 563 change because we didn't feel we had enough time to come to a good decision one way or the other before 3.10.
Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses.
Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in.
-- Eric V. Smith
On Mon, Aug 9, 2021 at 8:31 AM Eric V. Smith <eric@trueblade.com> wrote:
I'd like to revive the discussion of PEP 649 [https://www.python.org/dev/peps/pep-0649/] that started shortly before development of 3.10 features was closed. This is Larry's PEP to defer evaluation of __annotations__ until it's actually accessed. During the discussion the decision was made to back out the change that made "from __future__ import annotations" (PEP 563 [https://www.python.org/dev/peps/pep-0563/]) the default behavior for 3.10.
My understanding is that PEP 649 is currently in front of the SC.
Correct.
But do we need to have any additional discussion here?
I think it's worth discussing whether PEP 649 is the solution we want to see as the solution to this problem or not?
My recollection is that we backed out the PEP 563 change because we didn't feel we had enough time to come to a good decision one way or the other before 3.10.
Correct.
Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses.
Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in.
I think the question is whether we have general consensus around PEP 649? -Brett
-- Eric
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2MEOWHCV... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith <eric@trueblade.com> wrote:
Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses.
I don't think there is much difference. PEP 563 is not default. And PEP 563 is not the only source of stringified annotation. So Accepting PEP 649 doesn't mean "they'll all get their wish". We need to say "won't fix" anyway.
Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in.
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact. As far as I remember, the reference implementation created a function object for each methods. It means doubles function objects. It has major impact to memory usage, startup time, and GC time. There was an idea to avoid creating function objects for most cases. But it was not implemented. Regards, -- Inada Naoki <songofacandy@gmail.com>
On 10/08/2021 4:25 am, Inada Naoki wrote:
On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith <eric@trueblade.com> wrote:
Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses.
I don't think there is much difference.
PEP 563 is not default. And PEP 563 is not the only source of stringified annotation. So Accepting PEP 649 doesn't mean "they'll all get their wish". We need to say "won't fix" anyway.
Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in.
My only comment concerned performance. I won't claim that the cost of PEP 649 will be zero, but it won't be significantly more than PEP 563 if annotations are unused. The cost of unmarshalling a code object will be greater than a string, but it won't be significant. Not only that, but we are actively looking to reduce startup in 3.11 which will reduce the overhead further. If annotations *are* used, then PEP 649 should be cheaper as it relies on the interpreter to do the evaluation in an efficient fashion. For users of dataclasses and Pydantic, I expect PEP 649 to outperform PEP 563.
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact.
As far as I remember, the reference implementation created a function object for each methods.
No function object is created under normal circumstances. __annotations__ is a property that calls the underlying __co_annotations__ property, which lazily creates a callable. I'll leave it to Larry to explain why __co_annotations__ isn't just a code object.
It means doubles function objects. It has major impact to memory usage, startup time, and GC time.
Only if __annotations__ are widely used, in which case PEP 563 is probably worse. Cheers, Mark.
There was an idea to avoid creating function objects for most cases. But it was not implemented.
Regards,
On Tue, Aug 10, 2021 at 5:11 PM Mark Shannon <mark@hotpy.org> wrote:
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact.
As far as I remember, the reference implementation created a function object for each methods.
No function object is created under normal circumstances. __annotations__ is a property that calls the underlying __co_annotations__ property, which lazily creates a callable.
I'll leave it to Larry to explain why __co_annotations__ isn't just a code object.
I am talking about methods. As far as I remember, function objects are created for each method to keep class namespace. Larry explained it and possible optimization. That's what I am waiting for. https://mail.python.org/archives/list/python-dev@python.org/message/2OOCEE6O... Regards, -- Inada Naoki <songofacandy@gmail.com>
On 8/9/2021 11:25 PM, Inada Naoki wrote:
On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith <eric@trueblade.com> wrote:
Personally, I'd like to see PEP 649 accepted. There are a number of issues on bpo where people expect dataclass's field.type to be an actual Python object representing a type, and not a string. This field is copied directly from __annotations__. If we stick with PEP 563's string annotations, I'll probably just close these issues as "won't fix", and tell the caller they need to convert the strings to Python objects themselves. If 649 is accepted, then they'll all get their wish, and in addition I can remove some ugly logic from dataclasses.
I don't think there is much difference.
PEP 563 is not default. But the intent is/was to make it the default. That was the case in 3.10, up until right before the release of beta 1. And PEP 563 is not the only source of stringified annotation. So Accepting PEP 649 doesn't mean "they'll all get their wish". We need to say "won't fix" anyway.
If 649 is accepted, there will be few places where stringified annotations will be needed. For example, forward references that are defined before __annotations__ is examined will not need to be specified as strings. From the PEP: "Actually, annotations would become much easier to use, as they would now also handle forward references.
Do we need to do anything here to move forward on this issue? I've chatted with Larry and Mark Shannon, who have some additional thoughts and I'm sure will chime in.
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact.
Agreed. -- Eric
On 10 Aug 2021, at 13:05, Eric V. Smith <eric@trueblade.com> wrote:
If 649 is accepted, there will be few places where stringified annotations will be needed. For example, forward references that are defined before __annotations__ is examined will not need to be specified as strings. From the PEP: "Actually, annotations would become much easier to use, as they would now also handle forward references.
In general, yes, agreed. However, a popular special case that you're intimately familiar with are class decorators. Since those are executed quite eagerly, you'll have to account for failing evaluation of forward references. Example: from dataclasses import dataclass @dataclass class C: a: A class A: ... In this case PEP 649 doesn't help. I don't see evaluation failures discussed explicitly in the PEP but I assume it would be a NameError. It would also be interesting to see if we can somehow make it handle annotations that refer to the class they're used in. Example: @dataclass class Node: parent: Node - Ł
In this case PEP 649 doesn't help.
Sorry for the naive question but why doesn't PEP 649 help here? Is there something fundamental about the dataclass that needs to inspect the type of C.a to create the dataclass? - Damian (he/him) On Tue, Aug 10, 2021 at 1:10 PM Łukasz Langa <lukasz@langa.pl> wrote:
On 10 Aug 2021, at 13:05, Eric V. Smith <eric@trueblade.com> wrote:
If 649 is accepted, there will be few places where stringified annotations will be needed. For example, forward references that are defined before __annotations__ is examined will not need to be specified as strings. From the PEP: "Actually, annotations would become much easier to use, as they would now also handle forward references.
In general, yes, agreed. However, a popular special case that you're intimately familiar with are class decorators. Since those are executed quite eagerly, you'll have to account for failing evaluation of forward references. Example:
from dataclasses import dataclass
@dataclass class C: a: A
class A: ...
In this case PEP 649 doesn't help. I don't see evaluation failures discussed explicitly in the PEP but I assume it would be a NameError. It would also be interesting to see if we can somehow make it handle annotations that refer to the class they're used in. Example:
@dataclass class Node: parent: Node
- Ł _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FVJAUJ4T... Code of Conduct: http://python.org/psf/codeofconduct/
dataclasses need to check for ClassVar
Interesting, so the use case we are talking about is: 1) You are using annotations to mean actual types, 2) But you also have to inspect them at runtime, 3) For some of the types the name might not be defined at runtime yet In this example doesn't the current behavior, PEP 649, and PEP 563 (using get_type_hints) all throw an exception? Could PEP 649 be modified to say that if a NameError is raised the result is not cached and therefore you can inspect it later at runtime to get the real type once it is defined? Wouldn't that then allow users to write code that allows for all use cases under this scenario? - Damian (he/him) On Tue, Aug 10, 2021 at 1:55 PM Thomas Grainger <tagrain@gmail.com> wrote:
dataclasses need to check for ClassVar _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WOTCUBAO... Code of Conduct: http://python.org/psf/codeofconduct/
Damian Shaw wrote: > > dataclasses need to check for ClassVar > > Interesting, so the use case we are talking about is: 1) You are using > annotations to mean actual types, 2) But you also have to inspect them at > runtime, 3) For some of the types the name might not be defined at runtime > yet > In this example doesn't the current behavior, PEP 649, and PEP 563 (using > get_type_hints) all throw an exception? > Could PEP 649 be modified to say that if a NameError is raised the result > is not cached and therefore you can inspect it later at runtime to get the > real type once it is defined? Wouldn't that then allow users to write code > that allows for all use cases under this scenario? > - Damian (he/him) > On Tue, Aug 10, 2021 at 1:55 PM Thomas Grainger tagrain@gmail.com wrote: > > dataclasses need to check for ClassVar > > _______________________________________________ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-leave@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > https://mail.python.org/archives/list/python-dev@python.org/message/WOTCUBAO... > > Code of Conduct: http://python.org/psf/codeofconduct/ > > Yep it would need a note on the pep. It's a narrow usecase and everyone who does use the feature (attrs, pydantic etc) are watching this PEP carefully so it shouldn't be too much of a comparability concern
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
El mar, 10 ago 2021 a las 11:20, Thomas Grainger (<tagrain@gmail.com>) escribió:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
I implemented a version of this in https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't like it.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/46OW5FQP... Code of Conduct: http://python.org/psf/codeofconduct/
Jelle Zijlstra wrote:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name I implemented a version of this in https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't
El mar, 10 ago 2021 a las 11:20, Thomas Grainger (tagrain@gmail.com) escribió: like it.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/46OW5FQP... Code of Conduct: http://python.org/psf/codeofconduct/
I'd like the updated PEP to mention this approach and describe if/why it wasn't included
On Tue, Aug 10, 2021 at 11:41 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El mar, 10 ago 2021 a las 11:20, Thomas Grainger (<tagrain@gmail.com>) escribió:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
I implemented a version of this in https://github.com/larryhastings/co_annotations/pull/3 but Larry didn't like it.
However, I do like it, despite all Larry's arguing against it. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 8/10/21 11:15 AM, Thomas Grainger wrote:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
No, it should raise a NameError, just like any other Python code. Annotations aren't special enough to break the rules. I worry about Python-the-language enshrining design choices made by the typing module. Python is now on its fourth string interpolation technology, and it ships with three command-line argument parsing libraries; in each of these cases, we were adding a New Thing that was viewed at the time as an improvement over the existing thing(s). It'd be an act of hubris to assert that the current "typing" module is the ultimate, final library for expressing type information in Python. But if we tie the language too strongly to the typing module, I fear we could strangle its successors in their cribs. //arry/
What about using a coroutine of `v = yield (name, scope)` so the caller can choose how and when names are resolved?
Oh, I agree it shouldn’t reference the typing module. But it should not raise NameError. This whole thing already is a special case. We can debate what else it should, e.g. skip the name, return a fixed error token, return an error token that includes the name that failed (this is part if the NameError), return a string, etc… FWIW the “tiny” change to decorator syntax looks untenable. On Tue, Aug 10, 2021 at 23:09 Larry Hastings <larry@hastings.org> wrote:
On 8/10/21 11:15 AM, Thomas Grainger wrote:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
No, it should raise a NameError, just like any other Python code. Annotations aren't special enough to break the rules.
I worry about Python-the-language enshrining design choices made by the typing module. Python is now on its fourth string interpolation technology, and it ships with three command-line argument parsing libraries; in each of these cases, we were adding a New Thing that was viewed at the time as an improvement over the existing thing(s). It'd be an act of hubris to assert that the current "typing" module is the ultimate, final library for expressing type information in Python. But if we tie the language too strongly to the typing module, I fear we could strangle its successors in their cribs.
*/arry* _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5NUVYOLI... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Would: ``` @dataclass class Node: global_node: __class__ | None ``` "Just work" with co_annotations?
On Wed, Aug 11, 2021 at 2:56 PM Thomas Grainger <tagrain@gmail.com> wrote:
Would: ``` @dataclass class Node: global_node: __class__ | None ```
"Just work" with co_annotations?
This feels too specialized to me. It would be great to also handle forward references to other classes and cyclic references, which are also somewhat common: @dataclass class NodeA: component: NodeB | None @dataclass class NodeB: component: NodeA | None Another, slightly more complex example would be cyclic references within two modules in an import cycle. For example, NodeA and NodeB could be defined in different modules. The common thing is that the dependency cycle can only be fully resolved after we have created both type objects. The import cycle case is probably less common but I've seen it in real-world code. Jukka
On 8/11/2021 11:07 AM, Jukka Lehtosalo wrote:
On Wed, Aug 11, 2021 at 2:56 PM Thomas Grainger <tagrain@gmail.com <mailto:tagrain@gmail.com>> wrote:
Would: ``` @dataclass class Node: global_node: __class__ | None ```
"Just work" with co_annotations?
This feels too specialized to me.
It would be great to also handle forward references to other classes and cyclic references, which are also somewhat common:
@dataclass class NodeA: component: NodeB | None
@dataclass class NodeB: component: NodeA | None
Note that for dataclasses itself, you can just use: @dataclass class NodeA: component: "NodeB | None" @dataclass class NodeB: component: "NodeA | None" It's only looking for typing.ClassVar or dataclasses.InitVar. Anything else, including a string, just means "this is a normal field". This will result in dataclasses.fields(NodeA)[0].type being the string "NodeB | None". But that's okay with dataclasses. However, there are a number of open bpo issues where people want dataclasses to resolve the .type value to be the actual type object instead of a string, especially if using 'from __future__ import annotations'. I guess dataclasses.fields() could assist there, optionally. Eric
Another, slightly more complex example would be cyclic references within two modules in an import cycle. For example, NodeA and NodeB could be defined in different modules. The common thing is that the dependency cycle can only be fully resolved after we have created both type objects. The import cycle case is probably less common but I've seen it in real-world code.
On Wed, Aug 11, 2021 at 7:09 AM Larry Hastings <larry@hastings.org> wrote:
On 8/10/21 11:15 AM, Thomas Grainger wrote:
Although the co_annoations code could intercept the NameError and replace return a ForwardRef object instead of the resolved name
No, it should raise a NameError, just like any other Python code. Annotations aren't special enough to break the rules.
I worry about Python-the-language enshrining design choices made by the typing module. Python is now on its fourth string interpolation technology, and it ships with three command-line argument parsing libraries; in each of these cases, we were adding a New Thing that was viewed at the time as an improvement over the existing thing(s). It'd be an act of hubris to assert that the current "typing" module is the ultimate, final library for expressing type information in Python. But if we tie the language too strongly to the typing module, I fear we could strangle its successors in their cribs.
Which would be unfortunate given the (explicit?) assurances that annotations would be optional; they are casting their shadow over the whole language.
On 8/10/21 11:09 AM, Damian Shaw wrote:
Could PEP 649 be modified to say that if a NameError is raised the result is not cached and therefore you can inspect it later at runtime to get the real type once it is defined? Wouldn't that then allow users to write code that allows for all use cases under this scenario?
The PEP doesn't say so explicitly, but that was the intent of the design, yes. If you look at the pseudo-code in the "__co_annotations__" section: https://www.python.org/dev/peps/pep-0649/#co-annotations you'll see that it doesn't catch NameError, allowing it to bubble up to user code. The prototype intentionally behaves the same way. Certainly it wouldn't hurt to mention that explicitly in the PEP. //arry/
I think as long as there's a test case for something like ``` @dataclass class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] ``` the bug https://bugs.python.org/issue33453 and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6... shows this is a tricky problem
On 8/11/21 12:02 AM, Thomas Grainger wrote:
I think as long as there's a test case for something like
``` @dataclass class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] ```
the bug https://bugs.python.org/issue33453 and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6... shows this is a tricky problem
The most straightforward workaround for this is to skip the decorator syntax. With PEP 649 active, this code should work: class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] Node = dataclass(Node) //arry/
Larry Hastings wrote:
On 8/11/21 12:02 AM, Thomas Grainger wrote:
I think as long as there's a test case for something like @dataclass class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None]
the bug https://bugs.python.org/issue33453 and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6... shows this is a tricky problem The most straightforward workaround for this is to skip the decorator syntax. With PEP 649 active, this code should work: class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] Node = dataclass(Node) //arry/
the decorator version simply has to work
On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger <tagrain@gmail.com> wrote:
Larry Hastings wrote:
On 8/11/21 12:02 AM, Thomas Grainger wrote:
I think as long as there's a test case for something like @dataclass class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None]
the bug https://bugs.python.org/issue33453 and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6... shows this is a tricky problem The most straightforward workaround for this is to skip the decorator syntax. With PEP 649 active, this code should work: class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] Node = dataclass(Node) //arry/
the decorator version simply has to work
I also think that it would be unfortunate if the decorator version wouldn't work. This is a pretty basic use case. If we go through a lot of trouble to redesign how type annotations behave, it would be great if we could address as many common pain points as possible. Otherwise I see a risk that we'll have another (third!) redesign in Python 3.12 or 3.13. Jukka
On 8/11/21 2:48 AM, Jukka Lehtosalo wrote:
On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger <tagrain@gmail.com <mailto:tagrain@gmail.com>> wrote:
Larry Hastings wrote: > On 8/11/21 12:02 AM, Thomas Grainger wrote: > > I think as long as there's a test case for something like > > @dataclass > > class Node: > > global_node: ClassVar[Node | None] > > left: InitVar[Node | None] > > right: InitVar[None | None] > > > > the bug https://bugs.python.org/issue33453 <https://bugs.python.org/issue33453> and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6. <https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6.>.. shows this is a tricky problem > > The most straightforward workaround for this is to skip the decorator > syntax. With PEP 649 active, this code should work: > class Node: > global_node: ClassVar[Node | None] > left: InitVar[Node | None] > right: InitVar[None | None] > Node = dataclass(Node) > //arry/
the decorator version simply has to work
I also think that it would be unfortunate if the decorator version wouldn't work. This is a pretty basic use case.
So, here's an idea, credit goes to Eric V. Smith. What if we tweak how decorators work, /juuuust sliiiightly/, so that they work like the workaround code above? Specifically: currently, decorators are called just after the function or class object is created, before it's bound to a variable. But we could change it so that we first bind the variable to the initial value, then call the decorator, then rebind. That is, this code: @dekor8 class C: ... would become equivalent to this code: class C: ... C = dekorate(C) This seems like it would solve the class self-reference problem--the "Node" example above--when PEP 649 is active. This approach shouldn't break reasonable existing code. That said, this change would be observable from Python, and pathological code could notice and break. For example: def ensure_Foo_is_a_class(o): assert isinstance(Foo, type) return o class Foo: ... @ensure_Foo_is_a_class def Foo(): ... This terrible code currently would not raise an assertion. But if we made the proposed change to the implementation of decorators, it would. I doubt anybody does this sort of nonsense, I just wanted to fully flesh out the topic. If this approach seems interesting, here's one wrinkle to iron out. When an object has multiple decorators, would we want to re-bind after each decorator call? That is, would @dekor1 @dekor2 @dekor3 class C: ... turn into approach A: class C: ... C = dekor1(dekor2(dekor3(C))) or approach B: class C: ... C = dekor3(C) C = dekor2(C) C = dekor1(C) I definitely think "approach B" makes more sense. //arry/
On Wed, Aug 11, 2021 at 10:03 PM Larry Hastings <larry@hastings.org> wrote:
Specifically: currently, decorators are called just after the function or class object is created, before it's bound to a variable. But we could change it so that we first bind the variable to the initial value, then call the decorator, then rebind. That is, this code:
@dekor8 class C: ...
would become equivalent to this code:
class C: ... C = dekorate(C)
This seems like it would solve the class self-reference problem--the "Node" example above--when PEP 649 is active.
Critically, the decorator itself would have to be evaluated *before* that assignment. So it would actually have to work out more like: _tmp = dekor8 class C: ... C = _tmp(C) Otherwise things like "@foo.setter" would break.
This approach shouldn't break reasonable existing code. That said, this change would be observable from Python, and pathological code could notice and break. For example:
def ensure_Foo_is_a_class(o): assert isinstance(Foo, type) return o
class Foo: ...
@ensure_Foo_is_a_class def Foo(): ...
This terrible code currently would not raise an assertion. But if we made the proposed change to the implementation of decorators, it would. I doubt anybody does this sort of nonsense, I just wanted to fully flesh out the topic.
You would be here declaring that a @monkeypatch decorator is terrible code. I'm not sure whether you're right or wrong. You may very well be right. def monkeypatch(cls): basis = globals()[cls.__name__] for attr in dir(cls): setattr(basis, attr, getattr(cls, attr)) return basis @monkeypatch class SomeClass: def new_method(self): ... Currently this works, since SomeClass doesn't get assigned yet. This could be made to work across versions by writing it as @monkeypatch(SomeClass) instead (and then the actual class name would become immaterial).
If this approach seems interesting, here's one wrinkle to iron out. When an object has multiple decorators, would we want to re-bind after each decorator call? That is, would
@dekor1 @dekor2 @dekor3 class C: ...
turn into approach A:
class C: ... C = dekor1(dekor2(dekor3(C)))
or approach B:
class C: ... C = dekor3(C) C = dekor2(C) C = dekor1(C)
I definitely think "approach B" makes more sense.
If you're going to do it at all, best go the whole way. Each decorator functions independently, and a decorated class is a class like any other, so approach B makes far more sense to me. ChrisA
On 8/11/21 5:15 AM, Chris Angelico wrote:
On Wed, Aug 11, 2021 at 10:03 PM Larry Hastings <larry@hastings.org> wrote:
This approach shouldn't break reasonable existing code. That said, this change would be observable from Python, and pathological code could notice and break. For example:
def ensure_Foo_is_a_class(o): assert isinstance(Foo, type) return o
class Foo: ...
@ensure_Foo_is_a_class def Foo(): ...
This terrible code currently would not raise an assertion. But if we made the proposed change to the implementation of decorators, it would. I doubt anybody does this sort of nonsense, I just wanted to fully flesh out the topic.
You would be here declaring that a @monkeypatch decorator is terrible code. I'm not sure whether you're right or wrong. You may very well be right.
def monkeypatch(cls): basis = globals()[cls.__name__] for attr in dir(cls): setattr(basis, attr, getattr(cls, attr)) return basis
@monkeypatch class SomeClass: def new_method(self): ...
Currently this works, since SomeClass doesn't get assigned yet. This could be made to work across versions by writing it as @monkeypatch(SomeClass) instead (and then the actual class name would become immaterial).
Golly! I've never seen that. Is that a common technique? If we need to preserve that behavior, then this idea is probably a non-starter. //arry/
I don't think I've seen code like this. It would be incredibly useful to have a cpython-primer (like mypy-primer and black-primer) that ran as many test suites as possible from pypi with every cPython commit
On Thu, Aug 12, 2021 at 3:01 AM Larry Hastings <larry@hastings.org> wrote:
On 8/11/21 5:15 AM, Chris Angelico wrote:
On Wed, Aug 11, 2021 at 10:03 PM Larry Hastings <larry@hastings.org> wrote:
This approach shouldn't break reasonable existing code. That said, this change would be observable from Python, and pathological code could notice and break. For example:
def ensure_Foo_is_a_class(o): assert isinstance(Foo, type) return o
class Foo: ...
@ensure_Foo_is_a_class def Foo(): ...
This terrible code currently would not raise an assertion. But if we made the proposed change to the implementation of decorators, it would. I doubt anybody does this sort of nonsense, I just wanted to fully flesh out the topic.
You would be here declaring that a @monkeypatch decorator is terrible code. I'm not sure whether you're right or wrong. You may very well be right.
def monkeypatch(cls): basis = globals()[cls.__name__] for attr in dir(cls): setattr(basis, attr, getattr(cls, attr)) return basis
@monkeypatch class SomeClass: def new_method(self): ...
Currently this works, since SomeClass doesn't get assigned yet. This could be made to work across versions by writing it as @monkeypatch(SomeClass) instead (and then the actual class name would become immaterial).
Golly! I've never seen that. Is that a common technique?
If we need to preserve that behavior, then this idea is probably a non-starter.
This specific thing? No, it's not common. But it's a natural consequence of the current behaviour, so if it does change, there'll be a variety of things that will break. It's hard to know how much of that is good code and how much is bad code. I have frequently made decorators that do unusual things (like returning non-functions) and I'm sure that some of them would be considered abuse of functionality :) ChrisA
Larry Hastings wrote:
On Wed, Aug 11, 2021 at 10:32 AM Thomas Grainger <tagrain@gmail.com mailto:tagrain@gmail.com> wrote: Larry Hastings wrote:
On 8/11/21 12:02 AM, Thomas Grainger wrote:
I think as long as there's a test case for something like @dataclass class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None]
the bug https://bugs.python.org/issue33453 <https://bugs.python.org/issue33453> and the current implementation https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6. <https://github.com/python/cpython/blob/bfc2d5a5c4550ab3a2fadeb9459b4bd948ff6.>.. shows this is a tricky problem The most straightforward workaround for this is to skip the decorator syntax. With PEP 649 active, this code should work: class Node: global_node: ClassVar[Node | None] left: InitVar[Node | None] right: InitVar[None | None] Node = dataclass(Node) //arry/
the decorator version simply has to work
I also think that it would be unfortunate if the decorator version wouldn't work. This is a pretty basic use case. So, here's an idea, credit goes to Eric V. Smith. What if we tweak how decorators work, /juuuust sliiiightly/, so that they work like the workaround code above? Specifically: currently, decorators are called just after the function or class object is created, before it's bound to a variable. But we could change it so that we first bind the variable to the initial value,
On 8/11/21 2:48 AM, Jukka Lehtosalo wrote: then call the decorator, then rebind. That is, this code: @dekor8 class C: ... would become equivalent to this code: class C: ... C = dekorate(C) This seems like it would solve the class self-reference problem--the "Node" example above--when PEP 649 is active. This approach shouldn't break reasonable existing code. That said, this change would be observable from Python, and pathological code could notice and break. For example: def ensure_Foo_is_a_class(o): assert isinstance(Foo, type) return o class Foo: ... @ensure_Foo_is_a_class def Foo(): ... This terrible code currently would not raise an assertion. But if we made the proposed change to the implementation of decorators, it would. I doubt anybody does this sort of nonsense, I just wanted to fully flesh out the topic. If this approach seems interesting, here's one wrinkle to iron out. When an object has multiple decorators, would we want to re-bind after each decorator call? That is, would @dekor1 @dekor2 @dekor3 class C: ... turn into approach A: class C: ... C = dekor1(dekor2(dekor3(C))) or approach B: class C: ... C = dekor3(C) C = dekor2(C) C = dekor1(C) I definitely think "approach B" makes more sense. //arry/
You mention that you wanted this to work also for non-type hint usage of annotations, and so a ForwardRef won't work. As such, would you also change this for function decorators so you can do this? ``` @decorator @typing.no_type_check def ham(spam: ham): ... So it means: ``` def ham(spam: ham): ... ham = typing.no_type_check(ham) ham = decorator(ham) ``` Obviously it's meaningless as a type hint, hence the no_type_check opt out
On 8/11/2021 7:56 AM, Larry Hastings wrote:
So, here's an idea, credit goes to Eric V. Smith. What if we tweak how decorators work, /juuuust sliiiightly/, so that they work like the workaround code above?
Specifically: currently, decorators are called just after the function or class object is created, before it's bound to a variable. But we could change it so that we first bind the variable to the initial value, then call the decorator, then rebind. That is, this code:
@dekor8 class C: ...
would become equivalent to this code:
class C: ... C = dekorate(C)
This is how function decorators were originally defined. Before the 2016 (3.5) revision of https://docs.python.org/3/reference/compound_stmts.html#function-definitions by https://bugs.python.org/issue26576 --- @f1(arg) @f2 def func(): pass is equivalent to def func(): pass func = f1(arg)(f2(func)) --- After --- @f1(arg) @f2 def func(): pass is roughly equivalent to def func(): pass func = f1(arg)(f2(func)) except that the original function is not temporarily bound to the name func. --- I questioned on the issue whether the non-binding optimization "should it be documented as a guaranteed language feature or as just an optional optimization?" -- Terry Jan Reedy
On Mon, Aug 9, 2021 at 9:31 PM Inada Naoki <songofacandy@gmail.com> wrote:
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact.
I volunteered to check performance impact in practice on the Instagram codebase, which is almost fully annotated. However, when I tried, I found that the reference implementation of PEP 649 wasn't even able to import its own test file without a crash. I detailed the problem in https://github.com/larryhastings/co_annotations/issues/12 a couple months ago, but haven't gotten any response. Still willing to do this testing on the IG codebase, but it seems like the implementation needs some additional work before that will be possible. Carl
On 8/9/21 8:25 PM, Inada Naoki wrote:
Currently, reference implementation of PEP 649 has been suspended. We need to revive it and measure performance/memory impact.
Perhaps this sounds strange--but I don't actually agree. PEP 563 was proposed to solve a forward-references problem for the typing community. If you read the PEP, you'll note it contains no discussion about its impact on CPU performance or memory consumption. The performance of its prototype was fine, and in any case, the important thing was to solve the problem in the language. I think PEP 649 should be considered in the same way. In my opinion, the important thing is to figure out what semantics we want for the language. Once we figure out what semantics we want, we should implement them, and only /then/ should we start worrying about performance. Fretting about performance at this point is premature and a distraction. I assert PEP 649's performance and memory use is already acceptable, particularly for a prototype. And I'm confident that if PEP 649 is accepted, the core dev community will find endless ways to optimize the implementation.
As far as I remember, the reference implementation created a function object for each methods.
First, it's only for methods with annotations. Second, it only stores a code object on the annotated function; the co_annotations function is lazily created. The exception to this is when co_annotations needs a closure; in that case, it currently creates and binds the co_annotation function non-lazily. Obviously this could be done lazily too, I didn't bother for the prototype. //arry/
On Wed, 11 Aug 2021 at 07:37, Larry Hastings <larry@hastings.org> wrote:
[...]
I think PEP 649 should be considered in the same way. In my opinion, the important thing is to figure out what semantics we want for the language.
I think this is a very important point. As originally implemented, annotations allowed attaching arbitrary data to function arguments and return values. That was a simple language capability, with no particular nuances, although the intended use case was very much type annotations. Later, the decision was made that typing would be the primary and "official" use for annotations - but even at that point, the semantics were still impartial. String annotations are changing the fundamental semantics of annotations (introspection returns the source code string, not a Python object) and there's even discussion of how this allows the possibility for annotations to not even conform to standard Python syntax. Surely, before we start changing the fundamental syntax and semantics of a language feature for a single (albeit officially blessed) use case, we should look at what capabilities *in the abstract* are missing from the current behaviour, and work out how or if we can implement them. Proposers on python-ideas are routinely asked to demonstrate how their suggestion affects cases other than their particular use case. Why shouldn't we hold typing to a similar PEP 563 states that it's addressing two issues: * Forward references for type hints * Module import time cost of having type hints at all Both of those are real and valid problems that deserve a solution. PEP 563 is asserting that no solution is possible within the existing semantics, and it's necessary to significantly break backward compatibility in terms of semantics, and also opens up the possibility of annotations not even being normal Python syntax in future (the PEP says that any such change would need a further PEP, but definitely does *not* say that the possibility is out of the question). PEP 649, in my opinion, is exploring the option of addressing these two issues *within Python's existing syntax and semantics*. How we look at it IMO depends significantly on how we view the whole question of typing and what it means for typing to be "optional". Personally, my experience is that typing is indeed optional for small or personal projects. But for larger projects, where there's a team of people involved, it's very hard to argue against typing. Even if you're not convinced personally that the benefits justify the costs, it's hard to make that case persuasive, so typing pretty much always wins. With that in mind, if we *are* saying (as PEP 563 is, in effect) that we're at the breaking point where we cannot support the typing use case with existing Python syntax (note that PEP 563 essentially makes annotations a new syntax for string literals[1]) then I would far prefer it if Python gave up on this ambivalent attitude towards annotations, and made a proper commitment to having a way of expressing types within the language. That means defining type syntax as Python *language* syntax and treating type syntax as subject to all of the compatibility guarantees that the rest of the language adheres to. It may still mean that we define new, type-specific syntax, but we can put it in the language definition, and say that it's only valid in annotations. That's not a showstopper. And it means that we don't open ourselves up to tool-specific interpretation of core Python syntax. What that would mean is: 1. What is valid as an annotation would probably need to be specified once and for all at the language level. That's a compatibility break, but we'd be making it with our eyes open, having found that the "a single Python expression" approach isn't sufficient for the typing use case. 2. The forward reference issue is treated as a *python language* issue, not just a typing issue. It may be that we decide to only *solve* it for typing, but we can still look to the wider context for ideas and inspiration. 3. The idea of having "special syntax" for type expressions would be dropped - it's not "special syntax" if it's part of the language, any more than assignment expressions or conditional expressions are "special syntax". It's possible that this is something that's already been discussed in the typing community. But if so, then that's a discussion that should have involved python-dev, as it's fairly basic to how Python as a language is evolving. On the other hand, PEP 647 is saying that we think that typing can still be supported with existing syntax, and we don't need to (yet) have that discussion. Although maybe the typing experts have further plans (the whole "support non-Python syntax" hint suggests that they might) that mean new syntax is an inevitability - but if so, let's get that proposal out in the open where it can be discussed, and not use PEP 563 as a "back door" to avoid that discussion. [...]
Once we figure out what semantics we want, we should implement them, and only then should we start worrying about performance. Fretting about performance at this point is premature and a distraction.
For the forward reference question, definitely. But PEP 563 also states that it's addressing the issue that annotations have a runtime cost - and as an alternative to PEP 563, PEP 649 needs to be clear on what it's position is on that issue. I'd have sympathy for the argument "any non-trivial type annotations need the user to import the typing module, so make the runtime cost of *that* import go away, and then we'll talk". Also, I don't think that improving performance is a justification for a non-trivial backward compatibility break (I don't recall a case where we've taken that view in the past) so "PEP 649 solves forward references without a backward compatibility impact, and performance is a small issue in the face of that" is a reasonable position to take. Paul
2021/08/11 19:22、Paul Moore <p.f.moore@gmail.com>のメール:
Also, I don't think that improving performance is a justification for a non-trivial backward compatibility break (I don't recall a case where we've taken that view in the past) so "PEP 649 solves forward references without a backward compatibility impact, and performance is a small issue in the face of that" is a reasonable position to take.
OK. I will stop talking about import time. But memory footprint and GC time is still an issue. Annotations in PEP 649 semantics can be much heavier than docstrings. So I want to measure them before deprecating 563. IMHO, I don't think we can make PEP 563 default anymore. FastAPI ecosystem changed how annotations are used after PEP 563 was accepted. So I think two or three options are on the table: a. Keep Python 3.10 state. PEP 563 is opt-in in foreseeable future. b. Accept PEP 649 and deprecate both of old semantics and PEP 563. c?. Accept PEP 649 and deprecate old semantics, but keep PEP 563 as opt-in. (I exclude "Accept PEP 649 but keep old semantics" option because backward compatibility is not a problem if we keep old semantics.) Of course, (b) is the most simple solution. But I am not sure that (b) don't have significant memory usage and GC time regression. We can keep (a) in Python 3.11 so we don't have to harry about accepting PEP 649. Regards,
On 8/11/21 5:21 AM, Inada Naoki wrote:
But memory footprint and GC time is still an issue. Annotations in PEP 649 semantics can be much heavier than docstrings.
I'm convinced that, if we accept PEP 649 (or something like it), we can reduce its CPU and memory consumption. Here's a slightly crazy idea I had this morning: what if we didn't unmarshall the code object for co_annotation during the initial import, but instead lazily loaded it on demand? The annotated object would retain knowledge of what .pyc file to load, and what offset the co_annotation code object was stored at. (And, if the co_annotations function had to be a closure, a reference to the closure tuple.) If the user requested __annotations__ (or __co_annotations__), the code would open the .pyc file, unmarshall it, bind it, etc. Obviously this would only work for code loaded from .pyc (etc) files. To go even crazier, the runtime could LRU cache N (maybe == 1) open .pyc file handles as a speed optimization, perhaps closing them after some wall-clock timeout. I doubt we'd do exactly this--it's easy to find problems with the approach. But maybe this idea will lead to a better one? //arry/
As it happens, I have a working prototype of lazy in marshaling that would work well for this. On Wed, Aug 11, 2021 at 06:07 Larry Hastings <larry@hastings.org> wrote:
On 8/11/21 5:21 AM, Inada Naoki wrote:
But memory footprint and GC time is still an issue. Annotations in PEP 649 semantics can be much heavier than docstrings.
I'm convinced that, if we accept PEP 649 (or something like it), we can reduce its CPU and memory consumption.
Here's a slightly crazy idea I had this morning: what if we didn't unmarshall the code object for co_annotation during the initial import, but instead lazily loaded it on demand? The annotated object would retain knowledge of what .pyc file to load, and what offset the co_annotation code object was stored at. (And, if the co_annotations function had to be a closure, a reference to the closure tuple.) If the user requested __annotations__ (or __co_annotations__), the code would open the .pyc file, unmarshall it, bind it, etc. Obviously this would only work for code loaded from .pyc (etc) files. To go even crazier, the runtime could LRU cache N (maybe == 1) open .pyc file handles as a speed optimization, perhaps closing them after some wall-clock timeout.
I doubt we'd do exactly this--it's easy to find problems with the approach. But maybe this idea will lead to a better one?
*/arry* _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JMLXOEV6... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Lazy loading code object solves only a half of the problem. I am worrying about function objects for annotation too. Function objects are heavier than code objects. And they are GC-tracked objects. I want to know how we can reduce the function objects created for annotation in PEP 649, before deprecating PEP 563. -- Inada Naoki <songofacandy@gmail.com>
Maybe we could specialize the heck out of this and not bother with a function object? In the end we want to execute the code, the function object is just a convenient way to bundle defaults, free variables (cells) and globals. But co_annotation has no arguments or defaults, and is only called once. It does need to have access to the globals of the definition site (the call site may be in another module) and sometimes there are cells (not sure). On Thu, Aug 12, 2021 at 1:14 AM Inada Naoki <songofacandy@gmail.com> wrote:
Lazy loading code object solves only a half of the problem. I am worrying about function objects for annotation too.
Function objects are heavier than code objects. And they are GC-tracked objects.
I want to know how we can reduce the function objects created for annotation in PEP 649, before deprecating PEP 563.
-- Inada Naoki <songofacandy@gmail.com>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 8/12/21 8:25 AM, Guido van Rossum wrote:
Maybe we could specialize the heck out of this and not bother with a function object? In the end we want to execute the code, the function object is just a convenient way to bundle defaults, free variables (cells) and globals. But co_annotation has no arguments or defaults, and is only called once. It does need to have access to the globals of the definition site (the call site may be in another module) and sometimes there are cells (not sure).
Yes, there are sometimes cells. def foo(): my_type = int def bar(a:my_type): return a return bar Both bar() and the co_annotations function for bar() are nested functions inside foo, and the latter has to refer to the cell for my_type. Also, co_annotations on a class may keep a reference to the locals dict, permitting annotations to refer to values defined in the class. I don't know if it's worth making a specialized object as you suggest. I'd forgotten I did this, but late in the development of the 649 prototype, I changed it so the co_annotations function is /always/ lazy-bound, only constructed on demand. There are three possible blobs of information the descriptor might need when binding the co_annotations function: * the co_annotations code object, * the tuple of cells, when co_annotations is a closure, and * the locals dict, when co_annotations is defined inside a class and might refer to names defined in the class. The code object is always necessary, the other two are optional. If we only need the code object, we store that code object inside the function object and we're done. If we need either or both of the other two blobs, we throw all the blobs we need in a tuple and store the tuple instead. At the point that someone asks for the annotations on that object, the descriptor does PyType_ checks to determine which blobs of data it has, binds the function appropriately, calls it, and returns the result. I think this approach is reasonable and I'm not sure what a custom callable object would get us. //arry/
I will try to find time to review the code. On Thu, Aug 12, 2021 at 08:56 Larry Hastings <larry@hastings.org> wrote:
On 8/12/21 8:25 AM, Guido van Rossum wrote:
Maybe we could specialize the heck out of this and not bother with a function object? In the end we want to execute the code, the function object is just a convenient way to bundle defaults, free variables (cells) and globals. But co_annotation has no arguments or defaults, and is only called once. It does need to have access to the globals of the definition site (the call site may be in another module) and sometimes there are cells (not sure).
Yes, there are sometimes cells.
def foo(): my_type = int def bar(a:my_type): return a return bar
Both bar() and the co_annotations function for bar() are nested functions inside foo, and the latter has to refer to the cell for my_type. Also, co_annotations on a class may keep a reference to the locals dict, permitting annotations to refer to values defined in the class.
I don't know if it's worth making a specialized object as you suggest. I'd forgotten I did this, but late in the development of the 649 prototype, I changed it so the co_annotations function is *always* lazy-bound, only constructed on demand.
There are three possible blobs of information the descriptor might need when binding the co_annotations function:
- the co_annotations code object, - the tuple of cells, when co_annotations is a closure, and - the locals dict, when co_annotations is defined inside a class and might refer to names defined in the class.
The code object is always necessary, the other two are optional. If we only need the code object, we store that code object inside the function object and we're done. If we need either or both of the other two blobs, we throw all the blobs we need in a tuple and store the tuple instead. At the point that someone asks for the annotations on that object, the descriptor does PyType_ checks to determine which blobs of data it has, binds the function appropriately, calls it, and returns the result. I think this approach is reasonable and I'm not sure what a custom callable object would get us.
*/arry*
-- --Guido (mobile)
On Aug 9, 2021, at 08:27, Eric V. Smith <eric@trueblade.com> wrote:
My understanding is that PEP 649 is currently in front of the SC. But do we need to have any additional discussion here? My recollection is that we backed out the PEP 563 change because we didn't feel we had enough time to come to a good decision one way or the other before 3.10.
My recollection for why the SC deferred this (other than running out of time and not wanting to make a hasty decision for 3.10) is two fold: * Should the type annotation syntax always follow Python syntax, or should we allow the type annotation syntax to deviate from “standard” Python? This also came up in the context of PEP 646, which introduces syntax for type annotations that may or may not be useful or desired for regular Python. * Should we codify current practice so that type annotations are useful and supported at both compile time and run time (e.g. the pydantic use case)? These are still open questions. While it’s interesting to entertain the separation of general Python syntax from type annotation syntax — allowing the latter to evolve more quickly and independently from the former -- ultimately I think (and my gauge of the SC sentiment is) that there’s no way this will work in practice. That means that type annotation syntax can’t be delegated and so we have to consider type-inspired syntax changes within the full context of the Python language. As for the second question, it means that we have to be very careful that the folks who use type annotation at compile/static checking time (e.g. mypy and friends) explicitly consider the existing use cases and needs of the runtime type community. These two constituents have to work together to avoid backward incompatible changes. When the SC was considering all these PEPs for 3.10, we felt like we needed clarity on the direction and purpose of type annotations before we could make decisions we’d have to live with essentially forever. Cheers, -Barry
On Wed, Aug 11, 2021 at 2:13 PM Barry Warsaw <barry@python.org> wrote:
As for the second question, it means that we have to be very careful that the folks who use type annotation at compile/static checking time (e.g. mypy and friends) explicitly consider the existing use cases and needs of the runtime type community. These two constituents have to work together to avoid backward incompatible changes.
Very well put, thank you Barry. As I see it, the companies who did most of the investment in typing want the benefits static typing (as most of us do), but widespread use of type hints in large codebases created another problem: static checking only happens in developer workstations and CI servers, but *imports* happen all the time in code running in production, on countless servers. That became a real issue for those very same companies operating at Web scale. So they have a strong incentive to focus on the use of annotations for static checking only, while many of us also want type hints to address use cases where Python is used as a *dynamic* language, which is its nature, and likely a strong reason for its popularity in the first place—despite the inherent runtime costs of being a dynamic language. Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
participants (18)
-
Barry Warsaw -
Brett Cannon -
Carl Meyer -
Chris Angelico -
Damian Shaw -
Eric V. Smith -
Guido van Rossum -
Inada Naoki -
Jelle Zijlstra -
Jukka Lehtosalo -
Larry Hastings -
Luciano Ramalho -
Mark Shannon -
Paul Moore -
Steve Holden -
Terry Reedy -
Thomas Grainger -
Łukasz Langa