Proposal for @typing.override
Thanks to everyone who provided feedback about an override decorator in last week’s meetup!
We have finished an initial draft proposal [1]. We’re happy to take comments and suggestions on this doc; we also looking for a sponsor so we can put a PR together.
We also specifically wanted to call out some edge cases that might be worth addressing in the “Specification” section related to how override interacts with regular attributes. The main override-specific question is whether we care about a way to mark attribute assignments as overrides, since they are subject to the same renaming bugs.
But that brings up a few more edge cases such as properties and methods that override attributes. We’ve drawn up a doc [2] highlighting all these edge cases and would love to hear feedback.
- Joshua Xu and Steven Troxler
------------------
[1] Draft PEP for @typing.override https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH...
[2] Edge cases related to attributes https://docs.google.com/document/d/1dhlcKKuYtGN9Eftr1hu9U6EKzsS0NPuQm01b7brj...
[3] Slides from last week's presentation https://drive.google.com/drive/folders/1sGZ0WBLaVBkCb0AT7tneCEwbATxgisI2
[4] Original thread regarding override https://mail.python.org/archives/list/typing-sig@python.org/thread/V23I4D6DE...
Hi all -
Thanks for all the helpful suggestions on this proposal! One question has come up that I wanted to bring back to the full typing-sig list:
We are thinking about what kind of runtime logic `@typing.override` should have.
Jelle suggested that it should set an `__override__` attribute that would allow introspection. This seems like an obviously good idea and we've added it to our reference implementation [1][2] as well as the draft proposal.
Several folks also suggested that we consider adapting the existing runtime enforcement in `@overrides.overrides`.
This could be very cool, because override safety for renames makes sense even for folks who don't use static typing, plus it may help folks who run unit test more often than type checking get feedback faster.
The main downside is that it would add a bit of import-time overhead; my estimate is something like 120 microseconds per override [3]. This is relatively minor, although for a million line codebase we think it might introduce something like 0.75 seconds of fixed cost import time. It may be possible to reduce this significantly by moving the implementation to C.
From the Pyre team's side, I'm not sure we would want runtime enforcement on our largest codebases since the type checker can catch errors just as well, but we already have @pyre_extensions.override so this is not a major concern.
Do y'all have opinions on this? For now, I've moved the idea into the "Open Issues" section of the draft proposal [4]
---
[1] Add `__override__` attribute in `pyre_extensions` https://github.com/facebook/pyre-check/blob/main/pyre_extensions/__init__.py...
[2] Unit tests for `__override__` behavior https://github.com/facebook/pyre-check/blob/main/pyre_extensions/tests/basic...
[3] Simple perf estimate for `@overrides.overrides` https://github.com/stroxler/python_open_source_examples/tree/main/timing_ove...
[4] Draft PEP for @typing.override https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH...
El jue, 18 ago 2022 a las 19:15, Steven Troxler (steven.troxler@gmail.com) escribió:
Hi all -
Thanks for all the helpful suggestions on this proposal! One question has come up that I wanted to bring back to the full typing-sig list:
We are thinking about what kind of runtime logic `@typing.override` should have.
Jelle suggested that it should set an `__override__` attribute that would allow introspection. This seems like an obviously good idea and we've added it to our reference implementation [1][2] as well as the draft proposal.
Several folks also suggested that we consider adapting the existing runtime enforcement in `@overrides.overrides`.
This could be very cool, because override safety for renames makes sense even for folks who don't use static typing, plus it may help folks who run unit test more often than type checking get feedback faster.
The main downside is that it would add a bit of import-time overhead; my estimate is something like 120 microseconds per override [3]. This is relatively minor, although for a million line codebase we think it might introduce something like 0.75 seconds of fixed cost import time. It may be possible to reduce this significantly by moving the implementation to C.
From the Pyre team's side, I'm not sure we would want runtime enforcement on our largest codebases since the type checker can catch errors just as well, but we already have @pyre_extensions.override so this is not a major concern.
Do y'all have opinions on this? For now, I've moved the idea into the "Open Issues" section of the draft proposal [4]
I would oppose runtime validation because I see no clean way to implement it. The @override decorator runs during the execution of the class namespace, which is before the class actually exists, so it's not obvious how validation could work. The overrides library does it by introspecting the bytecode of the calling frame to figure out the class's bases ( https://github.com/mkorpela/overrides/blob/9d0356b5476489bb4fbe7179e30c3d47f...), and that's not something I'd want to see in typing.py. Bytecode introspection is only going to get more fragile in the future as the faster CPython team tinkers with the interpreter.
There are alternative approaches, but I don't like them either. A metaclass (or __init_subclass__ override) could run code after the class is created, introspect whether any class member has the __override__ attribute, and check that it is indeed an override. But I wouldn't want to force every user of the feature to add a special metaclass or __init_subclass__ to their class to use @overrides. We could also implement this behavior in the core language in the default metaclass (type), but then we'd slow down the creation of *all* classes, and I'm not sure the feature is sufficiently compelling to justify a change in the language core.
[1] Add `__override__` attribute in `pyre_extensions`
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/__init__.py...
[2] Unit tests for `__override__` behavior
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/tests/basic...
[3] Simple perf estimate for `@overrides.overrides`
https://github.com/stroxler/python_open_source_examples/tree/main/timing_ove...
[4] Draft PEP for @typing.override
https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH. .. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Well, ‘@abstract’ requires a meta class, so there’s some precedent. I won’t insist however.
On Thu, Aug 18, 2022 at 19:31 Jelle Zijlstra jelle.zijlstra@gmail.com wrote:
El jue, 18 ago 2022 a las 19:15, Steven Troxler (steven.troxler@gmail.com) escribió:
Hi all -
Thanks for all the helpful suggestions on this proposal! One question has come up that I wanted to bring back to the full typing-sig list:
We are thinking about what kind of runtime logic `@typing.override` should have.
Jelle suggested that it should set an `__override__` attribute that would allow introspection. This seems like an obviously good idea and we've added it to our reference implementation [1][2] as well as the draft proposal.
Several folks also suggested that we consider adapting the existing runtime enforcement in `@overrides.overrides`.
This could be very cool, because override safety for renames makes sense even for folks who don't use static typing, plus it may help folks who run unit test more often than type checking get feedback faster.
The main downside is that it would add a bit of import-time overhead; my estimate is something like 120 microseconds per override [3]. This is relatively minor, although for a million line codebase we think it might introduce something like 0.75 seconds of fixed cost import time. It may be possible to reduce this significantly by moving the implementation to C.
From the Pyre team's side, I'm not sure we would want runtime enforcement on our largest codebases since the type checker can catch errors just as well, but we already have @pyre_extensions.override so this is not a major concern.
Do y'all have opinions on this? For now, I've moved the idea into the "Open Issues" section of the draft proposal [4]
I would oppose runtime validation because I see no clean way to implement it. The @override decorator runs during the execution of the class namespace, which is before the class actually exists, so it's not obvious how validation could work. The overrides library does it by introspecting the bytecode of the calling frame to figure out the class's bases ( https://github.com/mkorpela/overrides/blob/9d0356b5476489bb4fbe7179e30c3d47f...), and that's not something I'd want to see in typing.py. Bytecode introspection is only going to get more fragile in the future as the faster CPython team tinkers with the interpreter.
There are alternative approaches, but I don't like them either. A metaclass (or __init_subclass__ override) could run code after the class is created, introspect whether any class member has the __override__ attribute, and check that it is indeed an override. But I wouldn't want to force every user of the feature to add a special metaclass or __init_subclass__ to their class to use @overrides. We could also implement this behavior in the core language in the default metaclass (type), but then we'd slow down the creation of *all* classes, and I'm not sure the feature is sufficiently compelling to justify a change in the language core.
[1] Add `__override__` attribute in `pyre_extensions`
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/__init__.py...
[2] Unit tests for `__override__` behavior
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/tests/basic...
[3] Simple perf estimate for `@overrides.overrides`
https://github.com/stroxler/python_open_source_examples/tree/main/timing_ove...
[4] Draft PEP for @typing.override
https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH. .. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
On Thu, Aug 18, 2022 at 7:31 PM Jelle Zijlstra jelle.zijlstra@gmail.com wrote:
El jue, 18 ago 2022 a las 19:15, Steven Troxler (steven.troxler@gmail.com) escribió:
Hi all -
Thanks for all the helpful suggestions on this proposal! One question has come up that I wanted to bring back to the full typing-sig list:
We are thinking about what kind of runtime logic `@typing.override` should have.
Jelle suggested that it should set an `__override__` attribute that would allow introspection. This seems like an obviously good idea and we've added it to our reference implementation [1][2] as well as the draft proposal.
Several folks also suggested that we consider adapting the existing runtime enforcement in `@overrides.overrides`.
This could be very cool, because override safety for renames makes sense even for folks who don't use static typing, plus it may help folks who run unit test more often than type checking get feedback faster.
The main downside is that it would add a bit of import-time overhead; my estimate is something like 120 microseconds per override [3]. This is relatively minor, although for a million line codebase we think it might introduce something like 0.75 seconds of fixed cost import time. It may be possible to reduce this significantly by moving the implementation to C.
From the Pyre team's side, I'm not sure we would want runtime enforcement on our largest codebases since the type checker can catch errors just as well, but we already have @pyre_extensions.override so this is not a major concern.
Do y'all have opinions on this? For now, I've moved the idea into the "Open Issues" section of the draft proposal [4]
I would oppose runtime validation because I see no clean way to implement it. The @override decorator runs during the execution of the class namespace, which is before the class actually exists, so it's not obvious how validation could work. The overrides library does it by introspecting the bytecode of the calling frame to figure out the class's bases ( https://github.com/mkorpela/overrides/blob/9d0356b5476489bb4fbe7179e30c3d47f...), and that's not something I'd want to see in typing.py. Bytecode introspection is only going to get more fragile in the future as the faster CPython team tinkers with the interpreter.
There are alternative approaches, but I don't like them either. A metaclass (or __init_subclass__ override) could run code after the class is created, introspect whether any class member has the __override__ attribute, and check that it is indeed an override. But I wouldn't want to force every user of the feature to add a special metaclass or __init_subclass__ to their class to use @overrides. We could also implement this behavior in the core language in the default metaclass (type), but then we'd slow down the creation of *all* classes, and I'm not sure the feature is sufficiently compelling to justify a change in the language core.
FWIW: I agree that runtime enforcement isn't worth it.
Here at Google, someone created a library that uses the metaclass approach. Usage of it never really caught on.
Implementation wise, there's a decent amount of subtle edge cases. I ended up owner of it, and I don't like going into that code because it's a deep dive into low-level details of the Python object model and various CPython details that show up. It didn't try to enforce type compatibility, so only enforced the signature sans type annotations. I'm not saying its *impossible *(well...there might be a couple cases for which there isn't a single good answer...), just not pleasant.
As a user, the runtime enforcement was actually rather annoying when you're trying to work out changes to an interface. I'd often just disable enforcement entirely; it was faster and easier than copy/pasting the changes to N other files that didn't need the changes while I experimented. This sort of thing isn't a problem with a type checker because that runs separately. When you're reading to check types, you run the type checker; until then you can hack away as you please.
Runtime enforcement could, of course, be added later. I am skeptical that the runtime behaviors/features can be on par with the static-analysis behaviors/features. For example, things like enforcing type compatibility have the long-tail issues stemming from getting type annotations at runtime.
[1] Add `__override__` attribute in `pyre_extensions`
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/__init__.py...
[2] Unit tests for `__override__` behavior
https://github.com/facebook/pyre-check/blob/main/pyre_extensions/tests/basic...
[3] Simple perf estimate for `@overrides.overrides`
https://github.com/stroxler/python_open_source_examples/tree/main/timing_ove...
[4] Draft PEP for @typing.override
https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH. .. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: richardlev@gmail.com
The proposal looks good to me. I think this feature adds clear value, provides easy-to-understand semantics, follows a precedent established in many other languages, composes well with existing features, and should be straightforward for Python type checkers, linters and other tools to implement.
-Eric
The number of animals kept by farmers and ranchers is substantial. They are primarily comprised of cows, pigs, sheep, and goats. The majority of the animals are used in the food sector, with the hides serving as a byproduct and a source of raw materials for the <a href="https://www.exporthub.com/textile-leather-products/">Leather Products Manufacturers</a>.
The proposal as it stands now only works for methods (and closely related concepts like properties), but a common pattern I have seen is to use class variables to configure behavior in child classes. For example, this pattern shows up in the standard library's webbrowser module (simplified from https://github.com/python/cpython/blob/f177f6f29b069f522a0b3ba44eaae19852b6d... ):
class UnixBrowser(BaseBrowser): remote_args = ['%action', '%s']
def open(self, url): args = [arg.replace("%s", url) for arg in self.remote_args] self._invoke(args)
class Netscape(UnixBrowser): remote_args = ['-remote', 'openURL(%s%action)']
This could give rise to the same kind of issues that your draft mentions for methods: if you rename the base class attribute, you may get hard-to-find bugs. However, you can't decorate an attribute.
A solution could be to add a typing.Override special form, similar to Final:
class Netscape(UnixBrowser): remote_args: Override = ['-remote', 'openURL(%s%action)'] # also allowed: remote_args: Override[list[str]] = ['-remote', 'openURL(%s%action)']
El mié, 17 ago 2022 a las 10:23, Steven Troxler (steven.troxler@gmail.com) escribió:
Thanks to everyone who provided feedback about an override decorator in last week’s meetup!
We have finished an initial draft proposal [1]. We’re happy to take comments and suggestions on this doc; we also looking for a sponsor so we can put a PR together.
We also specifically wanted to call out some edge cases that might be worth addressing in the “Specification” section related to how override interacts with regular attributes. The main override-specific question is whether we care about a way to mark attribute assignments as overrides, since they are subject to the same renaming bugs.
But that brings up a few more edge cases such as properties and methods that override attributes. We’ve drawn up a doc [2] highlighting all these edge cases and would love to hear feedback.
- Joshua Xu and Steven Troxler
[1] Draft PEP for @typing.override
https://docs.google.com/document/d/1dBkHKaOCqttnqzPo7kw1dzvbuFzNoOdx7OPyfbJH...
[2] Edge cases related to attributes
https://docs.google.com/document/d/1dhlcKKuYtGN9Eftr1hu9U6EKzsS0NPuQm01b7brj...
[3] Slides from last week's presentation https://drive.google.com/drive/folders/1sGZ0WBLaVBkCb0AT7tneCEwbATxgisI2
[4] Original thread regarding override
https://mail.python.org/archives/list/typing-sig@python.org/thread/V23I4D6DE... _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Yes, we noticed this problem. It's the first question we called out in our Edge Cases doc: https://docs.google.com/document/d/1dhlcKKuYtGN9Eftr1hu9U6EKzsS0NPuQm01b7brj...
Guido and I weren't convinced that attribute overrides cause enough bugs for this to be a problem, but we could add an Override special form. We did note that `override / Override` would be a similar pattern to `final / Final`. I'd be okay with adding that.
One problem we noticed is that it seems to be underspecified how attributes and methods interact - e.g., is it legal to override an attribute with a method when the types are compatible? The answers vary by type checker. I think this PEP should avoid taking a stance on that, and simply say that overrides must override *something*, and compatibility rules are a separate question that should either remain divergent per type checker or be spelled out in separate proposals.
- Steven
One thing I remember noticing that concerned me about Override on attributes is whether we'd force them in strict mode.
An @override decorator does add some verbosity to code, but it lives on its own line and is syntactically quite separate from the important logic, I'm guessing there won't be strong objections to it (especially considering that most other popular object-oriented languages already require it).
An `Override[...]` special form, on the other hand, will add quite a bit of inline verbosity to types, plus the interaction with type inference in `__init__` may be strange, if `__init__` doesn't call `super` (which is a good practice but not required by the runtime or as far as I know by most type checkers).
So I could see people being less thrilled about a strict mode for attributes, I'm not sure how that would play out. It's certainly true that the same classes of bugs can be caught for attributes as for methods.
participants (6)
-
booblatin69＠gmail.com -
Eric Traut -
Guido van Rossum -
Jelle Zijlstra -
Richard Levasseur -
Steven Troxler