PEP 563 and expensive backwards compatibility
I am generally supportive of leaving the type annotations unprocessed by default, but there are use cases where they should be processed (and even cases where doing it at the right time matters, because of a side effect). I am concerned that the backwards compatibility story for non-typing cases be not just possible, but reasonable. (1) The PEP suggests opting out with @typing.no_type_hints ... The closest I could find was @typing.no_type_check, which has to be called on each object. It should be possible to opt out for an entire module, and it should be possible to do so *without* first importing typing. Telling type checkers to ignore scopes (including modules) with a # typing.no_type_hints comment would be sufficient for me. If that isn't possible, please at least create a nontyping or minityping module so that the marker can be imported without the full overhead of the typing module. (2) Getting the annotations processed (preferably at the currently correct time) should also be possible on a module-wide basis, and should also not require importing the entire typing apparatus. It would be a bit messy (like the old coding cookie), but recognizing a module-wide # typing.no_type_hints comment and then falling back to the current behavior would be enough for me. Alternatively, it would be acceptable to run something like typing.get_type_hints, if that could be done in a single pass at the end of the module (callable from both within the module and from outside) ... but again, such a cleanup function should be in a smaller module that doesn't require loading all of typing. -jJ
On Wed, Sep 13, 2017 at 11:56 AM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
It should be possible to opt out for an entire module, and it should be possible to do so *without* first importing typing.
PEP 484 has a notation for this -- put # type: ignore at the top of your file and the file won't be type-checked. (Before you test this, mypy doesn't yet support this. But it could.) IIUC functions and classes will still have an __annotations__ attribute (except when it would be empty) so even with the __future__ import (or in Python 4.0) you could still make non-standard use of annotations pretty easily -- you'd just get a string rather than an object. (And a simple eval() will turn the string into an object -- the PEP has a lot of extra caution because currently the evaluation happens in the scope where the annotation is encountered, but if you don't care about that everything's easy.) -- --Guido van Rossum (python.org/~guido)
What is the "right time" you're speaking of?
This was a typo on my part. Yes, no_type_check is what I meant.
This is already possible. PEP 484 specifies that "A # type: ignore comment on a line by itself is equivalent to adding an inline # type: ignore to each line until the end of the current indented block. At top indentation level this has effect of disabling type checking until the end of file."
Again, what is the "correct time" you're speaking of?
Do you know of any other per-module feature toggle of this kind? __future__ imports are not feature toggles, they are timed deprecations. Finally, the non-typing use cases that you're worried about, what are they? From the research I've done, none of the actual use cases in existence would be rendered impossible by postponed evaluation. So far the concerns about side effects and local scope in annotations aren't supported by any strong evidence that this change would be disruptive. Don't get me wrong, I'm not being dismissive. I just don't think it's reasonable to get blocked on potential and obscure use cases that no real world code actually employs. - Ł
On Wed, Sep 13, 2017 at 3:12 PM, Lukasz Langa <lukasz@langa.pl> wrote:
On Sep 13, 2017, at 2:56 PM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
What is the "right time" you're speaking of?
The "right time" is whenever they are currently evaluated. (Definition time, I think, but won't swear.) For example, the "annotation" might really be a call to a logger, showing the current environment, including names that will be rebound before the module finishes loading. I'm perfectly willing to agree that even needing this much control over timing is a code smell, but it is currently possible, and I would rather it not become impossible. At a minimum, it seems like "just run this typing function that you should already be using" should either save the right context, or the PEP should state explicitly that this functionality is being withdrawn. (And go ahead and suggest a workaround, such as running the code before the method definition, or as a decorator.)
(1) The PEP suggests opting out with @typing.no_type_hints ...
This is already possible. PEP 484 specifies that
Great! Please mention this as well as (or perhaps instead of) typing.no_type_check.
It would be a bit messy (like the old coding cookie), but recognizing a module-wide
# typing.no_type_hints
comment and then falling back to the current behavior would be enough for me.
Do you know of any other per-module feature toggle of this kind?
No, thus the comment about it being messy. But it does offer one way to ensure that annotations are evaluated within the proper environment, even without having to save those environments. -jJ
2017-09-13 13:01 GMT-07:00 Jim J. Jewett <jimjjewett@gmail.com>:
Is this just a theoretical concern? Unless there is significant real-world code doing this sort of thing, I don't see much of a problem in deprecating such code using the normal __future__-based deprecation cycle.
On 14 September 2017 at 06:01, Jim J. Jewett <jimjjewett@gmail.com> wrote:
I think it would be useful for the PEP to include a definition of an "eager annotations" decorator that did something like: def eager_annotations(f): ns = f.__globals__ annotations = f.__annotations__ for k, v in annotations.items(): annotations[k] = eval(v, ns) return f And pointed out that you can create variants of that which also pass in the locals() namespace (or use sys._getframes() to access it dynamically). That way, during the "from __future__ import lazy_annotations" period, folks will have clearer guidance on how to explicitly opt-in to eager evaluation via function and class decorators. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I like this idea! For classes it would have to be a function that you call post factum. The way class decorators are implemented, they cannot evaluate annotations that contain forward references. For example: class Tree: left: Tree right: Tree def __init__(self, left: Tree, right: Tree): self.left = left self.right = right This is true today, get_type_hints() called from within a class decorator will fail on this class. However, a function performing postponed evaluation can do this without issue. If a class decorator knew what name a class is about to get, that would help. But that's a different PEP and I'm not writing that one ;-) - Ł
On 14 September 2017 at 09:43, Lukasz Langa <lukasz@langa.pl> wrote:
The class decorator case is indeed a bit more complicated, but there are a few tricks available to create a forward-reference friendly evaluation environment. 1. To get the right globals namespace, you can do: global_ns = sys.modules[cls.__module__].__dict__ 2. Define the evaluation locals as follows: local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__) 3. Evaluate the variable and method annotations using "eval(expr, global_ns, local_ns)" If you make the eager annotation evaluation recursive (so the decorator can be applied to the outermost class, but also affects all inner class definitions), then it would even be sufficient to allow nested classes to refer to both the outer class as well as other inner classes (regardless of definition order). To prevent inadvertent eager evaluation of annotations on functions and classes that are merely referenced from a class attribute, the recursive descent would need to be conditional on "attr.__qualname__ == cls.__qualname__ + '.' + attr.__name__". So something like: def eager_class_annotations(cls): global_ns = sys.modules[cls.__module__].__dict__ local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__) annotations = cls.__annotations__ for k, v in annotations.items(): annotations[k] = eval(v, global_ns, local_ns) for attr in cls.__dict__.values(): name = getattr(attr, "__name__", None) if name is None: continue qualname = getattr(attr, "__qualname__", None) if qualname is None: continue if qualname != f"{cls.__qualname}.{name}": continue if isinstance(attr, type): eager_class_annotations(attr) else: eager_annotations(attr) return cls You could also hide the difference between eager annotation evaluation on a class or a function inside a single decorator: def eager_annotations(obj): if isinstance(obj, type): _eval_class_annotations(obj) # Class elif hasattr(obj, "__globals__"): _eval_annotations(obj, obj.__globals__) # Function else: _eval_annotations(obj, obj.__dict__) # Module return obj Given the complexity of the class decorator variant, I now think it would actually make sense for the PEP to propose *providing* these decorators somewhere in the standard library (the lower level "types" module seems like a reasonable candidate, but we've historically avoided having that depend on the full collections module) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Using cls.__name__ and the ChainMap is clever, I like it. It might prove useful for Eric's data classes later. However, there's more to forward references than self-references: class A: b: B class B: ... In this scenario evaluation of A's annotations has to happen after the module is fully loaded. This is the general case. No magic decorator will solve this. The general solution is running eval() later, when the namespace is fully populated. I do agree with you that a default implementation of a typing-agnostic variant of `get_type_hints()` would be nice. If anything, implementing this might better surface limitations of postponed annotations. That function won't be recursive though as your example. And I'll leave converting the function to a decorator as an exercise for the reader, especially given the forward referencing caveats. - Ł
On Wed, Sep 13, 2017 at 11:56 AM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
It should be possible to opt out for an entire module, and it should be possible to do so *without* first importing typing.
PEP 484 has a notation for this -- put # type: ignore at the top of your file and the file won't be type-checked. (Before you test this, mypy doesn't yet support this. But it could.) IIUC functions and classes will still have an __annotations__ attribute (except when it would be empty) so even with the __future__ import (or in Python 4.0) you could still make non-standard use of annotations pretty easily -- you'd just get a string rather than an object. (And a simple eval() will turn the string into an object -- the PEP has a lot of extra caution because currently the evaluation happens in the scope where the annotation is encountered, but if you don't care about that everything's easy.) -- --Guido van Rossum (python.org/~guido)
What is the "right time" you're speaking of?
This was a typo on my part. Yes, no_type_check is what I meant.
This is already possible. PEP 484 specifies that "A # type: ignore comment on a line by itself is equivalent to adding an inline # type: ignore to each line until the end of the current indented block. At top indentation level this has effect of disabling type checking until the end of file."
Again, what is the "correct time" you're speaking of?
Do you know of any other per-module feature toggle of this kind? __future__ imports are not feature toggles, they are timed deprecations. Finally, the non-typing use cases that you're worried about, what are they? From the research I've done, none of the actual use cases in existence would be rendered impossible by postponed evaluation. So far the concerns about side effects and local scope in annotations aren't supported by any strong evidence that this change would be disruptive. Don't get me wrong, I'm not being dismissive. I just don't think it's reasonable to get blocked on potential and obscure use cases that no real world code actually employs. - Ł
On Wed, Sep 13, 2017 at 3:12 PM, Lukasz Langa <lukasz@langa.pl> wrote:
On Sep 13, 2017, at 2:56 PM, Jim J. Jewett <jimjjewett@gmail.com> wrote:
What is the "right time" you're speaking of?
The "right time" is whenever they are currently evaluated. (Definition time, I think, but won't swear.) For example, the "annotation" might really be a call to a logger, showing the current environment, including names that will be rebound before the module finishes loading. I'm perfectly willing to agree that even needing this much control over timing is a code smell, but it is currently possible, and I would rather it not become impossible. At a minimum, it seems like "just run this typing function that you should already be using" should either save the right context, or the PEP should state explicitly that this functionality is being withdrawn. (And go ahead and suggest a workaround, such as running the code before the method definition, or as a decorator.)
(1) The PEP suggests opting out with @typing.no_type_hints ...
This is already possible. PEP 484 specifies that
Great! Please mention this as well as (or perhaps instead of) typing.no_type_check.
It would be a bit messy (like the old coding cookie), but recognizing a module-wide
# typing.no_type_hints
comment and then falling back to the current behavior would be enough for me.
Do you know of any other per-module feature toggle of this kind?
No, thus the comment about it being messy. But it does offer one way to ensure that annotations are evaluated within the proper environment, even without having to save those environments. -jJ
2017-09-13 13:01 GMT-07:00 Jim J. Jewett <jimjjewett@gmail.com>:
Is this just a theoretical concern? Unless there is significant real-world code doing this sort of thing, I don't see much of a problem in deprecating such code using the normal __future__-based deprecation cycle.
On 14 September 2017 at 06:01, Jim J. Jewett <jimjjewett@gmail.com> wrote:
I think it would be useful for the PEP to include a definition of an "eager annotations" decorator that did something like: def eager_annotations(f): ns = f.__globals__ annotations = f.__annotations__ for k, v in annotations.items(): annotations[k] = eval(v, ns) return f And pointed out that you can create variants of that which also pass in the locals() namespace (or use sys._getframes() to access it dynamically). That way, during the "from __future__ import lazy_annotations" period, folks will have clearer guidance on how to explicitly opt-in to eager evaluation via function and class decorators. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I like this idea! For classes it would have to be a function that you call post factum. The way class decorators are implemented, they cannot evaluate annotations that contain forward references. For example: class Tree: left: Tree right: Tree def __init__(self, left: Tree, right: Tree): self.left = left self.right = right This is true today, get_type_hints() called from within a class decorator will fail on this class. However, a function performing postponed evaluation can do this without issue. If a class decorator knew what name a class is about to get, that would help. But that's a different PEP and I'm not writing that one ;-) - Ł
On 14 September 2017 at 09:43, Lukasz Langa <lukasz@langa.pl> wrote:
The class decorator case is indeed a bit more complicated, but there are a few tricks available to create a forward-reference friendly evaluation environment. 1. To get the right globals namespace, you can do: global_ns = sys.modules[cls.__module__].__dict__ 2. Define the evaluation locals as follows: local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__) 3. Evaluate the variable and method annotations using "eval(expr, global_ns, local_ns)" If you make the eager annotation evaluation recursive (so the decorator can be applied to the outermost class, but also affects all inner class definitions), then it would even be sufficient to allow nested classes to refer to both the outer class as well as other inner classes (regardless of definition order). To prevent inadvertent eager evaluation of annotations on functions and classes that are merely referenced from a class attribute, the recursive descent would need to be conditional on "attr.__qualname__ == cls.__qualname__ + '.' + attr.__name__". So something like: def eager_class_annotations(cls): global_ns = sys.modules[cls.__module__].__dict__ local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__) annotations = cls.__annotations__ for k, v in annotations.items(): annotations[k] = eval(v, global_ns, local_ns) for attr in cls.__dict__.values(): name = getattr(attr, "__name__", None) if name is None: continue qualname = getattr(attr, "__qualname__", None) if qualname is None: continue if qualname != f"{cls.__qualname}.{name}": continue if isinstance(attr, type): eager_class_annotations(attr) else: eager_annotations(attr) return cls You could also hide the difference between eager annotation evaluation on a class or a function inside a single decorator: def eager_annotations(obj): if isinstance(obj, type): _eval_class_annotations(obj) # Class elif hasattr(obj, "__globals__"): _eval_annotations(obj, obj.__globals__) # Function else: _eval_annotations(obj, obj.__dict__) # Module return obj Given the complexity of the class decorator variant, I now think it would actually make sense for the PEP to propose *providing* these decorators somewhere in the standard library (the lower level "types" module seems like a reasonable candidate, but we've historically avoided having that depend on the full collections module) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Using cls.__name__ and the ChainMap is clever, I like it. It might prove useful for Eric's data classes later. However, there's more to forward references than self-references: class A: b: B class B: ... In this scenario evaluation of A's annotations has to happen after the module is fully loaded. This is the general case. No magic decorator will solve this. The general solution is running eval() later, when the namespace is fully populated. I do agree with you that a default implementation of a typing-agnostic variant of `get_type_hints()` would be nice. If anything, implementing this might better surface limitations of postponed annotations. That function won't be recursive though as your example. And I'll leave converting the function to a decorator as an exercise for the reader, especially given the forward referencing caveats. - Ł
participants (5)
-
Guido van Rossum -
Jelle Zijlstra -
Jim J. Jewett -
Lukasz Langa -
Nick Coghlan