PEP: Deferred Evaluation Of Annotations Using Descriptors
I've written a new PEP. Please find it below. Happy reading! //arry/ ---------- PEP: XXXX Title: Deferred Evaluation Of Annotations Using Descriptors Version: $Revision$ Last-Modified: $Date$ Author: Larry Hastings <larry@hastings.org> Discussions-To: Python-Dev <python-dev@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Jan-2021 Abstract ======== As of Python 3.9, Python supports two different behaviors for annotations: * original Python semantics, in which annotations are evaluated at the time they are bound, and * PEP 563 semantics, currently enabled per-module by ``from __future__ import annotations``, in which annotations are converted back into strings and must be parsed by ``eval()`` to be used. Original Python semantics created a circular references problem for static typing analysis. PEP 563 solved that problem, but its novel semantics introduced new problems. This PEP proposes a third way that embodies the best of both previous approaches. It solves the same circular reference problems solved by PEP 563, while preserving Python's original straightforward runtime semantics for annotations. In this new approach, the code to generate the annotations dict is written to its own callable, and ``__annotations__`` is a "data descriptor" which calls the callable once and preserves the result. If accepted, these new semantics for annotations would initially be gated behind ``from __future__ import co_annotations``. However, these semantics would eventually be promoted to be the default behavior. Thus this PEP would *supercede* PEP 563, and PEP 563's behavior would be deprecated and eventually removed. Overview ======== .. note:: The code presented in this section is highly simplified for clarity. The intention is to communicate the high-level concepts involved without getting lost in with the details. The actual details are often quite different. See the Implementation_ section later in this PEP for a much more accurate description of how this PEP works. Consider this example code:: def foo(x: int = 3, y: MyType = None) -> float: ... class MyType: ... foo_y_type = foo.__annotations__['y'] As we see here, annotations are available at runtime through an ``__annotations__`` attribute on functions, classes, and modules. When annotations are specified on one of these objects, ``__annotations__`` is a dictionary mapping the names of the fields to the value specified as that field's annotation. The default behavior in Python 3.9 is to evaluate the expressions for the annotations, and build the annotations dict, at the time the function, class, or module is bound. At runtime the above code actually works something like this:: annotations = {'x': int, 'y': MyType, 'return': float} def foo(x = 3, y = "abc"): ... foo.__annotations__ = annotations class MyType: ... foo_y_type = foo.__annotations__['y'] The crucial detail here is that the values ``int``, ``MyType``, and ``float`` are looked up at the time the function object is bound, and these values are stored in the annotations dict. But this code doesn't run—it throws a ``NameError`` on the first line, because ``MyType`` hasn't been defined yet. PEP 563's solution is to decompile the expressions back into strings, and store those *strings* in the annotations dict. The equivalent runtime code would look something like this:: annotations = {'x': 'int', 'y': 'MyType', 'return': 'float'} def foo(x = 3, y = "abc"): ... foo.__annotations__ = annotations class MyType: ... foo_y_type = foo.__annotations__['y'] This code now runs successfully. However, ``foo_y_type`` is no longer a reference to ``MyType``, it is the *string* ``'MyType'``. The code would have to be further modified to call ``eval()`` or ``typing.get_type_hints()`` to convert the string into a useful reference to the actual ``MyType`` object. This PEP proposes a third approach, delaying the evaluation of the annotations by computing them in their own function. If this PEP was active, the generated code would work something like this:: class function: @property # __annotations__ on a function object is already a # "data descriptor", we're just changing what it does def __annotations__(self): return self.__co_annotations__() # ... def foo_annotations_fn(): return {'x': int, 'y': MyType, 'return': float} def foo(x = 3, y = "abc"): ... foo.__co_annotations__ = foo_annotations_fn class MyType: ... foo_y_type = foo.__annotations__['y'] The important change is that the code constructing the annotations dict now lives in a function—here, called `` foo_annotations__fn()``. But this function isn't called until we ask for the value of ``foo.__annotations__``, and we don't do that until *after* the definition of ``MyType``. So this code also runs successfully, and ``foo_y_type`` now has the correct value, the class ``MyType``. Motivation ========== Python's original semantics for annotations made its use for static type analysis painful due to forward reference problems. This was the main justification for PEP 563, and we need not revisit those arguments here. However, PEP 563's solution was to de-compile code for Python annotations back into strings at compile time, requiring users of annotations to ``eval()`` those strings to turn them back into Python values. This has several drawbacks: * It requires Python implementations to stringize their annotations. This is surprising—unprecedented behavior for a language-level feature. Also, adding this feature to CPython was complicated, and this complicated code would need to be reimplemented independently by every other Python implementation. * It requires a code change every time existing code uses an annotation, to handle converting the stringized annotation back into a useful value. * ``eval()`` is slow. * ``eval()`` isn't always available; it's sometimes removed from Python for space reasons. * In order to evaluate the annotations stored with a class, it requires obtaining a reference to that class's globals, which PEP 563 suggests should be done by looking up that class by name in ``sys.modules``—another surprising requirement for a language-level feature. * It adds an ongoing maintenance burden to Python implementations. Every time the language adds a new feature available in expressions, the implementation's stringizing code must be updated in tandem to support decompiling it. This PEP also solves the forward reference problem outlined in PEP 563 while avoiding the problems listed above: * Python implementations would generate annotations as code objects. This is simpler than stringizing, and is something Python implementations are already quite good at. This means: * alternate implementations would need to write less code to implement this feature, and * the implementation would be simpler overall, which should reduce its ongoing maintenance cost. * Code examining annotations at runtime would no longer need to use ``eval()`` or anything else—it would automatically get the correct values. This is easier, almost certainly faster, and removes the dependency on ``eval()``. Backwards Compatibility ======================= PEP 563 changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated in *module-level* scope. They may no longer refer directly to local variables or class attributes. This PEP retains that semantic change, also requiring that annotations be evaluated in *module-level* scope. Thus, code changed so its annotations are compatible with PEP 563 should *already* compatible with this aspect of this PEP and would not need further change. Modules still using stock semantics would have to be revised so its annotations evaluate properly in module-level scope, in the same way they would have to be to achieve compatibility with PEP 563. PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()`` to examine annotations. Code updated to work with PEP 563 that calls ``eval()`` directly would have to be updated simply to remove the ``eval()`` call. Code using ``typing.get_type_hints()`` would continue to work unchanged, though future use of that function would become optional in most cases. Because this PEP makes the same backwards-compatible change to annotation scoping as PEP 563, this PEP will be initially gated with a per-module ``from __future__ import co_annotations`` before it eventually becomes the default behavior. Apart from these two changes already discussed: * the evaluation of values in annotation dicts will be delayed until the ``__annotations__`` attribute is evaluated, and * annotations are now evaluated in module-level scope, this PEP preserves nearly all existing behavior of annotations dicts. Specifically: * Annotations dicts are mutable, and any changes to them are preserved. * The ``__annotations__`` attribute can be explicitly set, and any value set this way will be preserved. * The ``__annotations__`` attribute can be deleted using the ``del`` statement. However, there are two uncommon interactions possible with class and module annotations that work today—both with stock semantics, and with PEP 563 semantics—that would no longer work when this PEP was active. These two interactions would have to be prohibited. The good news is, neither is common, and neither is considered good practice. In fact, they're rarely seen outside of Python's own regression test suite. They are: * *Code that sets annotations from inside any kind of flow control statement.* It's currently possible to set module and class attributes with annotations inside an ``if`` or ``try`` statement, and it works as one would expect. It's untenable to support this behavior when this PEP is active. * *Code in module or class scope that references or modifies the local* ``__annotations__`` *dict directly.* Currently, when setting annotations on module or class attributes, the generated code simply creates a local ``__annotations__`` dict, then sets mappings in it as needed. It's also possible for user code to directly modify this dict, though this doesn't seem like it's an intentional feature. Although it'd be possible to support this after a fashion when this PEP was active, the semantics would likely be surprising and wouldn't make anyone happy. Note that these are both also pain points for static type checkers, and are unsupported by those checkers. It seems reasonable to declare that both are at the very least unsupported, and their use results in undefined behavior. It might be worth making a small effort to explicitly prohibit them with compile-time checks. There's one more idiom that's actually somewhat common when dealing with class annotations, and which will become more problematic when this PEP is active: code often accesses class annotations via ``cls.__dict__.get("__annotations__", {})`` rather than simply ``cls.__annotations__``. It's due to a flaw in the original design of annotations themselves. This topic will be examined in a separate discussion; the outcome of that discussion will likely guide the future evolution of this PEP. Mistaken Rejection Of This Approach In November 2017 ==================================================== During the early days of discussion around PEP 563, using code to delay the evaluation of annotations was briefly discussed, in a November 2017 thread in ``comp.lang.python-dev``. At the time the technique was termed an "implicit lambda expression". Guido van Rossum—Python's BDFL at the time—replied, asserting that these "implicit lambda expression" wouldn't work, because they'd only be able to resolve symbols at module-level scope: IMO the inability of referencing class-level definitions from annotations on methods pretty much kills this idea. https://mail.python.org/pipermail/python-dev/2017-November/150109.html This led to a short discussion about extending lambda-ized annotations for methods to be able to refer to class-level definitions, by maintaining a reference to the class-level scope. This idea, too, was quickly rejected. PEP 563 summarizes the above discussion here: https://www.python.org/dev/peps/pep-0563/#keeping-the-ability-to-use-functio... What's puzzling is PEP 563's own changes to the scoping rules of annotations—it *also* doesn't permit annotations to reference class-level definitions. It's not immediately clear why an inability to reference class-level definitions was enough to reject using "implicit lambda expressions" for annotations, but was acceptable for stringized annotations. In retrospect there was probably a pivot during the development of PEP 563. It seems that, early on, there was a prevailing assumption that PEP 563 would support references to class-level definitions. But by the time PEP 563 was finalized, this assumption had apparently been abandoned. And it looks like "implicit lambda expressions" were never reconsidered in this new light. PEP 563 semantics have shipped in three major Python releases. These semantics are now widely used in organizations depending on static type analysis. Evaluating annotations at module-level scope is clearly acceptable to all interested parties. Therefore delayed evaluation of annotations with code using the same scoping rules is obviously also completely viable. .. _Implementation: Implementation ============== There's a prototype implementation of this PEP, here: https://github.com/larryhastings/co_annotations/ As of this writing, all features described in this PEP are implemented, and there are some rudimentary tests in the test suite. There are still some broken tests, and the repo is many months behind. from __future__ import co_annotations ------------------------------------- In the prototype, the semantics presented in this PEP are gated with: from __future__ import co_annotations __co_annotations__ ------------------ Python supports runtime metadata for annotations for three different types: function, classes, and modules. The basic approach to implement this PEP is much the same for all three with only minor variations. With this PEP, each of these types adds a new attribute, ``__co_annotations__``, with the following semantics: * ``__co_annotations__`` is always set, and may contain either ``None`` or a callable. * ``__co_annotations__`` cannot be deleted. * ``__annotations__`` and ``__co_annotations__`` can't both be set to a useful value simultaneously: * If you set ``__annotations__`` to a dict, this also sets ``__co_annotations__`` to None. * If you set ``__co_annotations__`` to a callable, this also deletes ``__annotations__`` Internally, ``__co_annotations__`` is a "data descriptor", where functions are called whenever user code gets, sets, or deletes the attribute. In all three cases, the object has a separate internal place to store the current value of the ``__co_annotations__`` attribute. ``__annotations__`` is also reimplemented as a data descriptor, with its own separate internal storage for its internal value. The code implementing the "get" for ``__annotations__`` works something like this:: if (the internal value is set) return the internal annotations dict if (__co_annotations__ is not None) call the __co_annotations__ function if the result is a dict: store the result as the internal value set __co_annotations__ to None return the internal value do whatever this object does when there are no annotations Unbound code objects -------------------- When Python code defines one of these three objects with annotations, the Python compiler generates a separate code object which builds and returns the appropriate annotations dict. The "annotation code object" is then stored *unbound* as the internal value of ``__co_annotations__``; it is then bound on demand when the user asks for ``__annotations__``. This is an important optimization, for both speed and memory consumption. Python processes rarely examine annotations at runtime. Therefore, pre-binding these code objects to function objects would be a waste of resources in nearly all cases. Note that user code isn't permitted to see these unbound code objects. If the user gets the value of ``__co_annotations__``, and the internal value of ``__co_annotations__`` is an unbound code object, it is bound, and the resulting function object is stored as the new value of ``__co_annotations__``. The annotations function ------------------------ Annotations functions take no arguments and must return a dict (or subclass of dict). The bytecode generated for annotations code objects always uses the ``BUILD_CONST_KEY_MAP`` opcode to build the dict. Stock and PEP 563 semantics only uses this bytecode for function annotations; for class and module annotations, they generate a longer and slightly-less-efficient stanza of bytecode. Also, when generating the bytecode for an annotations code object, all ``LOAD_*`` opcodes are forced to be ``LOAD_GLOBAL``. Function Annotations -------------------- When compiling a function, the CPython bytecode compiler visits the annotations for the function all in one place, starting with ``compiler_visit_annotations()``. If there are any annotations, they create the scope for the annotations function on demand, and ``compiler_visit_annotations()`` assembles it. The code object is passed in in place of the annotations dict for the ``MAKE_FUNCTION`` bytecode. ``MAKE_FUNCTION`` supports a new bit in its oparg bitfield, ``0x10``, which tells it to expect a ``co_annotations`` code object on the stack. The bitfields for ``annotations`` (``0x04``) and ``co_annotations`` (``0x10``) are mutually exclusive. When binding an unbound annotation code object, a function will use its own ``__globals__`` as the new function's globals. One quirk of Python: you can't actually remove the annotations from a function object. If you delete the ``__annotations__`` attribute of a function, then get its ``__annotations__`` member, it will create an empty dict and use that as its ``__annotations__``. Naturally the implementation of this PEP maintains this quirk. Class Annotations ----------------- When compiling a class body, the compiler maintains two scopes: one for the normal class body code, and one for annotations. (This is facilitated by four new functions: ``compiler.c`` adds ``compiler_push_scope()`` and ``compiler_pop_scope()``, and ``symtable.c`` adds ``symtable_push_scope()`` and ``symtable_pop_scope()``.) Once the code generator reaches the end of the class body, but before it generates the bytecode for the class body, it assembles the bytecode for ``__co_annotations__``, then assigns that to ``__co_annotations__`` using ``STORE_NAME``. It also sets a new ``__globals__`` attribute. Currently it does this by calling ``globals()`` and storing the result. (Surely there's a more elegant way to find the class's globals--but this was good enough for the prototype.) When binding an unbound annotation code object, a class will use the value of this ``__globals__`` attribute. When the class drops its reference to the unbound code object--either because it has bound it to a function, or because ``__annotations__`` has been explicitly set--it also deletes its ``__globals__`` attribute. As discussed above, examination / modification of ``__annotations__`` from within the class body is no longer supported. Also, any flow control (``if`` / ``try``) around declarations of members with annotations is unsupported. If you delete the ``__annotations__`` attribute of a class, then get its ``__annotations__`` member, it will return the annotations dict of the first base class with annotations set. If no base classes have annotations set, it will raise ``AttributeError``. Although it's an implementation-specific detail, currently classes store the internal value of ``__co_annotations__`` in their ``tp_dict`` under the same name. Module Annotations ------------------ Module annotations work much the same as class annotations. The main difference is, a module uses its own dict as the ``__globals__`` when binding the function. If you delete the ``__annotations__`` attribute of a class, then get its ``__annotations__`` member, the module will raise ``AttributeError``. Interactive REPL Shell ---------------------- Everything works the same inside Python's interactive REPL shell, except for module annotations in the interactive module (``__main__``) itself. Since that module is never "finished", there's no specific point where we can compile the ``__co_annotations__`` function. For the sake of simplicity, in this case we forego delayed evaluation. Module-level annotations in the REPL shell will continue to work exactly as they do today, evaluating immediately and setting the result directly inside the ``__annotations__`` dict. (It might be possible to support delayed evaluation here. But it gets complicated quickly, and for a nearly-non-existent use case.) Local Annotations Inside Functions ---------------------------------- Python supports syntax for local variable annotations inside functions. However, these annotations have no runtime effect. Thus this PEP doesn't need to do anything to support them. Performance ----------- Performance with this PEP should be favorable. In general, resources are only consumed on demand—"you only pay for what you use". There are three scenarios to consider: * the runtime cost when annotations aren't defined, * the runtime cost when annotations are defined but *not* referenced, and * the runtime cost when annotations are defined *and* referenced. We'll examine each of these scenarios in the context of all three semantics for annotations: stock, PEP 563, and this PEP. When there are no annotations, all three semantics have the same runtime cost: zero. No annotations dict is created and no code is generated for it. This requires no runtime processor time and consumes no memory. When annotations are defined but not referenced, the runtime cost of Python with this PEP should be slightly faster than either original Python semantics or PEP 563 semantics. With those, the annotations dicts are built but never examined; with this PEP, the annotations dicts won't even be built. All that happens at runtime is the loading of a single constant (a simple code object) which is then set as an attribute on an object. Since the annotations are never referenced, the code object is never bound to a function, the code to create the dict is never executed, and the dict is never constructed. When annotations are both defined and referenced, code using this PEP should be much faster than code using PEP 563 semantics, and roughly the same as original Python semantics. PEP 563 semantics requires invoking ``eval()`` for every value inside an annotations dict, which is much slower. And, as already mentioned, this PEP generates more efficient bytecode for class and module annotations than either stock or PEP 563 semantics. Memory use should also be comparable in all three scenarios across all three semantic contexts. In the first and third scenarios, memory usage should be roughly equivalent in all cases. In the second scenario, when annotations are defined but not referenced, using this PEP's semantics will mean the function/class/module will store one unused code object; with the other two semantics, they'll store one unused dictionary. For Future Discussion ===================== __globals__ ----------- Is it permissable to add the ``__globals__`` reference to class objects as proposed here? It's not clear why this hasn't already been done; PEP 563 could have made use of class globals, but instead makes do with looking up classes inside ``sys.modules``. Yet Python seems strangely allergic to adding a ``__globals__`` reference to class objects. If adding ``__globals__`` to class objects is indeed a bad idea (for reasons I don't know), here are two alternatives as to how classes could get a reference to their globals for the implementation of this PEP: * The generate code for a class could bind its annotations code object to a function at the time the class is bound, rather than waiting for ``__annotations__`` to be referenced, making them an exception to the rule (even though "special cases aren't special enough to break the rules"). This would result in a small additional runtime cost when annotations were defined but not referenced on class objects. Honestly I'm more worried about the lack of symmetry in semantics. (But I wouldn't want to pre-bind all annotations code objects, as that would become much more costly for function objects, even as annotations are rarely used at runtime.) * Use the class's ``__module__`` attribute to look up its module by name in ``sys.modules``. This is what PEP 563 advises. While this is passable for userspace or library code, it seems like a little bit of a code smell for this to be defined semantics baked into the language itself. Also, the prototype gets globals for class objects by calling ``globals()`` then storing the result. I'm sure there's a much faster way to do this, I just didn't know what it was when I was prototyping. I'm sure we can revise this to something much faster and much more sanitary. I'd prefer to make it completely internal anyway, and not make it visible to the user (via this new __globals__ attribute). There's possibly already a good place to put it anyway--``ht_module``. Bikeshedding the name --------------------- During most of the development of this PEP, user code actually could see the raw annotation code objects. ``__co_annotations__`` could only be set to a code object; functions and other callables weren't permitted. In that context the name ``co_annotations`` makes a lot of sense. But with this last-minute pivot where ``__co_annotations__`` now presents itself as a callable, perhaps the name of the attribute and the name of the ``from __future__ import`` needs a re-think. Acknowledgements ================ Thanks to Barry Warsaw, Eric V. Smith, and Mark Shannon for feedback and encouragement. Thanks in particular to Mark Shannon for two key suggestions—build the entire annotations dict inside a single code object, and only bind it to a function on demand—that quickly became among the best aspects of this proposal. Copyright ========= This document is placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Tue, Jan 12, 2021 at 4:22 AM Larry Hastings <larry@hastings.org> wrote:
I've written a new PEP. Please find it below. Happy reading!
Can this get added to the PEPs repo and assigned a number and such? BTW, the currently preferred wording for the copyright blurb is slightly different. If you're the sole author of this text, can you please consider the license terms shown in PEP 12? ChrisA PEP editor - if you need a hand, I'm here to help
Certainly. I'm just another victim in the copy-and-paste wars. I actually have write access to the PEPs repo (I'm a former release manager) so I'd be happy to check it in myself once it gets a number, however that happens. Before I do so I'll study PEP 12 as if it was gonna be on tomorrow's midterms. //arry/ On 1/11/21 9:59 AM, Chris Angelico wrote:
On Tue, Jan 12, 2021 at 4:22 AM Larry Hastings <larry@hastings.org> wrote:
I've written a new PEP. Please find it below. Happy reading!
Can this get added to the PEPs repo and assigned a number and such?
BTW, the currently preferred wording for the copyright blurb is slightly different. If you're the sole author of this text, can you please consider the license terms shown in PEP 12?
ChrisA PEP editor - if you need a hand, I'm here to help
On Tue, Jan 12, 2021 at 5:10 AM Larry Hastings <larry@hastings.org> wrote:
Certainly. I'm just another victim in the copy-and-paste wars.
Ah yes, the Battle of the Clipboard. Iconic, epic, such a glorious engagement! But the casualties were steep. Fortunately we can rebuild.
I actually have write access to the PEPs repo (I'm a former release manager) so I'd be happy to check it in myself once it gets a number, however that happens. Before I do so I'll study PEP 12 as if it was gonna be on tomorrow's midterms.
Number allocation is pretty informal. Go ahead and grab PEP 649; in the unlikely event that someone else pushes a commit creating that PEP before you get to it, grab PEP 650 instead :) I'm happy to help out if you need me, but it sounds like you got this! ChrisA
On 1/11/21 10:16 AM, Chris Angelico wrote:
Number allocation is pretty informal. Go ahead and grab PEP 649;
It's now checked in as PEP 649, with a modern header, modern copyright, and I went ahead and grabbed the formatting stanza from the end too. Welcome to the world, baby 649! //arry/
I'm very much in favour of the this concept. A few points come to mind right away: 1. Backwards Compatibility
PEP 563 changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated inmodule-level scope. They may no longer refer directly to local variables or class attributes.
Given get_type_hints can be provided localns argument, this statement is not exactly true. Using localns is how I currently address scoping issues when affixing type hints. Maybe it could be argued that I'm abusing this feature, but then I would ask what the intent of localns is if not to provide additional (missing) scope during type hint evaluation? Under PEP 649, when __co_annotations__ is called (presumably by calling get_type_hints), would localns effectively be ignored? 2. __co_annotations__ scope? I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground. On Mon, 2021-01-11 at 10:27 -0800, Larry Hastings wrote:
On 1/11/21 10:16 AM, Chris Angelico wrote:
Number allocation is pretty informal. Go ahead and grab PEP 649; It's now checked in as PEP 649, with a modern header, modern copyright, and I went ahead and grabbed the formatting stanza from the end too. Welcome to the world, baby 649!
/arry _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OKOQEAEP... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for your feedback! I'll reply piecemeal. On 1/11/21 12:32 PM, Paul Bryan wrote:
*1. Backwards Compatibility*
PEP 563 <https://www.python.org/dev/peps/pep-0563> changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated in /module-level/ scope. They may no longer refer directly to local variables or class attributes.
Given get_type_hints can be provided localns argument, this statement is not exactly true.
PEP 563 states: For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form. So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
Under PEP 649, when __co_annotations__ is called (presumably by calling get_type_hints), would localns effectively be ignored?
Yes. You can experiment with this in Python 3.9--just turn off annotation stringizing. It seems that you can still use strings as annotations and typing.get_type_hints() will evaluate them--and I assume it'll use localns at that point, just as it does today.
*2. __co_annotations__ scope?*
I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground.
This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations": This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again. https://www.python.org/dev/peps/pep-0563/#keeping-the-ability-to-use-functio... Doing this automatically would indeed incur a sizeable runtime cost, for a feature that is already rarely used at runtime. I guess it would be remotely possible? to add this as an optional feature? But this gets crazy quickly--what if it's defined inside a function inside another function inside a class inside another function?--and the use cases seem few, and TOOWTDI. I've never understood how closures work in Python, so I'm not the guy to ask how possible / hard this would be. Then again, the implementation of closures is obscure enough that I've never been able to understand them, so that seems to establish at least a base level of difficulty. Anyway, one of the concepts my PEP is built on is that "annotations are always evaluated at module-level scope". I'd be against changing that unless it could be achieved without runtime cost--which AFAIK is impossible. Cheers, //arry/
On 1/11/21 1:16 PM, Larry Hastings wrote:
On 1/11/21 12:32 PM, Paul Bryan wrote:
*1. Backwards Compatibility*
PEP 563 <https://www.python.org/dev/peps/pep-0563> changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated in /module-level/ scope. They may no longer refer directly to local variables or class attributes.
Given get_type_hints can be provided localns argument, this statement is not exactly true.
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.
So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
Whoops! Let me walk that back a little. I'd been assuming that PEP 563 used the terms "annotations" and "type hints" to mean the exact same thing. But careful reading of PEP 484 suggests that they're distinct concepts; all "type hints" are annotations, but not all annotations are "type hints". So: if you're using annotations for something besides "type hints", such that you have a use for a non-None localns, I guess you have two options with my PEP: either a) use strings for your annotations where you need localns to work for you, or b) skip using annotations syntax and instead write your own custom __co_annotations__ function. Or, you could mix and match, using annotations syntax where it was convenient, and overriding only the troublesome spots in your custom __co_annotations__ function: def foo(a:int=3, b): ... foo_static_annotations = foo.__annotations__ def foo_dynamic_closure_annotations(): annotations = dict(foo_static_annotations) annotations['b'] = MyExcitingLocallyDefinedClosureTypeThing return annotations foo.__co_annotations = foo_dynamic_closure_annotations Cheers, //arry/
On 1/11/21 6:29 PM, Greg Ewing wrote:
On 12/01/21 10:41 am, Larry Hastings wrote:
So: if you're using annotations for something besides "type hints",
Didn't Guido decree that using annotations for anything other than type hints is no longer supported?
It was never an official decree. And, for what it's worth, Guido mostly walked that back, last month: https://mail.python.org/archives/list/python-dev@python.org/message/OSA7VKZS... Cheers, //arry/
On Mon, 2021-01-11 at 13:16 -0800, Larry Hastings wrote:
Thanks for your feedback! I'll reply piecemeal.
On 1/11/21 12:32 PM, Paul Bryan wrote:
1. Backwards Compatibility
PEP 563 changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated in module-level scope. They may no longer refer directly to local variables or class attributes.
Given get_type_hints can be provided localns argument, this statement is not exactly true. PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form. So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
I acknowledge that this will not fly with static type checkers. I also want to make sure that annotations can continue to serve runtime type validation. PEP 563 does go on to state:
For code which uses annotations for other purposes, a regulareval(ann, globals, locals) call is enough to resolve the annotation.
And I believe this would no longer be true under PEP 649; further, localns (and maybe globalns) parameters in get_type_hints would become meaningless. This passage in PEP 563 appears not true in Python 3.9 with __future__ annotations, emphasis mine:
The get_type_hints() function automatically resolves the correct value of globalns for functions and classes. It also automatically provides the correct localns for classes.
If this passage was true, I believe the issue that resulted in my affixing type hints could have been averted.
Under PEP 649, when __co_annotations__ is called (presumably by calling get_type_hints), would localns effectively be ignored? Yes. You can experiment with this in Python 3.9--just turn off annotation stringizing. It seems that you can still use strings as annotations and typing.get_type_hints() will evaluate them--and I assume it'll use localns at that point, just as it does today. OK, would string representations of type hints continue be supported under PEP 649 if strings are used as annotations? And, would get_type_hints continue evaluate the annotations in that case?
2. __co_annotations__ scope?
I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground. This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations":
I wasn't thinking the function local state of that being annotated (agree, this would be prohibitive), but rather the scope in which the annotated function, class, module, etc. are being defined.
This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again.
https://www.python.org/dev/peps/pep-0563/#keeping-the-ability-to-use-functio... Doing this automatically would indeed incur a sizeable runtime cost, for a feature that is already rarely used at runtime. I guess it would be remotely possible? to add this as an optional feature? But this gets crazy quickly--what if it's defined inside a function inside another function inside a class inside another function?--and the use cases seem few, and TOOWTDI.
I think this exactly the case for closures today.
I've never understood how closures work in Python, so I'm not the guy to ask how possible / hard this would be. Then again, the implementation of closures is obscure enough that I've never been able to understand them, so that seems to establish at least a base level of difficulty. Anyway, one of the concepts my PEP is built on is that "annotations are always evaluated at module-level scope". I'd be against changing that unless it could be achieved without runtime cost--which AFAIK is impossible.
Cheers,
/arry
On 1/11/21 3:02 PM, Paul Bryan wrote:
PEP 563 does go on to state:
For code which uses annotations for other purposes, a regular eval(ann, globals, locals) call is enough to resolve the annotation.
And I believe this would no longer be true under PEP 649; further, localns (and maybe globalns) parameters in get_type_hints would become meaningless.
[...] And, would get_type_hints continue evaluate [string] annotations in that case?
I don't work on typing.get_type_hints() so someone else will have to answer this question. All I can say is, with PEP 649 semantics, if you set an annotation to a string, you'll get a string back. And in 3.9 (and my out-of-date 3.10) I observe that typing.get_type_hints() will eval() string annotations for you, and localns is significant.
This passage in PEP 563 appears not true in Python 3.9 with __future__ annotations, emphasis mine:
The get_type_hints() function automatically resolves the correct value of globalns for functions and classes. *It also automatically provides the correct localns for classes.*
If this passage was true, I believe the issue that resulted in my affixing type hints could have been averted.
As you've discovered, this is one of the places where PEP 563 seems to be out-of-date with respect to its implementation. I sifted through the source code to typing.get_type_hints() twice, and near as I can figure out, localns is literally only ever set to None unless you override it with the parameter.
OK, would string representations of type hints continue be supported under PEP 649 if strings are used as annotations?
PEP 649 is itself totally agnostic as to what value you use as an annotation. It disallows a couple funky things (yield, walrus operator), but beyond that it doesn't care. Any Python expression or value is fine.
*2. __co_annotations__ scope?*
I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground.
This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations":
I wasn't thinking the function local state of that being annotated (agree, this would be prohibitive), but rather the scope in which the annotated function, class, module, etc. are being defined.
That's what PEP 563 is referring to. If you read the thread from November 2017 where the idea was discussed, they were talking about referring to e.g. "class-level definitions", as in, things defined inside class scope. Which is prohibitive. (If I understand you correctly, you thought it was referring to the scope inside the function when it runs? Because I can't imagine how that would ever work. What if the function hasn't been called yet? What if it's been called a thousand times? What if it's running right now in various stages of completeness in five threads and you inspect the annotation from a sixth thread?) Cheers, //arry/
Some more questions... "Binding"," bound" and "unbound" code objects: Is your use of "binding" terminology in the PEP identical to the binding of a function to an object instance as a method during object creation? Function Annotations:
When binding an unbound annotation code object, a function will use its own __globals__ as the new function's globals. I'm having trouble parsing this. Does this mean the newly bound __co_annotations__ function will inherit __globals__ from the function it's annotating?
Exceptions: It's quite possible for a __co_annotation__ function call to raise an exception (e.g. NameError). When accessing __annotations__, if such an exception is raised during the call to __co_annotations__, what is the expected behavior? s/__co_//?: I'm probably naive, but is there a reason that one could not just store a callable in __annotations__, and use the descriptor to resolve it to a dictionary and store it when it is accessed? It would be one less dunder in the Python data model. On Mon, 2021-01-11 at 15:46 -0800, Larry Hastings wrote:
On 1/11/21 3:02 PM, Paul Bryan wrote:
PEP 563 does go on to state:
For code which uses annotations for other purposes, a regular eval(ann, globals, locals) call is enough to resolve the annotation.
And I believe this would no longer be true under PEP 649; further, localns (and maybe globalns) parameters in get_type_hints would become meaningless.
[...] And, would get_type_hints continue evaluate [string] annotations in that case? I don't work on typing.get_type_hints() so someone else will have to answer this question. All I can say is, with PEP 649 semantics, if you set an annotation to a string, you'll get a string back. And in 3.9 (and my out-of-date 3.10) I observe that typing.get_type_hints() will eval() string annotations for you, and localns is significant.
This passage in PEP 563 appears not true in Python 3.9 with __future__ annotations, emphasis mine:
The get_type_hints() function automatically resolves the correct value of globalns for functions and classes. It also automatically provides the correct localns for classes.
If this passage was true, I believe the issue that resulted in my affixing type hints could have been averted. As you've discovered, this is one of the places where PEP 563 seems to be out-of-date with respect to its implementation. I sifted through the source code to typing.get_type_hints() twice, and near as I can figure out, localns is literally only ever set to None unless you override it with the parameter.
OK, would string representations of type hints continue be supported under PEP 649 if strings are used as annotations? PEP 649 is itself totally agnostic as to what value you use as an annotation. It disallows a couple funky things (yield, walrus operator), but beyond that it doesn't care. Any Python expression or value is fine.
2. __co_annotations__ scope?
I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground. This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations":
I wasn't thinking the function local state of that being annotated (agree, this would be prohibitive), but rather the scope in which the annotated function, class, module, etc. are being defined.
That's what PEP 563 is referring to. If you read the thread from November 2017 where the idea was discussed, they were talking about referring to e.g. "class-level definitions", as in, things defined inside class scope. Which is prohibitive. (If I understand you correctly, you thought it was referring to the scope inside the function when it runs? Because I can't imagine how that would ever work. What if the function hasn't been called yet? What if it's been called a thousand times? What if it's running right now in various stages of completeness in five threads and you inspect the annotation from a sixth thread?)
Cheers,
/arry
On 1/11/21 5:02 PM, Paul Bryan wrote:
Some more questions...
"Binding"," bound" and "unbound" code objects: Is your use of "binding" terminology in the PEP identical to the binding of a function to an object instance as a method during object creation?
I'm not. In PEP 649 I think every reference of "binding" is talking about binding a code object to a globals dict to produce a function object. The process of binding a function to an object instance to make a method is conceptually very similar, but distinct. (and btw, functions aren't bound to their object to make methods during object creation, it's done lazily at the time you ask for it--that's what the "descriptor protocol" is all about!)
Function Annotations:
When binding an unbound annotation code object, a function will use its own __globals__ as the new function's globals. I'm having trouble parsing this. Does this mean the newly bound __co_annotations__ function will inherit __globals__ from the function it's annotating?
Yes. Though I wouldn't use "inherit", I'd just say it "uses" the __globals__ from the function.
Exceptions: It's quite possible for a __co_annotation__ function call to raise an exception (e.g. NameError). When accessing __annotations__, if such an exception is raised during the call to __co_annotations__, what is the expected behavior?
If the function fails for any reason--throws an exception, or just doesn't return an acceptable value--then the getter immediately exits, and the internal state of the object is unchanged. If you wanted to, you could catch the exception, fix the error, and get __annotations__ again, and it'd work.
s/__co_//?: I'm probably naive, but is there a reason that one could not just store a callable in __annotations__, and use the descriptor to resolve it to a dictionary and store it when it is accessed? It would be one less dunder in the Python data model.
That would work, but I think the API is a bit of a code smell. __annotations__ would no longer be stable: a.__annotations__ = o assert a.__annotations__ == o Would that assert fail? It depends on what type(o) is, which is surprising. Cheers, //arry/
On Mon, 2021-01-11 at 17:56 -0800, Larry Hastings wrote:
On 1/11/21 5:02 PM, Paul Bryan wrote:
Some more questions...
"Binding"," bound" and "unbound" code objects: Is your use of "binding" terminology in the PEP identical to the binding of a function to an object instance as a method during object creation?
I'm not. In PEP 649 I think every reference of "binding" is talking about binding a code object to a globals dict to produce a function object. The process of binding a function to an object instance to make a method is conceptually very similar, but distinct. (and btw, functions aren't bound to their object to make methods during object creation, it's done lazily at the time you ask for it-- that's what the "descriptor protocol" is all about!) I think it'd be worth briefly describing binding, to avoid confusion. I'm now more educated on function/method binding lifecycle though, so bonus! 🙂
Function Annotations:
When binding an unbound annotation code object, a function will use its own __globals__ as the new function's globals. I'm having trouble parsing this. Does this mean the newly bound __co_annotations__ function will inherit __globals__ from the function it's annotating? Yes. Though I wouldn't use "inherit", I'd just say it "uses" the __globals__ from the function.
Then I think this solves my particular scoping problem with 3.10 string-evaluated annotations!
Exceptions: It's quite possible for a __co_annotation__ function call to raise an exception (e.g. NameError). When accessing __annotations__, if such an exception is raised during the call to __co_annotations__, what is the expected behavior? If the function fails for any reason--throws an exception, or just doesn't return an acceptable value--then the getter immediately exits, and the internal state of the object is unchanged. If you wanted to, you could catch the exception, fix the error, and get __annotations__ again, and it'd work. OK.
s/__co_//?: I'm probably naive, but is there a reason that one could not just store a callable in __annotations__, and use the descriptor to resolve it to a dictionary and store it when it is accessed? It would be one less dunder in the Python data model. That would work, but I think the API is a bit of a code smell. __annotations__ would no longer be stable: a.__annotations__ = o assert a.__annotations__ == o Would that assert fail? It depends on what type(o) is, which is surprising.
Equally surprising?: a.__co_annotations__ = o a.__annotations__ assert a.__co_annotations__ == o Paul
On 1/11/21 6:34 PM, Paul Bryan wrote:
a.__annotations__ = o assert a.__annotations__ == o
Would that assert fail? It depends on what type(o) is, which is surprising.
Equally surprising?:
a.__co_annotations__ = o a.__annotations__ assert a.__co_annotations__ == o
Well, since you asked, no. It's a good point, but I think my example is a touch more "surprising". In my example, a.__annotations__ has two different True values; in yours, a.__co_annotations__ goes from True to False. It's acting more like a cache. Also consider: if you set o.__annotations__ to a function, what if you want to examine the function later? What if you want to examine the function built by Python? def enhance_annotations(co_annotations): def enhance(): d = co_annotations() d['extra'] = ... return d o.__co_annotations__ = enhance_annotations(o.__co_annotations__) Finally, it's just a code smell to have one attribute support such a bewildering variety of types. Of course, this /works,/ but it's bad hygiene to store such violently different types in what is ostensibly the same attribute. Cheers, //arry/
On 1/11/21 6:34 PM, Paul Bryan wrote:
On Mon, 2021-01-11 at 17:56 -0800, Larry Hastings wrote:
On 1/11/21 5:02 PM, Paul Bryan wrote:
I'm probably naive, but is there a reason that one could not just store a callable in __annotations__, and use the descriptor to resolve it to a dictionary and store it when it is accessed? It would be one less dunder in the Python data model.
That would work, but I think the API is a bit of a code smell. __annotations__ would no longer be stable:
a.__annotations__ = o assert a.__annotations__ == o
Would that assert fail? It depends on what type(o) is, which is surprising.
Equally surprising?:
a.__co_annotations__ = o a.__annotations__ assert a.__co_annotations__ == o
I've ruminated about this a bit over the past few days, and I finally realized exactly why, yes, I think behavior is more surprising. It's because __annotations__ is now 12 years old (!), and never in that entire time has it silently changed its value. It's always been completely stable, and we have twelve years' worth of installed base that may rely on that assumption. In comparison, __co_annotations__ is a new attribute. While it's also surprising that __co_annotations__ can be automatically unset, at least this would be a documented part of its behavior from day 1. Relatedly, __co_annotations__ is behaving somewhat like a cached value, in that cached values get deleted when they're out-of-date. (An observation that may provide some guidance if we decide to rename __co_annotations__.) This idiom may be familiar to the user--unlike your proposed semantics, which I don't recall ever seeing used in an API. I admit it's only a small difference between what you proposed and what I propose, but in the end I definitely prefer my approach. Cheers, //arry/
OK. Makes sense to think of __annotations__ as being the location of the final, stable, "affixed" type hints. I wonder if then the __co_annotations__ call and overwriting of __annotations__ should be explicitly caused by a to get_type_hints instead of (mysteriously) occurring on an attempt to getattr __annotations__. I know this changes the descriptor behavior you documented, but at least it would occur explicitly in a function call and may be easier for developers to reason about? It would also address my other question of trying to access __annotations__, only to be confronted with an exception raised within __co_annotations__. On Fri, 2021-01-15 at 09:47 -0800, Larry Hastings wrote:
On 1/11/21 6:34 PM, Paul Bryan wrote:
On Mon, 2021-01-11 at 17:56 -0800, Larry Hastings wrote:
On 1/11/21 5:02 PM, Paul Bryan wrote:
I'm probably naive, but is there a reason that one could not just store a callable in __annotations__, and use the descriptor to resolve it to a dictionary and store it when it is accessed? It would be one less dunder in the Python data model. That would work, but I think the API is a bit of a code smell. __annotations__ would no longer be stable:
a.__annotations__ = o assert a.__annotations__ == o Would that assert fail? It depends on what type(o) is, which is surprising.
Equally surprising?:
a.__co_annotations__ = o a.__annotations__ assert a.__co_annotations__ == o
I've ruminated about this a bit over the past few days, and I finally realized exactly why, yes, I think behavior is more surprising. It's because __annotations__ is now 12 years old (!), and never in that entire time has it silently changed its value. It's always been completely stable, and we have twelve years' worth of installed base that may rely on that assumption. In comparison, __co_annotations__ is a new attribute. While it's also surprising that __co_annotations__ can be automatically unset, at least this would be a documented part of its behavior from day 1. Relatedly, __co_annotations__ is behaving somewhat like a cached value, in that cached values get deleted when they're out-of-date. (An observation that may provide some guidance if we decide to rename __co_annotations__.) This idiom may be familiar to the user--unlike your proposed semantics, which I don't recall ever seeing used in an API. I admit it's only a small difference between what you proposed and what I propose, but in the end I definitely prefer my approach. Cheers,
/arry
On 1/15/21 10:12 AM, Paul Bryan wrote:
I wonder if then the __co_annotations__ call and overwriting of __annotations__ should be explicitly caused by a to get_type_hints instead of (mysteriously) occurring on an attempt to getattr __annotations__.
I would say: absolutely not. While all "type hints" are annotations, not all annotations are "type hints". As mentioned previously in this thread, typing.get_type_hints() is opinionated in ways that users of annotations may not want. And personally I bristle at the idea of gating a language feature behind a library function. Besides, most users will never know or care about __co_annotations__. If you're not even aware that it exists, it's not mysterious ;-) Cheers, //arry/
I knew I was missing something. Agree, annotations are not necessarily type hints. On Fri, 2021-01-15 at 10:56 -0800, Larry Hastings wrote:
On 1/15/21 10:12 AM, Paul Bryan wrote:
I wonder if then the __co_annotations__ call and overwriting of __annotations__ should be explicitly caused by a to get_type_hints instead of (mysteriously) occurring on an attempt to getattr __annotations__.
I would say: absolutely not. While all "type hints" are annotations, not all annotations are "type hints". As mentioned previously in this thread, typing.get_type_hints() is opinionated in ways that users of annotations may not want. And personally I bristle at the idea of gating a language feature behind a library function. Besides, most users will never know or care about __co_annotations__. If you're not even aware that it exists, it's not mysterious ;-)
Cheers,
/arry
On 16/01/21 7:56 am, Larry Hastings wrote:
As mentioned previously in this thread, typing.get_type_hints() is opinionated in ways that users of annotations may not want.
This brings us back to my idea of introducing a new annotations() function to hide the details. It wouldn't be the same as get_type_hints(), since it wouldn't make any assumptions about what the annotations mean. -- Greg
Would annotations() just access the dunder, like other builtins (and then result in the descriptor resolving __co_annotations__ as proposed), or would calling it be required to actually resolve __co_annotations__? I think it should probably be the former. On Sat, 2021-01-16 at 12:29 +1300, Greg Ewing wrote:
On 16/01/21 7:56 am, Larry Hastings wrote:
As mentioned previously in this thread, typing.get_type_hints() is opinionated in ways that users of annotations may not want.
This brings us back to my idea of introducing a new annotations() function to hide the details. It wouldn't be the same as get_type_hints(), since it wouldn't make any assumptions about what the annotations mean.
On 1/15/21 3:29 PM, Greg Ewing wrote:
On 16/01/21 7:56 am, Larry Hastings wrote:
As mentioned previously in this thread, typing.get_type_hints() is opinionated in ways that users of annotations may not want.
This brings us back to my idea of introducing a new annotations() function to hide the details. It wouldn't be the same as get_type_hints(), since it wouldn't make any assumptions about what the annotations mean.
I think it's simpler and nicer for the user to preserve the existing interface, so I'm sticking with that approach. If you feel strongly about this, I encourage you to write your own competing PEP. //arry/
On 12/01/21 2:56 pm, Larry Hastings wrote:
In PEP 649 I think every reference of "binding" is talking about binding a code object to a globals dict to produce a function object. The process of binding a function to an object instance to make a method is conceptually very similar, but distinct.
Maybe "bare code object" would be a less confusing term? -- Greg
On Mon, Jan 11, 2021 at 3:51 PM Larry Hastings <larry@hastings.org> wrote:
[...]
This passage in PEP 563 appears not true in Python 3.9 with __future__ annotations, emphasis mine:
The get_type_hints() function automatically resolves the correct value of globalns for functions and classes. *It also automatically provides the correct localns for classes.*
If this passage was true, I believe the issue that resulted in my affixing type hints could have been averted.
As you've discovered, this is one of the places where PEP 563 seems to be out-of-date with respect to its implementation. I sifted through the source code to typing.get_type_hints() twice, and near as I can figure out, localns is literally only ever set to None unless you override it with the parameter.
This seems to be a bug in get_type_hints() for which someone should file a bug on bpo, please! -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Will do. On Mon, 2021-01-11 at 20:55 -0800, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 3:51 PM Larry Hastings <larry@hastings.org> wrote:
This passage in PEP 563 appears not true in Python 3.9 with __future__ annotations, emphasis mine:
The get_type_hints() function automatically resolves the correct value of globalns for functions and classes. It also automatically provides the correct localns for classes.
If this passage was true, I believe the issue that resulted in my affixing type hints could have been averted. As you've discovered, this is one of the places where PEP 563 seems to be out-of-date with respect to its implementation. I sifted
[...] through the source code to typing.get_type_hints() twice, and near as I can figure out, localns is literally only ever set to None unless you override it with the parameter.
This seems to be a bug in get_type_hints() for which someone should file a bug on bpo, please!
Another thought about this PEP (hopefully my last one tonight). The section on backwards compatibility doesn't mention what should happen with annotations that are stringified by the user (as is needed for forward references in code that hasn't been PEP-563-ified yet). That's a PEP 484 feature. Should we start deprecating that at the same time? Static checkers support it but don't need it (for example, stubs in typeshed don't use it since their code is never evaluated). At the very least I think your PEP should mention what happens for these -- presumably `__annotations__` will just contain the string literal, so get_type_hints() would be needed to evaluate these. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Yes, PEP 649 is completely agnostic about what values you put in as annotations. You can put in strings, complex objects, expressions--whatever you put in, you get back out later. I'm happy to add some text to the PEP if this needs clarifying; I just thought it was obvious. Cheers, //arry/ On 1/11/21 9:11 PM, Guido van Rossum wrote:
Another thought about this PEP (hopefully my last one tonight).
The section on backwards compatibility doesn't mention what should happen with annotations that are stringified by the user (as is needed for forward references in code that hasn't been PEP-563-ified yet).
That's a PEP 484 feature. Should we start deprecating that at the same time? Static checkers support it but don't need it (for example, stubs in typeshed don't use it since their code is never evaluated).
At the very least I think your PEP should mention what happens for these -- presumably `__annotations__` will just contain the string literal, so get_type_hints() would be needed to evaluate these.
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Given all the effort that get_type_hints() puts into evaluating those strings it seems important to spell out explicitly that they're not special. (IIRC they *are* special in PEP 563.) On Tue, Jan 12, 2021 at 8:56 AM Larry Hastings <larry@hastings.org> wrote:
Yes, PEP 649 is completely agnostic about what values you put in as annotations. You can put in strings, complex objects, expressions--whatever you put in, you get back out later.
I'm happy to add some text to the PEP if this needs clarifying; I just thought it was obvious.
Cheers,
*/arry* On 1/11/21 9:11 PM, Guido van Rossum wrote:
Another thought about this PEP (hopefully my last one tonight).
The section on backwards compatibility doesn't mention what should happen with annotations that are stringified by the user (as is needed for forward references in code that hasn't been PEP-563-ified yet).
That's a PEP 484 feature. Should we start deprecating that at the same time? Static checkers support it but don't need it (for example, stubs in typeshed don't use it since their code is never evaluated).
At the very least I think your PEP should mention what happens for these -- presumably `__annotations__` will just contain the string literal, so get_type_hints() would be needed to evaluate these.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 12/01/21 10:16 am, Larry Hastings wrote:
This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations":
This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again.
I'm not sure what that's supposed to mean. Firstly, functions that reference nonlocal names don't keep whole frames alive, only the particular objects they reference. Secondly, if an annotation references something at module level, that something will also be kept alive unless it is explicitly removed from the module -- which could also be done at a local level if you didn't want to keep those things around. So I don't really see any difference between global and local state when it comes to things being kept alive by annotations. -- Greg
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <larry@hastings.org> wrote:
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.
So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
I think you're misreading PEP 563 here. The mention of globalns=None, localns=None refers to the fact that these parameters have defaults, not that you must pass None. Note that the next paragraph in that PEP mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}). There is considerable discussion below in that same section, but it doesn't come closer than stating "using local state in annotations is no longer possible in general", and it states that get_type_hints() correctly provides the right localns for classes. Apart from how PEP 563 should be interpreted (maybe we should ask Lukasz) I would say that basically all feedback I've seen over the changed semantics of annotations in 3.10 is about this problem, and when I first saw your proposal I thought that this would solve those issues. I'd be much less enthusiastic if we're still going to force annotations to only reference globals. My read on PEP 563 is that it *primarily* solves the issue of having to put forward references in string quotes (the most common case probably being methods that accept or return an instance of the class in which they are being defined). You have come up with a better solution for that -- the solution that eluded us all when we were debating PEP 563. But I think that you should try for maximal backward compatibility with the state of the world *before* PEP 563. In particular, static type checkers have no problem with annotations referencing types defined in the local scope. This has always worked (hence the complaints about PEP 563) and it would actually be extra work to forbid this in certain situations but not others. [...]
*2. __co_annotations__ scope?*
I'm wondering why __co_annotations__ function could not be scoped (within a closure?) such that it can access the values where the function, method, class or module are being declared? I acknowledge that I'm railing against PEP 563 again, trying to reclaim lost ground.
This is addressed in PEP 563, when it rejected the idea of using "function local state when defining annotations":
This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again.
https://www.python.org/dev/peps/pep-0563/#keeping-the-ability-to-use-functio...
But that's a strawman for your PEP. In PEP 563, they would have to keep the whole frame alive. But with your PEP only the cells containing objects that are *actually* referenced by annotations, i.e. exactly the objects that the user wants to see preserved in `__annotations__`. (Well, at Instagram they probably don't want to see any annotations survive the compilation stage, but they can hack their own Python compiler -- we know they do that anyway. :-) Later in that same section, PEP 563 points out a problem with annotations that reference class-scoped variables, and claims that the implementation would run into problems because methods can't "see" the class scope. This is indeed a problem for PEP 563, but *you* can easily generate correct code, assuming the containing class exists in the global scope (and your solution requires that anyway). So in this case ``` class Outer: class Inner: ... def method(self, a: Inner, b: Outer) -> None: ... ``` The generated code for the `__annotations__` property could just have a reference to `Outer.Inner` for such cases: ``` def __annotations__(): return {"a": Outer.Inner, "b": Outer, "return": None} ``` (Note that for *function* locals you don't have to do anything, they just work, because of closures -- you may not understand them, but they are important here. :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Sorry it took me 3+ days to reply--I had a lot to think about here. But I have good things to report! On 1/11/21 8:42 PM, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.
So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
I think you're misreading PEP 563 here. The mention of globalns=None, localns=None refers to the fact that these parameters have defaults, not that you must pass None. Note that the next paragraph in that PEP mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).
I think that's misleading, then. The passage is telling you how to "correctly evaluate[s] expressions", and how I read it was, it's telling me I have to supply globalns=None and localns=None for it to work correctly--which, I had to discover on my own, were the default values. I don't understand why PEP 563 feels compelled to define a function that it's not introducing, and in fact had already shipped with Python two versions ago.
Later in that same section, PEP 563 points out a problem with annotations that reference class-scoped variables, and claims that the implementation would run into problems because methods can't "see" the class scope. This is indeed a problem for PEP 563, but *you* can easily generate correct code, assuming the containing class exists in the global scope (and your solution requires that anyway). So in this case ``` class Outer: class Inner: ... def method(self, a: Inner, b: Outer) -> None: ... ``` The generated code for the `__annotations__` property could just have a reference to `Outer.Inner` for such cases: ``` def __annotations__(): return {"a": Outer.Inner, "b": Outer, "return": None} ```
This suggestion was a revelation for me. Previously, a combination of bad experiences early on when hacking on compile and symtable, and my misunderstanding of exactly what was being asserted in the November 2017 thread, led me to believe that all I could support was globals. But I've been turning this over in my head for several days now, and I suspect I can support... just about anything. I can name five name resolution scenarios I might encounter. I'll discuss them below, in increasing order of difficulty. *First* is references to globals / builtins. That's already working, it's obvious how it works, and I need not elaborate further. *Second* is local variables in an enclosing function scope: def outer_fn(): class C: pass def inner_fn(a:C=None): pass return inner_fn As you pointed out elsewhere in un-quoted text, I could make the annotation a closure, so it could retain a reference to the value of (what is from its perspective) the free variable "C". *Third* is local variables in an enclosing class scope, as you describe above: class OuterCls: class InnerCls: def method(a:InnerCls=None): pass If I understand what you're suggesting, I could notice inside the compiler that Inner is being defined in a class scope, walk up the enclosing scopes until I hit the outermost class, then reconstruct the chain of pulling out attributes until it resolves globally. Thus I'd rewrite this example to: class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass We've turned the local reference into a global reference, and we already know globals work fine. *Fourth* is local variables in an enclosing class scope, which are themselves local variables in an enclosing function scope: def outerfn(): class OuterCls: class InnerCls: def method(a:InnerCls=None): pass return OuterCls.InnerCls Even this is solvable, I just need to combine the "second" and "third" approaches above. I walk up the enclosing scopes to find the outermost class scope, and if that's a function scope, I create a closure and retain a reference to /that/ free variable. Thus this would turn into def outerfn(): class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass and method.__co_annotations__ would reference the free variable "OuterCls" defined in outerfn. *Fifth* is the nasty one. Note that so far every definition we've referred to in an annotation has been /before/ the definition of the annotation. What if we want to refer to something defined /after/ the annotation? def outerfn(): class OuterCls: class InnerCls: def method(a:zebra=None): pass ... We haven't seen the definition of "zebra" yet, so we don't know what approach to take. It could be any of the previous four scenarios. What do we do? This is solvable too: we simply delay the compilation of __co_annotations__ code objects until the very last possible moment. First, at the time we bind the class or function, we generate a stub __co_annotations__ object, just to give the compiler what it expects. The compiler inserts it into the const table for the enclosing construct (function / class / module), and we remember what index it went into. Then, after we've finished processing the entire AST tree for this module, but before we we exit the compiler, we reconstruct the required context for evaluating each __co_annotations__ function--the nested chain of symbol tables, the compiler blocks if needed, etc--and evaluate the annotations for real. We assemble the correct __co_annotations__ code object and overwrite the stub in the const table with this now-correct value. I can't think of any more scenarios. So, I think I can handle basically anything! However, there are two scenarios where the behavior of evaluations will change in a way the user might find surprising. The first is when they redefine a variable used in an annotation: x = str def fn(a:x="345"): pass x = int With stock semantics, the annotation to "a" will be "str". With PEP 563 or my PEP, the annotation to "a" will be "int". (It gets even more exciting if you said "del x".) Similarly, delaying the annotations so that we make everything visible means defining variables with the same name in multiple scopes may lead to surprising behavior. x = str class Outer: def method(a:x="345"): pass x = int Again, stock gets you an annotation of "str", but PEP 563 and my PEP gets you "str", because they'll see the /final/ result of evaluating the body of Outer. Sadly this is the price you pay for delayed evaluation of annotations. Delaying the evaluation of annotations is the goal, and the whole point is to make changes, observable by the user, in how annotations are evaluated. All we can do is document these behaviors and hope our users forgive us. I think this is a vast improvement over the first draft of my PEP, and assuming nobody points out major flaws in this approach (and, preferably, at least a little encouragement), I plan to redesign my prototype along these lines. (Though not right away--I want to take a break and attend to some other projects first.) Thanks for the mind-blowing suggestions, Guido! I must say, you're pretty good at this Python stuff. Cheers, //arry/
On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <larry@hastings.org> wrote:
Sorry it took me 3+ days to reply--I had a lot to think about here. But I have good things to report!
On 1/11/21 8:42 PM, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <larry@hastings.org> wrote:
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.
So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
I think you're misreading PEP 563 here. The mention of globalns=None, localns=None refers to the fact that these parameters have defaults, not that you must pass None. Note that the next paragraph in that PEP mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).
I think that's misleading, then. The passage is telling you how to "correctly evaluate[s] expressions", and how I read it was, it's telling me I have to supply globalns=None and localns=None for it to work correctly--which, I had to discover on my own, were the default values. I don't understand why PEP 563 feels compelled to define a function that it's not introducing, and in fact had already shipped with Python two versions ago.
I suppose PEP 563 is ambiguous because on the one hand global symbols are the only things that work out of the box, on the other hand you can make other things work by passing the right scope (and there's lots of code now that does so), and on the third hand, it claims that get_type_hints() adds the class scope, which nobody noticed or implemented until this week (there's a PR, can't recall the number). But I think all this is irrelevant given what comes below.
Later in that same section, PEP 563 points out a problem with annotations that reference class-scoped variables, and claims that the implementation would run into problems because methods can't "see" the class scope. This is indeed a problem for PEP 563, but *you* can easily generate correct code, assuming the containing class exists in the global scope (and your solution requires that anyway). So in this case ``` class Outer: class Inner: ... def method(self, a: Inner, b: Outer) -> None: ... ``` The generated code for the `__annotations__` property could just have a reference to `Outer.Inner` for such cases: ``` def __annotations__(): return {"a": Outer.Inner, "b": Outer, "return": None} ```
This suggestion was a revelation for me. Previously, a combination of bad experiences early on when hacking on compile and symtable, and my misunderstanding of exactly what was being asserted in the November 2017 thread, led me to believe that all I could support was globals. But I've been turning this over in my head for several days now, and I suspect I can support... just about anything.
I can name five name resolution scenarios I might encounter. I'll discuss them below, in increasing order of difficulty.
*First* is references to globals / builtins. That's already working, it's obvious how it works, and I need not elaborate further.
Yup.
*Second* is local variables in an enclosing function scope:
def outer_fn(): class C: pass def inner_fn(a:C=None): pass return inner_fn
As you pointed out elsewhere in un-quoted text, I could make the annotation a closure, so it could retain a reference to the value of (what is from its perspective) the free variable "C".
Yup.
*Third* is local variables in an enclosing class scope, as you describe above:
class OuterCls: class InnerCls: def method(a:InnerCls=None): pass
If I understand what you're suggesting, I could notice inside the compiler that Inner is being defined in a class scope, walk up the enclosing scopes until I hit the outermost class, then reconstruct the chain of pulling out attributes until it resolves globally. Thus I'd rewrite this example to:
class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we already know globals work fine.
I think this is going too far. A static method defined in InnerCls does not see InnerCls (even after the class definitions are complete). E.g. ``` class Outer: class Inner: @staticmethod def foo(): return Inner ``` If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not defined".
*Fourth* is local variables in an enclosing class scope, which are themselves local variables in an enclosing function scope:
def outerfn(): class OuterCls: class InnerCls: def method(a:InnerCls=None): pass return OuterCls.InnerCls
Even this is solvable, I just need to combine the "second" and "third" approaches above. I walk up the enclosing scopes to find the outermost class scope, and if that's a function scope, I create a closure and retain a reference to *that* free variable. Thus this would turn into
def outerfn(): class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
and method.__co_annotations__ would reference the free variable "OuterCls" defined in outerfn.
Probably also not needed.
*Fifth* is the nasty one. Note that so far every definition we've referred to in an annotation has been *before* the definition of the annotation. What if we want to refer to something defined *after* the annotation?
def outerfn(): class OuterCls: class InnerCls: def method(a:zebra=None): pass ...
We haven't seen the definition of "zebra" yet, so we don't know what approach to take. It could be any of the previous four scenarios. What do we do?
If you agree with me that (3) and (4) are unnecessary (or even undesirable), the options here are either that zebra is a local in outerfn() (then just make it a closure), and if it isn't you should treat it as a global.
This is solvable too: we simply delay the compilation of __co_annotations__ code objects until the very last possible moment. First, at the time we bind the class or function, we generate a stub __co_annotations__ object, just to give the compiler what it expects. The compiler inserts it into the const table for the enclosing construct (function / class / module), and we remember what index it went into. Then, after we've finished processing the entire AST tree for this module, but before we we exit the compiler, we reconstruct the required context for evaluating each __co_annotations__ function--the nested chain of symbol tables, the compiler blocks if needed, etc--and evaluate the annotations for real. We assemble the correct __co_annotations__ code object and overwrite the stub in the const table with this now-correct value.
I can't think of any more scenarios. So, I think I can handle basically anything!
However, there are two scenarios where the behavior of evaluations will change in a way the user might find surprising. The first is when they redefine a variable used in an annotation:
x = str def fn(a:x="345"): pass x = int
With stock semantics, the annotation to "a" will be "str". With PEP 563 or my PEP, the annotation to "a" will be "int". (It gets even more exciting if you said "del x".)
This falls under the Garbage in, Garbage out principle. Mypy doesn't even let you do this. Another type checker which is easy to install, pyright, treats it as str. I wouldn't worry too much about it. If you strike the first definition of x, the pyright complains and mypy treats it as int.
Similarly, delaying the annotations so that we make everything visible means defining variables with the same name in multiple scopes may lead to surprising behavior.
x = str class Outer: def method(a:x="345"): pass x = int
Again, stock gets you an annotation of "str", but PEP 563 and my PEP gets you "str", because they'll see the *final* result of evaluating the body of Outer.
Sadly this is the price you pay for delayed evaluation of annotations. Delaying the evaluation of annotations is the goal, and the whole point is to make changes, observable by the user, in how annotations are evaluated. All we can do is document these behaviors and hope our users forgive us.
Agreed.
I think this is a vast improvement over the first draft of my PEP, and assuming nobody points out major flaws in this approach (and, preferably, at least a little encouragement), I plan to redesign my prototype along these lines. (Though not right away--I want to take a break and attend to some other projects first.)
Thanks for the mind-blowing suggestions, Guido! I must say, you're pretty good at this Python stuff.
You're not so bad yourself -- without your wakeup call we would have immortalized PEP 563's limitations. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 16/01/21 9:38 am, Guido van Rossum wrote:
On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we already know globals work fine.
I think this is going too far. A static method defined in InnerCls does not see InnerCls (even after the class definitions are complete). E.g. ``` class Outer: class Inner: @staticmethod def foo(): return Inner ``` If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not defined".
I'm not so sure about that. Conceptually, annotations are evaluated in the environment existing when the class scope is being constructed. The fact that we're moving them into a closure is an implementation detail that I don't think should be exposed.
What if we want to refer to something defined /after/ the annotation?
def outerfn(): class OuterCls: class InnerCls: def method(a:zebra=None): pass ...
We haven't seen the definition of "zebra" yet, so we don't know what approach to take.
I don't think that should be a problem. The compiler already knows about all the assignments occurring in a scope before starting to generate code for it. -- Greg
On Fri, Jan 15, 2021 at 4:45 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 16/01/21 9:38 am, Guido van Rossum wrote:
On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we already know globals work fine.
[Above was what Larry wrote, the rest is me. I guess Greg's mailer is having trouble with the GMail-style quoting. :-( ]
I think this is going too far. A static method defined in InnerCls does
not see InnerCls (even after the class definitions are complete). E.g. ``` class Outer: class Inner: @staticmethod def foo(): return Inner ``` If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not defined".
[Greg]
I'm not so sure about that. Conceptually, annotations are evaluated in the environment existing when the class scope is being constructed. The fact that we're moving them into a closure is an implementation detail that I don't think should be exposed.
Yeah, that wasn't very clear, and I'm not 100% sure I got it right. But consider this: ``` class Outer: foo = 1 class Inner: print(foo) ``` This gives "NameError: name 'foo' is not defined". And here there is no forward reference involved, and foo lives in the exactly the same scope/namespace as Inner. The reason for the NameError is that class scopes don't participate in the closure game (an intentional design quirk to avoid methods referencing unqualified class variables). So I still think that Larry's example shouldn't (have to) work. (I agree with Greg on the 'zebra' example.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 16/01/21 2:09 pm, Guido van Rossum wrote:
Yeah, that wasn't very clear, and I'm not 100% sure I got it right. But consider this: ``` class Outer: foo = 1 class Inner: print(foo)
That's true. So maybe the user should have to be explicit in cases like this: class Outer: class Inner: def f(x: Outer.Inner): ... However, I think cases like this should work: class C: t = List[int] def f(x: t): ... even though the closure placed in C.__co_annotations__ wouldn't normally have access to t without qualification. -- Greg
On Fri, Jan 15, 2021 at 18:15 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 16/01/21 2:09 pm, Guido van Rossum wrote:
Yeah, that wasn't very clear, and I'm not 100% sure I got it right. But consider this: ``` class Outer: foo = 1 class Inner: print(foo)
That's true. So maybe the user should have to be explicit in cases like this:
class Outer: class Inner: def f(x: Outer.Inner): ...
However, I think cases like this should work:
class C: t = List[int] def f(x: t): ...
even though the closure placed in C.__co_annotations__ wouldn't normally have access to t without qualification.
Yes, the immediately surrounding scope should be accessible for annotations even if it’s a class.
--
--Guido (mobile)
Given your comments below, I'd summarize the semantics you want as: Looking up names for annotations should work exactly as it does today with "stock" semantics, except annotations should also see names that haven't been declared yet. Thus an annotation should be able to see names set in the following scopes, in order of most-preferred to least-preferred: * names in the current scope (whether the current scope is a class body, function body, or global), * names in enclosing /function/ scopes, up to but not including the first enclosing /class/ scope, and * global scope, whether they are declared before or after the annotation. If the same name is defined multiple times, annotations will prefer the definition from the "nearest" scope, even if that definition hasn't been evaluated yet. For example: x = int def foo(): def bar(a:x): pass x = str Here a would be annotated with "str". Ambiguous conditions (referring to names that change value, referring to names that may be deleted) will result in undefined behavior. Does that sound right? Thanks for the kind words, //arry/ On 1/15/21 12:38 PM, Guido van Rossum wrote:
On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
Sorry it took me 3+ days to reply--I had a lot to think about here. But I have good things to report!
On 1/11/21 8:42 PM, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.
So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.
I think you're misreading PEP 563 here. The mention of globalns=None, localns=None refers to the fact that these parameters have defaults, not that you must pass None. Note that the next paragraph in that PEP mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).
I think that's misleading, then. The passage is telling you how to "correctly evaluate[s] expressions", and how I read it was, it's telling me I have to supply globalns=None and localns=None for it to work correctly--which, I had to discover on my own, were the default values. I don't understand why PEP 563 feels compelled to define a function that it's not introducing, and in fact had already shipped with Python two versions ago.
I suppose PEP 563 is ambiguous because on the one hand global symbols are the only things that work out of the box, on the other hand you can make other things work by passing the right scope (and there's lots of code now that does so), and on the third hand, it claims that get_type_hints() adds the class scope, which nobody noticed or implemented until this week (there's a PR, can't recall the number).
But I think all this is irrelevant given what comes below.
Later in that same section, PEP 563 points out a problem with annotations that reference class-scoped variables, and claims that the implementation would run into problems because methods can't "see" the class scope. This is indeed a problem for PEP 563, but *you* can easily generate correct code, assuming the containing class exists in the global scope (and your solution requires that anyway). So in this case ``` class Outer: class Inner: ... def method(self, a: Inner, b: Outer) -> None: ... ``` The generated code for the `__annotations__` property could just have a reference to `Outer.Inner` for such cases: ``` def __annotations__(): return {"a": Outer.Inner, "b": Outer, "return": None} ```
This suggestion was a revelation for me. Previously, a combination of bad experiences early on when hacking on compile and symtable, and my misunderstanding of exactly what was being asserted in the November 2017 thread, led me to believe that all I could support was globals. But I've been turning this over in my head for several days now, and I suspect I can support... just about anything.
I can name five name resolution scenarios I might encounter. I'll discuss them below, in increasing order of difficulty.
*First* is references to globals / builtins. That's already working, it's obvious how it works, and I need not elaborate further.
Yup.
*Second* is local variables in an enclosing function scope:
def outer_fn(): class C: pass def inner_fn(a:C=None): pass return inner_fn
As you pointed out elsewhere in un-quoted text, I could make the annotation a closure, so it could retain a reference to the value of (what is from its perspective) the free variable "C".
Yup.
*Third* is local variables in an enclosing class scope, as you describe above:
class OuterCls: class InnerCls: def method(a:InnerCls=None): pass
If I understand what you're suggesting, I could notice inside the compiler that Inner is being defined in a class scope, walk up the enclosing scopes until I hit the outermost class, then reconstruct the chain of pulling out attributes until it resolves globally. Thus I'd rewrite this example to:
class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we already know globals work fine.
I think this is going too far. A static method defined in InnerCls does not see InnerCls (even after the class definitions are complete). E.g. ``` class Outer: class Inner: @staticmethod def foo(): return Inner ``` If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not defined".
*Fourth* is local variables in an enclosing class scope, which are themselves local variables in an enclosing function scope:
def outerfn(): class OuterCls: class InnerCls: def method(a:InnerCls=None): pass return OuterCls.InnerCls
Even this is solvable, I just need to combine the "second" and "third" approaches above. I walk up the enclosing scopes to find the outermost class scope, and if that's a function scope, I create a closure and retain a reference to /that/ free variable. Thus this would turn into
def outerfn(): class OuterCls: class InnerCls: def method(a:OuterCls.InnerCls=None): pass
and method.__co_annotations__ would reference the free variable "OuterCls" defined in outerfn.
Probably also not needed.
*Fifth* is the nasty one. Note that so far every definition we've referred to in an annotation has been /before/ the definition of the annotation. What if we want to refer to something defined /after/ the annotation?
def outerfn(): class OuterCls: class InnerCls: def method(a:zebra=None): pass ...
We haven't seen the definition of "zebra" yet, so we don't know what approach to take. It could be any of the previous four scenarios. What do we do?
If you agree with me that (3) and (4) are unnecessary (or even undesirable), the options here are either that zebra is a local in outerfn() (then just make it a closure), and if it isn't you should treat it as a global.
This is solvable too: we simply delay the compilation of __co_annotations__ code objects until the very last possible moment. First, at the time we bind the class or function, we generate a stub __co_annotations__ object, just to give the compiler what it expects. The compiler inserts it into the const table for the enclosing construct (function / class / module), and we remember what index it went into. Then, after we've finished processing the entire AST tree for this module, but before we we exit the compiler, we reconstruct the required context for evaluating each __co_annotations__ function--the nested chain of symbol tables, the compiler blocks if needed, etc--and evaluate the annotations for real. We assemble the correct __co_annotations__ code object and overwrite the stub in the const table with this now-correct value.
I can't think of any more scenarios. So, I think I can handle basically anything!
However, there are two scenarios where the behavior of evaluations will change in a way the user might find surprising. The first is when they redefine a variable used in an annotation:
x = str def fn(a:x="345"): pass x = int
With stock semantics, the annotation to "a" will be "str". With PEP 563 or my PEP, the annotation to "a" will be "int". (It gets even more exciting if you said "del x".)
This falls under the Garbage in, Garbage out principle. Mypy doesn't even let you do this. Another type checker which is easy to install, pyright, treats it as str. I wouldn't worry too much about it. If you strike the first definition of x, the pyright complains and mypy treats it as int.
Similarly, delaying the annotations so that we make everything visible means defining variables with the same name in multiple scopes may lead to surprising behavior.
x = str class Outer: def method(a:x="345"): pass x = int
Again, stock gets you an annotation of "str", but PEP 563 and my PEP gets you "str", because they'll see the /final/ result of evaluating the body of Outer.
Sadly this is the price you pay for delayed evaluation of annotations. Delaying the evaluation of annotations is the goal, and the whole point is to make changes, observable by the user, in how annotations are evaluated. All we can do is document these behaviors and hope our users forgive us.
Agreed.
I think this is a vast improvement over the first draft of my PEP, and assuming nobody points out major flaws in this approach (and, preferably, at least a little encouragement), I plan to redesign my prototype along these lines. (Though not right away--I want to take a break and attend to some other projects first.)
Thanks for the mind-blowing suggestions, Guido! I must say, you're pretty good at this Python stuff.
You're not so bad yourself -- without your wakeup call we would have immortalized PEP 563's limitations.
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 1/11/2021 1:10 PM, Larry Hastings wrote:
Certainly. I'm just another victim in the copy-and-paste wars.
I actually have write access to the PEPs repo (I'm a former release manager) so I'd be happy to check it in myself once it gets a number, however that happens. Before I do so I'll study PEP 12 as if it was gonna be on tomorrow's midterms.
It gets a number by you renaming your file to the next available number and checking it in. There's a race condition, so act fast! Eric
//arry/
On 1/11/21 9:59 AM, Chris Angelico wrote:
On Tue, Jan 12, 2021 at 4:22 AM Larry Hastings<larry@hastings.org> wrote:
I've written a new PEP. Please find it below. Happy reading!
Can this get added to the PEPs repo and assigned a number and such?
BTW, the currently preferred wording for the copyright blurb is slightly different. If you're the sole author of this text, can you please consider the license terms shown in PEP 12?
ChrisA PEP editor - if you need a hand, I'm here to help
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AHFBMKWO... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for this detailed PEP and analysis, and for the interesting discussion in your separate thread. I’m glad to see this work that we chatted about all that time before has coalesced into a PEP. FYI: For those with write access to the PEPs repo, PEP number assignments are self-serve. Just grab the next available one and manage any push race conditions accordingly. Question:
On Jan 11, 2021, at 09:21, Larry Hastings <larry@hastings.org> wrote:
from __future__ import co_annotations -------------------------------------
In the prototype, the semantics presented in this PEP are gated with:
from __future__ import co_annotations
Given that PEP 563 is now the default in unreleased Python 3.10, does it make sense to introduce yet another __future__ import? What would happen if you just piggybacked your idea onto that change? -Barry
On 1/11/21 10:29 AM, Barry Warsaw wrote:
Given that PEP 563 is now the default in unreleased Python 3.10, does it make sense to introduce yet another __future__ import? What would happen if you just piggybacked your idea onto that change?
Part of my proposal is to deprecate PEP 563's semantics. If -> PEP 649 <- was accepted, we'd undo making PEP 563 the default behavior in 3.10; the behavior would instead remain gated behind the "from __future__ import annotations". It'd then go through a standard deprecation cycle (which is, what, three versions?) before finally being removed. (If you look at the revision history of my repo, you'll see that my first checkin was to reverse Batuhan's checkin from October 6, restoring the "from __future__" gate for annotations. Sorry, Batuhan!) Frankly I'd be astonished if -> PEP 649 <- received such unanimous acceptance that it become the new default Python semantics without a "from __future__" introductory period. You'd need a bigger brain than I have to think through all the ramifications of that sort of radical decision! But if the steering committee requested it, I don't expect I'd put a fight. Cheers, //arry/
Could you be more explicit about what is banned by the control-flow exclusion? I'm assuming that: class A: bar=float if FOO: bar=int def a(x:int, y:int)->int # function defined with annotations inside control flow return x+y def b(x:bar) # function annotated with value that depends on control flow is OK, and you're just talking about direct access to (the unfinished class or module).__annotations__ but I'm not certain. -jJ
The control-flow exclusion is for /module//attribute/ or /class attribute/ annotations: class C: if random.random() > 0.5: my_attr:int=3 else: my_attr2:float=3.5 Your example doesn't define any module attributes or class attributes inside flow control statements, so that code should work fine. (Defining functions/methods inside flow control statements isn't a problem.) Cheers, //arry/ On 1/11/21 1:39 PM, Jim J. Jewett wrote:
Could you be more explicit about what is banned by the control-flow exclusion?
I'm assuming that:
class A: bar=float if FOO: bar=int def a(x:int, y:int)->int # function defined with annotations inside control flow return x+y
def b(x:bar) # function annotated with value that depends on control flow
is OK, and you're just talking about direct access to (the unfinished class or module).__annotations__ but I'm not certain.
-jJ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VMPMQWCG... Code of Conduct: http://python.org/psf/codeofconduct/
Larry Hastings wrote:
The control-flow exclusion is for /module//attribute/ or /class attribute/ annotations: class C: if random.random() > 0.5: my_attr:int=3 else: my_attr2:float=3.5
That very example would be helpful in the FAQ, though I understand if you're concerned about making a minor sub-section seem too long. If I understand correctly, the problem is that you can't store multiple alternative annotations on my_attr. Therefore: class C: my_attr:(int if random.random > 0.5 else float) should be OK, because there is only a single annotation. What about optional attributes, like: class C: if random.random() > 0.5: my_attr:int=3 Also, would (conditionally defined) function variable attributes become a problem if they were actually stored? (Take Larry's class example, and make if a def instead of a class statement.) My (weakly held, personal) opinion is that these restrictions would be reasonable, and a single release of deprecation would be enough, but it would be better if that code could trigger a deprecation warning during that release, even for code that hasn't done the future import. It would also be OK to just say "implementation-defined behavior; CPython 3.x ignores the annotation" instead of banning them. -jJ
On 1/12/21 11:26 AM, Jim J. Jewett wrote:
If I understand correctly, the problem is that you can't store multiple alternative annotations on my_attr. Therefore:
class C: my_attr:(int if random.random > 0.5 else float)
should be OK, because there is only a single annotation.
Sure, that works fine. Any expression (except "yield" and ":=") is okay in an annotation.
What about optional attributes, like:
class C: if random.random() > 0.5: my_attr:int=3
Also, would (conditionally defined) function variable attributes become a problem if they were actually stored? (Take Larry's class example, and make if a def instead of a class statement.)
You mean attributions on function locals? def foo(): if random.random() > 0.5: x:int=3 else: x:float=3.5 As I mentioned in my PEP, attributions on function locals have no effect at runtime. If they did, this would cause the same problem that doing it in classes has. Cheers, //arry/
On Tue, Jan 12, 2021 at 11:31 AM Jim J. Jewett <jimjjewett@gmail.com> wrote:
Larry Hastings wrote:
The control-flow exclusion is for /module//attribute/ or /class attribute/ annotations: class C: if random.random() > 0.5: my_attr:int=3 else: my_attr2:float=3.5
That very example would be helpful in the FAQ, though I understand if you're concerned about making a minor sub-section seem too long.
This elucidates a crucial point to me: Larry's proposal looks at the source code of the annotations.
If I understand correctly, the problem is that you can't store multiple alternative annotations on my_attr. Therefore:
class C: my_attr:(int if random.random > 0.5 else float)
should be OK, because there is only a single annotation.
Does that mean that the generated function would contain the entire expression `(int if random.random > 0.5 else float)`? I guess that's what it has to mean. But the PEP only uses such simple examples that it's easy to miss this. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Hello, On Mon, 11 Jan 2021 13:44:45 -0800 Larry Hastings <larry@hastings.org> wrote:
The control-flow exclusion is for /module//attribute/ or /class attribute/ annotations:
class C: if random.random() > 0.5: my_attr:int=3 else: my_attr2:float=3.5
Ok, let's take "module attribute" as an example. Why do you think there's anything wrong with this code: ====== import config from .types import * if config.SUPPORT_BIGINT: var: bigint = 1 else: var: int64 = 1 ======
Your example doesn't define any module attributes or class attributes inside flow control statements, so that code should work fine. (Defining functions/methods inside flow control statements isn't a problem.)
PEP649 criticizes PEP563's approach with big words like "It requires Python implementations to stringize their annotations. This is surprising behavior — unprecedented for a language-level feature." But itself devolves to clauses like:
It seems reasonable to declare that both are at the very least unsupported, and their use results in undefined behavior. It might be worth making a small effort to explicitly prohibit them with compile-time checks.
Isn't the fact that PEP563 doesn't have problems with annotations in conditionals is a sign of PEP563 technical superiority? And its "unprecedented" measure of storing annotations as strings is actually a clever technical feat - it should have stored annotations as AST trees, but such trees would take quite a lot of memory. So, PEP563 smartly went to store those ASTs in a serialized format. So, those strings aren't strings, but serialized ASTs. Overall 2 comments/questions: 1. Was there an attempt to devise how to make PEP649 deal with existing Python language features (like conditionals)? 2. As a general comment, PEP649, by placing arbitrary restrictions on where annotations can appear, tries to dig under the foundations of Python as a dynamic language. Which is absolutely great, it just the PEP should be viewed as such - undermining Python's dynamic nature, as if there's something wrong with it. -- Best regards, Paul mailto:pmiscml@gmail.com
Paul Sokolovsky wrote:
Ok, let's take "module attribute" as an example. Why do you think there's anything wrong with this code: ====== import config from .types import * if config.SUPPORT_BIGINT: var: bigint = 1 else: var: int64 = 1
"Wrong" is too strong, but it would be better as mybigint = bigint if config.SUPPORT_BIGINT else int64 ... var:mybigint = 1 so asking people to rewrite it that way over the course of a major release is probably an acceptable price. -jJ
Hello, On Wed, 13 Jan 2021 05:04:36 -0000 "Jim J. Jewett" <jimjjewett@gmail.com> wrote:
Paul Sokolovsky wrote:
Ok, let's take "module attribute" as an example. Why do you think there's anything wrong with this code: ====== import config from .types import * if config.SUPPORT_BIGINT: var: bigint = 1 else: var: int64 = 1
"Wrong" is too strong, but it would be better as
mybigint = bigint if config.SUPPORT_BIGINT else int64 ... var:mybigint = 1
What's the explanation of why the above is better? It seems following is ok with PEP649: if config.LAYOUT_INT: @dataclass class MyData: val: int else: @dataclass class MyData: val: float So, how to explain to people that using the normal "if" is ok when defining classes/dataclasses, but suddenly not normal when defining just variables, and people should switch to the "if" expression?
so asking people to rewrite it that way over the course of a major release is probably an acceptable price.
But why haste to ask people to rewrite their code? Why not start with saying that PEP649 is not backward compatible, and ask it to explain why it has pretty arbitrary limitations and discrepancies like above? Then ask it how it can achieve backward compatibility? And that way is obvious - the smart code objects which PEP649 creates, they should store annotations just like PEP563 does, in a serialized form. Then those smart code objects would deserialize and evaluate them. They may even cache the end result. But wait, PEP563 already has all that! It provides public API to get annotations, typing.get_type_hints(), which already does all the deserialization (maybe it doesn't do caching - *yet*), and effectively treats __annotations__ as implementation detail. Because clearly, the format of information stored there already depends on a particular CPython version, and if you believe a thread running in parallel, going to change going forward. Seen like that, PEP649 is just a quest for making __annotations__ be the "public API", instead of the already defined public API, and which __annotations__ already can't be, as its format already varies widely (and likely will keep varying going forward). And while questing for that elusive goal, it even adds arbitrary restrictions for usage of annotations which never were there before, truly breaking backward compatibility and some annotation usages. -- Best regards, Paul mailto:pmiscml@gmail.com
On 11 Jan 2021, at 18:21, Larry Hastings <larry@hastings.org> wrote:
I've written a new PEP. Please find it below. Happy reading!
Interesting! I like the clever lazy-evaluation of the __annotations__ using a pre-set code object. My only real reservation is that the transition process will be weird but I don't have much to offer in terms of how to smooth it out. I have two questions though: 1. What do you anticipate the memory usage will look like for your solution compared to PEP 563? To give you an example, EdgeDB is a sizeable application with 100k SLOC of Python. It's got around 3,500 typed functions, all in all >71% type coverage. EdgeDB uses stringified annotations exclusively which minimizes runtime memory usage of annotations because those strings are pretty much all ASCII and many can be interned. Does it matter? It does, actually. Let's look at 20 most popular annotations in the codebase and how often they appear: 946 -> s_schema.Schema 362 -> str 298 -> sd.CommandContext 118 -> base.PLBlock 107 -> irast.Set 99 -> CommandContext 95 -> Any 86 -> qlast.DDLOperation 85 -> s_types.Type 71 -> bool 70 -> irast.PathId 67 -> int 54 -> context.Environment 46 -> so.Object 45 -> pgast.Query 42 -> uuid.UUID 38 -> irast.Base 38 -> sn.Name 37 -> pgast.SelectStmt 33 -> context.ReplContext (this list tapers of with a long tail after) Turns out most annotations are simple and predictable. (As a side note: we could make interning even stronger for this purpose if we allowed periods and square brackets for interning.) 2. What is your expected startup performance of an annotated Python application using co_annotations? The stringification process which your PEP describes as costly only happens during compilation of a .py file to .pyc. Since pip-installing pre-compiles modules for the user at installation time, there is very little runtime penalty for a fully annotated application. Cheers, Ł
On 2021-01-11, Łukasz Langa wrote:
The stringification process which your PEP describes as costly only happens during compilation of a .py file to .pyc. Since pip-installing pre-compiles modules for the user at installation time, there is very little runtime penalty for a fully annotated application.
It should be possible to make Larry's approach cheap as well. I have an old experiment stashed away[1] where I made the code object for functions to be lazily created. I.e. when a module is first loaded, functions are not fully loaded until they are first executed. My goal was to reduce startup time. It didn't show a significant gain so I didn't pursue it further. In my experiment, I deferred the unmarshal of the code object. However, it occurs to me you could go a bit further and have the function object be mostly skeletal until someone runs it or tries to inspect it. The skeleton would really be nothing but a file offset (or memory offset, if using mmap) into the .pyc file. Of course this would be some work to implement but then all Python functions would benefit and likely Python startup time would be reduced. I think memory use would be reduced too since typically you import a lot of modules but only use some of the functions in them. I like the idea of Larry's PEP. I understand why the string-based annotations was done (I use the __future__ import for my own code). Using eval() is ugly though and Larry's idea seems like a nice way to remove the need to call eval(). [1] https://github.com/nascheme/cpython/commits/lazy_codeobject
On 1/11/21 2:32 PM, Łukasz Langa wrote:
1. What do you anticipate the memory usage will look like for your solution compared to PEP 563?
It depends on the scenario. I talk about three runtime scenarios in PEP 649. But the most relevant scenario is "annotations are defined but never examined", because it's by far the most common for people using annotations. So for now let's just talk about that. In this scenario, I expect PEP 649 to be on par with PEP 563. PEP 563 will define a small dict that nobody looks at; PEP 649 will define a simple code object that nobody runs. These objects consume pretty similar amounts of memory. A quick experiment: on my 64-bit Linux laptop, with a function that had three annotated parameters, sys.sizeof() of the resulting annotation dict was 232 bytes. PEP 649 generated a 176 byte code object--but we should also factor in its bytecode (45 bytes) and lnotab (33 bytes), giving us 257 bytes. (The other fields of the code object are redundant references to stuff we already had lying around.) In that case PEP 649 is slightly bigger. But if we change it to twelve annotated parameters, PEP 649 becomes a big win. The dict is now 640 bytes (!), but the code object only totals 280 bytes. It seems to flip at about five parameters; less than that, and the dict wins a little, greater than that and the code object starts winning by more and more.
2. What is your expected startup performance of an annotated Python application using co_annotations?
Again, the most relevant scenario is "annotations are defined but not referenced" so we'll stick with that. On balance it should be roughly equivalent to "PEP 563" semantics, and perhaps a teeny-tiny bit faster. With PEP 563 semantics, defining a function / class / module with annotations must build the annotations dict, then store it on the object. But all the keys and values are strings, so the bytecode isn't much--for functions, it's just a bunch of LOAD_CONSTs then a BUILD_CONST_KEY_MAP. For classes and modules it's a bit wordier, but if the bytecode performance was important here, we could probably convert it to use BUILD_CONST_KEY_MAP too. With my PEP, defining a function / class / module with annotations means you LOAD_CONST the code object, then store it on the object--and that's it. (Right now there's the whole __globals__ thing but I expect to get rid of that eventually). Of course, the code object isn't free, it has to be unmarshalled--but as code objects go these are pretty simple ones. Annotations code objects tend to have two custom bytestrings and a non-"small" int, and all the other attributes we get for free. "stock" Python semantics is a bit slower than either, because it also evaluates all the annotations at the time the function / class / module is bound. I'd love to hear real-world results from someone with a large annotated code base. Unfortunately, I'm pretty sure typing.py is broken in the prototype right now, so it's a bit early yet. (I honestly don't think it'll be that hard to get it working again, it was just one of a million things and I didn't want to hold up releasing this stuff to the world any longer.)
The stringification process which your PEP describes as costly only happens during compilation of a .py file to .pyc. Since pip-installing pre-compiles modules for the user at installation time, there is very little runtime penalty for a fully annotated application.
I never intended to suggest that the stringification process /itself/ is costly at runtime--and I don't think I did. Because, as you point out, it isn't. PEP 563 semantics writ large are costly at runtime only when annotations are examined, because you have to call eval(), and calling eval() is expensive. If the PEP does say that stringification is itself expensive at runtime, please point it out, and I'll fix it. Cheers, //arry/
On 12/01/21 11:32 am, Łukasz Langa wrote:
EdgeDB uses stringified annotations exclusively which minimizes runtime memory usage of annotations because those strings are pretty much all ASCII and many can be interned.
946 -> s_schema.Schema 362 -> str 298 -> sd.CommandContext
Seems to me the code objects for those would be identical wherever they're used, and so could be cached and re-used the same way as interned strings. -- Greg
On 12/01/21 6:21 am, Larry Hastings wrote:
Unbound code objects --------------------
...The "annotation code object" is then stored *unbound* as the internal value of ``__co_annotations__``; it is then bound on demand when the user asks for ``__annotations__``.
This seems like premature optimisation. Function objects are tiny compared to the code object, which is already a fairly complicated thing composed of a number of sub-objects. -- Greg
On 1/11/21 4:55 PM, Greg Ewing wrote:
On 12/01/21 6:21 am, Larry Hastings wrote:
Unbound code objects --------------------
...The "annotation code object" is then stored *unbound* as the internal value of ``__co_annotations__``; it is then bound on demand when the user asks for ``__annotations__``.
This seems like premature optimisation. Function objects are tiny compared to the code object, which is already a fairly complicated thing composed of a number of sub-objects.
I'll have to let people with large code bases speak up about this, but my understanding is that most people would prefer Python to use less memory. On my 64-bit Linux machine, a code object is 136 bytes, its empty __dict__ is 64 bytes, and the other stuff you get for free. So that's 200 bytes even. Multiply that by 1000 and the back of my envelope says you've wasted 200k. Is that a big deal? I dunno. On the other hand, the code to support dynamically binding the code object on demand wasn't a big deal. Cheers, //arry/
On 1/11/21 5:37 PM, Larry Hastings wrote:
I'll have to let people with large code bases speak up about this, but my understanding is that most people would prefer Python to use less memory. On my 64-bit Linux machine, a code object is 136 bytes, its empty __dict__ is 64 bytes, [...]
Oops: where I said "a code object is 136 bytes", I meant, a /function/ object is 136 bytes. We regret the error. //arry/
Performance -----------
Performance with this PEP should be favorable. In general, resources are only consumed on demand—"you only pay for what you use".
Nice!
There are three scenarios to consider:
* the runtime cost when annotations aren't defined, * the runtime cost when annotations are defined but *not* referenced, and * the runtime cost when annotations are defined *and* referenced.
Note: The first two cases are major. Many codes doesn't have annotations. Many codes use annotations just only for documentation or static checker. In the second scenario, the annotations must be very cheap. Its cost must be comparable with docstrings. Otherwise, people can not use annotation freely in a large codebase. Or we must provide an option like -OO to strip annotations.
We'll examine each of these scenarios in the context of all three semantics for annotations: stock, PEP 563, and this PEP.
When there are no annotations, all three semantics have the same runtime cost: zero. No annotations dict is created and no code is generated for it. This requires no runtime processor time and consumes no memory.
When annotations are defined but not referenced, the runtime cost of Python with this PEP should be slightly faster than either original Python semantics or PEP 563 semantics. With those, the annotations dicts are built but never examined; with this PEP, the annotations dicts won't even be built. All that happens at runtime is the loading of a single constant (a simple code object) which is then set as an attribute on an object. Since the annotations are never referenced, the code object is never bound to a function, the code to create the dict is never executed, and the dict is never constructed.
Note that PEP 563 semantics allows more efficient implementation. Annotation is just a single constant tuple, not a dict. We already have the efficient implementation for Python 3.10. The efficient implementation in 3.10 can share tuples. If there are hundreds of methods with the same signature, annotation is just a single tuple, not hundreds of tuples. This is very efficient for auto generated codebase. I think this PEP can share the code objects for same signature by removing co_firstlineno information too. Additionally, we should include the cost for loading annotations from PYC files, because most annotations are "load once, set once". Loading "simple code object" from pyc files is not so cheap. It may affect importing time of large annotated codebase and memory footprints. I think we need a reference application that has a large codebase and highly annotated. But we need to beware even if the large application is 100% annotated, libraries used are not 100% annotated. Many libraries are dropping Python 2 support and start annotating. The cost of the annotations will become much more important in next several years.
On 1/11/21 5:33 PM, Inada Naoki wrote:
Note that PEP 563 semantics allows more efficient implementation. Annotation is just a single constant tuple, not a dict. We already have the efficient implementation for Python 3.10.
The efficient implementation in 3.10 can share tuples. If there are hundreds of methods with the same signature, annotation is just a single tuple, not hundreds of tuples. This is very efficient for auto generated codebase. I think this PEP can share the code objects for same signature by removing co_firstlineno information too.
That's very clever! My co_annotations repo was branched from before this feature was added, and I haven't pulled and merged recently. So I hadn't seen it.
Additionally, we should include the cost for loading annotations from PYC files, because most annotations are "load once, set once". Loading "simple code object" from pyc files is not so cheap. It may affect importing time of large annotated codebase and memory footprints.
I did some analysis in a separate message. The summary is, the code object for a single annotation costs us 232 bytes; that includes the code object itself, the bytestring for the bytecode, and the bytestring for the lnotab. This grows slowly as you add new parameters; the code object for ten parameters is 360 bytes. It seems possible to create a hybrid of these two approaches! Here's my idea: instead of the compiler storing a code object as the annotations argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to /recreate/ the code object at runtime--bytecode, lnotab, names, consts, etc. func_get_annotations would create the code object from that, bind it to a function object, call it, and return the result. These code-object-tuples would then be automatically shared in the .pyc file and at runtime the same way that 3.10 shares the tuples of stringized annotations today. That said, I suggest PEP 649's memory consumption isn't an urgent consideration in choosing to accept or reject it. PEP 649 is competitive in terms of startup time and memory usage with PEP 563, and PEP 563 was accepted and shipped with several versions of Python. Cheers, //arry/
On 1/12/21 8:47 AM, Larry Hastings wrote:
It seems possible to create a hybrid of these two approaches! Here's my idea: instead of the compiler storing a code object as the annotations argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to /recreate/ the code object at runtime--bytecode, lnotab, names, consts, etc. func_get_annotations would create the code object from that, bind it to a function object, call it, and return the result. These code-object-tuples would then be automatically shared in the .pyc file and at runtime the same way that 3.10 shares the tuples of stringized annotations today.
Note that this only works in the current version of the PEP / prototype, where annotations are strictly evaluated in module scope. If we start supporting closures, those need "cell" objects, which IIUC can't be marshalled. //arry/
On Tue, 2021-01-12 at 09:54 -0800, Larry Hastings wrote:
Note that this only works in the current version of the PEP / prototype, where annotations are strictly evaluated in module scope. If we start supporting closures, those need "cell" objects, which IIUC can't be marshalled. Since the __co_annotations__ function will get globals from the function it annotates, doesn't it get more than just module scope?
Paul
On 1/12/21 12:16 PM, Paul Bryan wrote:
On Tue, 2021-01-12 at 09:54 -0800, Larry Hastings wrote:
Note that this only works in the current version of the PEP / prototype, where annotations are strictly evaluated in module scope. If we start supporting closures, those need "cell" objects, which IIUC can't be marshalled.
Since the __co_annotations__ function will get globals from the function it annotates, doesn't it get more than just module scope?
I'm not sure what you're asking. And, uh, what's the difference between "globals" and "module scope"? //arry/
On 13/01/21 6:54 am, Larry Hastings wrote:
Note that this only works in the current version of the PEP / prototype, where annotations are strictly evaluated in module scope. If we start supporting closures, those need "cell" objects, which IIUC can't be marshalled.
The cells belong to the enclosing scope, not the function that uses them. I don't think they would interfere with sharing parts of code objects between identical annotations. -- Greg
On 13/01/21 5:47 am, Larry Hastings wrote:
instead of the compiler storing a code object as the annotations argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to /recreate/ the code object at runtime--bytecode, lnotab, names, consts, etc.
Would that tuple be significantly smaller than the corresponding code object, though? -- Greg
On 1/12/21 3:53 PM, Greg Ewing wrote:
On 13/01/21 5:47 am, Larry Hastings wrote:
instead of the compiler storing a code object as the annotations argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to /recreate/ the code object at runtime--bytecode, lnotab, names, consts, etc.
Would that tuple be significantly smaller than the corresponding code object, though?
It would only be slightly smaller. The point of doing it would be to boil out fields that change per-object (e.g. co_name) so that functions with identical signatures would share the same tuple both in the .pyc and at runtime. This idea is predicated on Inada-san's assertion that this is an important memory optimization, that there are large heavily-annotated projects with lots of functions/methods with identical signatures where this memory savings is significant. //arry/
On Wed, Jan 13, 2021 at 1:47 AM Larry Hastings <larry@hastings.org> wrote:
On 1/11/21 5:33 PM, Inada Naoki wrote:
Note that PEP 563 semantics allows more efficient implementation. Annotation is just a single constant tuple, not a dict. We already have the efficient implementation for Python 3.10.
The efficient implementation in 3.10 can share tuples. If there are hundreds of methods with the same signature, annotation is just a single tuple, not hundreds of tuples. This is very efficient for auto generated codebase. I think this PEP can share the code objects for same signature by removing co_firstlineno information too.
That's very clever! My co_annotations repo was branched from before this feature was added, and I haven't pulled and merged recently. So I hadn't seen it.
Please see this pull request too. It merges co_code and co_consts. It will save more RAM and importing time of your implementation. https://github.com/python/cpython/pull/23056
Additionally, we should include the cost for loading annotations from PYC files, because most annotations are "load once, set once". Loading "simple code object" from pyc files is not so cheap. It may affect importing time of large annotated codebase and memory footprints.
I did some analysis in a separate message. The summary is, the code object for a single annotation costs us 232 bytes; that includes the code object itself, the bytestring for the bytecode, and the bytestring for the lnotab. This grows slowly as you add new parameters; the code object for ten parameters is 360 bytes.
It seems possible to create a hybrid of these two approaches! Here's my idea: instead of the compiler storing a code object as the annotations argument to MAKE_FUNCTION, store a tuple containing the fields you'd need to recreate the code object at runtime--bytecode, lnotab, names, consts, etc. func_get_annotations would create the code object from that, bind it to a function object, call it, and return the result. These code-object-tuples would then be automatically shared in the .pyc file and at runtime the same way that 3.10 shares the tuples of stringized annotations today.
It may be good idea if we can strip most code object members, like argcount, kwonlyargcount, nlocals, flags, freevars, cellvars, filename, name, firstlineno, linetable. It can be smaller than Python 3.9.
That said, I suggest PEP 649's memory consumption isn't an urgent consideration in choosing to accept or reject it. PEP 649 is competitive in terms of startup time and memory usage with PEP 563, and PEP 563 was accepted and shipped with several versions of Python.
I still want a real-world application/library with heavy annotation. My goal is to use annotations in the stdlib without caring about resource usage or importtime. But I agree with you if PEP 649 will be smaller than Python 3.9. Regards, -- Inada Naoki <songofacandy@gmail.com>
This PEP doesn't cover about what happened when __co_annotation__() failed (e.g. NameError). Forward reference is a major reason, but not a only reason for using string annotation. There are two other reasons: * Avoid importing heavy module. * Avoid circular imports. In these cases, this pattern is used: ``` from __future__ import annotations import typing from dataclasses import dataclass if typing.TYPE_CHECKING: import other_mod # do not want to import actually @dataclass class Foo: a: other_mod.spam b: other_mod.ham def fun(a: other_mod.spam, b: other_mod.ham) -> None: ... ``` Of course, mypy works well with string annotation because it is static checker. IPython shows signature well too: ``` In [3]: sample.Foo? Init signature: sample.Foo(a: 'other_mod.spam', b: 'other_mod.ham') -> None Docstring: Foo(a: 'other_mod.spam', b: 'other_mod.ham') ``` PEP 563 works fine in this scenario. How PEP 649 works? Regards,
If you never examine __annotations__, then you can refer to symbols that are never defined and nothing bad happens. It's just like writing a function that refers to undefined symbols, then never calling that function. If you examine __annotations__, and the annotations refer to values that aren't defined, the evaluation fails. This too works like you'd expect: the __co_annotation__ function raises NameError. So your IPython use case would raise a NameError. Note that the code is deliberately written to allow you to fix the name errors and try again. (The __co_annotations__ attribute is only cleared if calling it succeeds and it returns a legal value.) So, if you examine an annotation in IPython, and it fails with a NameError, you could import the missing module--or otherwise do what is needed to fix the problem--and try again. If your imports are complicated, you could always hide them in a function. I just tried this and it seems to work fine: def my_imports(): global other_mod import other_mod So, you could put all your imports in such a function, run it from inside a "if typing.TYPE_CHECKING" block, and you'd have a convenient way of doing all your imports from inside IPython too. One final note: with PEP 649, you can still use strings as annotations if you prefer. You just have to write them as strings manually. So the IPython use case could continue to work correctly that way. I realize that this itself causes minor problems--it means no syntax checking is done on the annotation, and it causes a little extra work for the user--but I assume this is a rare use case and most users won't need to bother. Cheers, //arry/ // On 1/16/21 11:43 PM, Inada Naoki wrote:
This PEP doesn't cover about what happened when __co_annotation__() failed (e.g. NameError).
Forward reference is a major reason, but not a only reason for using string annotation. There are two other reasons:
* Avoid importing heavy module. * Avoid circular imports.
In these cases, this pattern is used:
``` from __future__ import annotations import typing from dataclasses import dataclass
if typing.TYPE_CHECKING: import other_mod # do not want to import actually
@dataclass class Foo: a: other_mod.spam b: other_mod.ham
def fun(a: other_mod.spam, b: other_mod.ham) -> None: ... ```
Of course, mypy works well with string annotation because it is static checker. IPython shows signature well too:
``` In [3]: sample.Foo? Init signature: sample.Foo(a: 'other_mod.spam', b: 'other_mod.ham') -> None Docstring: Foo(a: 'other_mod.spam', b: 'other_mod.ham') ```
PEP 563 works fine in this scenario. How PEP 649 works?
Regards,
On Sun, Jan 17, 2021 at 7:33 AM Larry Hastings <larry@hastings.org> wrote:
If your imports are complicated, you could always hide them in a function. I just tried this and it seems to work fine:
def my_imports(): global other_mod import other_mod
So, you could put all your imports in such a function, run it from inside a "if typing.TYPE_CHECKING" block, and you'd have a convenient way of doing all your imports from inside IPython too.
But static type checkers won't understand such imports. (Or is this about annotations used for other purposes? Then I suppose it's fine, but only as long as you completely give up static type checks for modules that use this idiom.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 1/18/21 12:24 PM, Guido van Rossum wrote:
On Sun, Jan 17, 2021 at 7:33 AM Larry Hastings <larry@hastings.org <mailto:larry@hastings.org>> wrote:
If your imports are complicated, you could always hide them in a function. I just tried this and it seems to work fine:
def my_imports(): global other_mod import other_mod
So, you could put all your imports in such a function, run it from inside a "if typing.TYPE_CHECKING" block, and you'd have a convenient way of doing all your imports from inside IPython too.
But static type checkers won't understand such imports. (Or is this about annotations used for other purposes? Then I suppose it's fine, but only as long as you completely give up static type checks for modules that use this idiom.)
Oh, okay. I haven't used the static type checkers, so it's not clear to me what powers they do and don't have. It was only a minor suggestion anyway. Perhaps PEP 649 will be slightly inconvenient to people exploring their code inside IPython. Or maybe it'd work if they gated the if statement on running in ipython? if typing.TYPE_CHECKING or os.path.split(sys.argv[0])[1] == "ipython3": import other_mod Cheers, //arry/
On Tue, Jan 19, 2021 at 6:02 AM Larry Hastings <larry@hastings.org> wrote:
Oh, okay. I haven't used the static type checkers, so it's not clear to me what powers they do and don't have. It was only a minor suggestion anyway. Perhaps PEP 649 will be slightly inconvenient to people exploring their code inside IPython.
Not only IPython, but many REPLs. Especially, Jupyter notebook is the same to IPython. We can see string annotations even in CPython REPL via pydoc. ```
def func(a: "Optional[int]") -> "Optional[str]": ... ... ... help(func)
func(a: 'Optional[int]') -> 'Optional[str]' ``` Since this signature with type hints came from inspect.signature(func), all tools using inspect.signature() will be affected too. I think Sphinx autodoc will be affected, but I am not sure.
Or maybe it'd work if they gated the if statement on running in ipython?
if typing.TYPE_CHECKING or os.path.split(sys.argv[0])[1] == "ipython3": import other_mod
It is possible for heavy modules, but not possible to avoid circular imports. Additionally, there are some cases modules are not runtime importable. * Optional dependency, user may not install it. * Dummy modules having only "pyi" files. If PEP 563 becomes the default, we can provide a faster way to get the text signature without eval() annotated string. So eval() performance is not a problem here. Many type hinting use cases don't need type objects in runtime. So I think PEP 563 is better for type hinting user experience. Regards, -- Inada Naoki <songofacandy@gmail.com>
On 1/18/21 3:42 PM, Inada Naoki wrote:
Many type hinting use cases don't need type objects in runtime. So I think PEP 563 is better for type hinting user experience.
You mean, in situations where the user doesn't want to import the types, because of heavyweight imports or circular imports? I didn't think those were very common. //arry/
On Tue, Jan 19, 2021 at 8:54 AM Larry Hastings <larry@hastings.org> wrote:
On 1/18/21 3:42 PM, Inada Naoki wrote:
Many type hinting use cases don't need type objects in runtime. So I think PEP 563 is better for type hinting user experience.
You mean, in situations where the user doesn't want to import the types, because of heavyweight imports or circular imports? I didn't think those were very common.
Personally, I dislike any runtime overhead caused by type hints. That is one reason I don't use type hinting much for now. I don't want to import modules used only in type hints. I don't want to import even "typing". I planned to use type hinting after I can drop Python 3.6 support and use `from __future__ import annotations`. And I love lightweight function annotation implementation (*) very much. (*) https://github.com/python/cpython/pull/23316 I expect we can start to write type hints even in stdlibs, because it doesn't require extra imports and overhead become very cheap. Maybe, I am a minority. But I dislike any runtime overhead and extra dependencies. Regards, -- Inada Naoki <songofacandy@gmail.com>
On 19/01/21 1:13 pm, Inada Naoki wrote:
I don't want to import modules used only in type hints. I don't want to import even "typing".
How about having a pseudo-module called __typing__ that is ignored by the compiler: from __typing__ import ... would be compiled to a no-op, but recognised by type checkers. If you want to do run-time typing stuff, you would use from typing import ... -- Greg
Hello, On Tue, 19 Jan 2021 14:31:49 +1300 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 19/01/21 1:13 pm, Inada Naoki wrote:
I don't want to import modules used only in type hints. I don't want to import even "typing".
How about having a pseudo-module called __typing__ that is ignored by the compiler:
from __typing__ import ...
would be compiled to a no-op, but recognised by type checkers. If you want to do run-time typing stuff, you would use
Please don't limit it to just "typing". There's a need for a module which would handle "language-level" features, to not put newly added things in the global namespace. By analogy with __future__, such a module could be named __present__. Alternative names would be "lang" or "python". But analogy with __future__ is helpful, as there should be a place for "pragma imports", which would change behavior of the programs, like imports from __future__ do, except that features in __future__ are destined to be "switchable" only temporary and become default later. Breaking backward compatibility with each version has already become a norm, but going further, even more radical changes would be required, and so it should be possible to either enable or disable them, as part of the standard, not temporary, language semantics, hence the idea of __present__ as alternative to __future__.
from typing import ...
-- Greg
[] -- Best regards, Paul mailto:pmiscml@gmail.com
How about having a pseudo-module called __typing__ that is ignored by the compiler:
from __typing__ import ...
would be compiled to a no-op, but recognised by type checkers.
If you want to do run-time typing stuff, you would use There is already a way of doing that: `if typing.TYPE_CHECKING: ...` https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING But yes, the issue with it is that this constant is defined in the `typing` module … However, I think this is a part of the solution. Indeed, the language could define another builtin constants, let's name it `__static__`, which would simply be always false (at runtime), while linters/type checkers would use it the same way `typing.TYPE_CHECKING` is used: ```python if __static__: import typing import expensive_module ```
By the way, without adding an other constant, `__debug__` can also be used. It discards runtime overhead when it matters, in optimized mode.
On Mon, Feb 15, 2021 at 10:20 AM Joseph Perez <joperez@hotmail.fr> wrote:
How about having a pseudo-module called __typing__ that is ignored by the compiler:
from __typing__ import ...
would be compiled to a no-op, but recognised by type checkers.
If you want to do run-time typing stuff, you would use There is already a way of doing that: `if typing.TYPE_CHECKING: ...` https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING But yes, the issue with it is that this constant is defined in the `typing` module …
However, I think this is a part of the solution. Indeed, the language could define another builtin constants, let's name it `__static__`, which would simply be always false (at runtime), while linters/type checkers would use it the same way `typing.TYPE_CHECKING` is used: ```python if __static__: import typing import expensive_module ```
Please note that this is a thread about PEP 649. If PEP 649 accepted and PEP 563 dies, all such idioms breaks annotation completely. Users need to import all heavy modules and circular references used only type hints, or user can not get even string form annotation which is very useful for REPLs. -- Inada Naoki <songofacandy@gmail.com>
On Sun, Feb 14, 2021 at 7:17 PM Inada Naoki <songofacandy@gmail.com> wrote:
On Mon, Feb 15, 2021 at 10:20 AM Joseph Perez <joperez@hotmail.fr> wrote:
How about having a pseudo-module called __typing__ that is ignored by the compiler:
from __typing__ import ...
would be compiled to a no-op, but recognised by type checkers.
If you want to do run-time typing stuff, you would use There is already a way of doing that: `if typing.TYPE_CHECKING: ...`
https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING
But yes, the issue with it is that this constant is defined in the `typing` module …
However, I think this is a part of the solution. Indeed, the language could define another builtin constants, let's name it `__static__`, which would simply be always false (at runtime), while linters/type checkers would use it the same way `typing.TYPE_CHECKING` is used: ```python if __static__: import typing import expensive_module ```
Please note that this is a thread about PEP 649.
If PEP 649 accepted and PEP 563 dies, all such idioms breaks annotation completely.
Users need to import all heavy modules and circular references used only type hints, or user can not get even string form annotation which is very useful for REPLs.
Hm, that's a rather serious problem with Larry's PEP 649 compared to `from __future__ import annotations`, actually. Larry, what do you think? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
I don't work on these sorts of codebases, and I don't use type hints or static type checking. So I'm not really qualified to judge how bad / widespread a problem this is. It's my hope that the greater Python core dev / user community can ascertain how serious this is. My main observation is that, for users facing this problem, they still have options. Off the top of my head, they could: * maintain a lightweight "mock" version of expensive_module, or * stringize their type hints by hand, or * perhaps use with some hypothetical stringizing support library that makes it less-painful to maintain stringized annotations. (I assume that static type checkers could continue to support stringized type hints even if PEP 649 was accepted.) I admit I'd be very surprised if PEP 649 was judged to be unworkable, given how similar it is to stock Python semantics for annotations at runtime. Cheers, //arry/ On 2/15/21 8:14 PM, Guido van Rossum wrote:
On Sun, Feb 14, 2021 at 7:17 PM Inada Naoki <songofacandy@gmail.com <mailto:songofacandy@gmail.com>> wrote:
On Mon, Feb 15, 2021 at 10:20 AM Joseph Perez <joperez@hotmail.fr <mailto:joperez@hotmail.fr>> wrote: > > > How about having a pseudo-module called __typing__ that is > > ignored by the compiler: > > > > from __typing__ import ... > > > > would be compiled to a no-op, but recognised by type checkers. > > If you want to do run-time typing stuff, you would use > There is already a way of doing that: `if typing.TYPE_CHECKING: ...` https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING <https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING> > But yes, the issue with it is that this constant is defined in the `typing` module … > > However, I think this is a part of the solution. Indeed, the language could define another builtin constants, let's name it `__static__`, which would simply be always false (at runtime), while linters/type checkers would use it the same way `typing.TYPE_CHECKING` is used: > ```python > if __static__: > import typing > import expensive_module > ```
Please note that this is a thread about PEP 649.
If PEP 649 accepted and PEP 563 dies, all such idioms breaks annotation completely.
Users need to import all heavy modules and circular references used only type hints, or user can not get even string form annotation which is very useful for REPLs.
Hm, that's a rather serious problem with Larry's PEP 649 compared to `from __future__ import annotations`, actually.
Larry, what do you think?
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Please note that this is a thread about PEP 649.
If PEP 649 accepted and PEP 563 dies, all such idioms breaks annotation completely.
Users need to import all heavy modules and circular references used only type hints, or user can not get even string form annotation which is very useful for REPLs.
I don't see why `if TYPE_CHECKING:` idiom breaks annotations with PEP 649. There will be no error as long as `__annotations__` descriptor is not called. And currently in 3.9 (with or without `from __future__ import annotations`), the issue is the same: you `get_type_hints` fails if some of the types in the annotations have been imported in a `if TYPE_CHECKING:` block.
Hm, that's a rather serious problem with Larry's PEP 649 compared to from __future__ import annotations, actually.
As I've written above, this is not a new issue, and neither this PEP nor PEP 563 can fix it.
It is an issue if you use `__annotations__` directly and you are using PEP 563's `from __future__ import annotations`. This currently gives some strings that may or may not refer to existing globals. With Larry's PEP 649 it will raise an error. On Tue, Feb 16, 2021 at 9:35 AM Joseph Perez <joperez@hotmail.fr> wrote:
Please note that this is a thread about PEP 649.
If PEP 649 accepted and PEP 563 dies, all such idioms breaks annotation completely.
Users need to import all heavy modules and circular references used only type hints, or user can not get even string form annotation which is very useful for REPLs.
I don't see why `if TYPE_CHECKING:` idiom breaks annotations with PEP 649. There will be no error as long as `__annotations__` descriptor is not called. And currently in 3.9 (with or without `from __future__ import annotations`), the issue is the same: you `get_type_hints` fails if some of the types in the annotations have been imported in a `if TYPE_CHECKING:` block.
Hm, that's a rather serious problem with Larry's PEP 649 compared to from __future__ import annotations, actually.
As I've written above, this is not a new issue, and neither this PEP nor PEP 563 can fix it. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PVPCJV6G... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
PEP 649 doesn't prevent to use stringified annotations (Larry has previously mentioned it in its response to Paul Bryan), and they seem to be still required when `if TYPE_CHECKING:` is used, despite the PEP claim. And my last message bring some use cases where strings are also required (notably, in recursive dataclasses).
I certainly wouldn't want to keep `from __future__ import annotations` in the language forever if Larry's PEP is accepted. Of course you can still use explicit string literals in annotations. Your observation about the @dataclass decorator is significant. Thanks for that. On Tue, Feb 16, 2021 at 10:36 AM Joseph Perez <joperez@hotmail.fr> wrote:
PEP 649 doesn't prevent to use stringified annotations (Larry has previously mentioned it in its response to Paul Bryan), and they seem to be still required when `if TYPE_CHECKING:` is used, despite the PEP claim.
And my last message bring some use cases where strings are also required (notably, in recursive dataclasses). _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7QA3Z4CN... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Mon, 18 Jan 2021 15:54:32 -0800 Larry Hastings <larry@hastings.org> wrote:
On 1/18/21 3:42 PM, Inada Naoki wrote:
Many type hinting use cases don't need type objects in runtime. So I think PEP 563 is better for type hinting user experience.
You mean, in situations where the user doesn't want to import the types, because of heavyweight imports or circular imports? I didn't think those were very common.
Probably not very common, but annoying anyway. For example, a library (say PyArrow) may expose a function for importing Pandas data without mandating a Pandas dependency. Note: I don't use type hinting, I'm just responding to this particular aspect (optional / heavy dependencies). Regards Antoine.
If I've understood the PEP correctly, it would cause the following simple example to fail: ```python from dataclasses import dataclass @dataclass class User: name: str friends: list[User] ``` In fact, when the `dataclass` decorator is called, `User` class is not yet added to the module namespace, so when class `__annotations__` descriptor will be called inside the decorator, it will raise a `NameError` because of `friends` recursive annotation. By the way, in the example given by the PEP: ```python def foo(x: int = 3, y: MyType = None) -> float: ... class MyType: ... ``` if `foo` is decorated with a decorator calling `__annotations__` or `get_type_hints`, it will fail too. Using stringified annotations would prevent `NameError` to be raised, but it really mitigates the PEP claim that
This PEP also solves the forward reference problem
Not only this PEP doesn't solve (again, if I understand it correctly) the forward reference problem, but also it makes it a lot more tricky. And I think my first example is not so uncommon.
participants (14)
-
Antoine Pitrou -
Barry Warsaw -
Chris Angelico -
Eric V. Smith -
Greg Ewing -
Guido van Rossum -
Inada Naoki -
Jim J. Jewett -
Joseph Perez -
Larry Hastings -
Neil Schemenauer -
Paul Bryan -
Paul Sokolovsky -
Łukasz Langa