[Python-Dev] PEP 563: Postponed Evaluation of Annotations (Draft 3)

Guido van Rossum guido at python.org
Mon Dec 4 11:42:38 EST 2017


I am hereby accepting your PEP. This will be a great improvement in the
experience of users annotating large complex codebases. Congrats on the
design and implementation and on your shepherding the PEP through the
discussion phase. Also a special thanks to Serhiy for thoroughly reviewing
and contributing to the ast-expr-stringification code.


PS. I have some editorial quibbles (mostly suggestions to make the
exposition clearer in a few places) but they don't affect acceptance of the
PEP and I will contact you at a later time with these.

On Tue, Nov 21, 2017 at 4:26 PM, Lukasz Langa <lukasz at langa.pl> wrote:

> Based on the feedback I gather in early November,
> I'm publishing the third draft for consideration on python-dev.
> I hope you like it!
> A nicely formatted rendering is available here:
> https://www.python.org/dev/peps/pep-0563/
> The full list of changes between this version and the previous draft
> can be found here:
> https://github.com/ambv/static-annotations/compare/
> python-dev1...python-dev2
> - Ł
> PEP: 563
> Title: Postponed Evaluation of Annotations
> Version: $Revision$
> Last-Modified: $Date$
> Author: Łukasz Langa <lukasz at langa.pl>
> Discussions-To: Python-Dev <python-dev at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 8-Sep-2017
> Python-Version: 3.7
> Post-History: 1-Nov-2017, 21-Nov-2017
> Resolution:
> Abstract
> ========
> PEP 3107 introduced syntax for function annotations, but the semantics
> were deliberately left undefined.  PEP 484 introduced a standard meaning
> to annotations: type hints.  PEP 526 defined variable annotations,
> explicitly tying them with the type hinting use case.
> This PEP proposes changing function annotations and variable annotations
> so that they are no longer evaluated at function definition time.
> Instead, they are preserved in ``__annotations__`` in string form.
> This change is going to be introduced gradually, starting with a new
> ``__future__`` import in Python 3.7.
> Rationale and Goals
> ===================
> PEP 3107 added support for arbitrary annotations on parts of a function
> definition.  Just like default values, annotations are evaluated at
> function definition time.  This creates a number of issues for the type
> hinting use case:
> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string
>   literal;
> * type hints are executed at module import time, which is not
>   computationally free.
> Postponing the evaluation of annotations solves both problems.
> Non-goals
> ---------
> Just like in PEP 484 and PEP 526, it should be emphasized that **Python
> will remain a dynamically typed language, and the authors have no desire
> to ever make type hints mandatory, even by convention.**
> This PEP is meant to solve the problem of forward references in type
> annotations.  There are still cases outside of annotations where
> forward references will require usage of string literals.  Those are
> listed in a later section of this document.
> Annotations without forced evaluation enable opportunities to improve
> the syntax of type hints.  This idea will require its own separate PEP
> and is not discussed further in this document.
> Non-typing usage of annotations
> -------------------------------
> While annotations are still available for arbitrary use besides type
> checking, it is worth mentioning that the design of this PEP, as well
> as its precursors (PEP 484 and PEP 526), is predominantly motivated by
> the type hinting use case.
> In Python 3.8 PEP 484 will graduate from provisional status.  Other
> enhancements to the Python programming language like PEP 544, PEP 557,
> or PEP 560, are already being built on this basis as they depend on
> type annotations and the ``typing`` module as defined by PEP 484.
> In fact, the reason PEP 484 is staying provisional in Python 3.7 is to
> enable rapid evolution for another release cycle that some of the
> aforementioned enhancements require.
> With this in mind, uses for annotations incompatible with the
> aforementioned PEPs should be considered deprecated.
> Implementation
> ==============
> In Python 4.0, function and variable annotations will no longer be
> evaluated at definition time.  Instead, a string form will be preserved
> in the respective ``__annotations__`` dictionary.  Static type checkers
> will see no difference in behavior, whereas tools using annotations at
> runtime will have to perform postponed evaluation.
> The string form is obtained from the AST during the compilation step,
> which means that the string form might not preserve the exact formatting
> of the source.  Note: if an annotation was a string literal already, it
> will still be wrapped in a string.
> Annotations need to be syntactically valid Python expressions, also when
> passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
> Annotations can only use names present in the module scope as postponed
> evaluation using local names is not reliable (with the sole exception of
> class-level names resolved by ``typing.get_type_hints()``).
> Note that as per PEP 526, local variable annotations are not evaluated
> at all since they are not accessible outside of the function's closure.
> Enabling the future behavior in Python 3.7
> ------------------------------------------
> The functionality described above can be enabled starting from Python
> 3.7 using the following special import::
>     from __future__ import annotations
> A reference implementation of this functionality is available
> `on GitHub <https://github.com/ambv/cpython/tree/string_annotations>`_.
> Resolving Type Hints at Runtime
> ===============================
> To resolve an annotation at runtime from its string form to the result
> of the enclosed expression, user code needs to evaluate the string.
> For code that uses type hints, the
> ``typing.get_type_hints(obj, globalns=None, localns=None)`` function
> correctly evaluates expressions back from its string form.  Note that
> all valid code currently using ``__annotations__`` should already be
> doing that since a type annotation can be expressed as a string literal.
> For code which uses annotations for other purposes, a regular
> ``eval(ann, globals, locals)`` call is enough to resolve the
> annotation.
> In both cases it's important to consider how globals and locals affect
> the postponed evaluation.  An annotation is no longer evaluated at the
> time of definition and, more importantly, *in the same scope* where it
> was defined.  Consequently, using local state in annotations is no
> longer possible in general.  As for globals, the module where the
> annotation was defined is the correct context for postponed evaluation.
> The ``get_type_hints()`` function automatically resolves the correct
> value of ``globalns`` for functions and classes.  It also automatically
> provides the correct ``localns`` for classes.
> When running ``eval()``,
> the value of globals can be gathered in the following way:
> * function objects hold a reference to their respective globals in an
>   attribute called ``__globals__``;
> * classes hold the name of the module they were defined in, this can be
>   used to retrieve the respective globals::
>     cls_globals = vars(sys.modules[SomeClass.__module__])
>   Note that this needs to be repeated for base classes to evaluate all
>   ``__annotations__``.
> * modules should use their own ``__dict__``.
> The value of ``localns`` cannot be reliably retrieved for functions
> because in all likelihood the stack frame at the time of the call no
> longer exists.
> For classes, ``localns`` can be composed by chaining vars of the given
> class and its base classes (in the method resolution order).  Since slots
> can only be filled after the class was defined, we don't need to consult
> them for this purpose.
> Runtime annotation resolution and class decorators
> --------------------------------------------------
> Metaclasses and class decorators that need to resolve annotations for
> the current class will fail for annotations that use the name of the
> current class.  Example::
>     def class_decorator(cls):
>         annotations = get_type_hints(cls)  # raises NameError on 'C'
>         print(f'Annotations for {cls}: {annotations}')
>         return cls
>     @class_decorator
>     class C:
>         singleton: 'C' = None
> This was already true before this PEP.  The class decorator acts on
> the class before it's assigned a name in the current definition scope.
> Runtime annotation resolution and ``TYPE_CHECKING``
> ---------------------------------------------------
> Sometimes there's code that must be seen by a type checker but should
> not be executed.  For such situations the ``typing`` module defines a
> constant, ``TYPE_CHECKING``, that is considered ``True`` during type
> checking but ``False`` at runtime.  Example::
>   import typing
>   if typing.TYPE_CHECKING:
>       import expensive_mod
>   def a_func(arg: expensive_mod.SomeClass) -> None:
>       a_var: expensive_mod.SomeClass = arg
>       ...
> This approach is also useful when handling import cycles.
> Trying to resolve annotations of ``a_func`` at runtime using
> ``typing.get_type_hints()`` will fail since the name ``expensive_mod``
> is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime).
> This was already true before this PEP.
> Backwards Compatibility
> =======================
> This is a backwards incompatible change.  Applications depending on
> arbitrary objects to be directly present in annotations will break
> if they are not using ``typing.get_type_hints()`` or ``eval()``.
> Annotations that depend on locals at the time of the function
> definition will not be resolvable later.  Example::
>     def generate():
>         A = Optional[int]
>         class C:
>             field: A = 1
>             def method(self, arg: A) -> None: ...
>         return C
>     X = generate()
> Trying to resolve annotations of ``X`` later by using
> ``get_type_hints(X)`` will fail because ``A`` and its enclosing scope no
> longer exists.  Python will make no attempt to disallow such annotations
> since they can often still be successfully statically analyzed, which is
> the predominant use case for annotations.
> Annotations using nested classes and their respective state are still
> valid.  They can use local names or the fully qualified name.  Example::
>     class C:
>         field = 'c_field'
>         def method(self) -> C.field:  # this is OK
>             ...
>         def method(self) -> field:  # this is OK
>             ...
>         def method(self) -> C.D:  # this is OK
>             ...
>         def method(self) -> D:  # this is OK
>             ...
>         class D:
>             field2 = 'd_field'
>             def method(self) -> C.D.field2:  # this is OK
>                 ...
>             def method(self) -> D.field2:  # this is OK
>                 ...
>             def method(self) -> field2:  # this is OK
>                 ...
>             def method(self) -> field:  # this FAILS, class D doesn't
>                 ...                     # see C's attributes,  This was
>                                         # already true before this PEP.
> In the presence of an annotation that isn't a syntactically valid
> expression, SyntaxError is raised at compile time.  However, since names
> aren't resolved at that time, no attempt is made to validate whether
> used names are correct or not.
> Deprecation policy
> ------------------
> Starting with Python 3.7, a ``__future__`` import is required to use the
> described functionality.  No warnings are raised.
> In Python 3.8 a ``PendingDeprecationWarning`` is raised by the
> compiler in the presence of type annotations in modules without the
> ``__future__`` import.
> Starting with Python 3.9 the warning becomes a ``DeprecationWarning``.
> In Python 4.0 this will become the default behavior.  Use of annotations
> incompatible with this PEP is no longer supported.
> Forward References
> ==================
> Deliberately using a name before it was defined in the module is called
> a forward reference.  For the purpose of this section, we'll call
> any name imported or defined within a ``if TYPE_CHECKING:`` block
> a forward reference, too.
> This PEP addresses the issue of forward references in *type annotations*.
> The use of string literals will no longer be required in this case.
> However, there are APIs in the ``typing`` module that use other syntactic
> constructs of the language, and those will still require working around
> forward references with string literals.  The list includes:
> * type definitions::
>     T = TypeVar('T', bound='<type>')
>     UserId = NewType('UserId', '<type>')
>     Employee = NamedTuple('Employee', [('name', '<type>', ('id',
> '<type>')])
> * aliases::
>     Alias = Optional['<type>']
>     AnotherAlias = Union['<type>', '<type>']
>     YetAnotherAlias = '<type>'
> * casting::
>     cast('<type>', value)
> * base classes::
>     class C(Tuple['<type>', '<type>']): ...
> Depending on the specific case, some of the cases listed above might be
> worked around by placing the usage in a ``if TYPE_CHECKING:`` block.
> This will not work for any code that needs to be available at runtime,
> notably for base classes and casting.  For named tuples, using the new
> class definition syntax introduced in Python 3.6 solves the issue.
> In general, fixing the issue for *all* forward references requires
> changing how module instantiation is performed in Python, from the
> current single-pass top-down model.  This would be a major change in the
> language and is out of scope for this PEP.
> Rejected Ideas
> ==============
> Keeping the ability to use function local state when defining annotations
> -------------------------------------------------------------------------
> With postponed evaluation, this would require keeping a reference to
> the frame in which an annotation got created.  This could be achieved
> for example by storing all annotations as lambdas instead of strings.
> This would be prohibitively expensive for highly annotated code as the
> frames would keep all their objects alive. That includes predominantly
> objects that won't ever be accessed again.
> To be able to address class-level scope, the lambda approach would
> require a new kind of cell in the interpreter.  This would proliferate
> the number of types that can appear in ``__annotations__``, as well as
> wouldn't be as introspectable as strings.
> Note that in the case of nested classes, the functionality to get the
> effective "globals" and "locals" at definition time is provided by
> ``typing.get_type_hints()``.
> If a function generates a class or a function with annotations that
> have to use local variables, it can populate the given generated
> object's ``__annotations__`` dictionary directly, without relying on
> the compiler.
> Disallowing local state usage for classes, too
> ----------------------------------------------
> This PEP originally proposed limiting names within annotations to only
> allow names from the model-level scope, including for classes.  The
> author argued this makes name resolution unambiguous, including in cases
> of conflicts between local names and module-level names.
> This idea was ultimately rejected in case of classes.  Instead,
> ``typing.get_type_hints()`` got modified to populate the local namespace
> correctly if class-level annotations are needed.
> The reasons for rejecting the idea were that it goes against the
> intuition of how scoping works in Python, and would break enough
> existing type annotations to make the transition cumbersome.  Finally,
> local scope access is required for class decorators to be able to
> evaluate type annotations. This is because class decorators are applied
> before the class receives its name in the outer scope.
> Introducing a new dictionary for the string literal form instead
> ----------------------------------------------------------------
> Yury Selivanov shared the following idea:
> 1. Add a new special attribute to functions: ``__annotations_text__``.
> 2. Make ``__annotations__`` a lazy dynamic mapping, evaluating
>    expressions from the corresponding key in ``__annotations_text__``
>    just-in-time.
> This idea is supposed to solve the backwards compatibility issue,
> removing the need for a new ``__future__`` import.  Sadly, this is not
> enough.  Postponed evaluation changes which state the annotation has
> access to.  While postponed evaluation fixes the forward reference
> problem, it also makes it impossible to access function-level locals
> anymore.  This alone is a source of backwards incompatibility which
> justifies a deprecation period.
> A ``__future__`` import is an obvious and explicit indicator of opting
> in for the new functionality.  It also makes it trivial for external
> tools to recognize the difference between a Python files using the old
> or the new approach.  In the former case, that tool would recognize that
> local state access is allowed, whereas in the latter case it would
> recognize that forward references are allowed.
> Finally, just-in-time evaluation in ``__annotations__`` is an
> unnecessary step if ``get_type_hints()`` is used later.
> Dropping annotations with -O
> ----------------------------
> There are two reasons this is not satisfying for the purpose of this
> PEP.
> First, this only addresses runtime cost, not forward references, those
> still cannot be safely used in source code.  A library maintainer would
> never be able to use forward references since that would force the
> library users to use this new hypothetical -O switch.
> Second, this throws the baby out with the bath water. Now *no* runtime
> annotation use can be performed.  PEP 557 is one example of a recent
> development where evaluating type annotations at runtime is useful.
> All that being said, a granular -O option to drop annotations is
> a possibility in the future, as it's conceptually compatible with
> existing -O behavior (dropping docstrings and assert statements).  This
> PEP does not invalidate the idea.
> Pass string literals in annotations verbatim to ``__annotations__``
> -------------------------------------------------------------------
> This PEP originally suggested directly storing the contents of a string
> literal under its respective key in ``__annotations__``.  This was
> meant to simplify support for runtime type checkers.
> Mark Shannon pointed out this idea was flawed since it wasn't handling
> situations where strings are only part of a type annotation.
> The inconsistency of it was always apparent but given that it doesn't
> fully prevent cases of double-wrapping strings anyway, it is not worth
> it.
> Make the name of the future import more verbose
> -----------------------------------------------
> Instead of requiring the following import::
>     from __future__ import annotations
> the PEP could call the feature more explicitly, for example
> ``string_annotations``, ``stringify_annotations``,
> ``annotation_strings``, ``annotations_as_strings``, ``lazy_anotations``,
> ``static_annotations``, etc.
> The problem with those names is that they are very verbose.  Each of
> them besides ``lazy_annotations`` would constitute the longest future
> feature name in Python.  They are long to type and harder to remember
> than the single-word form.
> There is precedence of a future import name that sounds overly generic
> but in practice was obvious to users as to what it does::
>     from __future__ import division
> Prior discussion
> ================
> In PEP 484
> ----------
> The forward reference problem was discussed when PEP 484 was originally
> drafted, leading to the following statement in the document:
>     A compromise is possible where a ``__future__`` import could enable
>     turning *all* annotations in a given module into string literals, as
>     follows::
>       from __future__ import annotations
>       class ImSet:
>           def add(self, a: ImSet) -> List[ImSet]: ...
>       assert ImSet.add.__annotations__ == {
>           'a': 'ImSet', 'return': 'List[ImSet]'
>       }
>     Such a ``__future__`` import statement may be proposed in a separate
>     PEP.
> python/typing#400
> -----------------
> The problem was discussed at length on the typing module's GitHub
> project, under `Issue 400 <https://github.com/python/typing/issues/400>`_.
> The problem statement there includes critique of generic types requiring
> imports from ``typing``.  This tends to be confusing to
> beginners:
>     Why this::
>         from typing import List, Set
>         def dir(o: object = ...) -> List[str]: ...
>         def add_friends(friends: Set[Friend]) -> None: ...
>     But not this::
>         def dir(o: object = ...) -> list[str]: ...
>         def add_friends(friends: set[Friend]) -> None ...
>     Why this::
>         up_to_ten = list(range(10))
>         friends = set()
>     But not this::
>         from typing import List, Set
>         up_to_ten = List[int](range(10))
>         friends = Set[Friend]()
> While typing usability is an interesting problem, it is out of scope
> of this PEP.  Specifically, any extensions of the typing syntax
> standardized in PEP 484 will require their own respective PEPs and
> approval.
> Issue 400 ultimately suggests postponing evaluation of annotations and
> keeping them as strings in ``__annotations__``, just like this PEP
> specifies.  This idea was received well.  Ivan Levkivskyi supported
> using the ``__future__`` import and suggested unparsing the AST in
> ``compile.c``.  Jukka Lehtosalo pointed out that there are some cases
> of forward references where types are used outside of annotations and
> postponed evaluation will not help those.  For those cases using the
> string literal notation would still be required.  Those cases are
> discussed briefly in the "Forward References" section of this PEP.
> The biggest controversy on the issue was Guido van Rossum's concern
> that untokenizing annotation expressions back to their string form has
> no precedent in the Python programming language and feels like a hacky
> workaround.  He said:
>     One thing that comes to mind is that it's a very random change to
>     the language.  It might be useful to have a more compact way to
>     indicate deferred execution of expressions (using less syntax than
>     ``lambda:``).  But why would the use case of type annotations be so
>     all-important to change the language to do it there first (rather
>     than proposing a more general solution), given that there's already
>     a solution for this particular use case that requires very minimal
>     syntax?
> Eventually, Ethan Smith and schollii voiced that feedback gathered
> during PyCon US suggests that the state of forward references needs
> fixing.  Guido van Rossum suggested coming back to the ``__future__``
> idea, pointing out that to prevent abuse, it's important for the
> annotations to be kept both syntactically valid and evaluating correctly
> at runtime.
> First draft discussion on python-ideas
> --------------------------------------
> Discussion happened largely in two threads, `the original announcement
> <https://mail.python.org/pipermail/python-ideas/2017-
> September/thread.html#47031>`_
> and a follow-up called `PEP 563 and expensive backwards compatibility
> <https://mail.python.org/pipermail/python-ideas/2017-
> September/thread.html#47108>`_.
> The PEP received rather warm feedback (4 strongly in favor,
> 2 in favor with concerns, 2 against). The biggest voice of concern on
> the former thread being Steven D'Aprano's review stating that the
> problem definition of the PEP doesn't justify breaking backwards
> compatibility.  In this response Steven seemed mostly concerned about
> Python no longer supporting evaluation of annotations that depended on
> local function/class state.
> A few people voiced concerns that there are libraries using annotations
> for non-typing purposes.  However, none of the named libraries would be
> invalidated by this PEP.  They do require adapting to the new
> requirement to call ``eval()`` on the annotation with the correct
> ``globals`` and ``locals`` set.
> This detail about ``globals`` and ``locals`` having to be correct was
> picked up by a number of commenters.  Nick Coghlan benchmarked turning
> annotations into lambdas instead of strings, sadly this proved to be
> much slower at runtime than the current situation.
> The latter thread was started by Jim J. Jewett who stressed that
> the ability to properly evaluate annotations is an important requirement
> and backwards compatibility in that regard is valuable.  After some
> discussion he admitted that side effects in annotations are a code smell
> and modal support to either perform or not perform evaluation is
> a messy solution.  His biggest concern remained loss of functionality
> stemming from the evaluation restrictions on global and local scope.
> Nick Coghlan pointed out that some of those evaluation restrictions from
> the PEP could be lifted by a clever implementation of an evaluation
> helper, which could solve self-referencing classes even in the form of a
> class decorator.  He suggested the PEP should provide this helper
> function in the standard library.
> Second draft discussion on python-dev
> -------------------------------------
> Discussion happened mainly in the `announcement thread <
> https://mail.python.org/pipermail/python-dev/2017-November/150062.html>`_,
> followed by a brief discussion under `Mark Shannon's post
> <https://mail.python.org/pipermail/python-dev/2017-November/150637.html
> >`_.
> Steven D'Aprano was concerned whether it's acceptable for typos to be
> allowed in annotations after the change proposed by the PEP.  Brett
> Cannon responded that type checkers and other static analyzers (like
> linters or programming text editors) will catch this type of error.
> Jukka Lehtosalo added that this situation is analogous to how names in
> function bodies are not resolved until the function is called.
> A major topic of discussion was Nick Coghlan's suggestion to store
> annotations in "thunk form", in other words as a specialized lambda
> which would be able to access class-level scope (and allow for scope
> customization at call time).  He presented a possible design for it
> (`indirect attribute cells
> <https://mail.python.org/pipermail/python-dev/2017-November/150141.html
> >`_).
> This was later seen as equivalent to "special forms" in Lisp.  Guido van
> Rossum expressed worry that this sort of feature cannot be safely
> implemented in twelve weeks (i.e. in time before the Python 3.7 beta
> freeze).
> After a while it became clear that the point of division between
> supporters of the string form vs. supporters of the thunk form is
> actually about whether annotations should be perceived as a general
> syntactic element vs. something tied to the type checking use case.
> Finally, Guido van Rossum declared he's rejecting the thunk idea
> based on the fact that it would require a new building block in the
> interpreter.  This block would be exposed in annotations, multiplying
> possible types of values stored in ``__annotations__`` (arbitrary
> objects, strings, and now thunks).  Moreover, thunks aren't as
> introspectable as strings.  Most importantly, Guido van Rossum
> explicitly stated interest in gradually restricting the use of
> annotations to static typing (with an optional runtime component).
> Nick Coghlan got convinced to PEP 563, too, promptly beginning
> the mandatory bike shedding session on the name of the ``__future__``
> import.  Many debaters agreed that ``annotations`` seems like
> an overly broad name for the feature name.  Guido van Rossum briefly
> decided to call it ``string_annotations`` but then changed his mind,
> arguing that ``division`` is a precedent of a broad name with a clear
> meaning.
> The final improvement to the PEP suggested in the discussion by Mark
> Shannon was the rejection of the temptation to pass string literals
> through to ``__annotations__`` verbatim.
> A side-thread of discussion started around the runtime penalty of
> static typing, with topic like the import time of the ``typing``
> module (which is comparable to ``re`` without dependencies, and
> three times as heavy as ``re`` when counting dependencies).
> Acknowledgements
> ================
> This document could not be completed without valuable input,
> encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and
> Ivan Levkivskyi.
> Copyright
> =========
> This document has been placed in the public domain.
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org

--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171204/115f6761/attachment-0001.html>

More information about the Python-Dev mailing list