[Python-ideas] PEP 563: Postponed Evaluation of Annotations, first draft

[Steven D'Aprano]

On Mon, Sep 11, 2017 at 11:58:45AM -0400, Lukasz Langa wrote:
> PEP: 563
> Title: Postponed Evaluation of Annotations

A few comments, following the quoted passages as needed.


> Rationale and Goals
> ===================
> 
> PEP 3107 added support for arbitrary annotations on parts of a function
> definition.  Just like default values, annotations are evaluated at
> function definition time.  This creates a number of issues for the type
> hinting use case:
> 
> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string
>   literal;
> 
> * type hints are executed at module import time, which is not
>   computationally free.
> 
> Postponing the evaluation of annotations solves both problems.

You haven't justified that these are problems large enough to need 
fixing, let alone fixing in a backwards-incompatible way.

Regarding forward references: I see no problem with quoting forward 
references. Some people complain about the quotation marks, but frankly 
I don't think that's a major imposition. I'm not sure that going from a 
situation where only forward references are strings, to one where all 
annotations are strings, counts as a positive solution.

Regarding the execution time at runtime: this sounds like premature 
optimization. If it is not, if you have profiled some applications found 
that there is more than a trivial amount of time used by generating the 
annotations, you should say so.

In my opinion, a less disruptive solution to the execution time 
(supposed?) problem is a switch to disable annotations altogether, 
similar to the existing -O optimize switch, turning them into no-ops. 
That won't make any difference to static type-checkers.

Some people will probably want such a switch anyway, if they care about 
the memory used by the __annotations__ dictionaries.


> Implementation
> ==============
> 
> In a future version of Python, function and variable annotations will no
> longer be evaluated at definition time.  Instead, a string form will be
> preserved in the respective ``__annotations__`` dictionary.  Static type
> checkers will see no difference in behavior, whereas tools using
> annotations at runtime will have to perform postponed evaluation.
> 
> If an annotation was already a string, this string is preserved
> verbatim.  In other cases, the string form is obtained from the AST
> during the compilation step, which means that the string form preserved
> might not preserve the exact formatting of the source.

Can you give an example of how the AST may distort the source?

My knee-jerk reaction is that anything which causes the annotation to 
differ from the source is likely to cause confusion. 


> Annotations need to be syntactically valid Python expressions, also when
> passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
> Annotations can only use names present in the module scope as postponed
> evaluation using local names is not reliable.

And that's a problem. Forcing the use of global variables seems harmful. 
Preventing the use of locals seems awful. Can't we use some sort of 
closure-like mechanism?

This restriction is going to break, or prevent, situations like this:

def decorator(func):
    kind = ... # something generated at decorator call time
    @functools.wraps(func)
    def inner(arg: kind):
        ...
    return inner


Even if static typecheckers have no clue what the annotation on the 
inner function is, it is still useful for introspection and 
documentation.


> Resolving Type Hints at Runtime
> ===============================
[...]
> To get the correct module-level
> context to resolve class variables, use::
> 
>     cls_globals = sys.modules[SomeClass.__module__].__dict__

A small style issue: I think that's better written as:

    cls_globals = vars(sys.modules[SomeClass.__module__])

We should avoid directly accessing dunders unless necessary, and vars() 
exists specifically for the purpose of returning object's __dict__.


> Runtime annotation resolution and ``TYPE_CHECKING``
> ---------------------------------------------------
> 
> Sometimes there's code that must be seen by a type checker but should
> not be executed.  For such situations the ``typing`` module defines a
> constant, ``TYPE_CHECKING``, that is considered ``True`` during type
> checking but ``False`` at runtime.  Example::
> 
>   import typing
> 
>   if typing.TYPE_CHECKING:
>       import expensive_mod
> 
>   def a_func(arg: expensive_mod.SomeClass) -> None:
>       a_var: expensive_mod.SomeClass = arg
>       ...

I don't know whether this is important, but for the record the current 
documentation shows expensive_mod.SomeClass quoted.


> Backwards Compatibility
> =======================
[...]
> Annotations that depend on locals at the time of the function/class
> definition are now invalid.  Example::

As mentioned above, I think this is a bad loss of existing 
functionality.


[...]
> In the presence of an annotation that cannot be resolved using the
> current module's globals, a NameError is raised at compile time.

It is not clear what this means, or how it will work, especially given 
that the point of this is to delay evaluating annotations. How will the 
compiler know that an annotation cannot be resolved if it doesn't try to 
evaluate it?

Which brings me to another objection. In general, errors should be 
caught as early as possible. Currently, many (but not all) errors in 
annotations are caught at runtime because their evaluation fails:

class MyClass:
    ...

def function(arg: MyClsas) -> int:  # oops
    ...


That's a nice feature even if I'm not using a type-checker: I get 
immediate feedback as soon as I try running the code that my annotation 
is wrong.

It is true that forward references aren't evaluated at runtime, because 
they are strings, but "ordinary" annotations are evaluated, and that's a 
good thing! Losing that seems like a step backwards.


> Rejected Ideas
> ==============
> 
> Keep the ability to use local state when defining annotations
> -------------------------------------------------------------
> 
> With postponed evaluation, this is impossible for function locals.

Impossible seems a bit strong. Can you elaborate?


[...]
> This is brittle and doesn't even cover slots.  Requiring the use of
> module-level names simplifies runtime evaluation and provides the
> "one obvious way" to read annotations.  It's the equivalent of absolute
> imports.

I hardly think that "simplifies runtime evaluation" is true. At the 
moment annotations are already evaluated. *Anything* that you have to do 
by hand (like call eval) cannot be simpler than "do nothing".

I don't think the analogy with absolute imports is even close to useful, 
and far from being "one obvious way", this is a surprising, non-obvious, 
seemingly-arbitrary restriction on annotations.

Given that for *six versions* of Python 3 annotations could be locals, 
there is nothing obvious about restricting them to globals.



-- 
Steve