[Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 6 November 2017 at 16:36, Lukasz Langa <lukasz at langa.pl> wrote:
>
> On 5 Nov, 2017, at 9:55 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Python's name resolution rules are already ridiculously complicated,
> and PEP 563 is proposing to make them *even worse*, purely for the
> sake of an optional feature primarily of interest to large enterprise
> users.
>
>
> Solving forward references in type annotations is one of the two explicit
> goals of the PEP. That alone changes how name resolution works. It sounds
> like you're -1 on that idea alone?

That's just lazy evaluation, and no different from the way name
resolution works for globals and builtins in functions, and for
globals, builtins, and closure references in classes.

So while I do expect that's going to be confusing, it's also entirely
independent of how the lazy evaluation is implemented, and I think the
basic goal of deferring annotation evaluation is a good one.

[snip]

> Since your example that illustrates the problem shows that those things fail
> for regular attribute lookup, too, that can be simply fixed in the PEP. I
> did that here:
> https://github.com/ambv/static-annotations/blob/master/pep-0563.rst#backwards-compatibility

Unfortunately, it isn't quite that trivial to fix. To see the
remaining problem, take your nested class example from the PEP and
move it inside a function.

Today, that makes no difference, since "C" will transparently switch
from being accessed via LOAD_NAME to instead being accessed with
LOAD_CLASSDEREF.

By contrast, without the ability to access the outer "C" class
definition via a closure reference, you'll no longer have a generally
applicable way to evaluate *any* of the annotations that reference it,
since you won't have access to C from the module namespace any more.

I've been persuaded that a nested *function* isn't the right answer
for type annotations (since it doesn't let you tinker with locals()
prior to execution, which in turn means you can't readily allow access
to any class attributes at execution time), but that still leaves the
more exec-friendly logic of class body compilation available.

The difference between this and class body creation is that instead of
passing the compiled code object to MAKE_FUNCTION, and then passing
the resulting function to __build_class__, we'd instead introduce a
new MAKE_THUNK opcode that implemented __call__ differently from the
way regular functions implement it.

That way, the compiler changes would be limited to:

- compile annotations like a small nested class body (but returning
the expression result, rather than None)
- emit MAKE_THUNK instead of the expression's opcodes
- emit STORE_ANNOTATION as usual

>From a name resolution perspective, the new things folks would need to
learn are:

- annotations are now lazily evaluated (just like functions)
- but name resolution works the same way it does in class bodies
(unlike lambda expressions)

To allow typing.get_type_hints() to provide access to attributes
defined in the class, you'd need one final piece of the puzzle:

- rather than accepting regular function arguments (since thunks won't
have parameter lists) thunk.__call__ would instead accept an optional
pre-populated locals() namespace to use

With those changes, blindly calling annotations would usually just
work - the one case that couldn't be handled that way would be
annotations that implicitly accessed class level attrbutes, which
would require passing in "vars(class)" when calling the thunk.

Cheers,
Nick.

P.S. Back when I made the implicit scope change for list
comprehensions, I tried all sorts of potential tweaks to the
compiler's name resolution logic before finally giving up and deciding
that using a real nested function was the only way to make sure I
avoided introducing any weird new edge cases. Lexically nested
closures are generally great, but they make it *really* hard to
emulate Python's name resolution logic without direct assistance from
the compiler at the point where the name reference appears.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia