On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <larry@hastings.org> wrote:

Sorry it took me 3+ days to reply--I had a lot to think about here. But I have good things to report!

On 1/11/21 8:42 PM, Guido van Rossum wrote:

On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <larry@hastings.org> wrote:

PEP 563 states:

For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form.

So, if you are passing in a localns argument that isn't None, okay, but you're not using them "correctly" according to the language. Also, this usage won't be compatible with static type checkers.

I think you're misreading PEP 563 here. The mention of globalns=None, localns=None refers to the fact that these parameters have defaults, not that you must pass None. Note that the next paragraph in that PEP mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).

I think that's misleading, then. The passage is telling you how to "correctly evaluate[s] expressions", and how I read it was, it's telling me I have to supply globalns=None and localns=None for it to work correctly--which, I had to discover on my own, were the default values. I don't understand why PEP 563 feels compelled to define a function that it's not introducing, and in fact had already shipped with Python two versions ago.

I suppose PEP 563 is ambiguous because on the one hand global symbols are the only things that work out of the box, on the other hand you can make other things work by passing the right scope (and there's lots of code now that does so), and on the third hand, it claims that get_type_hints() adds the class scope, which nobody noticed or implemented until this week (there's a PR, can't recall the number).

But I think all this is irrelevant given what comes below.

Later in that same section, PEP 563 points out a problem with annotations that reference class-scoped variables, and claims that the implementation would run into problems because methods can't "see" the class scope. This is indeed a problem for PEP 563, but *you* can easily generate correct code, assuming the containing class exists in the global scope (and your solution requires that anyway). So in this case

```

class Outer:

    class Inner:

       ...

    def method(self, a: Inner, b: Outer) -> None:

        ...

```

The generated code for the `__annotations__` property could just have a reference to `Outer.Inner` for such cases:

```

def __annotations__():

    return {"a": Outer.Inner, "b": Outer, "return": None}

```

This suggestion was a revelation for me. Previously, a combination of bad experiences early on when hacking on compile and symtable, and my misunderstanding of exactly what was being asserted in the November 2017 thread, led me to believe that all I could support was globals. But I've been turning this over in my head for several days now, and I suspect I can support... just about anything.

I can name five name resolution scenarios I might encounter. I'll discuss them below, in increasing order of difficulty.

First is references to globals / builtins. That's already working, it's obvious how it works, and I need not elaborate further.

Yup.

Second is local variables in an enclosing function scope:

def outer_fn():
    class C: pass
    def inner_fn(a:C=None): pass
    return inner_fn

As you pointed out elsewhere in un-quoted text, I could make the annotation a closure, so it could retain a reference to the value of (what is from its perspective) the free variable "C".

Yup.

Third is local variables in an enclosing class scope, as you describe above:

class OuterCls:
    class InnerCls:
      def method(a:InnerCls=None): pass

If I understand what you're suggesting, I could notice inside the compiler that Inner is being defined in a class scope, walk up the enclosing scopes until I hit the outermost class, then reconstruct the chain of pulling out attributes until it resolves globally. Thus I'd rewrite this example to:

class OuterCls:
    class InnerCls:
      def method(a:OuterCls.InnerCls=None): pass

We've turned the local reference into a global reference, and we already know globals work fine.

I think this is going too far. A static method defined in InnerCls does not see InnerCls (even after the class definitions are complete). E.g.

```
class Outer:
class Inner:
@staticmethod
def foo(): return Inner

```

If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not defined".

Fourth is local variables in an enclosing class scope, which are themselves local variables in an enclosing function scope:

def outerfn():
    class OuterCls:
        class InnerCls:
          def method(a:InnerCls=None): pass
    return OuterCls.InnerCls

Even this is solvable, I just need to combine the "second" and "third" approaches above. I walk up the enclosing scopes to find the outermost class scope, and if that's a function scope, I create a closure and retain a reference to that free variable. Thus this would turn into

def outerfn():
    class OuterCls:
        class InnerCls:
          def method(a:OuterCls.InnerCls=None): pass

and method.__co_annotations__ would reference the free variable "OuterCls" defined in outerfn.

Probably also not needed.

Fifth is the nasty one. Note that so far every definition we've referred to in an annotation has been before the definition of the annotation. What if we want to refer to something defined after the annotation?

def outerfn():
    class OuterCls:
        class InnerCls:
          def method(a:zebra=None): pass
            ...

We haven't seen the definition of "zebra" yet, so we don't know what approach to take. It could be any of the previous four scenarios. What do we do?

If you agree with me that (3) and (4) are unnecessary (or even undesirable), the options here are either that zebra is a local in outerfn() (then just make it a closure), and if it isn't you should treat it as a global.

This is solvable too: we simply delay the compilation of __co_annotations__ code objects until the very last possible moment. First, at the time we bind the class or function, we generate a stub __co_annotations__ object, just to give the compiler what it expects. The compiler inserts it into the const table for the enclosing construct (function / class / module), and we remember what index it went into. Then, after we've finished processing the entire AST tree for this module, but before we we exit the compiler, we reconstruct the required context for evaluating each __co_annotations__ function--the nested chain of symbol tables, the compiler blocks if needed, etc--and evaluate the annotations for real. We assemble the correct __co_annotations__ code object and overwrite the stub in the const table with this now-correct value.

I can't think of any more scenarios. So, I think I can handle basically anything!

However, there are two scenarios where the behavior of evaluations will change in a way the user might find surprising. The first is when they redefine a variable used in an annotation:

x = str
def fn(a:x="345"): pass
x = int

With stock semantics, the annotation to "a" will be "str". With PEP 563 or my PEP, the annotation to "a" will be "int". (It gets even more exciting if you said "del x".)

This falls under the Garbage in, Garbage out principle. Mypy doesn't even let you do this. Another type checker which is easy to install, pyright, treats it as str. I wouldn't worry too much about it. If you strike the first definition of x, the pyright complains and mypy treats it as int.

Similarly, delaying the annotations so that we make everything visible means defining variables with the same name in multiple scopes may lead to surprising behavior.

x = str
class Outer:
def method(a:x="345"): pass
x = int

Again, stock gets you an annotation of "str", but PEP 563 and my PEP gets you "str", because they'll see the final result of evaluating the body of Outer.

Sadly this is the price you pay for delayed evaluation of annotations. Delaying the evaluation of annotations is the goal, and the whole point is to make changes, observable by the user, in how annotations are evaluated. All we can do is document these behaviors and hope our users forgive us.

Agreed.

I think this is a vast improvement over the first draft of my PEP, and assuming nobody points out major flaws in this approach (and, preferably, at least a little encouragement), I plan to redesign my prototype along these lines. (Though not right away--I want to take a break and attend to some other projects first.)

Thanks for the mind-blowing suggestions, Guido! I must say, you're pretty good at this Python stuff.

You're not so bad yourself -- without your wakeup call we would have immortalized PEP 563's limitations.

--Guido van Rossum (python.org/~guido)

Pronouns: he/him (why is my pronoun here?)