[Python-ideas] Delay evaluation of annotations

[Chris Angelico]

On Fri, Sep 23, 2016 at 11:58 PM, אלעזר <elazarg at gmail.com> wrote:
> What other context you see where the result of an expression is not intended
> to be used at all? Well there's Expression statements, which are evaluated
> for side effect. There's docstrings, which are a kind of annotations. What
> else? The only other that comes to mind is reveal_type(exp)... surely I
> don't need evaluation there.

Function annotations ARE used. They're stored as function attributes,
just as default argument values and docstrings are. (It's not the
language's problem if you never use them.)

>> The PEP that introduced them describes them as expressions:
>
> Syntactically, yes. Just like X in "a = lambda: X" is an expression, but you
> don't see it evaluated, do you? And this is an _actual_ expression,
> undeniably so, that is intended to be evaluated and used at runtime.

And the X in "if False: X" is a statement, but you don't see it
evaluated either. This is an actual expression that has to be
evaluated and used just like any other does.

>> I think its time to give up arguing that annotations aren't expressions.
>>
>
> I don't care if you call them expressions, delayed-expressions, or flying
> monkeys. The allowed syntax is exactly that of an expression (like inside a
> lambda). The time of binding of names to scope is the same (again like a
> lambda) but the evaluation time is unknown to the non-reflecting-developer.
> Decorators may promise time of evaluation, if they want to.

Thing is, literally every other expression in Python is evaluated at
the point where it's hit. You can guard an expression with control
flow statements or operators, but other than that, it will be hit when
execution reaches its line:

def func(x):
    expr # evaluated when function called

if cond:
    expr # evaluated if cond is true

[expr for x in range(n)] # evaluated if n > 0
(expr for x in [1]) # evaluated when genexp nexted
expr if cond else "spam" # evaluated if cond is true
lambda: expr # evaluated when function called

def func(x=expr): pass # evaluated when function defined
def func(x: expr): pass # evaluated when function defined

Default arguments trip some people up because they expect them to be
evaluated when the function's called, but it can easily be explained.
Function annotations are exactly the same. Making them magically
late-evaluate would have consequences for the grokkability of the
language - they would be special. Now, that can be done, but as
Rumplestiltskin keeps reminding us, all magic comes with a price, so
it has to be strongly justified. (For instance, the no-arg form of
super() is most definitely magical, but its justification is obvious
when you compare Py2 inheritance with Py3.)

> "Unknown evaluation time" is scary. _for expressions_, which might have side
> effects (one of which is running time). But annotations must be pure by
> convention (and tools are welcome to warn about it). I admit that I propose
> breaking the following code:
>
> def foo(x: print("defining foo!")): pass
>
> Do you know anyone who would dream about writing such code?

Yes, side effects make evaluation time scary. But so do rebindings,
and any other influences on expression evaluation. Good, readable code
generally follows the rule that the first instance of a name is its
definition. That's why we put imports up the top of the script, and so
on. Making annotations not work that way isn't going to improve
readability; you'd have to search the entire project for the class
being referenced. And since you can't probe them at definition time,
you have to wait until, uhh, SOME time, to do that search - you never
know where the actual name binding will come from. (It might even get
injected from another file, so you can't statically search the one
file.)

>> You want to make them fancy, give them super-powers, in order to solve
>> the forward reference problem. I don't think that the problem is serious
>> enough to justify changing the semantics of annotation evaluation and
>> make them non-standard, fancy, lazy-evaluated expressions.
>>
>
> My proposal solves the forward reference problem, but I believe in it
> because I believe it is aligned with what the programmer see.

This is on par with a proposal to make default argument values
late-bind, which comes up every now and then. It's just not worth
making these expressions magical.

>> > > class MyClass:
>> > >     pass
>> > >
>> > > def function(arg: MyCalss):
>> > >     ...
>> > >
>> > > I want to see an immediate NameError here, thank you very much
>> >
>> > Two things to note here:
>> > A. IDEs will point at this NameError
>>
>> Some or them might. Not everyone uses an IDE, it is not a requirement
>> for Python programmers. Runtime exceptions are still, and always will
>> be, the primary way of detecting such errors.
>
> How useful is the detection of this error at production?

The sooner you catch an error, the better. Always.

> Can you repeat that? NameError indeed happens at runtime, but the scope in
> which MyCalss was looked up for is determined at compile time - as far as I
> know. The bytecode-based typechecker I wrote rely on this information being
> accessible statically in the bytecode.
>
> def foo():
>     locals()['MyType'] = str
>     def bar(a : MyType): pass
>
>>>> foo()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 4, in foo
> NameError: name 'MyType' is not defined
>
> What do I miss?

That locals() is not editable (or rather, that mutations to it don't
necessarily change the actual locals). This is equivalent to:

def foo():
    locals()['MyType'] = str
    print(MyType)

>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in foo
NameError: name 'MyType' is not defined

In each case, you have to *call* foo() to see the NameError. It's
happening at run time.

> This way or the other, the very least that I hope, is explicitly forbidding
> reliance on side-effect or any other way to distinguish evaluation time of
> annotation expressions. Annotations must be pure, and the current promise of
> evaluation time should be deprecated.

Define "pure". Function decorator syntax goes to some lengths to
ensure that this is legal:

@deco(arg)
def f(): pass

PEP 484 annotations include subscripting, even nested:

def inproduct(v: Iterable[Tuple[T, T]]) -> T:

so you'd have to accept some measure of run-time evaluation.

It's worth reiterating, too, that function annotations have had the
exact same semantics since Python 3.0, in 2008. Changing that now
would potentially break up to eight years' worth of code, not all of
which follows PEP 484. When Steve mentioned 'not breaking other uses
of annotations', he's including this large body of code that might
well not even be visible to us, much less under python.org control.
Changing how annotations get evaluated is a *major, breaking change*,
so all you can really do is make a style guide recommendation that
"annotations should be able to be understood with minimal external
information" or something.

> Additionally, before making it impossible to go back, we should make the new
> variable annotation syntax add its annotations to a special object
> __reflect__, so that __reflect__.annotations__ will allow forcing evaluation
> (since there is no mechanism to do this in a variable).

Wow, lots of magic needed to make this work. Here's my
counter-proposal. In C++, you can pre-declare a class like this:

class Mutual2; //Pre-declare Mutual2
class Mutual1 {
    Mutual2 *ptr;
};
class Mutual2 {
    Mutual1 *ptr;
}

Here's how you could do it in Python:

Mutual2 = "Mutual2" # Pre-declare Mutual2
class Mutual1:
    def spam() -> Mutual2: pass
class Mutual2:
    def spam() -> Mutual1: pass

Problem solved, no magic needed.

ChrisA