[Python-ideas] Delay evaluation of annotations

Fri Sep 23 08:10:12 EDT 2016

On Fri, Sep 23, 2016 at 10:17:15AM +0000, אלעזר wrote:
> On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano <steve at pearwood.info> wrote:
> > On Thu, Sep 22, 2016 at 07:21:18PM +0000, אלעזר wrote:
> > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote:
> > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, אלעזר wrote:
> > > > > Hi all,
> > > > >
> > > > > Annotations of function parameters and variables are evaluated when
> > > > > encountered.
> > > >
> > > > Right, like all other Python expressions in general, and specifically
> > > > like function parameter default arguments.
> > > >
> > >
> > > Just because you call it "expression", when for most purposes it isn't -
> > > it is an annotation.
> >
> > It is *both*. It's an expression, because it's not a statement or a
> > block.
> 
> 
> Did you just use a false-trichotomy argument? :)

No.

You are the one trying to deny that annotations are expressions -- I'm 
saying that they are both annotations and expressions at the same time. 
There's no dichotomy here, since the two are not mutually exclusive. 
(The word here is dichotomy, not trichotomy, since there's only two 
things under discussion, not three.)

> > You cannot write:
> >
> > def func(arg1: while flag: sleep(1), arg2: raise ValueError):
> >     ...
> 
> > because the annotation must be a legal Python expression, not a code
> > block or a statement.
> 
> 
> This is the situation I'm asking to change

That's a much bigger change than what you suggested earlier, changing 
function annotations to lazy evaluation instead of eager.

Supporting non-expressions as annotations -- what's your use-case? Under 
what circumstances would you want to annotate an function parameter with 
a code block instead of an expression?

> > It's an annotation because that's the
> > specific *purpose* of the expression in that context.
>
> Exactly! Ergo, this is an annotation.

I've never denied that annotations are annotations, or that annotations 
are used to annotate function parameters. I'm not sure why you are 
giving a triumphant cry of "Exactly!" here -- it's not under dispute 
that annotations are annotations.

And it shouldn't be under dispute that annotations are expressions. 
They're not code blocks. They're not statements. What else could they be 
apart from expressions?

The PEP that introduced them describes them as expressions:

    Function annotations are nothing more than a way of associating 
    arbitrary Python EXPRESSIONS with various parts of a function at 
    compile-time. [Emphasis added.]

https://www.python.org/dev/peps/pep-3107/

and they are documented as an expression:

    parameter ::=  identifier [":" expression]

    Parameters may have annotations of the form “: expression” following 
    the parameter name. ... These annotations can be any valid Python 
    expression

https://docs.python.org/3/reference/compound_stmts.html#function-definitions

I think its time to give up arguing that annotations aren't expressions.

> > As an analogy: would you argue that it is wrong to call the for-loop
> > iterable an expression?
> >
> >     for <target-list> in <expression>:
> >         block
> >
> > I trust that you understand that the loop iterable can be any expression
> > that evaluates to an iterable. Well, annotations can be any expression
> > that evaluates to anything at all, but for the purposes of type
> > checking, are expected to evaluate to a string or a type object.
> >
> >
> for-loop iterable is an expression, evaluated at runtime, _for_ the
> resulting value to be used in computation. A perfectly standard expression.
> Nothing fancy.

Right. And so are annotations.

You want to make them fancy, give them super-powers, in order to solve 
the forward reference problem. I don't think that the problem is serious 
enough to justify changing the semantics of annotation evaluation and 
make them non-standard, fancy, lazy-evaluated expressions.

> > In the case of function annotations, remember that they can be any
> > legal Python expression. They're not even guaranteed to be type
> > annotations. Guido has expressed a strong preference that they are only
> > used as type annotations, but he hasn't yet banned other uses (and I
> > hope he doesn't), so any "solution" for a type annotation problem must
> > not break other uses.
> >
> >
> Must *allow* other use cases. My proposal allows: just evaluate them at the
> time of their use, instead at definition time.

I meant what I said. Changing the evaluation model for annotations is a 
big semantic change, a backwards-incompatible change. It's not just 
adding new syntax for something that was a syntax error before, it would 
be changing the meaning of existing Python code.

The transition from 3.6 to 3.7 is not like that from 2.x to 3.0 -- 
backwards compatibility is a hard requirement. Code that works a certain 
way in 3.6 is expected to work the same way in 3.7 onwards, unless we go 
through a deprecation period of at least one full release, and probably 
with a `from __future__ import ...` directive required. There may be a 
little bit of wiggle-room available for small changes in behaviour, 
under some circumstances -- but changing the evaluation model is 
unlikely to be judged to be a "small" change.

In any case, before such a backwards-incompatible change would be 
allowed, you would have to prove that it was needed.

[...]
> > class MyClass:
> >     pass
> >
> > def function(arg: MyCalss):
> >     ...
> >
> > I want to see an immediate NameError here, thank you very much
> 
> Two things to note here:
> A. IDEs will point at this NameError

Some or them might. Not everyone uses an IDE, it is not a requirement 
for Python programmers. Runtime exceptions are still, and always will 
be, the primary way of detecting such errors.

> B. Type checkers catch this NameError

Likewise for type checkers.

> C. Even the compiler can be made to catch this name error, since the name
> MyCalss is bound to builtins where it does not exist

How do you know it doesn't exist? Any module, any function, any class, 
any attribute access, might have added something called MyCalss to this 
module's namespace, or to the built-ins.

It's okay for a non-compulsory type-checker, linter or editor to make 
common-sense assumptions about built-ins. But the compiler cannot: it 
has no way of knowing *for sure* whether or not MyCalss exists until 
runtime. It has to actually do the name lookup, and see what happens.

> - you see, name lookup does happen at compile time anyway. 

It really doesn't.

You might be confusing function-definition time (which occurs at 
runtime) with compile time. When the function is defined, which occurs 
at runtime, the name MyCalss must exist or a NameError will occur. But 
that's not at compile time.

> D. Really, where's the error here? if no tool looks at this signature,
> there's nothing wrong with it - As a human I understand perfectly.

class CircuitDC:
    ...

class CircuitAC:
    ...

def func(arg: CircuitSC):
    ...

Do you still understand perfectly what I mean?

-- 
Steve