[Python-ideas] Delay evaluation of annotations

[אלעזר]

On Fri, Sep 23, 2016 at 3:11 PM Steven D'Aprano <steve at pearwood.info> wrote:

> On Fri, Sep 23, 2016 at 10:17:15AM +0000, אלעזר wrote:
> > On Fri, Sep 23, 2016 at 6:06 AM Steven D'Aprano <steve at pearwood.info>
> wrote:
> > > On Thu, Sep 22, 2016 at 07:21:18PM +0000, אלעזר wrote:
> > > > On Thu, Sep 22, 2016 at 9:43 PM Steven D'Aprano wrote:
> > > > > On Thu, Sep 22, 2016 at 05:19:12PM +0000, אלעזר wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Annotations of function parameters and variables are evaluated
> when
> > > > > > encountered.
> > > > >
> > > > > Right, like all other Python expressions in general, and
> specifically
> > > > > like function parameter default arguments.
> > > > >
> > > >
> > > > Just because you call it "expression", when for most purposes it
> isn't -
> > > > it is an annotation.
> > >
> > > It is *both*. It's an expression, because it's not a statement or a
> > > block.
> >
> >
> > Did you just use a false-trichotomy argument? :)
>
> No.
>
> You are the one trying to deny that annotations are expressions -- I'm
> saying that they are both annotations and expressions at the same time.
> There's no dichotomy here, since the two are not mutually exclusive.
> (The word here is dichotomy, not trichotomy, since there's only two
> things under discussion, not three.)
>
>
The argument "It's an expression, because it's not a statement or a block"
assumes that things must an expression, a statement or a block. Hence
"trichotomy". And it is false.

But I think we are getting lost in the terminology. Since I propose no
change in what is considered valid syntax,


> > > You cannot write:
> > >
> > > def func(arg1: while flag: sleep(1), arg2: raise ValueError):
> > >     ...
> >
> > > because the annotation must be a legal Python expression, not a code
> > > block or a statement.
> >
> >
> > This is the situation I'm asking to change
>
> That's a much bigger change than what you suggested earlier, changing
> function annotations to lazy evaluation instead of eager.
>
> Supporting non-expressions as annotations -- what's your use-case? Under
> what circumstances would you want to annotate an function parameter with
> a code block instead of an expression?
>
>
It indeed came out different than I meant. I don't suggest allowing
anything that is not already allowed, syntactically. I only propose giving
the current syntax a slightly different meaning, in a way that I'm sure
matches how Python coders already understand the code.

> > It's an annotation because that's the
> > > specific *purpose* of the expression in that context.
> >
> > Exactly! Ergo, this is an annotation.
>
> I've never denied that annotations are annotations, or that annotations
> are used to annotate function parameters. I'm not sure why you are
> giving a triumphant cry of "Exactly!" here -- it's not under dispute
> that annotations are annotations.
>
>
:( this kind of fighting over terminology takes us nowhere indeed.

What other context you see where the result of an expression is not
intended to be used at all? Well there's Expression statements, which are
evaluated for side effect. There's docstrings, which are a kind of
annotations. What else? The only other that comes to mind is
reveal_type(exp)... surely I don't need evaluation there.

And it shouldn't be under dispute that annotations are expressions.
> They're not code blocks. They're not statements. What else could they be
> apart from expressions?
>
> Now it is a false dichotomy as question. The answer is "annotation" as an
independent concept, closely related to expressions, but not evaluated when
encountered. Very similar to E in `lambda: E` except that lambda are there
mainly for the resulting value (hence "expression") and annotations are
there mainly for being there. In the code.


> The PEP that introduced them describes them as expressions:
>
>     Function annotations are nothing more than a way of associating
>     arbitrary Python EXPRESSIONS with various parts of a function at
>     compile-time. [Emphasis added.]
>
> https://www.python.org/dev/peps/pep-3107/
>
>
Syntactically, yes. Just like X in "a = lambda: X" is an expression, but
you don't see it evaluated, do you? And this is an _actual_ expression,
undeniably so, that is intended to be evaluated and used at runtime.


> and they are documented as an expression:
>
>     parameter ::=  identifier [":" expression]
>
>     Parameters may have annotations of the form “: expression” following
>     the parameter name. ... These annotations can be any valid Python
>     expression
>
>
> https://docs.python.org/3/reference/compound_stmts.html#function-definitions
>
> I think its time to give up arguing that annotations aren't expressions.
>
>
I don't care if you call them expressions, delayed-expressions, or flying
monkeys. The allowed syntax is exactly that of an expression (like inside a
lambda). The time of binding of names to scope is the same (again like a
lambda) but the evaluation time is unknown to the non-reflecting-developer.
Decorators may promise time of evaluation, if they want to.

"Unknown evaluation time" is scary. _for expressions_, which might have
side effects (one of which is running time). But annotations must be pure
by convention (and tools are welcome to warn about it). I admit that I
propose breaking the following code:

def foo(x: print("defining foo!")): pass

Do you know anyone who would dream about writing such code?


> > > As an analogy: would you argue that it is wrong to call the for-loop
> > > iterable an expression?
> > >
> > >     for <target-list> in <expression>:
> > >         block
> > >
> > > I trust that you understand that the loop iterable can be any
> expression
> > > that evaluates to an iterable. Well, annotations can be any expression
> > > that evaluates to anything at all, but for the purposes of type
> > > checking, are expected to evaluate to a string or a type object.
> > >
> > >
> > for-loop iterable is an expression, evaluated at runtime, _for_ the
> > resulting value to be used in computation. A perfectly standard
> expression.
> > Nothing fancy.
>
> Right. And so are annotations.
>
> You want to make them fancy, give them super-powers, in order to solve
> the forward reference problem. I don't think that the problem is serious
> enough to justify changing the semantics of annotation evaluation and
> make them non-standard, fancy, lazy-evaluated expressions.
>
>
My proposal solves the forward reference problem, but I believe in it
because I believe it is aligned with what the programmer see.


> > > In the case of function annotations, remember that they can be any
> > > legal Python expression. They're not even guaranteed to be type
> > > annotations. Guido has expressed a strong preference that they are only
> > > used as type annotations, but he hasn't yet banned other uses (and I
> > > hope he doesn't), so any "solution" for a type annotation problem must
> > > not break other uses.
> > >
> > >
> > Must *allow* other use cases. My proposal allows: just evaluate them at
> the
> > time of their use, instead at definition time.
>
> I meant what I said. Changing the evaluation model for annotations is a
> big semantic change, a backwards-incompatible change. It's not just
> adding new syntax for something that was a syntax error before, it would
> be changing the meaning of existing Python code.
>
> The transition from 3.6 to 3.7 is not like that from 2.x to 3.0 --
> backwards compatibility is a hard requirement. Code that works a certain
> way in 3.6 is expected to work the same way in 3.7 onwards, unless we go
> through a deprecation period of at least one full release, and probably
> with a `from __future__ import ...` directive required. There may be a
> little bit of wiggle-room available for small changes in behaviour,
> under some circumstances -- but changing the evaluation model is
> unlikely to be judged to be a "small" change.
>
> In any case, before such a backwards-incompatible change would be
> allowed, you would have to prove that it was needed.
>
>
I would like to see an example for a code that breaks under the Alexander's
suggestion of forcing evaluation at `.__annotations__` access time.


> [...]
> > > class MyClass:
> > >     pass
> > >
> > > def function(arg: MyCalss):
> > >     ...
> > >
> > > I want to see an immediate NameError here, thank you very much
> >
> > Two things to note here:
> > A. IDEs will point at this NameError
>
> Some or them might. Not everyone uses an IDE, it is not a requirement
> for Python programmers. Runtime exceptions are still, and always will
> be, the primary way of detecting such errors.


How useful is the detection of this error at production?


>
>
> B. Type checkers catch this NameError
>
> Likewise for type checkers.
>
>
> > C. Even the compiler can be made to catch this name error, since the name
> > MyCalss is bound to builtins where it does not exist
>
> How do you know it doesn't exist? Any module, any function, any class,
> any attribute access, might have added something called MyCalss to this
> module's namespace, or to the built-ins.
>
> It's okay for a non-compulsory type-checker, linter or editor to make
> common-sense assumptions about built-ins. But the compiler cannot: it
> has no way of knowing *for sure* whether or not MyCalss exists until
> runtime. It has to actually do the name lookup, and see what happens.
>
>
Yeah it was just a thought. I wouldn't really want the compiler to do that.


> > - you see, name lookup does happen at compile time anyway.
>
> It really doesn't.
>
> You might be confusing function-definition time (which occurs at
> runtime) with compile time. When the function is defined, which occurs
> at runtime, the name MyCalss must exist or a NameError will occur. But
> that's not at compile time.
>
>
Can you repeat that? NameError indeed happens at runtime, but the scope in
which MyCalss was looked up for is determined at compile time - as far as I
know. The bytecode-based typechecker I wrote rely on this information being
accessible statically in the bytecode.

def foo():
    locals()['MyType'] = str
    def bar(a : MyType): pass

>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in foo
NameError: name 'MyType' is not defined

What do I miss?

> D. Really, where's the error here? if no tool looks at this signature,
> > there's nothing wrong with it - As a human I understand perfectly.
>
> class CircuitDC:
>     ...
>
> class CircuitAC:
>     ...
>
> def func(arg: CircuitSC):
>     ...
>
>
> Do you still understand perfectly what I mean?
>
>
No.

def func(arg: CircuitAC):
    ...

Do you understand what I mean?

Code with small distance (hamming distance / edit distance) between
related-but-different entities is prone to such errors, and NameError gives
you very little assurance - if you erred this way, you get it; If you err
that way, you don't.

---

This way or the other, the very least that I hope, is explicitly forbidding
reliance on side-effect or any other way to distinguish evaluation time of
annotation expressions. Annotations must be pure, and the current promise
of evaluation time should be deprecated.

Additionally, before making it impossible to go back, we should make the
new variable annotation syntax add its annotations to a special object
__reflect__, so that __reflect__.annotations__ will allow forcing
evaluation (since there is no mechanism to do this in a variable).

Elazar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160923/c962fbe3/attachment-0001.html>