[Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

TL;DR version: I'm now +1 on a string-based PEP 563, with one
relatively small quibble regarding the future flag's name.

Putting that quibble first: could we adjust the feature flag to be
either "from __future__ import lazy_annotations" or "from __future__
import str_annotations"?

Every time I see "from __future__ import annotations" I think "But
we've had annotations since 3.0, why would they need a future
import?".

Adding the "lazy_" or "str_" prefix makes the feature flag
self-documenting: it isn't the annotations support that's new, it's
the fact the interpreter will avoid evaluating them at runtime by
treating them as implicitly quoted strings at compile time.

See inline comments for clarifications on what I was attempting to
propose in relation to thunks, and more details on why I changed my
mind :)

On 9 November 2017 at 14:16, Guido van Rossum <guido at python.org> wrote:
> On Wed, Nov 8, 2017 at 5:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 8 November 2017 at 16:24, Guido van Rossum <guido at python.org> wrote:
>> > I also don't like the idea that there's nothing you can do with a thunk
>> > besides calling it -- you can't meaningfully introspect it (not without
>> > building your own bytecode interpreter anyway).
>>
>> Wait, that wasn't what I was suggesting at all - with thunks exposing
>> their code object the same way a function does (i.e. as a `__code__`
>> attribute), the introspection functions in `dis` would still work on
>> them, so you'd be able to look at things like which variable names
>> they referenced, thus granting the caller complete control over *how*
>> they resolved those variable names (by setting them in the local
>> namespace passed to the call).
>
> I understood that they would be translated to `lambda: <expr>`. It seems you
> have a slightly more complex idea but if you're suggesting introspection
> through dis, that's too complicated for my taste.

Substituting in a lambda expression wouldn't work for the reasons you
gave when you objected to that idea (there wouldn't be any way for
typing.get_type_hints() to inject "vars(cls)" when evaluating the
annotations for method definitions, and enabling a cell-based
alternative would be a really intrusive change).

>> This is why they'd have interesting potential future use cases as
>> general purpose callbacks - every local, nonlocal, global, and builtin
>> name reference would implicitly be an optional parameter (or a
>> required parameter if the name couldn't be resolved as a nonlocal,
>> global, or builtin).
>
> Yeah, but that's scope creep for PEP 563. Łukasz and I are interested in
> gradually restricting the use of annotations to static typing with an
> optional runtime component. We're not interested in adding different use
> cases. (We're committed to backwards compatibility, but only until 4.0, with
> a clear deprecation path.)

Sorry, that was ambiguous wording on my part: the "potential future
use cases" there related to thunks in general, not their use for
annotations in particular.

APIs like pandas.query are a more meaningful example of where thunks
are potentially useful (and that's a problem I've been intermittently
pondering since Fernando Perez explained it to me at SciPy a few years
back - strings are an OK'ish workaround, but losing syntax
highlighting, precompiled code object caching, and other benefits of
real Python expressions means they *are* a workaround).

>> Instead, thunks would offer all the same introspection features as
>> lambda expressions do, they'd just differ in the following ways:
>>
>> * the parameter list on their code objects would always be empty
>> * the parameter list for their __call__ method would always be "ns=None"
>> * they'd be compiled without CO_OPTIMIZED (the same as a class namespace)
>> * they'd look up their closure references using LOAD_CLASSDEREF (the
>> same as a class namespace)
>
> I don't understand the __call__ with "ns-None" thing but I don't expect it
> matters.

It was an attempted shorthand for the way thunks could handle the
method annotations use case in a way that regular lambda expressions
can't: "thunk(vars(cls))" would be roughly equivalent to
"exec(thunk.__code__, thunk.__globals__, vars(cls))", similar to the
way class body evaluations works like "exec(body.__code__,
body.__globals__, mcl.__prepare__())"

That doesn't make a difference to your decision in relation to PEP 563, though.

>> That leaves the door open to a future PEP that proposes thunk-based
>> annotations as part of proposing thunks as a new low level delayed
>> evaluation primitive.
>
> Sorry, that's not a door I'd like to leave open.

At this point, I'd expect any successful PEP for the thunks idea to
offer far more compelling use cases than type annotations - the key
detail for me is that even if PEP 563 says "Lazy evaluation as strings
means that type annotations do not support lexical closures",
injecting attributes into class namespaces will still offer a way for
devs to emulate closure references if they really want them.

It's also the case that not supporting lexical closures would likely
be *better* for potential thunk use cases like pandas.query, as in
those kinds of use cases, the desired name resolution sequence is
"table columns, module globals, builtins" - picking up random function
locals as a closure reference and hence keeping them alive
indefinitely isn't actually desirable.

So I've reached a point where I'm happy that name resolution in simple
string-based delayed evaluation in annotations can be mapped to
existing name resolution concepts ("it's like a nested class body,
just without lexical closure support, and with method definitions
receiving their defining class namespace as a read-only locals()"),
*and* that it won't prevent us from considering potential future
enhancements to our syntactic support for ad hoc query APIs.

Those are the two major things that were worrying me about the current
draft, so with those concerns resolved, +1 from me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia