[Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim[
>> ... If a comprehension C is in class scope S, apparently the
>> class locals are _not_ in C's environment.  Since C doesn't
>>> even have read access to S's locals, it seems to me bizarre
>> that ":=" could _create_ a local in S.

[Ethan Furman <ethan at stoneleaf.us>]
> Python 3.7.0b3+ (heads/bpo-33217-dirty:28c1790, Apr  5 2018, 13:10:10)
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> --> class C:
> ...   huh = 7
> ...   hah = [i for i in range(huh)]
> ...
> --> C.hah
> [0, 1, 2, 3, 4, 5, 6]
>
> Same results clear back to 3.3 (the oldest version of 3 I have).  Are the
> docs wrong?
>
> Or maybe they just refer to functions:
>
> --> class C:
> ...   huh = 7
> ...   hah = [i for i in range(huh)]
> ...   heh = lambda: [i for i in range(huh)]
> ...
> --> C.hah
> [0, 1, 2, 3, 4, 5, 6]
> --> C.heh()
> Traceback (most recent call last):
>   File "test_class_comp.py", line 7, in <module>
>     print(C.heh())
>   File "test_class_comp.py", line 4, in <lambda>
>     heh = lambda: [i for i in range(huh)]
> NameError: global name 'huh' is not defined

As Chris already explained (thanks!), the expression defining the
iterable for the outermost `for` (which, perhaps confusingly, is the
_leftmost_ `for`) is treated specially in a comprehension (or genexp),
evaluated at once _in_ the scope containing the comprehension, not in
the comprehension's own scope.  Everything else in the comprehension
is evaluated in the comprehension's scope.

I just want to add that it's really the same thing as your lambda
example.  Comprehensions are also implemented as lambdas (functions),
but invisible functions created by magic.  The synthesized function
takes one argument, which is the expression defining the iterable for
the outermost `for`.  So, skipping irrelevant-to-the-point details,
your original example is more like:

    class C:
        huh = 7

        def _magic(it):
            return [i for i in it]

        hah = _magic(range(huh))

Since the `range(huh)` part is evaluated _in_ C's scope, no problem.
For a case that blows up, as Chris did you can add another `for` as
"outermost", or just try to reference a class local in the body of the
comprehension:

    class C2:
        huh = 7
        hah = [huh for i in range(5)]

That blows up (NameError on `huh`) for the same reason your lambda
example blows up, because it's implemented like:

    class C:
        huh = 7

        def _magic(it):
            return [huh for i in it]

        hah = _magic(range(5))

and C's locals are not in the environment seen by any function called
from C's scope.

A primary intent of the proposed ":= in comprehensions" change is that
you _don't_ have to learn this much about implementation cruft to
guess what a comprehension will do when it contains an assignment
expression.  The intent of

    total = 0
    sums = [total := total + value for value in data]

is obvious - until you think too much about it ;-)  Because there's no
function in sight, there's no reason to guess that the `total` in
`total = 0` has nothing to do with the instances of `total` inside the
comprehension.  The point of the change is to make them all refer to
the same thing, as they already do in (the syntactically similar, but
useless):

    total = 0
    sums = [total == total + value for value in data]

Except even _that_ doesn't work "as visually expected" in class scope
today.  The `total` inside the comprehension refers to the closest (if
any) scope (_containing_ the `class` statement) in which `total` is
local (usually the module scope, but may be a function scope if the
`class` is inside nested functions).

In function and module scopes, the second `total` example does work in
"the obvious" way, so in those scopes I'd like to see the first
`total` example do so too.