[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Steven D'Aprano]

On Tue, Jun 26, 2018 at 05:42:43AM +1000, Chris Angelico wrote:

> So..... sublocal scopes, like in the earliest versions of PEP 572?
> 
> The wheel turns round and round, and the same spokes come up.

It isn't as if comprehensions (and generator expressions) run in a 
proper separate scope. It is more half-and-half, sometimes it is 
seperate, sometimes it isn't:


py> def show_subscope():
...     a, b = 1, 2
...     print("Comprehension scope, Part A")
...     print(next(locals() for x in (1,)))
...     print("Comprehension scope, Part B")
...     print(next(obj for obj in (locals(),)))
...
py> show_subscope()
Comprehension scope, Part A
{'x': 1, '.0': <tuple_iterator object at 0xb799cf8c>}
Comprehension scope, Part B
{'b': 2, 'a': 1}


Comprehensions already run partly in the surrounding scope.

I tried to take a survey of people on the Python-List mailing list, so 
see what their expectations of comprehension scope was. Disappointingly, 
not many people responded, but those who did, invariably think in terms 
of comprehensions running inside their enclosing scope, like any other 
expression:

https://mail.python.org/pipermail/python-list/2018-June/734838.html

(Please excuse the doubled-up posts, some misconfigured news server is 
periodically sending duplicate posts.)

(Oh and ignore my comment about Python 2 -- I was thinking of 
something else.)

Given the code shown:

def test():
    a = 1
    b = 2
    result = [value for key, value in locals().items()]
    return result


nobody suggested that the result ought to be the empty list, which is 
what you should get if the comprehension ran in its own scope. Instead, 
they all expected some variation of [1, 2], which is what you would get 
if the comprehension ran in the enclosing scope.

A decade or more since generator expressions started running in their 
own half-local-half-sublocal scope, people still think of scoping in 
terms of LEGB and don't think of comprehensions as running in their own 
scope *except* to the very limited degree that sometimes they are either 
surprised or pleased that "the loop variable doesn't leak".

For example:

http://nbviewer.jupyter.org/github/rasbt/python_reference/blob/master/tutorials/scope_resolution_legb_rule.ipynb

doesn't mention comprehensions until the very end, almost in passing, 
and doesn't describe them as a separate scope at all. Rather, they are 
described as using closures "to prevent the for-loop variable to cut 
[sic] into the global namespace."

This doesn't mention comprehension subscope at all:

https://www.python-course.eu/python3_global_vs_local_variables.php

Even the official documentation doesn't explicitly state that 
comprehensions are a separate scope:

https://docs.python.org/3/reference/executionmodel.html#resolution-of-names

rather leaving it to an after thought, to mention in passing almost as 
if it were an implementation-dependent accident, that comprehensions 
cannot see variables defined in any surrounding class scope.

Aside from the loop variable (which PEP 572 will not change!) I see no 
evidence that the average non-core developer Python programmer considers 
comprehensions as a separate scope, or wants them to be a separate 
scope. Regardless of comprehensions being implicitly wrapped in a 
function or not, the average developer doesn't want the loop variable to 
"leak", and that's as far as their consideration has needed to go until 
now. But when pressed to explicitly consider the scope inside a 
comprehension, the evidence I have seen is that they consider it the 
same as the local scope surrounding it.

Which is not wrong, as can be seen from the example above.

Unlike the loop variable, I don't believe that assignment-expression 
bindings quote-unquote "leaking" from comprehensions will come as a 
surprise. On the contrary -- given that Nick et al have gone to great 
lengths to ensure that as a first approximation, comprehensions are 
equivalent to a simple for-loop running in the current scope:

    result = [expr for a in seq]

    # is almost the same as
    result = []
    for a in seq:
        result.append(expr)

I expect that people will be surprised if explicit, non-loop 
variable assignments *don't* occur in the current scope.

If all that takes to implement is something like an implicit "nonlocal", 
that's hardly worse than the implicit functions already used.


-- 
Steve