On 27 June 2018 at 07:54, Steven D'Aprano email@example.com wrote:
Comprehensions already run partly in the surrounding scope.
I tried to take a survey of people on the Python-List mailing list, so see what their expectations of comprehension scope was. Disappointingly, not many people responded, but those who did, invariably think in terms of comprehensions running inside their enclosing scope, like any other expression:
(Please excuse the doubled-up posts, some misconfigured news server is periodically sending duplicate posts.)
(Oh and ignore my comment about Python 2 -- I was thinking of something else.)
Given the code shown:
def test(): a = 1 b = 2 result = [value for key, value in locals().items()] return result
nobody suggested that the result ought to be the empty list, which is what you should get if the comprehension ran in its own scope. Instead, they all expected some variation of [1, 2], which is what you would get if the comprehension ran in the enclosing scope.
A decade or more since generator expressions started running in their own half-local-half-sublocal scope, people still think of scoping in terms of LEGB and don't think of comprehensions as running in their own scope *except* to the very limited degree that sometimes they are either surprised or pleased that "the loop variable doesn't leak".
But test() returns [1, 2]. So does that say (as you claim above) that "the comprehension ran in the enclosing scope"? Doesn't it just say that the outermost iterable runs in the enclosing scope?
So everybody expected the actual behaviour? (Disclaimer: in my response, I said that I had no clear expectation, which I stand by - locals() exposes implementation details that I don't normally feel that I need to know - but certainly the majority of respondents expected 1 and 2 to appear).
On the other hand,
... a = 1 ... b = 2 ... result = [locals().items() for v in 'a'] ... return result ...
[dict_items([('v', 'a'), ('.0', <str_iterator object at 0x0000015AA0BDE8D0>)])]
and I bet no-one would have expected that if you'd posed that question (I certainly wouldn't). Although some might have said [('v', 'a')]. I suspect some would have expected a and b to appear there too, but that's just a guess...
So yes, it's likely that people would have found the current behaviour unexpected in respect of locals(). But I imagine most people only care about the effective results when referencing variables, and
... a = 1 ... b = 2 ... result = [a for v in (1,)] ... return result ...
i.e., thanks to scope nesting, you can still reference locals from the enclosing scope.
The problem is that := allows you to *change* values in a scope, and at that point you need to know *which* scope. So to that extent, the locals() question is important. However, I still suspect that most people would answer that they would like := to assign values *as if* they were in the enclosing scope, which is not really something that I think people would express in answer to a question about locals(). This can be achieved with an implicit "nonlocal" (and some extra shenanigans if the enclosing scope has a nonlocal or global declaration itself). Which, AIUI, is what the current proposal tries to do.
IMO, the big question over the current PEP 572 proposal is whether it goes too far in the direction of "do what I mean". Superficially, the semantics are pretty clearly "what people would expect", and indeed that's been the whole focus recently to capture and satisfy *expected* behaviour. But there are edge cases (there always are when you work from messy real-world requirements rather than nice clean mathematical definitions ;-)) and the question is essentially whether any of those are bad enough to be an issue.
I'm starting to feel that they aren't, and I'm moving towards a cautious +0 (or even +1) on the proposal. Paul