[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 08:00:20AM -0400, Eric V. Smith wrote:
> On 6/27/2018 7:08 AM, Chris Angelico wrote:
> >It gets funnier with nested loops. Or scarier. I've lost the ability
> >to distinguish those two.
> >
> >def test():
> >     spam = 1
> >     ham = 2
> >     vars = [key1+key2 for key1 in locals() for key2 in locals()]
> >     return vars
> >
> >Wanna guess what that's gonna return?
> 
> I'm not singling out Chris here, but these discussions would be easier 
> to follow and more illuminating if the answers to such puzzles were 
> presented when they're posed.

You can just copy and paste the function into the interactive 
interpreter and run it :-)

But where's the fun in that? The point of the exercise is to learn first 
hand just how complicated it is to try to predict the *current* scope 
behaviour of comprehensions. Without the ability to perform assignment 
inside them, aside from the loop variable, we've managed to avoid 
thinking too much about this until now.

It also demonstrates the unrealisticness of treating comprehensions as a 
separate scope -- they're hybrid scope, with parts of the comprehension 
running in the surrounding local scope, and parts running in an sublocal 
scope.

Earlier in this thread, Nick tried to justify the idea that 
comprehensions run in their own scope, no matter how people think of 
them -- but that's an over-simplification, as Chris' example above 
shows. Parts of the comprehension do in fact behave exactly as the naive 
model would suggest (even if Nick is right that other parts don't).

As complicated and hairy as the above example is, (1) it is a pretty 
weird thing to do, so most of us will almost never need to consider it; 
and (2) backwards compatibility requires that we live with it now (at 
least unless we introduce a __future__ import).

If we can't simplify the scope of comprehensions, we can at least 
simplify the parts that actually matters. What matters are the loop 
variables (already guaranteed to be sublocal and not "leak" out of the 
comprehension) and the behaviour of assignment expressions (open to 
discussion).

Broadly speaking, there are two positions we can take:

1. Let the current implementation of comprehensions as an implicit 
hidden function drive the functionality; that means we duplicate the 
hairiness of the locals() behaviour seen above, although it won't be 
obvious at first glance.

What this means in practice is that assignments will go to different 
scopes depending on *where* they are in the comprehension:

    [ expr   for x in iter1  for y in iter2  if cond   ...]
    [ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...]

Assignments in the section marked "AAAAAA" will be in the local scope; 
assignments in the BBBBBB sections will be in the sublocal scope. That's 
not too bad, up to the point you try to assign to the same name in 
AAAAAA and BBBBBB. And then you are likely to get confusing hard to 
debug UnboundLocalErrors.


2. Or we can keep the current behaviour for locals and the loop 
variables, but we can keep assignment expressions simple by ensuring 
they always bind to the enclosing scope. Compared to the complexity of 
the above, we have the relatively straight forward:

    [ AAAAAA for x in AAAAAA for y in AAAAAA if AAAAAA ...]

The loop variables continue to be hidden away in the invisible, implicit 
comprehension function, where they can't leak out, while explicit 
assignments to variables (using := or given or however it is spelled) 
will always go into the surrounding local scope, like they do in every 
other expression.

Does it matter that the implementation of this requires an implicit 
nonlocal declaration for each assignment? No more than it matters that 
comprehensions themselves require an implicit function.

And what we get out of this is simpler semantics at the Python level:

- Unless previous declared global, assignment expressions always bind to 
the current scope, even if they're inside a comprehension;

- and we don't have to deal with the oddity that different bits of a 
comprehension run in different scopes (unless we go out of our way to 
use locals()); merely using assignment expressions will just work 
consistently and simply, and loop variables will still be confined to 
the comprehension as they are now.


-- 
Steve