[Python-ideas] A "local" pseudo-function

[Steven D'Aprano]

On Sat, Apr 28, 2018 at 12:16:16PM -0500, Tim Peters wrote:
> [Steven D'Aprano <steve at pearwood.info>]
> > Chris' PEP 572 started off with the concept that binding expressions
> > would create a "sub-local" scope, below function locals. After some
> > debate on Python-Ideas, Chris, Nick and Guido took the discussion off
> > list and decided to drop the sub-local scope idea as confusing and hard
> > to implement.
> 
> Enormously harder to implement than binding expressions, and the
> latter (to my eyes) capture many high-value use cases "good enough".

And yet you're suggesting an alternative which is harder and more 
confusing. What's the motivation here for re-introducing sublocal 
scopes, if they're hard to do and locals are "good enough"?

That's not a rhetorical question: why have you suggested this sublocal 
scoping idea? PEP 572 stopped talking about sublocals back in revision 2 
or so, and as far as I can see, *not a single objection* since has 
been that the variables weren't sublocal.

For what it is worth, if we ever did introduce a sublocal scope, I 
don't hate Nick's "given" block statement:

https://www.python.org/dev/peps/pep-3150/


[...]
> It was also the case that nesting scopes _at all_ was very
> controversial in Python's earliest years, and Guido resisted it
> mightily (with my full support).  The only scopes at first were
> function-local, module-global, and builtin, and while functions could
> _textually_ nest, they had no access to enclosing local scopes.

While I started off with Python 1.5, I wasn't part of the discussions 
about nested scopes. But I'm astonished that you say that nested scopes 
were controversial. *Closures* I would completely believe, but mere 
lexical scoping? Astonishing.

Even when I started, as a novice programmer who wouldn't have recognised 
the term "lexical scoping" if it fell on my head from a great height, I 
thought it was strange that inner functions couldn't see their 
surrounding function's variables. Nested scopes just seemed intuitively 
obvious: if a function sees the variables in the module surrounding it, 
then it should also see the variables in any function surrounding it.

This behaviour in Python 1.5 made functions MUCH less useful:


>>> def outer():
...     x = 1
...     def inner():
...             return x
...     return inner()
...
>>> outer()
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 5, in outer
  File "<stdin>", line 4, in inner
NameError: x


I think it is fair to say that inner functions in Python 1.5 were 
crippled to the point of uselessness.


> Adding nested local scopes was also "confusing" at the time, and
> indeed made the scoping rules far harder to explain to newbies, and
> complicated the implementation.  Then again, experienced programmers
> overwhelmingly (unanimously?) welcomed the change after it was done.

I agree with the above regarding closures, which are harder to explain, 
welcomed by experienced programmers, and often a source of confusion for 
newbies and experts alike:

https://stackoverflow.com/questions/7546285/creating-lambda-inside-a-loop

http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/comment-page-1/

but I disagree that lexical scoping alone is or ever was confusing. 
Neither did Niklaus Wirth, who included it in Pascal, a language 
intended to be friendly for beginners *wink*


> Since then, Python has gone down a pretty bizarre path, inventing
> sublocal scopes on an ad hoc basis by "pure magic" when their absence
> in some specific context seemed just too unbearable to live with
> (e.g., in comprehensions).  So we already have sublocal scopes, but in
> no explicit form that can be either exploited or explained.

I'm not entirely sure that comprehensions (including generator 
expressions) alone counts as "a path" :-) but I agree with this. I'm not 
a fan of comprehensions being their own scope. As far as I am concerned, 
leakage of comprehension variables was never a problem that needed to be 
solved, and was (very occasionally) a useful feature. Mostly for 
introspection and debugging.

But the decision was made for generator comprehensions to be in their 
own scope, and from there I guess it was inevitable that list 
comprehensions would have to match.


> > But the biggest problem is that this re-introduces exactly the same
> > awful C mistake that := was chosen to avoid. Which of the following two
> > contains the typo?
> >
> >     local(spam=expression, eggs=expression, cheese = spam+eggs)
> >
> >     local(spam=expression, eggs=expression, cheese == spam+eggs)
> 
> Neither :-)  I don't expect that to be a real problem.

I'm sure the C designers didn't either.

You miss the point that looking at the above, it is impossible to tell 
whether I meant assignment or an equality test. Typos of = for == do 
happen, even in Python, for whatever reason typos occur. Regardless of 
whether this makes them more likely or not (I didn't make that claim) 
once made, it is a bug that can fail silently in a way that is hard to 
see and debug.

Most = for == typos in Python give an instant SyntaxError, but there are 
two places where they don't:

- a statement like "spam == eggs" called for its side-effects only;

- in a function call, func(spam==eggs, spam=eggs) are both legal.

The first is so vanishingly rare that we can forget it. If you see 

    spam = eggs

as a statement, we can safely assume it means exactly what it says.

Inside function calls, it's a bit less cut and dried:

   func(foo=bar)

*could* be a typoed positional argument (foo == bar) but in practice a 
couple of factors mitigate that risk:

- PEP 8 style conventions: we expect to see func(foo=bar) for the 
  keyword argument case and func(foo == bar) for the positional
  argument case;

- if we mess it up, unless there happens to be a parameter called foo
  we'll get a TypeError, not a silent bug.

But with your suggested local() pseudo-function, neither mitigating 
factor applies and we can't tell or even guess which meaning is 
intended just by sight.


> In C I'm
> _thinking_ "if a equals b" and type "if (a=b)" by mistake in haste.
> In a "local" I'm _ thinking_ "I want to create these names with these
> values" in the former case, and in the latter case also "and I want to
> to test whether cheese equals spam + eggs".  But having already typed
> "=" to mean "binding" twice in the same line, "but the third time I
> type it it will mean equality instead" just doesn't seem likely.

Of course people won't *consciously* think that the operator for 
equality testing is = but they'll be primed to hit the key once, not 
twice, and they'll be less likely to notice their mistake. I never make 
more = instead of == typos than after I've just spent a lot of time 
working on maths problems, even though I am still consciously aware that 
I should be using == I simply don't notice the error.


> The original C mistake is exceedingly unlikely on the face of it:  if
> what I'm thinking is "if a equals b", or "while a equals b", I'm not
> going to use "local()" _at all_.

Given that while ... is one of the major motivating use-cases for 
binding expressions, I think you are mistaken to say that people won't 
use this local() pseudo-function in while statements.


[...]
> Still, if people are scared of that, a variation of Yury's alternative
> avoids it:  the last "argument" must be an expression (not a binding).
> In that case your first line above is a compile-time error.
> 
> I didn't like that because I really dislike the textual redundancy in the common
> 
>     if local(matchobject=re.match(regexp, line), matchobject):
> 
> compared to
> 
>     if local(matchobject=re.match(regexp, line)):

Indeed. That sort of "Repeat Yourself To Satisfy The Compiler" will be 
an ugly anti-pattern.


> But I could compromise ;-)
> 
> - There must be at least one argument.
> - The first argument must be a binding.
> - All but the last argument must also be bindings.
> - If there's more than one argument, the last argument must be an expression.

That's not really a complete specification of the pseudo-function 
though, since sometimes the sublocals it introduces extend past the 
final parenthesis and into the subsequent block.

What will, for example, this function return?

spam = eggs = "global"
def func(arg=local(spam="sublocal", eggs="sublocal", 1)):
    eggs = "local"    
    return (spam, eggs)


Even if you think nobody will be tempted to write such "clever" (dumb?) 
code, the behaviour still has to be specified.



[...]
> > Once you drop those two flaws, you're basically left with PEP 572 :-)
> 
> Which is fine by me, but do realize that since PEP 572 dropped any
> notion of sublocal scopes, that recurring issue remains wholly
> unaddressed regardless.

I don't think that sublocal scopes is a recurring issue, nor that we 
need address it now.



-- 
Steve