[Python-ideas] PEP 572: Statement-Local Name Bindings, take three!

[Steven D'Aprano]

On Sat, Mar 24, 2018 at 08:49:08PM +1100, Chris Angelico wrote:

> >     [(spam, spam+1) for x in values for spam in (func(x),)]
> >
> >     [(spam, spam+1) for spam in (func(x) for x in values)]
> >
> > They are the equivalent to "just add another assignment statement" for
> > comprehensions.
> 
> They might be mechanically equivalent. They are not syntactically
> equivalent. This PEP is not about "hey let's do something in Python
> that's utterly impossible to do". It's "here's a much tidier way to
> spell something that currently has to be ugly".

For the record, I don't think either of those are ugly. The first is a 
neat trick, but the second in particular is a natural, elegant and 
beautiful way of doing it in a functional style. And the beauty of it 
is, if it ever becomes too big and unwieldy for a single expression, it 
is easy to *literally* "just add another assignment statement":

        eggs = (long_and_complex_expression_with(x) for x in values)
        [(spam, spam+1) for spam in eggs)]

So I stand by my claim that even for comprehensions, "just add another 
assignment statement" is always an alternative.


> > while ("spam" as x):
> >     assert x == "spam"
> >     while ("eggs" as x):
> >         assert x == "eggs"
> >         break
> >     assert x == "eggs"
> 
> That means that sometimes, ``while ("eggs" as x):`` creates a new
> variable, and sometimes it doesn't. Why should that be?

I'm not following you.

If we talk implementation for a moment, my proposal is that x is just a 
regular local variable. So the CPython compiler sees (... as x) in the 
code and makes a slot for it in the function. (Other implementations may 
do differently.) Whether or not that local slot gets filled with a value 
depends on whether or not the specific (... as x) actually gets executed 
or not. That's no different from any other binding operation.

If x is defined as global, then (... as x) will bind to the global, not 
the local, but otherwise will behave the same.


[...]
> Function-local names give the same confidence. It doesn't matter what
> names you use inside a function (modulo 'global' or 'nonlocal'
> declarations) - they quietly shadow anything from the outside.

Yes, I get functions, and I think function-scope is a sweet spot between 
too few scopes and too many. Remember the bad old days of BASIC when all 
variables were application-global? Even if you used GOSUB as a 
second-rate kind of function, all the variables were still global.

On the other hand, introducing sub-function scopes is, I strongly 
believe, too many.


[...]
> > I think the rule should be either:
> >
> > - statement-locals actually *are* locals and so behave like locals;
> >
> > - statement-locals introduce a new scope, but still behave like
> >   locals with respect to closures.
> >
> > No need to introduce two separate modes of behaviour. (Or if there is
> > such a need, then the PEP should explain it.)
> 
> That would basically mean locking in some form of semantics.
> For your first example, you're locking in the rule that "(g(i) as x)" 
> is exactly the same as "x = g(i)", and you HAVE to then allow that 
> this will potentially assign to global or nonlocal names as well 
> (subject to the usual rules). In other words, you have 
> assignment-as-expression without any form of subscoping. This is a 
> plausible stance and may soon be becoming a separate PEP.

Well, if we really wanted to, we could ban (expression as name) where 
name was declared global, but why bother?


> But for your second, you're locking in the same oddities that a 'with'
> block has: that a variable is being "created" and "destroyed", yet it
> sticks around for the rest of the function, just in case.

Where is it documented that with blocks destroy variables? They don't. 
`with expression as name` is a name-binding operation no different from 
`name = expression` and the others. With the sole special case of except 
blocks auto-magically deleting the exception name, the only way to 
unbind a name is to call `del`.

What you're describing is not an oddity, but the standard way variables 
work in Python, and damn useful too. I have code that requires that the 
`with` variable is not unbound at the end of the block.


> It's a source of some confusion to people that the name used in a 
> 'with' statement is actually still valid afterwards.

The difference between "import foo" and "from foo import bar" is source 
of some confusion to some people. I should know, because I went through 
that period myself.

Just because "some people" make unjustified assumptions about the 
semantics of a language feature doesn't necessarily mean the language 
feature is wrong or harmful.


> Or does it only stick
> around if there is a function to close over it?

No, there's no need for a closure:

py> with open("/tmp/foo", "w") as f:
...     pass
...
py> f.closed
True
py> f.name
'/tmp/foo'

 
> Honestly, I really want to toss this one into the "well don't do that"
> basket, and let the semantics be dictated by simplicity and
> cleanliness even if it means that a closure doesn't see that variable.

If `(expression as name)` just bounds to a local, this is a non-problem.


[...]
> > Indeed. In case it isn't obvious, you should define the acronym the
> > first time you use it in the PEP.
> 
> Once again, I assumed too much of people. Expected them to actually
> read the stuff they're discussing. And once again, the universe
> reminds me that people aren't like that. Ah well. Will fix that next
> round of edits.

Sorry I don't have time to read that paragraph, so I'll just assume you 
are thanking me for pointing out your terrible error and offering 
profuse apologies.

*wink*


>> >> * An SLNB cannot be the target of any form of assignment, including augmented.
> >> >>   Attempting to do so will remove the SLNB and assign to the fully-scoped name.
> >> >
> >> > What's the justification for this limitation?
> >>
> >> Not having that limitation creates worse problems, like that having
> >> "(1 as a)" somewhere can suddenly make an assignment fail. This is
> >> particularly notable with loop headers rather than simple statements.
> >
> > How and why would it fail?
> 
> a = (1 as a)
> 
> With current semantics, this is equivalent to "a = 1". If assignment
> went into the SLNB, it would be equivalent to "pass". Which do you
> expect it to do?

Sorry, I don't follow this. If assignment goes into the statement-local, 
then it would be equivalent to:

statement-local a = 1

not pass.

Anyway, this confusion disappears if a is just a local. Then it is just:

local a = 1  # the right hand side (1 as a)
local a = 1  # the left hand side a = ...

which presumably some interpreters could optimize down to a single 
assignment. If they can be bothered.

 
> > MUST NOT implies that if there is *any* measurable penalty, even a
> > nano-second, the feature must be rejected. I think that's excessive.
> > Surely a nanosecond cost for the normal case is a reasonable tradeoff
> > if it buys us better expressiveness?
> 
> Steve, you know how to time a piece of code. You debate these kinds of
> points on python-list frequently. Are you seriously trying to tell me
> that you could measure a single nanosecond in regular compiling and
> running of Python code?

On my computer? Not a hope.

But some current generation computers have sub-nanosecond CPU clock 
rates, and 3.7 is due to have new timers with nanosecond resolution:

https://www.python.org/dev/peps/pep-0564/

I have no difficulty in believing that soon, if not right now, people 
will have sufficiently fast computers that yes, a nanosecond difference 
could be reliably measured with sufficient care.


[...]
> > Of course we don't want to necessarily impose unreasonable performance
> > and maintence costs on any implementation. But surely performance
> > cost is a quality of implementation issue. It ought to be a matter of
> > trade-offs: is the benefit sufficient to make up for the cost?
> 
> I don't see where this comes in. Let's say that Jython can't implement
> this feature without a 10% slowdown in run-time performance even if
> these subscopes aren't used.

Unlikely, but for the sake of the argument, okay.


> What are you saying the PEP should say?
> That it's okay for this feature to hurt performance by 10%? Then it
> should be rightly rejected. Or that Jython is allowed to ignore this
> feature? Or what?

That's really for Guido to decide whether the benefit is worth the 
(hypothetical) cost.
 
But why single out this feature from every other syntactic feature added 
to Python over its history? We have never before, as far as I can tell, 
demanded that a feature prove that every Python implmentation be able to 
support the feature with ZERO performance cost before accepting the PEP.

Normally, we introduce a new feature, and expect that like any new code, 
the first version may not be the most efficient, but subsequent versions 
will be faster. The first few versions of Python 3 were significant 
slower than Python 2.

Normally we make performance a trade-off: it's okay to make certain 
things a bit slower if there are sufficient other benefits. I still 
don't understand why you think that sort of tradeoff doesn't apply here. 
Effectively you seem to be saying that the value of this proposed 
feature is so infinitesimally small that we shouldn't accept *any* 
runtime cost, no matter how small, to gain this feature.



-- 
Steve