On Wed, Jun 27, 2018 at 12:01 PM, Eric V. Smith eric@trueblade.com wrote:
def test(): spam = 1 ham = 2 vars = [key1+key2 for key1 in locals() for key2 in locals()] return vars
Wanna guess what that's gonna return?
Guessing aside, messing around with locals() isn't really helpful for the usual case of common code.
But the real problem I see with all of this the distinction between generator expressions and comprehensions:
""" an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as "comprehensions") binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope. """
It seems everyone agrees that scoping rules should be the same for generator expressions and comprehensions, which is a good reason for python3's non-leaking comprehensions:
(py2)
In [5]: i = 0
In [6]: l = [i for i in range(3)]
In [7]: i
Out[7]: 2
In [8]: i = 0
In [9]: g = (i for i in range(3))
In [10]: i
Out[10]: 0
In [11]: for j in g:
...: *pass*
...:
In [12]: i
Out[12]: 0
so comprehensions and generator expressions behave differently -- not great.
(py3)
In [4]: i = 0
In [5]: l = [i for i in range(3)]
In [6]: i
Out[6]: 0
In [7]: g = (i for i in range(3))
In [8]: i
Out[8]: 0
In [9]: list(g)
Out[9]: [0, 1, 2]
In [10]: i
Out[10]: 0
The loop name doesn't "leak" and comprehensions and generator expressions are the same this regard -- nice.
So what about:
l = [x:=i for i in range(3)]
vs
g = (x:=i for i in range(3))
Is there any way to keep these consistent if the "x" is in the regular local scope?
Note that this thread is titled "Informal educator feedback on PEP 572".
As an educator -- this is looking harder an harder to explain to newbies...
Though easier if any assignments made in a "comprehension" don't "leak out".
Which does not mean that we'd need a "proper" new local scope (i.e. locals() returning something new) -- as long as the common usage was "intuitive".
I'm not singling out Chris here, but these discussions would be easier
to follow and more illuminating if the answers to such puzzles were presented when they're posed.
well, I think the point there was that it wasn't obvious without running the code -- and that point is made regardless of the answer.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
[Chris Barker]
... So what about:
l = [x:=i for i in range(3)]
vs
g = (x:=i for i in range(3))
Is there any way to keep these consistent if the "x" is in the regular local scope?
I'm not clear on what the question is. The list comprehension would bind
l
to [0, 1, 2] and leave the local x
bound to 2. The second
example
binds g
to a generator object, which just sits there unexecuted. That
has nothing to do with the PEP, though.
If you go on to do, e.g.,
l = list(g)
then, same as the listcomp, l
will be bound to [0, 1, 2] and the local
x
will be left bound to 2.
The only real difference is in _when_ the x:=i for i in range(3)
part
gets executed. There's no new twist here due to the PEP. Put a body B in
a listcomp and any side effects due to executing B happen right away, but
put B in a genexp and they don't happen until you force the genexp to yield
results.
For example, do you think these two are "consistent" today?
l = [print(i) for i in range(3)] g = (print(i) for i in range(3))
? If so, nothing essential changes by replacing "print(i)" with "x := i" - in either case the side effects happen when the body is executed.
But if you don't think they're already consistent, then nothing gets less consistent either ;-)
Sent from my iPhone
So what about:
l = [x:=i for i in range(3)]
vs
g = (x:=i for i in range(3))
Is there any way to keep these consistent if the "x" is in the regular local scope?
I'm not clear on what the question is. The list comprehension would bind
l
to [0, 1, 2] and leave the local x
bound to 2. The second
example binds g
to a generator object, which just sits there unexecuted.
That has nothing to do with the PEP, though.
If you go on to do, e.g.,
l = list(g)
then, same as the listcomp, l
will be bound to [0, 1, 2] and the local
x
will be left bound to 2.
OK, it has been said that the priority is that
list(a_gen_expression)
Behave the same as
[the_same_expression]
So we’re good there. And maybe it’s correct that leaving the running of the gen_exp ‘till later is pretty uncommon, particularly for newbies, but:
If the execution of the gen_exp is put off, it really confuses things — that name being changed would happen at some arbitrary tone, and at least in theory, the gen_exp could be passed off to somewhere else in the code, and be run or not run completely remotely from where the name is used.
So while this is technically the same as the comprehension, it is not the same as a generator function which does get its own scope.
And we should be clear how it will work — after all, in py2, the handling of the looping name was handled differently in gen_exp vs comprehensions.
So I think a local scope for all comprehension-like things would be the way to go.
But getting back to the original thread topic — python has a number of places that you can only use expressions — adding the ability to bind a name in all these places complicates the language significantly.
Put a body B in a listcomp and any side effects due to executing B
Maybe it’s just me, but re-binding a name seems like a whole new category of side effect.
-CHB
[Chris Barker]
So what about:
l = [x:=i for i in range(3)]
vs
g = (x:=i for i in range(3))
Is there any way to keep these consistent if the "x" is in the regular local scope?
[Tim]
I'm not clear
on what the question is. The list comprehension would
bind l
to [0, 1, 2] and leave the local x
bound to 2. The
second
example binds g
to a generator object, which just sits there
unexecuted. That has nothing to do with the PEP, though.
If you go on to do, e.g.,
l = list(g)
then, same as the listcomp, l
will be bound to [0, 1, 2] and the local
x
will
be left bound to 2.
[Chris]
OK, it has been said that the priority is that
list(a_gen_expression)
Behave the same as
[the_same_expression]
That's certainly desirable.
So we’re good there. And maybe it’s correct that leaving the running of the gen_exp ‘till later is pretty uncommon, particularly for newbies,
Common or not, I have no idea why anyone would write a genexp like the one you gave, except to contrive an example of silly behavior exhibited by silly code ;-)
It's really not interesting to me to make up code as goofy as you can conceive of - the interesting questions are about plausible code (including plausible coding errors).
but:
If the execution of the gen_exp is put off, it really confuses things — that name being changed would happen at some arbitrary tone, and at least in theory, the gen_exp could be passed off to somewhere else in the code, and be run or not run completely remotely from where the name is used.
Sure.
So while this is technically the same as the comprehension, it is not the same as a generator function which does get its own scope.
It is the same as a generator function with appropriate scope declarations
a generator expression is, after all, implemented _by_ a nested generator function. You can write a workalike to your code above today, but nobody worries about that because nobody does that ;-)
def f():
def bashx(outermost):
nonlocal x
for i in outermost:
x = i
yield i
x = 12
g = bashx(range(3))
print("x before", x)
L = list(g)
print("L", L)
print("x after", x)
Then calling f()
prints:
x before 12
L [0, 1, 2]
x after 2
And we should be clear how it will work — after all, in py2, the handling of the looping name was handled differently in gen_exp vs comprehensions.
The PEP specifies the semantics. If it's accepted, that will be folded into the docs.
So I think a local scope for all comprehension-like things would be the way to go.
But getting back to the original thread topic — python has a number of places that you can only use expressions — adding the ability to bind a name in all these places complicates the language significantly.
Did adding ternary if
(truepart if expression else falsepart)
complicate
the language significantly? Python has rarely expanded the number of
expression forms, but whenever it has the sky didn't actually fall despite
earnest warnings that disaster was inevitable ;-)
Put a body B in a listcomp and any side effects due to executing B
Maybe it’s just me, but re-binding a name seems like a whole new category of side effect.
With no trickery at all, you've always been able to rebind attributes, and
mutate containers, in comprehensions and genexps. Because for
targets
aren't limited to plain names; e.g.,
g = (x+y for object.attribute, a[i][j] in zip(range(3), range(3)))
is already "legal", and will stomp all over the complex for
targets when
executed - there's nothing "local" about them. But nobody worries about
that because nobody does stuff like that.
And as in my goofy code above, mucking with binding of plain names is also possible today. Indeed, straightforward if that's what you _want_ to do. But nobody does.
It's just not one of Python's goals to make it impossible to write useless code ;-)
On Thu, Jun 28, 2018 at 9:28 AM, Tim Peters tim.peters@gmail.com wrote:
g = (x:=i for i in range(3)) Common or not, I have no idea why anyone would write a genexp like the one you gave, except to contrive an example of silly behavior exhibited by silly code ;-)
yes, it was a contrived example, but the simplest one I could think of off the top of my head that re-bound a name in the loop -- which was what I thought was the entire point of this discussion?
If we think hardly anyone is ever going to do that -- then I guess it doesn't matter how it's handled.
So while this is technically the same as the comprehension, it is not
the same as a generator function which does get its own scope.
It is the same as a generator function with appropriate scope declarations
a generator expression is, after all, implemented _by_ a nested generator function. You can write a workalike to your code above today, but nobody worries about that because nobody does that ;-)
def f():
def bashx(outermost):
nonlocal x
for i in outermost:
x = i
yield i
but here the keyword "nonlocal" is used -- you are clearly declaring that you are messing with a nonlocal name here -- that is a lot more obvious than simply using a :=
And "nonlocal" is not used that often, and when it is it's for careful closure trickery -- I'm guessing := will be far more common. And, of course, when a newbie encounters it, they can google it and see what it means -- far different that seeing a := in a comprehension and understanding (by osmosis??) that it might make changes in the local scope.
And I don't think you can even do that with generator expressions now -- as they can only contain expressions. Which is my point -- this would allow the local namespace to be manipulated in places it never could before.
Maybe it's only comprehensions, and maybe it'll be rare to have a confusing version of those, so it'll be no big deal, but this thread started talking about educators' take on this -- and as an educator, I think this really does complicate the language.
Python got much of it's "fame" by being "executable pseudo code" -- its been moving farther and farther away from those roots. That's generally a good thing, as we've gain expressiveness in exchangel, but we shouldn't pretend it isn't happening, or that this proposal doesn't contribute to that trend.
Did adding ternary if
(truepart if expression else falsepart)
complicate
the language significantly?
I don't think so -- no. For two reasons:
1) the final chosen form is kind of verbose, but therefor more like "executable pseudo code" :-) As apposed to the C version, for instance.
2) it added one new construct, that if, when someone sees it for the first (or twenty fifth) time and doesn't understand it, they can look it up, and find out. and it only effects that line of code.
So adding ANYTHING does complicate the language, by simply making it a bit larger, but some things are far more complicating than others.
Python has rarely expanded the number of expression forms, but whenever it
has the sky didn't actually fall despite earnest warnings that disaster was inevitable ;-)
Well, I've been surprised by what confused students before, and I will again. But I dont hink there is any doubt that Python 3.7 is a notably harder to learn that Python 1.5 was...
Maybe it’s just me, but re-binding a name seems like a whole new category of side effect.
With no trickery at all, you've always been able to rebind attributes, and
mutate containers, in comprehensions and genexps. Because for
targets
aren't limited to plain names; e.g.,
g = (x+y for object.attribute, a[i][j] in zip(range(3), range(3)))
sure, but you are explicitly using the names "object" and "a" here -- so while side effects in comprehension are discouraged, it's not really a surprised that namespaces specifically named are changed.
and this:
In [55]: x = 0
In [56]: [x for x in range(3)]
Out[56]: [0, 1, 2]
In [57]: x
Out[57]: 0
doesn't change x in the local scope -- if that was a good idea, why is a good idea to have := in a comprehension effect the local scope??
But maybe it is just me.
And as in my goofy code above, mucking with binding of plain names is also possible today. Indeed, straightforward if that's what you _want_ to do. But nobody does.
It's just not one of Python's goals to make it impossible to write useless code ;-)
I suppose we need to go back and look at the "real" examples of where/how folks think they'll use := in comprehensions, and see how confusing it may be.
One of these conversations was started with an example something like this:
[(f(x), g(f(x))) for x in an_iterable]
The OP didn't like having to call f() twice. So that would become:
[ (temp:=f(x), g(temp)) for x in an_iterable]
so now the question is: should "temp" be created / changed in the enclosing local scope?
This sure looks a lot like letting the iteration name (x in this example) leak out -- so I'd say no.
And I don't think this kind of thing would be rare.
Someone mentions that one problem with letting the iteration name leak out is that people tend to use short, common names, like "i" -- Im thinking that would also be the case for this kind of temp variable.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
[Chris]
yes, it was a contrived example, but the simplest one I could think of off the top of my head that re-bound a name in the loop -- which was what I thought was the entire point of this discussion?
But why off the top of your head? There are literally hundreds & hundreds of prior messages about this PEP, not to mention that you could also find examples in the PEP. Why make up a senseless example?
If we think hardly anyone is ever going to do that -- then I guess it doesn't matter how it's handled.
So look at real examples. One that's been repeated at least a hundred times wants a local to "leak into" a listcomp:
total = 0 cumsums = [total ::= total + value for value in data]
As an educator, how are you going to explain that blowing up with
UnboundLocalError instead? Do you currently teach that comprehensions and
genexps are implemented via invisible magically generated lexically nested
functions? If not, you're going to have to start for people to even begin
to make sense of UnboundLocalError if total
_doesn't_ "leak into" that
example. My belief is that just about everyone who doesn't know "too much"
about the current implementation will be astonished & baffled if that
example doesn't "just work".
In other cases it's desired that targets "leak out":
while any(n % (divisor := p) == 0 for p in small_primes): n //= divisor
And in still other cases no leaking (neither in nor out) is desired.
Same as for
targets in that way,. but in the opposite direction: they
don't leak and there's no way to make them leak, not even when that's
wanted. Which _is_ wanted in the last example above, which would be
clearer still written as:
while any(n % p == 0 for p in small_primes): n //= p
But that ship has sailed.
... And "nonlocal" is not used that often, and when it is it's for careful closure trickery -- I'm guessing := will be far more common.
My guess (recorded in the PEP's Appendix A) is that assignment expressions
_overall_ will be used more often than ternary if
but significantly less
often than augmented assignment. I expect their use in genexps and
comprehensions will be minimal. There are real use cases for them, but the
vast majority of genexps and comprehensions apparently have no use for them
at all.
And, of course, when a newbie encounters it, they can google it and see what it means -- far different that seeing a := in a comprehension and understanding (by osmosis??) that it might make changes in the local scope.
Which relates to the above: how do you teach these things? The idea that "a newbie" even _suspects_ that genexps and listcomps have something to do with lexically nested scopes and invisible nested functions strikes me as hilarious ;-)
Regardless of how assignment expressions work in listcomps and genexps,
this example (which uses neither) _will_ rebind the containing block's
x
:
[x := 1]
How then are you going to explain that this seemingly trivial variation _doesn't_?
[x := 1 for ignore in "a"]
For all the world they both appear to be binding x
in the code block
containing the brackets. So let them.
Even worse,
[x for ignore in range(x := 1)]
will rebind x
in the containing block _regardless_ of how assignment
expression targets are treated in "most of" a comprehension, because the
expression defining the iterable of the outermost "for" _is_ evaluated in
the containing block (it is _not_ evaluated in the scope of the synthetic
function).
That's not a special case for targets if they all "leak", but is if they don't.
And I don't think you can even do that with generator expressions now -- as they can only contain expressions.
Expressions can invoke arbitrary functions, which in turn can do anything whatsoever.
Which is my point -- this would allow the local namespace to be manipulated in places it never could before.
As above, not true. However, it would make it _easier_ to write senseless code mucking with the local namespace - if that's what you want to do.
Maybe it's only comprehensions, and maybe it'll be rare to have a confusing version of those, so it'll be no big deal, but this thread started talking about educators' take on this -- and as an educator, I think this really does complicate the language.
I'll grant that it certainly doesn't simplify the language ;-)
Python got much of it's "fame" by being "executable pseudo code" -- its been moving farther and farther away from those roots. That's generally a good thing, as we've gain expressiveness in exchangel, but we shouldn't pretend it isn't happening, or that this proposal doesn't contribute to that trend.
I didn't say a word about that one way or the other. I mostly agree, but at the start Guido was aiming to fill a niche between shell scripting languages and C. It was a very "clean" language from the start, but not aimed at beginners. Thanks to his experience working on ABC, it carried over some key ideas that were beginner-friendly, though.
I view assignment expressions as being aimed at much the same audience as augmented assignments: experienced programmers who already know the pros and cons from vast experience with them in a large number of other widely used languages. That's also a key Python audience.
... Well, I've been surprised by what confused students before, and I will again. But I dont hink there is any doubt that Python 3.7 is a notably harder to learn that Python 1.5 was...
Absolutely. It doesn't much bother me, though - at this point the language and its widely used libraries are so sprawling that I doubt anyone is fluent in all of it. That's a sign of worldly success.
... and this:
In [55]: x = 0 In [56]: [x for x in range(3)] Out[56]: [0, 1, 2] In [57]: x Out[57]: 0
doesn't change x in the local scope --
In Python 3, yes; in Python 2 it rebinds x
to 2.
if that was a good idea, why is a good idea to have := in a comprehension effect the local scope??
Because you can't write a genexp or comprehension AT ALL without specifying
for
targets, and in the overwhelming majority of genexps and
comprehensions anyone ever looked at, "leaking" of for-targets was not
wanted. "So don't let them leak" was pretty much a no-brainer for Python 3.
But assignment expressions are NEVER required to write a genexp or comprehension, and there are only a handful of patterns known so far in which assignment expressions appear to be of real value in those contexts. In at least half those patterns, leaking _is_ wanted - indeed, essential. In the rest, leaking isn't.
So it goes. Also don't ignore other examples given before, showing how having assignment expressions _at all_ argues for "leaking" in order to be consistent with what assignment expressions do outside of comprehensions and genexps.
But maybe it is just me.
Nope. But it has been discussed so often before this is the last time I'm going to repeat it all again ;-)
... One of these conversations was started with an example something like this:
[(f(x), g(f(x))) for x in an_iterable]
The OP didn't like having to call f() twice. So that would become:
[ (temp:=f(x), g(temp)) for x in an_iterable]
so now the question is: should "temp" be created / changed in the enclosing local scope?
This sure looks a lot like letting the iteration name (x in this example) leak out - so I'd say no.
In that example, right, leaking temp
almost certainly isn't wanted.
So
it goes.
And I don't think this kind of thing would be rare.
I do. It's dead easy to make up examples to "prove" anything people like, but I'm unswayed unless examples come from real code, or are obviously compelling.
Since we're not going to get a way to explicitly say which targets (neither
for
nor assignment expression) do and don't leak, it's a reasonably
satisfying compromise to say that one kind never leaks and the other kind
always leaks. The pick your poison accordingly.
In the example above, note that they _could_ already do, e.g.,
[(fx, g(fx)) for x in an_iterable for fx in [f(x)]]
Then nothing leaks (well, unless f() or g() do tricky things). I
personally wouldn't care that temp
leaks - but then I probably would have
written that example as the shorter (& clearer to my eyes):
[(v. g(v)) for v in map(f, an_iterable)]
to begin with.
On 6/28/2018 11:21 PM, Tim Peters wrote:
[somewhere below] this is the last time I'm going to repeat it all again ;-)
For me, this is your most convincing exposition and summary of why the proposal is at least ok. Thank you.
[Chris]
yes, it was a contrived example, but the simplest one I could think of off the top of my head that re-bound a name in the loop -- which was what I thought was the entire point of this discussion?
But why off the top of your head? There are literally hundreds & hundreds of prior messages about this PEP, not to mention that you could also find examples in the PEP. Why make up a senseless example?
If we think hardly anyone is ever going to do that -- then I guess it doesn't matter how it's handled.
So look at real examples. One that's been repeated at least a hundred times wants a local to "leak into" a listcomp:
total = 0 cumsums = [total ::= total + value for value in data]
As an educator, how are you going to explain that blowing up with
UnboundLocalError instead? Do you currently teach that comprehensions
and genexps are implemented via invisible magically generated lexically
nested functions? If not, you're going to have to start for people to
even begin to make sense of UnboundLocalError if total
_doesn't_ "leak
into" that example. My belief is that just about everyone who doesn't
know "too much" about the current implementation will be astonished &
baffled if that example doesn't "just work".
In other cases it's desired that targets "leak out":
while any(n % (divisor := p) == 0 for p in small_primes): n //= divisor
And in still other cases no leaking (neither in nor out) is desired.
Same as for
targets in that way,. but in the opposite direction: they
don't leak and there's no way to make them leak, not even when that's
wanted. Which _is_ wanted in the last example above, which would be
clearer still written as:
while any(n % p == 0 for p in small_primes): n //= p
But that ship has sailed.
... And "nonlocal" is not used that often, and when it is it's for careful closure trickery -- I'm guessing := will be far more common.
My guess (recorded in the PEP's Appendix A) is that assignment
expressions _overall_ will be used more often than ternary if
but
significantly less often than augmented assignment. I expect their use
in genexps and comprehensions will be minimal. There are real use cases
for them, but the vast majority of genexps and comprehensions apparently
have no use for them at all.
And, of course, when a newbie encounters it, they can google it and see what it means -- far different that seeing a := in a comprehension and understanding (by osmosis??) that it might make changes in the local scope.
Which relates to the above: how do you teach these things? The idea that "a newbie" even _suspects_ that genexps and listcomps have something to do with lexically nested scopes and invisible nested functions strikes me as hilarious ;-)
Regardless of how assignment expressions work in listcomps and genexps,
this example (which uses neither) _will_ rebind the containing block's
x
:
[x := 1]
How then are you going to explain that this seemingly trivial variation _doesn't_?
[x := 1 for ignore in "a"]
For all the world they both appear to be binding x
in the code block
containing the brackets. So let them.
Even worse,
[x for ignore in range(x := 1)]
will rebind x
in the containing block _regardless_ of how assignment
expression targets are treated in "most of" a comprehension, because the
expression defining the iterable of the outermost "for" _is_ evaluated
in the containing block (it is _not_ evaluated in the scope of the
synthetic function).
That's not a special case for targets if they all "leak", but is if they don't.
And I don't think you can even do that with generator expressions now -- as they can only contain expressions.
Expressions can invoke arbitrary functions, which in turn can do anything whatsoever.
Which is my point -- this would allow the local namespace to be manipulated in places it never could before.
As above, not true. However, it would make it _easier_ to write senseless code mucking with the local namespace - if that's what you want to do.
Maybe it's only comprehensions, and maybe it'll be rare to have a confusing version of those, so it'll be no big deal, but this thread started talking about educators' take on this -- and as an educator, I think this really does complicate the language.
I'll grant that it certainly doesn't simplify the language ;-)
Python got much of it's "fame" by being "executable pseudo code" -- its been moving farther and farther away from those roots. That's generally a good thing, as we've gain expressiveness in exchangel, but we shouldn't pretend it isn't happening, or that this proposal doesn't contribute to that trend.
I didn't say a word about that one way or the other. I mostly agree, but at the start Guido was aiming to fill a niche between shell scripting languages and C. It was a very "clean" language from the start, but not aimed at beginners. Thanks to his experience working on ABC, it carried over some key ideas that were beginner-friendly, though.
I view assignment expressions as being aimed at much the same audience as augmented assignments: experienced programmers who already know the pros and cons from vast experience with them in a large number of other widely used languages. That's also a key Python audience.
... Well, I've been surprised by what confused students before, and I will again. But I dont hink there is any doubt that Python 3.7 is a notably harder to learn that Python 1.5 was...
Absolutely. It doesn't much bother me, though - at this point the language and its widely used libraries are so sprawling that I doubt anyone is fluent in all of it. That's a sign of worldly success.
... and this:
In [55]: x = 0 In [56]: [x for x in range(3)] Out[56]: [0, 1, 2] In [57]: x Out[57]: 0
doesn't change x in the local scope --
In Python 3, yes; in Python 2 it rebinds x
to 2.
if that was a good idea, why is a good idea to have := in a comprehension effect the local scope??
Because you can't write a genexp or comprehension AT ALL without
specifying for
targets, and in the overwhelming majority of genexps
and comprehensions anyone ever looked at, "leaking" of for-targets was
not wanted. "So don't let them leak" was pretty much a no-brainer for
Python 3.
But assignment expressions are NEVER required to write a genexp or comprehension, and there are only a handful of patterns known so far in which assignment expressions appear to be of real value in those contexts. In at least half those patterns, leaking _is_ wanted - indeed, essential. In the rest, leaking isn't.
So it goes. Also don't ignore other examples given before, showing how having assignment expressions _at all_ argues for "leaking" in order to be consistent with what assignment expressions do outside of comprehensions and genexps.
But maybe it is just me.
Nope. But it has been discussed so often before this is the last time I'm going to repeat it all again ;-)
... One of these conversations was started with an example something like this:
[(f(x), g(f(x))) for x in an_iterable]
The OP didn't like having to call f() twice. So that would become:
[ (temp:=f(x), g(temp)) for x in an_iterable]
so now the question is: should "temp" be created / changed in the enclosing local scope?
This sure looks a lot like letting the iteration name (x in this example) leak out - so I'd say no.
In that example, right, leaking temp
almost certainly isn't
wanted.
So it goes.
And I don't think this kind of thing would be rare.
I do. It's dead easy to make up examples to "prove" anything people like, but I'm unswayed unless examples come from real code, or are obviously compelling.
Since we're not going to get a way to explicitly say which targets
(neither for
nor assignment expression) do and don't leak, it's a
reasonably satisfying compromise to say that one kind never leaks and
the other kind always leaks. The pick your poison accordingly.
In the example above, note that they _could_ already do, e.g.,
[(fx, g(fx)) for x in an_iterable for fx in [f(x)]]
Then nothing leaks (well, unless f() or g() do tricky things). I
personally wouldn't care that temp
leaks - but then I probably would
have written that example as the shorter (& clearer to my eyes):
[(v. g(v)) for v in map(f, an_iterable)]
to begin with.
-- Terry Jan Reedy
On Jun 28, 2018, at 8:21 PM, Tim Peters tim.peters@gmail.com wrote:
Seems it’s all been said, and Tim’s latest response made an excellent case for consistency.
But:
Regardless of how assignment expressions work in
listcomps and genexps, this example (which uses neither) _will_ rebind the containing
block's x
:
[x := 1]
This reinforces my point that it’s not just about comprehensions, but rather that the local namespace can be altered anywhere an expression is used — which is everywhere.
That trivial example is unsurprising, but as soon as your line of code gets a bit longer, it could be far more hidden.
I’m not saying it’s not worth it, but it a more significant complication than simply adding a new feature like augmented assignment or terniary expressions, where the effect is seen only where it is used.
A key problem with thinking about this is that we can scan existing code to find places where this would improve the code, and decide if those use-cases would cause confusion.
But we really can’t anticipate all the places where it might get used (perhaps inappropriately) that would cause confusion. We can hope that people won’t tend to do that, but who knows?
Example: in a function argument:
result = call_a_func(arg1, arg2, kwarg1=x, kwarg2=x:=2*y)
Sure, there are always ways to write bad code, and most people wouldn’t do that, but someone, somewhere, that thinks shorter code is better code might well do it. Or something like it.
After all, expressions can be virtually anywhere in your code.
Is this a real risk? Maybe not, but it is a complication.
-CHB
[Tim]
Regardless of
how assignment expressions work in listcomps and genexps,
this example (which uses neither) _will_ rebind the containing block's
x
:
>
[x := 1]
> [Chris Barker]
This reinforces my point that it’s not just about comprehensions,
I agree, it's not at all - and I'm amazed at the over-the-top passion that minor issues of scope in comprehensions have ... inspired. It's the tip of the tail of the dog.
but rather that the local namespace can be altered anywhere an expression is used — which is everywhere.
Yes, everywhere. But what of it? Have you read the PEP? The examples are all simple and straightforward and "local". My example above was wholly contrived to make a specific point, and I expect we'll _never_ see that line in real code.
>
That trivial example is unsurprising, but as soon as your line of code gets a bit longer, it could be far more hidden.
It's not possible to prevent people from writing horrible code, and I'm hard pressed to think of _any_ programming feature that can't be so abused. From ridiculouslyLongVariableNamesWhoseVerbostiySeemsToBeAGoalInItself. massive overuse of globals, insanely deep nesting, horridly redundant parenthesization, functions with 20 undocumented arguments, creating Byzantine class structures spread over a directory full of modules to implement a concept that _could_ have been done faster and better with a list, ...
So on a scale of 1 ("wake me up when it's over") to 100 ("OMG! It's the end of the world!!!"), "but it can be horridly abused" rates about a 2 on my weighting scale. Do we really think so little of our fellow Pythoneers? Key point: absolutely nobody has expressed a fear that they _themself_ will abuse assignment expressions. It's always some seemingly existential dread that someone else will ;-)
I’m not saying it’s not worth it, but it a more significant
complication than simply adding a new feature like augmented
assignment or terniary expressions, where the effect is seen only
where it is used.
Which is good to keep in mind when using a feature like this. Python is an imperative language, and side effects are rampant. Controlling them is important.
A key problem with thinking about this is that we can scan existing code to find places where this would improve the code, and decide if those use-cases would cause confusion.
I went through that exercise for the PEP's Appendix A. I assume you haven't read it. I found many places where assignment expressions would make for a small improvement, and got surprised by concluding it was really the multitude of tiny, extremely-local improvements that "added up" to the real win overall, not the much rarer cases where assignment expressions really shine (such as in collapsing chains of semantically misleading ever-increasing indentation in long assign/f/else/assign/if/else/assign/if/else ...structures). I also gave examples of places where, despite being "small and local" changes, using assignment expressions appeared to be a _bad_ idea.
But we really can’t anticipate all the places where it might get used
perhaps inappropriately) that would cause confusion. We can hope that people won’t tend to do that, but who knows?
Having spent considerable time on it myself (see just above), I do not assume that other Pythonistas are incapable of reaching sane conclusions too ;-)
Example: in a function argument:
>
result = call_a_func(arg1, arg2, kwarg1=x, kwarg2=x:=2*y)
The PEP already calls that one a SyntaxError. I can't imagine why a sane programmer would want to do that, but if they really must the PEP _will_ allow it if they parenthesize the assignment expression in this context (so "kwarg2=(x:=2*y)" instead.
Sure, there are always ways to write bad code, and most people
wouldn’t do that, but someone, somewhere, that thinks shorter code is
better code might well do it. Or something like it.
Someone will! No doubt about it. But what of it? If someone is programming for their own amusement, why should I care? If they're working with a group, bad practice should be discouraged by the group's coding standards and enforced by the group's code review process. For this feature, and all others.
"Consenting adults" is a key Python principle too. And I believe in it, despite that I wrote tabnanny.py ;-)
On 30 June 2018 at 09:49, Chris Barker - NOAA Federal via Python-Dev
python-dev@python.org wrote:
On Jun 28, 2018, at 8:21 PM, Tim Peters tim.peters@gmail.com wrote:
Seems it’s all been said, and Tim’s latest response made an excellent case for consistency.
But:
Regardless of how assignment expressions work in
listcomps and genexps, this example (which uses neither) _will_ rebind the containing
block's x
:
[x := 1]
This reinforces my point that it’s not just about comprehensions, but rather that the local namespace can be altered anywhere an expression is used — which is everywhere.
That trivial example is unsurprising, but as soon as your line of code gets a bit longer, it could be far more hidden.
The significant semantic differences between "{x : 1}" and "{x := 1}" are also rather surprising :)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jun 30, 2018 at 06:30:56PM +1000, Nick Coghlan wrote:
The significant semantic differences between "{x : 1}" and "{x := 1}" are also rather surprising :)
Significant and obvious differences are good. It's the subtle differences that you don't notice immediately that really hurt:
{x+1} versus {x-1}
x > y versus x < y
x/y versus x//y
alist = [a, b]
alist = (a, b)
Sometimes small differences in punctuation or spelling make a big difference to semantics.
Punctuation Saves Lives!
"Let's eat, grandma!"
"Let's eat grandma!"
Unless you propose to ban all operators and insist on a minimum string distance between all identifiers:
https://docs.python.org/3/library/os.html#os.spawnl
picking out little differences in functionality caused by little differences in code is a game we could play all day.
At least we won't have the "=" versus "==" bug magnet from C, or the "==" versus "===" confusion from Javascript. Compared to that, the in-your-face obvious consequences of {x: 1} versus {x := 1} are pretty harmless.
-- Steve
On 29 June 2018 at 08:42, Chris Barker via Python-Dev
python-dev@python.org wrote:
On Thu, Jun 28, 2018 at 9:28 AM, Tim Peters tim.peters@gmail.com wrote:
Did adding ternary if
(truepart if
expression else falsepart) complicate
the language significantly?
I don't think so -- no. For two reasons:
1) the final chosen form is kind of verbose, but therefor more like "executable pseudo code" :-) As apposed to the C version, for instance.
2) it added one new construct, that if, when someone sees it for the first (or twenty fifth) time and doesn't understand it, they can look it up, and find out. and it only effects that line of code.
So adding ANYTHING does complicate the language, by simply making it a bit larger, but some things are far more complicating than others.
It's worth noting that without the bug prone "C and A or B" construct (which gives the wrong result when "not A" is True), we'd likely never have gotten "A if C else B" (which gives the right result regardless of the truth value of A). In the case of PEP 308, the new construction roughly matched the existing idiom in expressive power, it just handled it correctly by being able to exactly match the developer's intent.
"NAME := EXPR" exists on a different level of complexity, since it adds name binding in arbitrary expressions for the sake of minor performance improvement in code written by developers that are exceptionally averse to the use of vertical screen real estate, and making a couple of moderately common coding patterns (loop-and-a-half, if-elif-chains with target binding) more regular, and hence easier to spot.
I think the current incarnation of PEP 572 does an excellent job of making the case that says "If we add assignment expressions, we should add them this particular way" - there are a lot of syntactic and semantic complexities to navigate, and it manages to make its way through them and still come out the other side with a coherent and self-consistent proposal that copes with some thoroughly quirky existing scoping behaviour.
That only leaves the question of "Does the gain in expressive power match the increase in the cognitive burden imposed on newcomers to the language?", and my personal answer to that is still "No, I don't think it does". It isn't my opinion on that that matters, though: I think that's now going to be a conversation between Guido and folks that are actively teaching Python to new developers, and are willing to get their students involved in some experiments.
Cheers, Nick.
P.S. It does make me wonder if it would be possible to interest the folks behind https://quorumlanguage.com/evidence.html in designing and conducting fully controlled experiments comparing the comprehensibility of pre-PEP-572 code with post-PEP-572 code before the syntax gets added to the language :)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
[Nick Coghlan]
...
"NAME := EXPR" exists on a different level of complexity, since it
adds name binding in arbitrary expressions for the sake of minor
performance improvement in code written by developers that are
exceptionally averse to the use of vertical screen real estate, ...
Note that PEP 572 doesn't contain a single word about "performance" (neither that specific word nor any synonym), and I gave only one thought to it when writing Appendix A: "is this going to slow anything down significantly?". The answer was never "yes", which I thought was self-evident, so I never mentioned it. Neither did Chris or Guido.
Best I can recall, nobody has argued for it on the grounds of "performance". except in the indirect sense that sometimes it allows a more compact way of reusing an expensive subexpression by giving it a name. Which they already do by giving it a name in a separate statement, so the possible improvement would be in brevity rather than performance.
The attractions are instead in the areas of reducing redundancy, improving clarity, allowing to remove semantically pointless indentation levels in some cases, indeed trading away some horizontal whitespace in otherwise nearly empty lines for freeing up a bit of vertical screen space, and in the case of comprehensions/genexps adding straightforward ways to accomplish some conceptually trivial things that at best require trickery now (like emulating a cell object by hand).
Calling all that "for the sake of minor performance improvements" - which isn't even in the list - is sooooo far off base it should have left me speechless - but it didn't ;-)
But now that you mention it, ya, there will be a trivial performance improvement in some cases. I couldn't care less about that, and can confidently channel that Guido doesn't either. It would remain fine by me if assignment expressions ran trivially slower.
On Sat, Jun 30, 2018 at 9:43 AM Tim Peters tim.peters@gmail.com wrote:
The attractions are instead in the areas of reducing redundancy, improving clarity, allowing to remove semantically pointless indentation levels in some cases, indeed trading away some horizontal whitespace in otherwise nearly empty lines for freeing up a bit of vertical screen space, and in the case of comprehensions/genexps adding straightforward ways to accomplish some conceptually trivial things that at best require trickery now (like emulating a cell object by hand).
The examples you provided (some were new in this thread, I think) are compelling. While my initial reaction to the proposal was mild horror, I'm not troubled by the scoping questions.
Issues still bothering me:
To be fair, I felt a similar gut reaction to f-strings, and now I can't live without them. Have I become a cranky old man, resistant to change? Your examples have put me into the "on the fence, slightly worried" category instead of "clearly a bad idea".
On scoping, beginners seem more confused by UnboundLocalError than by variables bleeding between what they perceive as separate scopes. The concept of a scope can be tricky to communicate. Heck, I still make the mistake of looking up class attributes in instance methods as if they were globals. Same-scope is natural. Natural language is happy with ambiguity. Separate-scope is something programmers dreamed up. Only experienced C, Java, etc. programmers get surprised when they make assumptions about what syntax in Python creates separate scopes, and I'm not so worried about those folks. I remind them that the oldest versions of C didn't have block scopes (1975?) and they quiet down.
The PEP lists many exclusions of where the new := operator is invalid [0]. I unfortunately didn't have a chance to read the initial discussion over the operator. I'm sure it was thorough :-). What I can observe is that each syntactical exclusion was caused by a different confusion, probably teased out by that discussion. Many exclusions means many confusions.
My intuition is that the awkwardness stems from avoiding the replacement of = with :=. Languages that use := seem to avoid the Yoda-style comparison recommendation that is common to languages that use = for assignment expressions. I understand the reluctance for such a major change to the appearance of Python code, but it would avoid the laundry list of exclusions. There's some value in parsimony.
Anyway, we've got some time for testing the idea on live subjects.
Have a good weekend, everyone. -- Michael
PS. Pepe just tied it up for Portugal vs Uruguay. Woo! ... and now Cavani scored again :-(
[0] https://www.python.org/dev/peps/pep-0572/#exceptional-cases
On 1 July 2018 at 02:37, Tim Peters tim.peters@gmail.com wrote:
[Nick Coghlan]
...
"NAME := EXPR" exists on a different level of complexity, since it
adds name binding in arbitrary expressions for the sake of minor
performance improvement in code written by developers that are
exceptionally averse to the use of vertical screen real estate, ...
Note that PEP 572 doesn't contain a single word about "performance" (neither that specific word nor any synonym), and I gave only one thought to it when writing Appendix A: "is this going to slow anything down significantly?". The answer was never "yes", which I thought was self-evident, so I never mentioned it. Neither did Chris or Guido.
Best I can recall, nobody has argued for it on the grounds of "performance". except in the indirect sense that sometimes it allows a more compact way of reusing an expensive subexpression by giving it a name. Which they already do by giving it a name in a separate statement, so the possible improvement would be in brevity rather than performance.
The PEP specifically cites this example as motivation:
group = re.match(data).group(1) if re.match(data) else None
That code's already perfectly straightforward to read and write as a single line, so the only reason to quibble about it is because it's slower than the arguably less clear two-line alternative:
_m = re.match(data) group = _m.group(1) if _m else None
Thus the PEP's argument is that it wants to allow the faster version to remain a one-liner that preserves the overall structure of the version that repeats the subexpression:
group = _m.group(1) if _m := re.match(data) else None
That's a performance argument, not a readability one (as if you don't care about performance, you can just repeat the subexpression).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
[Nick Coghlan]
"NAME := EXPR" exists on a different level of complexity, since it adds name binding in arbitrary expressions for the sake of minor performance improvement in code written by developers that are exceptionally averse to the use of vertical screen real estate,
...
[Tim]
Note that PEP 572 doesn't contain a single word about "performance" (neither
that specific word nor any synonym), and I gave only one thought to it when
writing Appendix A: "is this going to slow anything down significantly?".
The answer was never "yes", which I thought was self-evident, so I never
mentioned it. Neither did Chris or Guido.
>
Best I can recall, nobody has argued for it on the grounds of "performance".
except in the indirect sense that sometimes it allows a more compact way of
reusing an expensive subexpression by giving it a name. Which they already
do by giving it a name in a separate statement, so the possible improvement
would be in brevity rather than performance.
[Nick]
The PEP specifically cites this example as motivation:
The PEP gives many examples. Your original was a strawman mischaracterization of the PEP's _motivations_ (note the plural: you only mentioned "minor performance improvement", and snipped my listing of the major motivations).
>
group = re.match(data).group(1) if re.match(data) else None
>
That code's already perfectly straightforward to read and write as a
single line,
I disagree. In any case of textual repetition, it's a visual pattern-matching puzzle to identify the common substrings (I have to visually scan that line about 3 times to be sure), and then a potentially difficult conceptual puzzle to figure out whether side effects may result in textually identical substrings evaluating to different objects. That's why "refererential transparency" is so highly valued in functional languages ("if subexpressions are spelled the same, they evaluate to the same result, period" - which isn't generally true in Python - to get that enormously helpful (to reasoning) guarantee in Python you have to ensure the subexpression is evaluated exactly once).
And as you of all people should be complaining about, textual repetition is also prone to "oops - forgot one!" and "oops! made a typo when changing the second one!" when code is later modified.
so the only reason to quibble about it
I gave you three better reasons to quibble about it just above ;-)
is because it's slower than the arguably less clear two-line alternative:
>
_m = re.match(data)
group = _m.group(1) if _m else None
>
I find that much clearer than the one-liner above: the visual pattern
matching is easier because the repeated substring is shorter and of much
simpler syntactic structure; it guarantees _by construction_ that the two
instances of _m
evaluate to the same object, so there's no possible
concern about that (it doesn't even matter if you bound re
to some
"non-standard" object that has nothing to do with Python's re
module);
and any later changes to the single instance of re.match(data)
don't have
to be repeated verbatim elsewhere. It's possible that it runs twice as
fast too, but that's the least of my concerns.
All of those advantages are retained in the one-liner too if an assignment expression can be used in it.
Thus the PEP's argument is that it wants to allow the faster version
to remain a one-liner that preserves the overall structure of the
version that repeats the subexpression:
>
group = _m.group(1) if _m := re.match(data) else None
>
That's a performance argument, not a readability one (as if you don't
care about performance, you can just repeat the subexpression).
> How does that differ from the part of what I said that you did retain above?
sometimes it allows a more compact way of reusing an expensive subexpression by giving it a name. Which they already do by giving it a name in a separate statement, so the possible improvement would be in brevity rather than performance.
You already realized the performance gain could be achieved by using two
statements. The _additional_ performance gain by using assignment
expressions is at best trivial (it may save a LOAD_FAST opcode to fetch the
object bound to _m
for the if
test).
So, no, gaining performance is _not_ the motivation here. You already had a way to make it "run fast'. The motivation is the _brevity_ assignment expressions allow while _retaining_ all of the two-statement form's advantages in easier readability, easier reasoning, reduced redundancy, and performance.
As Guido said, in the PEP, of the example you gave here:
Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code
It couldn't possibly be clearer that Guido thought the programmer's motivation was brevity ("in order to save one line of code"). Guido only happened to mention that they were willing to slow down the code to get that brevity, but, as above, they were also willing to make the code harder to read, reason about, and maintain. With the assignment expression, they don't have to give up any of the latter to get the brevity they mistakenly _think_ ;-) they care most about - and, indeed, they can make it even briefer.
I sure don't count it against the PEP that it may trick people overly concerned with brevity into writing code that's clearer and faster too, but that's a tiny indirect part of the PEP's motivation_s_ (note the plural again).
On 1 July 2018 at 14:32, Tim Peters tim.peters@gmail.com wrote:
[Nick]
The PEP specifically cites this example as motivation:
The PEP gives many examples. Your original was a strawman mischaracterization of the PEP's _motivations_ (note the plural: you only mentioned "minor performance improvement", and snipped my listing of the major motivations).
I listed two motivations, not one:
Technically, avoid repeated subexpressions without requiring a separate line also falls into the second category.
The subsequent interaction with comprehensions and generator expressions is an interesting side effect of extending the basic idea to a fully coherent and self-consistent proposal, not one of the original motivations for it.
group = re.match(data).group(1) if re.match(data) else None
That code's already perfectly straightforward to read and write as a
single line,
I disagree. In any case of textual repetition, it's a visual pattern-matching puzzle to identify the common substrings (I have to visually scan that line about 3 times to be sure), and then a potentially difficult conceptual puzzle to figure out whether side effects may result in textually identical substrings evaluating to different objects. That's why "refererential transparency" is so highly valued in functional languages ("if subexpressions are spelled the same, they evaluate to the same result, period" - which isn't generally true in Python - to get that enormously helpful (to reasoning) guarantee in Python you have to ensure the subexpression is evaluated exactly once).
And as you of all people should be complaining about, textual repetition is also prone to "oops - forgot one!" and "oops! made a typo when changing the second one!" when code is later modified.
That's a reasonable readability based argument, but it's not what the PEP currently gives as a motivation for this aspect of the proposal.
so the only reason to quibble about it
I gave you three better reasons to quibble about it just above ;-)
Then add them to the PEP, as what's currently there really isn't offering a compelling motivation for this aspect of the proposal :)
is because it's slower than the arguably less clear two-line alternative:
_m = re.match(data)
group = _m.group(1) if _m else None
I find that much clearer than the one-liner above: the visual pattern
matching is easier because the repeated substring is shorter and of much
simpler syntactic structure; it guarantees _by construction_ that the two
instances of _m
evaluate to the same object, so there's no possible
concern about that (it doesn't even matter if you bound re
to some
"non-standard" object that has nothing to do with Python's re
module); and
any later changes to the single instance of re.match(data)
don't have to
be repeated verbatim elsewhere. It's possible that it runs twice as fast
too, but that's the least of my concerns.
I agree with this, but also think the two-line form is a perfectly acceptable way of spelling it, and a perfectly acceptable refactoring of the one-line form with duplicated subexpressions to improve maintainability.
All of those advantages are retained in the one-liner too if an assignment expression can be used in it.
Sure, but the open design question is whether folks that would have written the one-liner with repeated subexpressions are going to be any more likely to use an assignment expression to avoid the repetition without prompting by a more experienced developer than they are to use a separate preceding assignment statement.
That assessment of "What is the increased chance that the repeated subexpression will be avoided when the code is first written?" then gets traded off against the overall increase in language complexity arising from allowing name bindings in arbitrary subexpressions.
Don't get me wrong, I now agree that the proposal in PEP 572 is the most coherent and self-consistent approach to assignment expressions that we could pursue given the existing scoping semantics of comprehensions and generator expressions.
The remaining point of contention is only the "Inevitable cost of change" one: given the level of disruption this will cause in the way that Python gets taught to new users, is it giving a commensurate pay-off in increased semantic expressiveness?
My answer to that question remains "No", while your answer is either "Yes" or "I don't see why that should matter" (I'm genuinely unsure which).
sometimes it allows a more compact way of reusing an expensive subexpression by giving it a name. Which they already do by giving it a name in a separate statement, so the possible improvement would be in brevity rather than performance.
You already realized the performance gain could be achieved by using two
statements. The _additional_ performance gain by using assignment
expressions is at best trivial (it may save a LOAD_FAST opcode to fetch the
object bound to _m
for the if
test).
So, no, gaining performance is _not_ the motivation here. You already had a way to make it "run fast'. The motivation is the _brevity_ assignment expressions allow while _retaining_ all of the two-statement form's advantages in easier readability, easier reasoning, reduced redundancy, and performance.
I never said the motivation was to gain performance relative to the two-statement version - I said the motivation given in the PEP is to gain performance relative to the repeated subexpression version, without making the transition to the already supported two-statement version.
As Guido said, in the PEP, of the example you gave here:
Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code
It couldn't possibly be clearer that Guido thought the programmer's motivation was brevity ("in order to save one line of code"). Guido only happened to mention that they were willing to slow down the code to get that brevity, but, as above, they were also willing to make the code harder to read, reason about, and maintain. With the assignment expression, they don't have to give up any of the latter to get the brevity they mistakenly _think_ ;-) they care most about - and, indeed, they can make it even briefer.
The quoted paragraph from the PEP clearly states that the reason the repeated subexpression is considered a problem is because it slows down the program, not because it repeats code.
As noted above, the PEP could certainly be updated to point out that repeating subexpressions is problematic for more reasons than just speed, but that isn't what it currently says.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I think I'll bow out of this now. It's just too tedious.
Like here:
[Nick]
I never said the motivation was to gain performance relative to the two-statement version - I said the motivation given in the PEP is to gain performance relative to the repeated subexpression version, without making the transition to the already supported two-statement version.
This is what you wrote:
"NAME := EXPR" exists on a different level of complexity, since it adds name binding in arbitrary expressions for the sake of minor performance improvement in code written by developers that are exceptionally averse to the use of vertical screen real estate,
I'm not telepathic, so took it for it what said. It didn't say a word about "repeated expressions", nor a word about "two-statement versions"
It did say "minor" performance improvements, which sure suggested to me that you had in mind the tiny changes in bytecode that can result from squashing a current two-statement spelling into a one-statement form with an embedded assignment. That fit both "minor performance improvement" and "vertical screen space", which were the only the two clues I had to go on.
In the example you eventually gave in a later message, the performance difference was more on the order of a factor 2 (regexp searches can be expensive indeed), which matches nobody's idea of "minor". So the use of "minor" wholly ruled out in my mind that you had _anything_ akin to your eventual example in mind.
For the rest, sure, words could be added to the PEP without end. At least some of the points I covered were telegraphically mentioned in my Appendix, although I'll grant that nobody else is telepathic either ;-)
For the question:
The remaining point of contention is only the "Inevitable cost of
change" one: given the level of disruption this will cause in the way
that Python gets taught to new users, is it giving a commensurate
pay-off in increased semantic expressiveness?
>
My answer to that question remains "No", while your answer is either
"Yes" or "I don't see why that should matter" (I'm genuinely unsure
which).
I don't know, and I'm not qualified to guess - I don't teach Python to new users for a living. Decades ago I tutored "advanced" engineering undergrads in a variety of science-y subjects, and was routinely surprised by what broke their brains.
I have noted that assignment expressions have been part of a great many languages for decades (this isn't cutting edge tech), questions about them are conspicuous by absence on StackOverflow (everyone else seems to teach them effectively), and skimming various online teaching materials for other languages convinced me that none of those professional educators thought it needed more than a page to teach (and to teach them with relatively few words). There's really not much to them: people have to learn what binding means in Python regardless, pretty much starting in the first hour, yes? "Binding" on its own is the hard part.
If they don't drop out and stick around for the 10-minute lesson on assignment expressions 3 weeks later, will _that_ finally break their brains? "Wow - it not only binds the name, but ALSO returns the object that was bound?! I just can't take it - please, let's go back to the lesson on metaclasses" just doesn't seem likely to me ;-)
At heart, ":=" could make a good case for being the simplest of all Python's operators. It does no computation at all, and you don't even have to worry about a dunder method changing its meaning depending on context.
So, ya, when someone claims they'll make Python significantly harder to teach, I'm skeptical of that claim. Which does not mean I believe it's wrong - it means I'm skeptical. I would also be skeptical of a claim that teaching them would be no trouble at all, except nobody has made such a claim ;-)
On Sun, Jul 1, 2018 at 12:39 AM Tim Peters tim.peters@gmail.com wrote:
So, ya, when someone claims [assignment expressions will] make Python significantly harder to teach, I'm skeptical of that claim.
I don't believe anyone is making that claim. My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
On Sun, Jul 1, 2018 at 2:19 AM Mark Dickinson dickinsm@gmail.com wrote:
There's a constant struggle to keep the Python portion of the course large enough to be coherent and useful, but small enough to allow time for the other topics.
On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik mike@selik.org wrote:
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword.
But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well.
Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
This cynical view on students is shocking! Everyone on this list has been a student or a learner for far longer than an educator, and the perspective from students and learners are far more important than educators to assess this angle regardless. Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities learners use to increase knowledge and skill, from the perspectives of policy, tool creation, and each of our time spent learning?
Shortest story: Teach not to re-use names.
Short story: 1) What about the full mosaic of learning vs. this myopic view on seat-based student-educator interaction? 2) What about smart, motivated, diligent and cautious students? 3) What weight should educator opinion be given with respect to providing convenience to professional Python programmers? 4) Who is this Student Stupid von Densemeister anyways? 5) Are assignment expressions convenience and is any danger the pose unmitagatble? 6) Consider adding an "Important Topics not Covered" or "Further Reading" reading section to your class description 7) Creating examples showing this effect is easy, especially when not actually re-using the name in the expression for explanatory purposes. it's the same as creating examples showing how re-use works in comprehensions.
Let's stop constructing these fake Students. They only work as appeals to the people we have come across whose lack of understanding has made our life painful. This construction is actively filtering all the good students for the sake of influencing this decision, yet again punishing or discounting the intelligent, quick, and diligent.
And what of this underlying premise that educator's should _significantly_ influence language development? Limiting Python's tools to Student Straw-man's ability to learn is just dissonant, they have nothing to do with each other, nor does this cause-effect relationship actually exist. Let's evaluate this reductionist statement: "I understand X, but this other person is not capable of understanding X, therefore X should not exist" Is has there ever been an X for which this is true, let alone the backwardation necessary to fully close the statement?
The actual argument is far less reductionist, yet even more ridiculous: "I understand X, this other person may take time to learn X, and may use X wrong, therefore X should not exist" "I understand assignment expressions, but this other class of person may take time to learn assignment expressions, and may use assignment expressions wrong, therefore assignment expressions should not be accepted"
Rhetorically I disagree with how teaching is being presented, to the point of near insult (for me lacking a better term). You are saying these statements about _my_ learning path, (though not personally of course.) Each of you occupied a role of student at some point, and each of these statements are being made about your path as well. Do these ring true of your student experience? What about your much broader experience as a _learner_? You think a tool shouldn't exist because it took you time to learn it and you wrote some hard to debug code, and possibly crashed production, got fired, lost your house and your pet snake, and crashed the planet into the sun?
Now I yield, I will accept this position: all/some students cannot learn this (or it's too complex to teach), but they must learn this during some class to quickly become effective python developers. How much weight should this position have in this decision? Let's appeal to the learner in us. How much of our learner's path, percentage of total time learning all things python related, has been in a seat listening to someone else, and that's the only place from which we gained the knowledge to meet the educator's objective? This time spent in a class, how does that compare to hours in other learning modalities? Is this percentage not exactly the weight assigned to that position? Are people hired from pure class-room based experience expected to require zero further learning? Are people more valuable based on classroom hours or work hours?
As for handling teaching the subject or not, this is easily remedied with how I do it: "Important Topics not Covered", with resources.
Anyone here can rightfully claim educator status by having taught another person something related to this language, which includes at-work mentoring, informal discussions, posting/replying on SO, blogging, etc. Are they not being solicited to comment as well? It's possible to answer this question while vehemently disagreeing with the PEP. This focus on people who are being ostensibly paid to teach is myopic.
Concretely, it's clear to me that parent-local effects can be dangerously non-obvious when reading and mimicking code without undertsanding. But when? And how to guard against? How about this: teach proper (i.e. not) re-using names. The name will still be ejected to the parent scope, but there won't be any use of it. Teach the explicit declare pattern first (as everyone does anyways), explain to not re-use. Regardless of when re-use is done, it is always (or only) as dangerous as the effect the bound value has anyways, and any re-use has the potential to trigger the same dangerous behaviors.
Must we continue this educator assessment of PEP 572? Educators are not gate-keepers, and the only measure of their success is what students learn, and students have a far larger and more fine-grained mosaic now than ever. Seat-based education plays a far smaller role in anyone's learning path than ever before, and even while in the seat they have access to Google. All this yield to tiling the outcome in the favor of the educator by assigning the goal meeting to them, not to the diligence of the student to learn. How unfair to the student.
I consider this position purposefully ignoring motivated self-learners of high ability and skill, or just plain old diligent programmers who learned to read specs before using tools.
I don't like posting to python-dev because it's not really my realm, but this topic is insanely tilted against PEP572 for the most ridiculous of reasons. I am Pro-572 , so I have decided to join critique of the educator position. I would rather do it on python-Ideas, and I want more types of educators solicited, as well as students and learners. And yes, lets assess your specific lesson plans if you will make claims it's not possible. Do you even teach the difference between assignments and expressions at all?
To have this raised in the this stage of this PEP, and on the dev list, illustrates how long it took educators to understand the tool to begin with, as opposed to those who understood even if they disagree. To have a room full of seat-based educators provide feedback on a tool they have had 30m to understand as critical to shaping the language is not defensible in these lower courts. This angle cannot withstand rigorous scrutiny because each premise is false and it rests the majority of students being dense. They aren't, and the vast majority of learners don't need class-room education anyways, so what is this weight being placed on the opinion of these educators? I agree it's important to hear it, but dimishingly.
This is not to say that PEP572 should be accepted otherwise. However, this educator angle, raised only now and depending on this platonically dumb student and a non-creative approach to education, is just pure straw-man to distract from the point at hand. And while I have repeated "straw-man" as the critique, I dislike leaning on such a debate-team crutch, and of course employing straw-man doesn't mean the point is invalid. it just means the rhetoric is bad. However, as the underlying premise is a severe minority opinion yet claiming to be broad, and the absolute percentage of solicited opinions is 0% (to the 17th place), I don't see any importance to the position of educators right now, especially since these educators in the thread are complaining about an increase in their personal work, for which it appears they were compensated (this is pretty bad straw-man, sorry).
And to repeat, what about the learners? Time spent in seat based education is so insanely small now. The fact that the name is ejected to the parent stop is not terribly difficult, and appropriately factored examples showing the the effect of scope ejection will be easy to construct, even if trivial for explanation purposes. So what, exactly, is the issue with learning it?
On Sun, Jul 1, 2018 at 4:35 PM Chris Barker via Python-Dev
python-dev@python.org wrote: >
On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik mike@selik.org wrote: >
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword.
But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well.
Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/marcidy%40gmail.com
Is this message some kind of joke or did you just send it to the wrong mailing-list/recipient?
On Sun, 1 Jul 2018 20:21:19 -0700 Matt Arcidy marcidy@gmail.com wrote:
This cynical view on students is shocking! Everyone on this list has been a student or a learner for far longer than an educator, and the perspective from students and learners are far more important than educators to assess this angle regardless. Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities learners use to increase knowledge and skill, from the perspectives of policy, tool creation, and each of our time spent learning?
Shortest story: Teach not to re-use names.
Short story: 1) What about the full mosaic of learning vs. this myopic view on seat-based student-educator interaction? 2) What about smart, motivated, diligent and cautious students? 3) What weight should educator opinion be given with respect to providing convenience to professional Python programmers? 4) Who is this Student Stupid von Densemeister anyways? 5) Are assignment expressions convenience and is any danger the pose unmitagatble? 6) Consider adding an "Important Topics not Covered" or "Further Reading" reading section to your class description 7) Creating examples showing this effect is easy, especially when not actually re-using the name in the expression for explanatory purposes. it's the same as creating examples showing how re-use works in comprehensions.
Let's stop constructing these fake Students. They only work as appeals to the people we have come across whose lack of understanding has made our life painful. This construction is actively filtering all the good students for the sake of influencing this decision, yet again punishing or discounting the intelligent, quick, and diligent.
And what of this underlying premise that educator's should _significantly_ influence language development? Limiting Python's tools to Student Straw-man's ability to learn is just dissonant, they have nothing to do with each other, nor does this cause-effect relationship actually exist. Let's evaluate this reductionist statement: "I understand X, but this other person is not capable of understanding X, therefore X should not exist" Is has there ever been an X for which this is true, let alone the backwardation necessary to fully close the statement?
The actual argument is far less reductionist, yet even more ridiculous: "I understand X, this other person may take time to learn X, and may use X wrong, therefore X should not exist" "I understand assignment expressions, but this other class of person may take time to learn assignment expressions, and may use assignment expressions wrong, therefore assignment expressions should not be accepted"
Rhetorically I disagree with how teaching is being presented, to the point of near insult (for me lacking a better term). You are saying these statements about _my_ learning path, (though not personally of course.) Each of you occupied a role of student at some point, and each of these statements are being made about your path as well. Do these ring true of your student experience? What about your much broader experience as a _learner_? You think a tool shouldn't exist because it took you time to learn it and you wrote some hard to debug code, and possibly crashed production, got fired, lost your house and your pet snake, and crashed the planet into the sun?
Now I yield, I will accept this position: all/some students cannot learn this (or it's too complex to teach), but they must learn this during some class to quickly become effective python developers. How much weight should this position have in this decision? Let's appeal to the learner in us. How much of our learner's path, percentage of total time learning all things python related, has been in a seat listening to someone else, and that's the only place from which we gained the knowledge to meet the educator's objective? This time spent in a class, how does that compare to hours in other learning modalities? Is this percentage not exactly the weight assigned to that position? Are people hired from pure class-room based experience expected to require zero further learning? Are people more valuable based on classroom hours or work hours?
As for handling teaching the subject or not, this is easily remedied with how I do it: "Important Topics not Covered", with resources.
Anyone here can rightfully claim educator status by having taught another person something related to this language, which includes at-work mentoring, informal discussions, posting/replying on SO, blogging, etc. Are they not being solicited to comment as well? It's possible to answer this question while vehemently disagreeing with the PEP. This focus on people who are being ostensibly paid to teach is myopic.
Concretely, it's clear to me that parent-local effects can be dangerously non-obvious when reading and mimicking code without undertsanding. But when? And how to guard against? How about this: teach proper (i.e. not) re-using names. The name will still be ejected to the parent scope, but there won't be any use of it. Teach the explicit declare pattern first (as everyone does anyways), explain to not re-use. Regardless of when re-use is done, it is always (or only) as dangerous as the effect the bound value has anyways, and any re-use has the potential to trigger the same dangerous behaviors.
Must we continue this educator assessment of PEP 572? Educators are not gate-keepers, and the only measure of their success is what students learn, and students have a far larger and more fine-grained mosaic now than ever. Seat-based education plays a far smaller role in anyone's learning path than ever before, and even while in the seat they have access to Google. All this yield to tiling the outcome in the favor of the educator by assigning the goal meeting to them, not to the diligence of the student to learn. How unfair to the student.
I consider this position purposefully ignoring motivated self-learners of high ability and skill, or just plain old diligent programmers who learned to read specs before using tools.
I don't like posting to python-dev because it's not really my realm, but this topic is insanely tilted against PEP572 for the most ridiculous of reasons. I am Pro-572 , so I have decided to join critique of the educator position. I would rather do it on python-Ideas, and I want more types of educators solicited, as well as students and learners. And yes, lets assess your specific lesson plans if you will make claims it's not possible. Do you even teach the difference between assignments and expressions at all?
To have this raised in the this stage of this PEP, and on the dev list, illustrates how long it took educators to understand the tool to begin with, as opposed to those who understood even if they disagree. To have a room full of seat-based educators provide feedback on a tool they have had 30m to understand as critical to shaping the language is not defensible in these lower courts. This angle cannot withstand rigorous scrutiny because each premise is false and it rests the majority of students being dense. They aren't, and the vast majority of learners don't need class-room education anyways, so what is this weight being placed on the opinion of these educators? I agree it's important to hear it, but dimishingly.
This is not to say that PEP572 should be accepted otherwise. However, this educator angle, raised only now and depending on this platonically dumb student and a non-creative approach to education, is just pure straw-man to distract from the point at hand. And while I have repeated "straw-man" as the critique, I dislike leaning on such a debate-team crutch, and of course employing straw-man doesn't mean the point is invalid. it just means the rhetoric is bad. However, as the underlying premise is a severe minority opinion yet claiming to be broad, and the absolute percentage of solicited opinions is 0% (to the 17th place), I don't see any importance to the position of educators right now, especially since these educators in the thread are complaining about an increase in their personal work, for which it appears they were compensated (this is pretty bad straw-man, sorry).
And to repeat, what about the learners? Time spent in seat based education is so insanely small now. The fact that the name is ejected to the parent stop is not terribly difficult, and appropriately factored examples showing the the effect of scope ejection will be easy to construct, even if trivial for explanation purposes. So what, exactly, is the issue with learning it?
On Sun, Jul 1, 2018 at 4:35 PM Chris Barker via Python-Dev
python-dev@python.org wrote: >
On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik mike@selik.org wrote:
>
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword.
But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well.
Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/marcidy%40gmail.com
On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy marcidy@gmail.com wrote:
[...] Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities [...]?
I hope this was an adequate explanation.
On Mon, Jul 2, 2018 at 2:34 AM Michael Selik mike@selik.org wrote: >
On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy marcidy@gmail.com wrote: >
[...] Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities [...]?
I hope this was an adequate explanation.
Absolutely, thank you. We agree it doesn't out weigh other methods. Clearly I disagree about the proxying.
On Sun, Jul 01, 2018 at 08:35:08AM -0700, Michael Selik wrote:
On Sun, Jul 1, 2018 at 12:39 AM Tim Peters tim.peters@gmail.com wrote:
So, ya, when someone claims [assignment expressions will] make Python significantly harder to teach, I'm skeptical of that claim.
I don't believe anyone is making that claim. My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
How do people who teach other languages deal with this?
Assignment expressions are hardly a new-fangled innovation of Python's. They're used in Java, Javascript, Ruby, Julia, R, PHP and of course pretty much the entire C family (C, C++, C# at least). What do teachers of those languages do?
R has a similar demographic of users (strong in the sciences, many beginners to programming, growing in popularity). Once R teachers have taught that you can assign values like this:
x = 1 + 2
does it take them 15-20 minutes to teach that you can do this as well?
y = (x = 1 + 2) + 3
Admittedly R has the advantage that they don't have to teach a distinct assignment syntax and explain why it ought to be distinct. But countering that, they have four different ways of doing assignment.
x <- expression
expression -> x
x = expression
assign('x', expression)
(all of which can be used as expressions).
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
Without knowing the details of your course, and who they are aimed at, we cannot possibly judge this comment. Decorators are widely used, but surely you don't teach them in a one day introductory class aimed at beginners?
Here is the syllabus for a ten week course:
https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus
Note that decorators and even regular expressions don't get touched until week ten. If you can't fit assignment expressions in a ten week course, you're doing something wrong. If you can't fit them in a two hour beginners course, there is so much more that you aren't covering that nobody will notice the lack.
-- Steve
On Sun, Jul 1, 2018 at 5:28 PM Steven D'Aprano steve@pearwood.info wrote:
On Sun, Jul 01, 2018 at 08:35:08AM -0700, Michael Selik wrote:
On Sun, Jul 1, 2018 at 12:39 AM Tim Peters tim.peters@gmail.com wrote:
So, ya, when someone claims [assignment expressions will] make Python significantly harder to teach, I'm skeptical of that claim.
I don't believe anyone is making that claim. My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
How do people who teach other languages deal with this?
Python may be in a unique situation in the history of programming. It wouldn't surprise me if more people learned Python last year than any other programming language.
Assignment expressions are hardly a new-fangled innovation of Python's. They're used in Java, Javascript, Ruby, Julia, R, PHP and of course pretty much the entire C family (C, C++, C# at least). What do teachers of those languages do?
Assignment expressions are not the issue. The real question is: How do open-source projects balance the addition of new features against the growth of complexity? It's the same as that "Remember the Vasa" thread.
[...] R [has] four different ways of doing assignment. >
I think that's a good explanation of why I teach Python and not R. The first time someone asked me to teach a data science course, Python wasn't the clear winner. In fact, R may have been more popular among statisticians. I picked Python for the same reason it's more popular in the industry -- it's the easiest* to use.
As Mark and Chris said (quoting Mark below), this is just one straw in the
struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
Without knowing the details of your course, and who they are aimed at, we cannot possibly judge this comment.
I disagree. I think the sentiment holds for a great variety of courses and audiences.
Decorators are widely used, but surely you don't teach them in a one day introductory class aimed at beginners?
Most of the time, no. Once, yes, because that's what the team needed. I was pretty proud of myself for handling that one. Because I had to teach decorators early, many other important topics were excluded.
Here is the syllabus for a ten week course:
https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus
Note that decorators and even regular expressions don't get touched until week ten. If you can't fit assignment expressions in a ten week course, you're doing something wrong. If you can't fit them in a two hour beginners course, there is so much more that you aren't covering that nobody will notice the lack.
It's not about any one particular topic, but the trade-offs between topics. A 10-week lecture course might be 30 hours of lecture, comparable to a 4-day "bootcamp" style course. I assure you that 4 days doesn't feel long enough when those last few hours are winding down. There's always more to say.
On Sun, Jul 1, 2018 at 5:25 PM, Steven D'Aprano steve@pearwood.info wrote:
>
an optional topic. Once widely used, it ain't optional.
Without knowing the details of your course, and who they are aimed at, we cannot possibly judge this comment. Decorators are widely used, but surely you don't teach them in a one day introductory class aimed at beginners?
Here is the syllabus for a ten week course:
https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus
That would be mine ;-)
Note that decorators and even regular expressions don't get touched until week ten.
I actually often don't ever get to regex -- why? because they are a topic all of their own, and not unique to Python. And if code uses a regex, it doesn't change anything about any other code anywhere. IN sort, regex's are a library, not a language feature.
Which brings us to decorators (and I'm going to add context managers, as its similar). Decorators (and even more so context managers) are used commonly enough that we certainly have to introduce them in a intro class. But there is a really big distinction between having some idea how to use them, and knowing how they work / how to write them yourself.
So I introduce:
with open(filename) as the_file: do_somethign_with(the_file)
early on, with only a hand-wavy explanation of what that with is all about.
And when i get to properties, I teach them:
@property def a_method(self): ....
with also a hand-wavy explanation of what that @ symbol is.
and if := catches on, I expect it will be far more common that either of those, especially in the simple script style codes that newbies are going to be reading/writing at first.
But in the end, I, at least, am not trying to make the case that assignment expressions are going to be particularly difficult to understand, but that they are a significant complication, and any new feature like that makes the language more complex. That's it.
I taught myself Python with version 1.5 in 1998 -- and have been teaching it for I think almost ten years. In that time, the language has grown substantially in complexity, and it does make it harder to teach.
We (UWPCE) recently revamped our 3-class program, and had a lot of debate about how early to introduce things -- one of the instructors wanted to leave off lambda and comprehensions 'till the second course in the sequence, as part of functional programming. I I think he was right, if we were teaching it in a more isolated environment, but both of those show up in example code all over the place, so I've found it's necessary to introduce a lot at a high level early:
decorators comprehensions lambda context managers *args and **kwargs (off the top of my head)
So there is basically no way that we won't have to add assignment expressions early on.
As someone else said: it's another straw on the haystack.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
On Mon, 2 Jul 2018 10:25:42 +1000 Steven D'Aprano steve@pearwood.info wrote:
How do people who teach other languages deal with this?
Assignment expressions are hardly a new-fangled innovation of Python's. They're used in Java, Javascript, Ruby, Julia, R, PHP and of course pretty much the entire C family (C, C++, C# at least).
Those other languages don't have two different assignment operators, AFAIK. That's the main point of complication PEP 572 introduces, not the fact that assignment can now be used in an expression.
Admittedly R has the advantage that they don't have to teach a distinct assignment syntax and explain why it ought to be distinct. But countering that, they have four different ways of doing assignment.
I don't think R is easy to understand. It depends on the demographics. For a software engineer like me, it's pretty hard to wrap my head around R's nonsense. I think R is only easy if you accept to use it in a "tinker aimlessly until my code works" manner.
Regards
Antoine.
[Tim]
... So, ya, when someone claims [assignment expressions will] make Python significantly harder to teach, I'm skeptical of that claim.
[Michael Selik]
I don't believe anyone is making that claim.
I haven't seen it in this specific thread, but the larger discussion has been going on for several months.
My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
So not intractable - which is my high-order bit ;-)
For those who want more bits of precision (perhaps Guido), while quantification is good, it needs context to provide insight. Like, out of how many class hours total? Is 15-20 minutes a little, a lot, par for the course ... compared to other topics? Will it require you to drop other topics? Would you _save_ twice as much class time if we got rid of "is"? ;-)
As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional.
If it's accepted, do read the PEP - while the syntax will _allow_ assignment expressions just about everywhere, the recommended examples are all simple & straightforward. "If it's not obviously better, don't use it" is excellent advice.
Because it's allowed almost everywhere, I expect that unanticipated "good uses" will pop up, but at the start a high majority of good uses seem to fall into a small number of patterns.
Meta: About the Vasa, I'm not concerned. C++ has around 150 people relentlessly writing an endless sequence of enhancement proposals largely aimed at highly esoteric expert applications of an already extraordinarily complex language. Bjarne Stroustrup is right to be concerned about that. His goal is to cut back on the complications striving for theoretical perfection in all conceivable applications, and go back to working on ideas:
that can be used by “ordinary programmers” whose main concern is to ship
great applications on time
http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p0977r0.pdf
If C++ hadn't inherited assignment expressions from C from the start(*), I
expect that's an idea he'd _want_ to consider now. They're well within the
grasp of "ordinary programmers" - if they can master j = 3
, they're 90%
of the way to mastering j := 3
(although it may be that "good taste"
can't be taught at all).
(*) Yes, I know about the stuff added to support yet another form of
assignment expression in if
and switch
headers in C++ 17. That
appeared to be more for "theoretical purity". Assignment expressions were
always allowed there, but previously only for
headers allowed _declaring_
a new variable inside the parens. Now if
and switch
allow that
too.
On Sun, Jul 1, 2018 at 11:36 PM Tim Peters tim.peters@gmail.com wrote:
[Michael Selik]
My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
So not intractable - which is my high-order bit ;-)
For those who want more bits of precision (perhaps Guido), while quantification is good, it needs context to provide insight. Like, out of how many class hours total?
Generally between 20 and 40 hours.
Is 15-20 minutes a little, a lot, par for the course ... compared to other
topics?
I guessed 15-20 minutes, because I'm mentally comparing it to things like ternary expressions. Odds and ends that make the code better, but not a major concept that deserves hours.
Will it require you to drop other topics? >
Yes. It might not seem like much, but every minute counts. I'd probably try to ignore := unless some pesky student brings it up. It's like someone saying, "Hey, I heard that Python can't do threads?!" I always say, "Good question," but internally I'm thinking, "there goes a half hour. What can I cut today?"
Would you _save_ twice as much class time if we got rid of "is"? ;-)
Ha. You joke, but is
takes about 5 minutes. About 5 or 10 minutes
more
if some clever student notices that 1 is 1
and I need to explain
Singletons and interpreter optimizations versus language spec.
If it's accepted, do read the PEP >
I've read it a few times now. I hope I didn't sound like I haven't read it. That'd be embarrassing.
Meta: About the Vasa, I'm not concerned. >
Matt Arcidy brought up an interesting point, which I'll quote here: "... I don't see any importance to the position of educators right now, especially since these educators in the thread are complaining about an increase in their personal work, for which it appears they were compensated."
From my brief observations, it seems that the nattering nabobs of negativism, such as myself, are mostly educators. I recently started to wonder if I'd care so much about the language if I didn't teach. I suspect that if I didn't worry about teaching new features, Python 4 could be announced tomorrow and I wouldn't really mind.
I suppose it is selfish. But I hope that you [Tim], Guido, and the so many others who have poured energy into this project will appreciate that it's not the current users, but the next billion (?!) Pythonistas that will really keep the language going. Maintaining popularity among educators is a big part of that.
[Michael Selik]
My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort.
[Tim]
So not intractable - which is my high-order bit ;-)
For those who want more bits of precision (perhaps Guido), while quantification is good, it needs context to provide insight. Like, out of how many class hours total?
[Michael]
Generally between 20 and 40 hours.
Ah - so I take it you're not teaching raw computer beginners, but people who already know how to program in some other language(s)? If so, perhaps you could leverage on that assignment expressions are already present in the most heavily used languages. Depends on the students' backgrounds, of course.
Is 15-20 minutes a little, a lot, par for the course ... compared to other topics?
I guessed 15-20 minutes, because I'm mentally comparing it to things like ternary expressions. Odds and ends that make the code better, but not a major concept that deserves hours.
That's where I see it fitting too. While I don't expect it will ever come up, if I were tasked with teaching assignment expressions, I can picture ways of doing it that would take anywhere from one minute to two hours, depending on the target audience's backgrounds, interests, and needs ("two hours" for those fascinated by computer language history).
Will it require you to drop other topics?
Yes. It might not seem like much,
Nope! It was a trick question. If you had answered "no", I would have known you were just making things up ;-)
but every minute counts. I'd probably try to ignore := unless some pesky student brings it up. It's like someone saying, "Hey, I heard that Python can't do threads?!" I always say, "Good question," but internally I'm thinking, "there goes a half hour. What can I cut today?"
Absolutely. 40 hours can't possibly cover more than a significant overview of high-order bits.
Would you _save_ twice as much class time if we got rid of "is"? ;-)
Ha. You joke, but is
takes about 5 minutes. About 5 or 10 minutes
more if some clever student notices that 1 is 1
and I need to explain
Singletons and interpreter optimizations versus language spec.
That surprised me! Educators have often said "is" was hard to teach, and it's one of the F'est of Python FAQs on StackOverflow. I always figured that's because it's trivial if you have deep understanding of Python's conceptual object model, but appears to be a random Boolean generator if you're just mucking around at a shell without that understanding. Still, something like this sometimes temporarily baffles even experts:
[] is [] # OK, []
always creates a new
list
False
id([]) == id([]) # or does it???
True
If it's accepted, do read the PEP
I've read it a few times now. I hope I didn't sound like I haven't read it. That'd be embarrassing.
Heh. No, I just wanted to be sure. It's just impossible to tell from a discussion that's entirely "meta".
... From my brief observations, it seems that the nattering nabobs of negativism, such as myself, are mostly educators.
In this specific thread, sure, but I expect that's because it has "educator feedback" in the Subject. There's been plenty of opposition from non-educators in other threads. I like this thread because it's been more civil than most :-)
I recently started to wonder if I'd care so much about the language if I didn't teach. I suspect that if I didn't worry about teaching new features, Python 4 could be announced tomorrow and I wouldn't really mind.
Sure - and I wouldn't care so much (or, indeed, at all) if Python wasn't my language of choice for most projects for over 20 years now.
I suppose it is selfish. But I hope that you [Tim], Guido, and the so many others who have poured energy into this project will appreciate that it's not the current users, but the next billion (?!) Pythonistas that will really keep the language going.
Which, perhaps paradoxically, is why I'm generally in favor of even small (but general) improvements. regardless of short-term costs: so the next billion Pythonistas can benefit.
Maintaining popularity among educators is a big part of that.
Oh, I have no idea about that. I'm at a loss for what accounts for longer-term language popularity, so I push for things that appeal to me as a programmer with broad and deep experiences. Everyone who cares about the language _should_ be heard, but their motivations don't even need to overlap.
When I started college, I took classes in every language for which there was a class: assembler, FORTRAN, LISP, and SNOBOL4. Three of those survive as niche languages now, and the last is long dead (damned shame, too! SNOBOL4 was brilliantly creative).
The last time I had significant contact with academia, Pascal was all the rage (yes, that dates me). Not only was it universally loved by educators, it was in exactly the right place at exactly the right time: the first compilers for PCs were for Pascal variants, and things like Turbo Pascal were hugely (relative to the much smaller user base at the time) popular and influential.
Try to get a job teaching Pascal now ;-)
So while I don't pretend to know what accounts for longer-term popularity, "evolve or die" captures a necessary condition. Any living language that doesn't continue evolving will die out. Fighting that is like yelling at clouds.
On Sat, Jun 30, 2018 at 11:32:03PM -0500, Tim Peters wrote:
[...]
So, no, gaining performance is _not_ the motivation here. You already had a way to make it "run fast'. The motivation is the _brevity_ assignment expressions allow while _retaining_ all of the two-statement form's advantages in easier readability, easier reasoning, reduced redundancy, and performance.
I think the two of you (Tim and Nick) are in violent agreement :-)
In fairness to Nick performance is _a_ motivation here, in the sense of "how can we avoid making redundant multiple calls to a function in an expression without splitting it into multiple statements?".
If performance were the only concern, then I agree with your point that we already have a solution. Simply refactor this:
process(expensive_call(arg), expensive_call(arg) + 1)
to this:
useful_value = expensive_call(arg)
process(useful_value, useful_value + 1)
But since performance is not the only concern, all the other factors you refer to (saving vertical space, DRY, side-effects, etc) count too.
Assignment expressions aren't valuable because they give us One Big Win. They're valuable because they give us Many Little Wins, which we (proponents of PEP 572) believe will outweigh the (minor) additional costs in complexity and opportunities for abuse.
The closest (in my opinion) assignment expressions comes to a One Big Win is to reduce the need for cascades of if... statements:
m = re.match(pattern, text)
if m:
...
else:
m = re.match(other_pattern, text)
if m:
...
else:
m = re.match(third_pattern, text)
if m: ...
which wastes both vertical and horizontal space.
If that were the only win, I'd consider dedicated syntax for if/elif statements alone, but it isn't. Little wins include:
use in while statement headers, "while x:= expr"
DRY inside comprehensions, as an alternative to the neat but somewhat obscure idiom:
[(x, x+1, x*2) for a in it for x in (func(a),) if x]
running totals and similar inside comprehensions, where we need a way to initialise the total on the first iteration
total = 0 [total := total + a for a in it]
and examples such as Tim's use of Heron's Formula:
area = sqrt((s := (a+b+c)/2)(s-a)(s-b)*(s-c))
This last one is, I think, the only one where the postfix form reads better, but maybe I only say that because I'm more familiar with it:
area = sqrt(s*(s-a)*(s-b)*(s-c)) where s = (a+b+c)/2
-- Steve
Nick Coghlan wrote:
That's a performance argument, not a readability one (as if you don't care about performance, you can just repeat the subexpression).
Repeated subexpressions can be a readability issue too, since you have to examine them to notice they are actually the same. They also provide an opportunity to make the error of not making them the same when they should be, and add the maintenance burden of ensuring they stay the same when changes are made.
-- Greg
On Thu, Jun 28, 2018 at 03:42:49PM -0700, Chris Barker via Python-Dev wrote:
If we think hardly anyone is ever going to do that -- then I guess it doesn't matter how it's handled.
That's how you get a language with surprising special cases, gotchas and landmines in its behaviour. (Cough PHP cough.)
It is one thing when gotchas occur because nobody thought of them, or because there is nothing you can do about them. But I do not think it is a good idea to intentionally leave gotchas lying around because "oh, I didn't think anyone would ever do that...".
wink
[...]
but here the keyword "nonlocal" is used -- you are clearly declaring that you are messing with a nonlocal name here -- that is a lot more obvious than simply using a :=
But from the point of view of somebody reading the code, there is no need for a nonlocal declaration, since the assignment is just a local assignment.
Forget the loop variable -- it is a special case where "practicality beats purity" and it makes sense to have it run in a sublocal scope. Everything else is just a local, regardless of whether it is in a comprehension or not.
And "nonlocal" is not used that often, and when it is it's for careful closure trickery -- I'm guessing := will be far more common. And, of course, when a newbie encounters it, they can google it and see what it means -- far different that seeing a := in a comprehension and understanding (by osmosis??) that it might make changes in the local scope.
I've given reasons why I believe that people will expect assignments in comprehensions to occur in the local scope. Aside from the special case of loop variables, people don't think of comprehensions as a separate scope.
There's no Comprehension Sublocal-Local-Enclosing Local-Global-Builtin scoping rule. (Do you want there to be?) Even class scope comes as an unfamiliar surprise to people.
I do not believe that people will "intuitively" expect assignments in a comprehension to disappear when the comprehension finishes -- I expect that most of the time they won't even think about it, but when they do, they'll expect it to hang around like every other use of assignment expressions.
And I don't think you can even do that with generator expressions now -- as they can only contain expressions.
It makes me cry to think of the hours I spent -- and the brownie points I lost with my wife -- showing how you can already simulate this with locals() or globals(). Did nobody read it? :-(
https://mail.python.org/pipermail/python-dev/2018-June/154114.html
Yes, you can do this right now. We just don't because playing around with locals() is a dodgy thing to do.
Maybe it's only comprehensions, and maybe it'll be rare to have a confusing version of those, so it'll be no big deal, but this thread started talking about educators' take on this -- and as an educator, I think this really does complicate the language.
See my recent post here:
https://mail.python.org/pipermail/python-dev/2018-June/154184.html
I strongly believe that the "comprehensions are local, like everything else" scenario is simpler and less surprising and easier to explain than hiding assignments inside a sublocal comprehension scope that hardly anyone even knows exists.
Especially if we end up doing it inconsistently and let variables sometimes leak.
Python got much of it's "fame" by being "executable pseudo code" -- its been moving farther and farther away from those roots. That's generally a good thing, as we've gain expressiveness in exchangel, but we shouldn't pretend it isn't happening, or that this proposal doesn't contribute to that trend.
I think there are two distinct forms of "complication" here.
Are assignment expressions in isolation complicated?
Given assignment expressions, can people write obfuscated, complex code?
Of course the answer to Q2 is yes, the opportunity will be there. Despite my general cynicism about my fellow programmers, I actually do believe that the Python community does a brilliant job of self-policing to prevent the worst excesses of obfuscatory one-liners. I don't think that will change.
So I think Q1 is the critical one. And I think the answer is, no, they're conceptually bloody simple. They evaluate the expression on the right, assign it to the name on the left, and return that value.
Here is a question and answer:
Question: after ``result = (x := 2) + 3``, what is the value of x?
Answer: 2.
Question: what if we put the assignment inside a function call?
``f((x:=2), x+3)``
Answer: still 2.
Question: how about inside a list display? ``[1, x:=2, 3]``
Answer: still 2.
Question: what about a dict display? ``{key: x:=2}`` A tuple? A set?
Answer: still 2 to all of them.
Question: how about a list comprehension?
Answer: ah, now, that's complicated, it depends on which bit of the
comprehension you put it in, the answer could be that x is 2 as you
would expect, or it could be undefined.
Yes, I can see why as an educator you would prefer that over my answer, which would be:
Answer: it's still ^&*@^!$ two, what did you think it would be???
wink
Maybe it’s just me, but re-binding a name seems like a whole new category of side effect.
With no trickery at all, you've always been able to rebind attributes, and
mutate containers, in comprehensions and genexps. Because for
targets
aren't limited to plain names; e.g.,
g = (x+y for object.attribute, a[i][j] in zip(range(3), range(3)))
sure, but you are explicitly using the names "object" and "a" here -- so while side effects in comprehension are discouraged, it's not really a surprised that namespaces specifically named are changed.
Trust me on this Chris, assuming the arguments over this PEP are finished by then (I give 50:50 odds wink) by the time 3.8 comes out you'll be saying
"sure, but you are explicitly assigning to a local variable
with the := operator, it's not really a surprise that the
local variable specifically named is being changed, it would
be weird if it wasn't..."
and you'll have forgotten that there was ever a time we seriously discussed comprehension-scope as a thing.
:-)
[example with the loop variable of a comprehension]
doesn't change x in the local scope -- if that was a good idea, why is a good idea to have := in a comprehension effect the local scope??
But maybe it is just me.
Some of us don't think that it was a good idea wink
But I know when I'm outvoted and when "practicality beats purity".
Loop variables are semantically special. They tend to have short, often one-character names, taken from a small set of common examples (i, j, x are especially common). We don't tend to think of them as quite the same as ordinary variables: once the loop is complete, we typically ignore the loop variable.
It takes a conscious effort to remember that it actually still hangs around. I've written code like this:
for x in sequence:
last_x = x
do_stuff()
# outside the loop
process(last_x)
and then looked at it a day later and figuratively kicked myself. What the hell was I thinking?
So it is hardly surprising that people would sometimes write:
for x in sequence:
L = [expression for x in something]
process(x)
and be unpleasantly surprised in Python 2.
But this is not likely to happen BY ACCIDENT with assignment expressions, and if it does, well, the answer is, change the variable name to something a little less generic, or at least different:
for x in sequence:
L = [x := a+1 for a in something]
process(x)
I am reluctantly forced to agree that, purity be damned, burying the loop variable inside a hidden implicit function scope is the right thing to do, even if it is sometimes annoying. But the reasons for doing do don't apply to explicit assignment expressions.
We have no obligation to protect people from every accidental name clobbering caused by carelessness and poor choice of names.
py> import math as module, string as module, sys as module py> module
<module 'sys' (built-in)>
I suppose we need to go back and look at the "real" examples of where/how folks think they'll use := in comprehensions, and see how confusing it may be.
One the simplest but most useful examples is as a debugging aide.
[function(x) for x in whatever]
crashes part of the way through and raises an exception. How to debug? If it were a for-loop, one simple solution would be to wrap it in a try...except and then print the last seen value of x.
try:
for x in whatever:
function(x)
except:
print(x)
Can't do that with a list comprehension, not in Python 3. (It works in Python 2.) Assignment expressions to the rescue!
try:
[function(a := x) for x in whatever]
except:
print(a)
That's simple and lightweight enough that with a bit of polishing, you could even leave it in production code to log a hard-to-track down error.
Oh, and it doesn't easily translate to a map(), since lambda functions do execute in their own scope:
map(lambda x: function(a := x), whatever)
Another reason to prefer comprehensions to map.
One of these conversations was started with an example something like this:
[(f(x), g(f(x))) for x in an_iterable]
The OP didn't like having to call f() twice. So that would become:
[ (temp:=f(x), g(temp)) for x in an_iterable]
so now the question is: should "temp" be created / changed in the enclosing local scope?
The comprehension here is a red herring.
result = (temp := f(x), g(temp))
Should temp be a local, or should tuple displays exist in their own, special, sublocal scope?
In either case, there's no specific advantage to letting temp "leak" (scare quotes) out of the expression. But there's no disadvantage either, and if we pick a less prejudicial name, it might actually be advantagous:
result = (useful_data := f(x), g(useful_data))
process(useful_data)
Using a comprehension isn't special enough to change the rule that assignment expressions are local (unless declared global).
-- Steve
On 30 June 2018 at 19:35, Steven D'Aprano steve@pearwood.info wrote:
So I think Q1 is the critical one. And I think the answer is, no, they're conceptually bloody simple. They evaluate the expression on the right, assign it to the name on the left, and return that value.
And the proposed parent local scoping in PEP 572 has the virtue of making all of the following do exactly the same thing, just as they would in the version without the assignment expression:
ref = object()
container = [item := ref]
container = [x for x in [item := ref]]
container = [x for x in [item := ref] for i in range(1)]
container = list(x for x in [item := ref])
container = list(x for x in [item := ref] for i in range(1))
# All variants pass this assertion, keeping the implicit sublocal
scope almost entirely hidden assert container == [ref] and item is ref and item is container[0]
My own objections were primarily based on the code-generation-centric concept of wanting Python's statement level scoping semantics to continue to be a superset of its expression level semantics, and Guido's offer to define "__parentlocal" in the PEP as a conventional shorthand for describing comprehension style assignment expression scoping when writing out their statement level equivalents as pseudo-code addresses that. (To put it another way: it turned out it wasn't really the semantics themselves that bothered me, since they have a lot of very attractive properties as shown above, it was the lack of a clear way of referring to those semantics other than "the way assignment expressions behave in implicit comprehension and generator expression scopes").
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6/30/2018 5:35 AM, Steven D'Aprano wrote:
I've given reasons why I believe that people will expect assignments in comprehensions to occur in the local scope. Aside from the special case of loop variables, people don't think of comprehensions as a separate scope.
I think this is because comprehensions other than generator expressions were originally defined in terms of equivalent code in the same local scope, are still easily thought of in those terms, and, as I explained in my response to Guido, could, at least in simple cases, still be implemented in the local scope, so that assignment expressions would be executed and affect the expected local scope without 'nonlocal'.
Generator expressions, on the other hand, have always been defined in terms of equivalent code in a nested scope, and must be implemented that way, so some re-definition and re-implementation is needed for assignment expressions to affect the local scope in which the g.e is defined and for that effect to be comprehensible. It might be enough to add something like "any names that are targets of assignment expressions are added to global or nonlocal declarations within the implementation generator function."
If the equality [generator-expression] == list(generator-expression] is preserved, then it could serve as the definition of the list comprehension. The same could be true for set and dict comprehensions, with the understanding that the equality is modified to {a:b for ...} == dict((a,b) for ...). It should also be mentioned that the defining equality is not necessarily the implementation.
-- Terry Jan Reedy
On Wed, Jun 27, 2018 at 09:52:43PM -0700, Chris Barker wrote:
It seems everyone agrees that scoping rules should be the same for generator expressions and comprehensions,
Yes. I dislike saying "comprehensions and generator expressions" over and over again, so I just say "comprehensions".
Principle One:
Principle Two:
Principle Three:
glossing over the builtin name look-up, calling list(genexpr) will remain equivalent to using a list comprehension;
similarly for set and dict comprehensions.
Principle Four:
So far, there should be (I hope!) no disagreement with those first four principles. With those four principles in place, teaching and using comprehensions (genexprs) in the absense of assignment expressions does not need to change one iota.
Normal cases stay normal; weird cases mucking about with locals() inside the comprehension are already weird and won't change.
So what about:
l = [x:=i for i in range(3)]
vs
g = (x:=i for i in range(3))
Is there any way to keep these consistent if the "x" is in the regular local scope?
Yes. That is what closures already do.
We already have such nonlocal effects in Python 3. Move the loop inside an inner (nested) function, and then either call it immediately to simulate the effect of a list comprehension, or delay calling it to behave more like a generator expression.
Of course the runtime effects depend on whether or not the generator expression is actually evaluated. But that's no mystery, and is precisely analogous to this case:
def demo(): x = 1 def inner(): nonlocal x x = 99 inner() # call the inner function print(x)
This prints 99. But if you comment out the call to the inner function, it prints 1. I trust that doesn't come as a surprise.
Nor should this come as a surprise:
def demo(): x = 1
# assuming assignment scope is local rather than sublocal
g = (x:= i for i in (99,))
L = list(g)
print(x)
The value of x printed will depend on whether or not you comment out the call to list(g).
Note that this thread is titled "Informal educator feedback on PEP 572".
As an educator -- this is looking harder an harder to explain to newbies...
Though easier if any assignments made in a "comprehension" don't "leak out".
Let me introduce two more principles.
Principle Five:
Principle Six:
Five is, I think, so intuitive that we forget about it in the same way that we forget about the air we breathe. It would be surprising, even shocking, if two expressions in the same context were executed in different scopes:
result = [x + 1, x - 2]
If the first x were local and the second was global, that would be disturbing. The same rule ought to apply if we include assignment expressions:
result = [(x := expr) + 1, x := x - 2]
It would be disturbing if the first assignment (x := expr) executed in the local scope, and the second (x := x - 2) failed with NameError because it was executed in the global scope.
Or worse, didn't fail with NameError, but instead returned something totally unexpected.
Now bring in a comprehension:
result = [(x := expr) + 1] + [x := x - 2 for a in (None,)]
Do you still want the x inside the comprehension to be a different x to the one outside the comprehension? How are you going to explain that UnboundLocalError to your students?
That's not actually a rhetorical question. I recognise that while Principle Five seems self-evidently desirable to me, you might consider it less important than the idea that "assignments inside comprehensions shouldn't leak".
I believe that these two expressions should give the same results even to the side-effects:
[(x := expr) + 1, x := x - 2]
[(x := expr) + 1] + [x := x - 2 for a in (None,)]
I think that is the simplest and most intuitive behaviour, the one which will be the least surprising, cause the fewest unexpected NameErrors, and be the simplest to explain.
If you still prefer the "assignments shouldn't leak" idea, consider this: under the current implementation of comprehensions as an implicit hidden function, the scope of a variable depends on where it is, violating Principle Six.
(That was the point of my introducing locals() into a previous post: to demonstrate that, today, right now, "comprehension scope" is a misnomer. Comprehensions actually execute in a hybrid of at least two scopes, the surrounding local scope and the sublocal hidden implicit function scope.)
Let me bring in another equivalency:
[(x := expr) + 1, x := x - 2]
[(x := expr) + 1] + [x := x - 2 for a in (None,)]
[(x := expr) + 1] + [a for a in (x := x - 2,)]
By Principle Six, the side-effect of assigning to x shouldn't depend on where inside the comprehension it is. The two comprehension expressions shown ought to be referring to the same "x" variable (in the same scope) regardless of whether that is the surrounding local scope, or a sublocal comprehension scope.
(In the case of it being a sublocal scope, the two comprehensions will raise UnboundLocalError.)
But -- and this is why I raised all that hoo-ha about locals() -- according to the current implementation, they don't. This version would assign to x in the sublocal scope:
# best viewed in a monospaced font
[x := x - 2 for a in (None,)]
^^^^^^^^^^ this is sublocal scope
but this would assign in the surrounding local scope:
[a for a in (x := x - 2,)]
^^^^^^^^^^^^^ this is local scope
I strongly believe that all three ought to be equivalent, including side-effects. (Remember that by Principle Two, we agree that the loop variable doesn't leak. The loop variable is invisible from the outside and doesn't count as a side-effect for this discussion.)
So here are three possibilities (assuming assignment expressions are permitted):
This torpedoes Princple Six, and leaves you having to explain why assignment sometimes "works" inside comprehensions and sometimes gives UnboundLocalError.
That rules out some interesting (but probably not critical) uses of assignment expressions inside comprehensions, such as using them as a side-channel to sneak out debugging information.
And it adds a great big screaming special case to Principle Five:
Number 3 is my strong preference. It complicates the implementation a bit (needing to add some implicit nonlocals) but not as much as needing to hide the otherwise-local scope beneath another implicit function. And it gives by far the most consistent, useful and obvious semantics out of the three options.
My not-very-extensive survey on the Python-List mailing lists suggests that, if you don't ask people explicitly about "assignment expressions", they already think of the inside of comprehensions as being part of the surrounding local scope rather than a hidden inner function. So I do not believe that this will be hard to teach.
These two expressions ought to give the same result with the same side-effect:
[x := 1]
[x := a for a in (1,)]
That, I strongly believe, is the inuitive behaviour to peope who aren't immersed in the implementation details of comprehensions, as well as being the most useful.
-- Steve