PEP 572: Statement-Local Name Bindings
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement. The nicely-rendered version will be visible here shortly: https://www.python.org/dev/peps/pep-0572/ ChrisA PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018 Abstract ======== Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions. Rationale ========= When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement. Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides. Example usage ============= These list comprehensions are all approximately equivalent:: # Calling the function twice stuff = [[f(x), f(x)] for x in range(5)] # Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)] # Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)] # Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]] # Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y]) # Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)] If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning. Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option. Open questions ============== 1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning? 2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing? 3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear? 4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters? 5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
I hope nobody will mind too much if I throw in my (relatively uninformed) 2c before some of the big guns respond. First: Well done, Chris, for all the work on this. IMHO this could be a useful Python enhancement (and reduce the newsgroup churn :-)). On 27/02/2018 22:27, Chris Angelico wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
https://www.python.org/dev/peps/pep-0572/
ChrisA
PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018
Abstract ========
Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions. It may be pedantic of me (and it *will* produce a more pedantic-sounding sentence) but I honestly think that "safe against name collisions" is clearer than "safe against
collisions", and that clarity matters. > > > Rationale > ========= > > When an expression is used multiple times in a list comprehension, there > are currently several suboptimal ways to spell this, and no truly good > ways. A statement-local name allows any expression to be temporarily > captured and then used multiple times. IMHO the first sentence is a bit of an overstatement (though of course it's a big part of the PEP's "sell"). How about "... there are currently several ways to spell this, none of them ideal."
Also, given that a list comprehension is an expression, which in turn could be part of a larger expression, would it be appropriate to replace "expression" by "sub-expression" in the 2 places where it occurs in the above paragraph?
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity,
I agree, pro tem (not that I am claiming that my opinion counts for much). I'm personally somewhat allergic to making parentheses mandatory where they really don't need to be, but trying to think about where they could be unambiguously omitted makes my head spin. At least, if we run with this for now, then making them non-mandatory in some contexts, at some future time, won't lead to backwards incompatibility.
and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name.
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement.
Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides.
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice # Calling the function twice (assuming that side effects can be ignored) stuff = [[f(x), f(x)] for x in range(5)]
# Helper function # External helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y]) Please feel free to ignore this, but (trying to improve on the above example): # Using a generator: def gen(): for x in range(5): y = f(x) yield y,y stuff = list(gen())
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning. Maybe add to last sentence "and of adding (at least conceptually) extra steps: building a 1-element list, then extracting the first element"
Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning? IMHO this is not only acceptable, but the (only) correct behaviour. Your crystal-clear statement "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement " is *critical* and IMO what should happen. Perhaps an extra example or two, to clarify that *execution order* is what matters, might help, e.g. y if (f() as y) > 0 else None will work as expected, because "(f() as y)" is evaluated before the initial "y" is (if it is).
[Parenthetical comment: Advanced use of this new feature would require knowledge of Python's evaluation order. But this is not an argument against the PEP, because the same could be said about almost any feature of Python, e.g. [ f(x), f(x) ] where evaluating f(x) has side effects.] 2. The current implementation [1] implements statement-local names using
a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
It's great that it works perfectly in functions and list comprehensions, but it sounds as if, at top level, in rare circumstances it could produce a hard-to-track-down bug, which is not exactly desirable. It's hard to say more without knowing more details. As a stab in the dark, is it possible to avoid it by including the module name in the mangling? Sorry if I'm talking rubbish.
3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear?
IMHO this is an implementation detail. IMO you should have some idea what you're doing when you use locals(). But I think consistency matters - either the temporary variable *always* gets into locals() "from the point where it is evaluated until the end of the statement", or it *never* gets into locals(). (Possibly the language spec should specify one or the other - I'm not sure, time may tell.)
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion.
This (4. and 5.) shows that we are using "as" in more than one sense, and in a perfect world we would use different keywords. But IMHO (admittedly, without having thought about it much) this isn't much of a problem. Again, perhaps some clarifying examples would help. Some pedantry: One issue not so far explicitly mentioned: IMHO it should be perfectly legal to assign a value to a temporary variable, and then not use that temporary variable (just as it is legal to assign to a variable in a regular assignment statement, and then not use that variable) though linters should IMO point it out. E.g. you might want to modify (perhaps only temporarily) a = [ (f() as b), b ] to a = [ (f() as b), c ] Also (and I'm relying on "In any context where arbitrary Python expressions can be used, a named expression can appear." ), linters should also IMO point to a = (42 as b) which AFAICT is a laborious synonym for a = 42 And here's a thought: What are the semantics of a = (42 as a) # Of course a linter should point this out too At first I thought this was also a laborious synonym for "a=42". But then I re-read your statement (the one I described above as crystal-clear) and realised that its exact wording was even more critical than I had thought: "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement" Note: "until the end of the *statement*". NOT "until the end of the *expression*". The distinction matters. If we take this as gospel, all this will do is create a temporary variable "a", assign the value 42 to it twice, then discard it. I.e. it effectively does nothing, slowly. Have I understood correctly? Very likely you have considered this and mean exactly what you say, but I am sure you will understand that I mean no offence by querying it. Best wishes Rob Cliffe
On Wed, Feb 28, 2018 at 2:47 PM, Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:
I hope nobody will mind too much if I throw in my (relatively uninformed) 2c before some of the big guns respond.
Not at all! Everyone's contributions are welcomed. Even after the "big guns" respond, other voices are definitely worth hearing. (One small tip though: Responding in plain text is appreciated, as it means information about who said what is entirely copy-and-pasteable.)
First: Well done, Chris, for all the work on this. IMHO this could be a useful Python enhancement (and reduce the newsgroup churn :-)).
Thanks :) It's one of those PEPs that can be immensely useful even if it's rejected.
On 27/02/2018 22:27, Chris Angelico wrote: Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions.
It may be pedantic of me (and it will produce a more pedantic-sounding sentence) but I honestly think that "safe against name collisions" is clearer than "safe against collisions", and that clarity matters.
Sure. I'm also aware that I'm using the same words over and over, but I can add that one.
Rationale =========
When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times.
IMHO the first sentence is a bit of an overstatement (though of course it's a big part of the PEP's "sell"). How about "... there are currently several ways to spell this, none of them ideal."
Hmm, I think I prefer the current wording, but maybe there's some other way to say it that's even better.
Also, given that a list comprehension is an expression, which in turn could be part of a larger expression, would it be appropriate to replace "expression" by "sub-expression" in the 2 places where it occurs in the above paragraph?
Thanks, done.
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity,
I agree, pro tem (not that I am claiming that my opinion counts for much). I'm personally somewhat allergic to making parentheses mandatory where they really don't need to be, but trying to think about where they could be unambiguously omitted makes my head spin. At least, if we run with this for now, then making them non-mandatory in some contexts, at some future time, won't lead to backwards incompatibility.
Yeah, definitely. There've been other times when a new piece of syntax is extra restrictive at first, and then gets opened up later. It's way easier than the alternative. (For the record, I had some trouble with this syntax at first, and was almost going to launch this PEP with a syntax of "( > expr as NAME)" to disambiguate. That was never the intention, though, and I'm grateful to the folks on core-mentorship for helping me get that sorted.)
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice
# Calling the function twice (assuming that side effects can be ignored)
That assumption should be roughly inherent in the problem. If the call has no side effects and low cost, none of this is necessary - just repeat the expression.
stuff = [[f(x), f(x)] for x in range(5)]
# Helper function
# External helper function
Not a big deal either way, can toss in the extra word but I'm not really sure it's needed.
Please feel free to ignore this, but (trying to improve on the above example): # Using a generator: def gen(): for x in range(5): y = f(x) yield y,y stuff = list(gen())
I think it's unnecessary; the direct loop is entirely better IMO. Since the point of these examples is just to contrast against the proposal, it's no biggie if there are EVEN MORE ways (and I haven't even mentioned the steak knives!) to do something, unless they're actually better.
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
Maybe add to last sentence "and of adding (at least conceptually) extra steps: building a 1-element list, then extracting the first element"
That's precisely the point that Serhiy's optimization is aiming at, with the intention of making "for x in [expr]" a standard idiom for list comp assignment. If we assume that this does become standard, it won't add the extra steps, but it does still push that expression out to the far end of the comprehension, whereas a named subexpression places it at first use.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
IMHO this is not only acceptable, but the (only) correct behaviour. Your crystal-clear statement "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement " is critical and IMO what should happen.
Regular function-locals don't work that way, though: x = "global" def f(): print(x) x = "local" print(x) This won't print "global" followed by "local" - it'll bomb with UnboundLocalError. I do still think this is correct behaviour, though; the only other viable option is for the SLNB to fail if it's shadowing anything at all, and even that has its weird edge cases.
Perhaps an extra example or two, to clarify that *execution order* is what matters, might help, e.g. y if (f() as y) > 0 else None will work as expected, because "(f() as y)" is evaluated before the initial "y" is (if it is).
Unnecessary in the "open questions" section, but if this proves to be a point of confusion and I make a FAQ, then yeah, I could put in some examples like that.
[Parenthetical comment: Advanced use of this new feature would require knowledge of Python's evaluation order. But this is not an argument against the PEP, because the same could be said about almost any feature of Python, e.g. [ f(x), f(x) ] where evaluating f(x) has side effects.]
Yeah, people should have no problem figuring this out.
2. The current implementation [1] implements statement-local names using
a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
It's great that it works perfectly in functions and list comprehensions, but it sounds as if, at top level, in rare circumstances it could produce a hard-to-track-down bug, which is not exactly desirable. It's hard to say more without knowing more details. As a stab in the dark, is it possible to avoid it by including the module name in the mangling? Sorry if I'm talking rubbish.
The problem is that it's all done through the special "cell" slots in a function's locals. To try to do that at module level would potentially mean polluting the global namespace, which could interfere with other functions and cause extreme confusion. Currently, attempting to use an SLNB at top level produces a bizarre UnboundLocalError, and I don't truly understand why. The disassembly shows the same name mangling that happens inside a function, but it doesn't get properly undone. But I'm sure there are many other implementation bugs too.
3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear?
IMHO this is an implementation detail. IMO you should have some idea what you're doing when you use locals(). But I think consistency matters - either the temporary variable *always* gets into locals() "from the point where it is evaluated until the end of the statement", or it *never* gets into locals(). (Possibly the language spec should specify one or the other - I'm not sure, time may tell.)
Yeah, and I would prefer the former, but that's still potentially confusing. Consider: y = "gg" def g(): x = 1 print(x, locals()) print((3 as x), x, locals()) print(y, (4 as y), y, locals()) print(x, locals()) del x print(locals()) Current output: 1 {} 3 3 {'x': 3} gg 4 4 {'y': 4} 1 {} {} Desired output: 1 {'x': 1} 3 3 {'x': 3} gg 4 4 {'x': 1, 'y': 4} 1 {'x': 1} {} Also acceptable (but depreferred) output: 1 {'x': 1} 3 3 {'x': 1} gg 4 4 {'x': 1} 1 {'x': 1} {} If the language spec mandates that "either this or that" happen, I'd be okay with that; it'd give other Pythons the option to implement this completely outside of locals() while still being broadly sane.
4. Syntactic confusion in `except` statements. 5. Similar confusion in `with` statements
This (4. and 5.) shows that we are using "as" in more than one sense, and in a perfect world we would use different keywords. But IMHO (admittedly, without having thought about it much) this isn't much of a problem. Again, perhaps some clarifying examples would help.
No, we want to keep using the same keywords - otherwise there are too many keywords in the language. The "except" case isn't a big deal IMO, but the "with" one is more serious, and the subtle difference between "with (x as y):" and "with x as y:" is sure to trip someone up. But maybe that's one for linters and code review.
Some pedantry:
One issue not so far explicitly mentioned: IMHO it should be perfectly legal to assign a value to a temporary variable, and then not use that temporary variable (just as it is legal to assign to a variable in a regular assignment statement, and then not use that variable) though linters should IMO point it out. E.g. you might want to modify (perhaps only temporarily) a = [ (f() as b), b ] to a = [ (f() as b), c ]
Yep, perfectly legal. Once linters learn that this is an assignment, they can flag this as "unused variable". Otherwise, it's not really hurting much.
Also (and I'm relying on "In any context where arbitrary Python expressions can be used, a named expression can appear." ), linters should also IMO point to a = (42 as b) which AFAICT is a laborious synonym for a = 42
Ditto - an unused variable. You could also write "a = b = 42" and then never use b.
And here's a thought: What are the semantics of a = (42 as a) # Of course a linter should point this out too At first I thought this was also a laborious synonym for "a=42". But then I re-read your statement (the one I described above as crystal-clear) and realised that its exact wording was even more critical than I had thought: "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement" Note: "until the end of the statement". NOT "until the end of the expression". The distinction matters. If we take this as gospel, all this will do is create a temporary variable "a", assign the value 42 to it twice, then discard it. I.e. it effectively does nothing, slowly. Have I understood correctly? Very likely you have considered this and mean exactly what you say, but I am sure you will understand that I mean no offence by querying it.
Actually, that's a very good point, and I had to actually go and do that to confirm. You're correct that the "a =" part is also affected, but there may be more complicated edge cases. Disassembly can help track down what the compiler's actually doing:
def f(): ... a = 1 ... a = (2 as a) ... print(a) ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 2 STORE_FAST 0 (a)
3 4 LOAD_CONST 2 (2) 6 DUP_TOP 8 STORE_FAST 1 (a) 10 STORE_FAST 1 (a) 12 DELETE_FAST 1 (a) 4 14 LOAD_GLOBAL 0 (print) 16 LOAD_FAST 0 (a) 18 CALL_FUNCTION 1 20 POP_TOP 22 LOAD_CONST 0 (None) 24 RETURN_VALUE If you're not familiar with the output of dis.dis(), the first column (largely blank) is line numbers in the source, the second is byte code offsets, and then we have the operation and its parameter (if any). The STORE_FAST and LOAD_FAST opcodes work with local names, which are identified by their indices; the first such operation sets slot 0 (named "a"), but the two that happen in line 3 (byte positions 8 and 10) are manipulating slot 1 (also named "a"). So you can see that line 3 never touches slot 0, and it is entirely operating within the SLNB scope. Identical byte code is produced from this function:
def f(): ... a = 1 ... b = (2 as b) ... print(a) ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 2 STORE_FAST 0 (a)
3 4 LOAD_CONST 2 (2) 6 DUP_TOP 8 STORE_FAST 1 (b) 10 STORE_FAST 1 (b) 12 DELETE_FAST 1 (b) 4 14 LOAD_GLOBAL 0 (print) 16 LOAD_FAST 0 (a) 18 CALL_FUNCTION 1 20 POP_TOP 22 LOAD_CONST 0 (None) 24 RETURN_VALUE I love dis.dis(), it's such an awesome tool :) I'll push PEP changes based on your suggestions shortly. Am also going to add a "performance considerations" section, as features like this are potentially costly. Thanks for your input! ChrisA
On 02/27/2018 09:23 PM, Chris Angelico wrote:
On Wed, Feb 28, 2018 at 2:47 PM, Rob Cliffe via Python-ideas wrote:
And here's a thought: What are the semantics of a = (42 as a) # Of course a linter should point this out too At first I thought this was also a laborious synonym for "a=42". But then I re-read your statement (the one I described above as crystal-clear) and realised that its exact wording was even more critical than I had thought: "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement" Note: "until the end of the statement". NOT "until the end of the expression". The distinction matters. If we take this as gospel, all this will do is create a temporary variable "a", assign the value 42 to it twice, then discard it. I.e. it effectively does nothing, slowly. Have I understood correctly? Very likely you have considered this and mean exactly what you say, but I am sure you will understand that I mean no offence by querying it.
Actually, that's a very good point, and I had to actually go and do that to confirm. You're correct that the "a =" part is also affected, but there may be more complicated edge cases. Disassembly can help track down what the compiler's actually doing:
def f(): ... a = 1 ... a = (2 as a) ... print(a) ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 2 STORE_FAST 0 (a)
3 4 LOAD_CONST 2 (2) 6 DUP_TOP 8 STORE_FAST 1 (a) 10 STORE_FAST 1 (a) 12 DELETE_FAST 1 (a)
4 14 LOAD_GLOBAL 0 (print) 16 LOAD_FAST 0 (a) 18 CALL_FUNCTION 1 20 POP_TOP 22 LOAD_CONST 0 (None) 24 RETURN_VALUE
If you're not familiar with the output of dis.dis(), the first column (largely blank) is line numbers in the source, the second is byte code offsets, and then we have the operation and its parameter (if any). The STORE_FAST and LOAD_FAST opcodes work with local names, which are identified by their indices; the first such operation sets slot 0 (named "a"), but the two that happen in line 3 (byte positions 8 and 10) are manipulating slot 1 (also named "a"). So you can see that line 3 never touches slot 0, and it is entirely operating within the SLNB scope.
dis.dis may be great, but so is running the function so everyone can see the output. ;) If I understood your explanation, `print(a)` produces `1` ? That seems wrong -- the point of statement-local name bindings is twofold: - give a name to a value - evaluate to that value Which is why your first example works: stuff = [[(f(x) as y), y] for x in range(5)] (f(x) as y), y evaluates as f(x), and also assigns that result to y, so in a = (2 as a) there is a temporary variable 'a', which gets assigned 2, and the SLNB is evaluated as 2, which should then get assigned back to the local variable 'a'. In other words, the final print from `f()` above should be 2, not 1. (Slightly different names would help avoid confusion when referencing different locations of the PEP.) -- ~Ethan~
On Thu, Mar 1, 2018 at 2:10 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
dis.dis may be great, but so is running the function so everyone can see the output. ;)
Oh, sorry.
f() 1
If I understood your explanation, `print(a)` produces `1` ? That seems wrong -- the point of statement-local name bindings is twofold:
- give a name to a value - evaluate to that value
Which is why your first example works:
stuff = [[(f(x) as y), y] for x in range(5)]
(f(x) as y), y
evaluates as f(x), and also assigns that result to y, so in
a = (2 as a)
there is a temporary variable 'a', which gets assigned 2, and the SLNB is evaluated as 2, which should then get assigned back to the local variable 'a'. In other words, the final print from `f()` above should be 2, not 1. (Slightly different names would help avoid confusion when referencing different locations of the PEP.)
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary. Honestly, though, it's like writing "a = a++" in C, and then being confused by the result. Why are you using the same name in two assignments? Normal code shouldn't do this. :) ChrisA
On 28 February 2018 at 15:18, Chris Angelico <rosuav@gmail.com> wrote:
a = (2 as a)
there is a temporary variable 'a', which gets assigned 2, and the SLNB is evaluated as 2, which should then get assigned back to the local variable 'a'. In other words, the final print from `f()` above should be 2, not 1. (Slightly different names would help avoid confusion when referencing different locations of the PEP.)
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Honestly, though, it's like writing "a = a++" in C, and then being confused by the result. Why are you using the same name in two assignments? Normal code shouldn't do this. :)
Eww. I can understand the logic here, but this sort of weird gotcha is precisely why people dislike C/C++ and prefer Python. I don't consider it a selling point that this proposal allows Python coders to make the sort of mistakes C coders have suffered from for years. Can you make sure that the PEP includes a section that covers weird behaviours like this as problems with the proposal? I'm happy if you just list them, or even say "while this is a potential issue, the author doesn't think it's a major problem". I just don't think it should be forgotten. Paul
On Thu, Mar 1, 2018 at 2:46 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 February 2018 at 15:18, Chris Angelico <rosuav@gmail.com> wrote:
a = (2 as a)
there is a temporary variable 'a', which gets assigned 2, and the SLNB is evaluated as 2, which should then get assigned back to the local variable 'a'. In other words, the final print from `f()` above should be 2, not 1. (Slightly different names would help avoid confusion when referencing different locations of the PEP.)
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Honestly, though, it's like writing "a = a++" in C, and then being confused by the result. Why are you using the same name in two assignments? Normal code shouldn't do this. :)
Eww. I can understand the logic here, but this sort of weird gotcha is precisely why people dislike C/C++ and prefer Python. I don't consider it a selling point that this proposal allows Python coders to make the sort of mistakes C coders have suffered from for years.
Can you make sure that the PEP includes a section that covers weird behaviours like this as problems with the proposal? I'm happy if you just list them, or even say "while this is a potential issue, the author doesn't think it's a major problem". I just don't think it should be forgotten.
Sure. Ultimately, it's like any other feature: it can be abused in ways that make no sense. You can write a list comprehension where you ignore the end result and work entirely with side effects; you can write threaded code that spawns threads and immediately joins them all; nobody's stopping you. In a non-toy example, assigning to the same name twice in one statement is almost certainly an error for other reasons, so I'm not too bothered by it here. I'll add something to the PEP about execution order. ChrisA
On 2018-02-28 07:18, Chris Angelico wrote:
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Wait, so you're saying that if I do a = (2 as a) The "a = " assignment assigns to the SLNB, and so is then discarded after the statement finishes? That seems very bad to me. If there are SLNBs with this special "as" syntax, I think the ONLY way to assign to an SLNB should be with the "as" syntax. You shouldn't be able to assign to an SLNB with regular assignment syntax, even if you created an SNLB with the same name as the LHS within the RHS. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Thu, Mar 1, 2018 at 6:35 AM, Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2018-02-28 07:18, Chris Angelico wrote:
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Wait, so you're saying that if I do
a = (2 as a)
The "a = " assignment assigns to the SLNB, and so is then discarded after the statement finishes?
That seems very bad to me. If there are SLNBs with this special "as" syntax, I think the ONLY way to assign to an SLNB should be with the "as" syntax. You shouldn't be able to assign to an SLNB with regular assignment syntax, even if you created an SNLB with the same name as the LHS within the RHS.
That seems a reasonable requirement on the face of it, but what about these variants? a = (x as a) a[b] = (x as a) b[a] = (x as a) a[b].c = (x as a) b[a].c = (x as a) Which of these should use the SLNB, which should be errors, which should use the previously-visible binding of 'a'? It wouldn't be too hard to put in a trap for assignment per se, but where do you draw the line? I think "a[b] =" is just as problematic as "a =", but "b[a] =" could be useful. Maybe the rule could be that direct assignment or mutation is disallowed, but using that value to assign to something else isn't? That would permit the last three and disallow only the first two. ChrisA
On 1 March 2018 at 06:00, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Mar 1, 2018 at 6:35 AM, Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2018-02-28 07:18, Chris Angelico wrote:
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Wait, so you're saying that if I do
a = (2 as a)
The "a = " assignment assigns to the SLNB, and so is then discarded after the statement finishes?
That seems very bad to me. If there are SLNBs with this special "as" syntax, I think the ONLY way to assign to an SLNB should be with the "as" syntax. You shouldn't be able to assign to an SLNB with regular assignment syntax, even if you created an SNLB with the same name as the LHS within the RHS.
That seems a reasonable requirement on the face of it, but what about these variants?
a = (x as a) a[b] = (x as a) b[a] = (x as a) a[b].c = (x as a) b[a].c = (x as a)
Which of these should use the SLNB, which should be errors, which should use the previously-visible binding of 'a'?
This is the kind of ambiguity of intent that goes away if statement locals are made syntactically distinct in addition to being semantically distinct: .a = (2 as .a) # Syntax error (persistent bindings can't target statement locals) a = (2 as .a) # Binds both ".a" (ephemerally) and "a" (persistently) to "2" .a[b] = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a] = (x as .a) # LHS references .a .a[b].c = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a].c = (x as .a) # LHS references .a We may still decide that even the syntactically distinct variant poses a net loss to overall readability, but I do think it avoids many of the confusability problems that arise when statement locals use the same reference syntax as regular variable names. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Mar 1, 2018 at 3:54 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 1 March 2018 at 06:00, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Mar 1, 2018 at 6:35 AM, Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2018-02-28 07:18, Chris Angelico wrote:
Except that assignment is evaluated RHS before LHS as part of a single statement. When Python goes to look up the name "a" to store it (as the final step of the assignment), the SLNB is still active (it's still the same statement - note that this is NOT expression-local), so it uses the temporary.
Wait, so you're saying that if I do
a = (2 as a)
The "a = " assignment assigns to the SLNB, and so is then discarded after the statement finishes?
That seems very bad to me. If there are SLNBs with this special "as" syntax, I think the ONLY way to assign to an SLNB should be with the "as" syntax. You shouldn't be able to assign to an SLNB with regular assignment syntax, even if you created an SNLB with the same name as the LHS within the RHS.
That seems a reasonable requirement on the face of it, but what about these variants?
a = (x as a) a[b] = (x as a) b[a] = (x as a) a[b].c = (x as a) b[a].c = (x as a)
Which of these should use the SLNB, which should be errors, which should use the previously-visible binding of 'a'?
This is the kind of ambiguity of intent that goes away if statement locals are made syntactically distinct in addition to being semantically distinct:
.a = (2 as .a) # Syntax error (persistent bindings can't target statement locals) a = (2 as .a) # Binds both ".a" (ephemerally) and "a" (persistently) to "2" .a[b] = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a] = (x as .a) # LHS references .a .a[b].c = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a].c = (x as .a) # LHS references .a
We may still decide that even the syntactically distinct variant poses a net loss to overall readability, but I do think it avoids many of the confusability problems that arise when statement locals use the same reference syntax as regular variable names.
Okay. I think I have the solution, then. One of two options: 1) SLNBs are not permitted as assignment (incl augassign) targets. Doing so is a SyntaxError. 2) SLNBs are ignored when compiling assignment targets. Doing so will bind to the "real" name. Using an SLNB to subscript another object is perfectly acceptable, as that's simply referencing. The only case that might slip between the cracks is "a[b].c" which technically is looking up a[b] and only assigning to *that* object (for instance, if 'a' is a tuple and 'b' is zero, it's perfectly legal to write "a[b].c = 1" even though tuples are immutable). Other than that, the intention given in all your examples would be sustained. Which of the two is preferable? I'm thinking of going with option 2, but there are arguments on both sides. ChrisA
On 01/03/2018 06:47, Chris Angelico wrote:
This is the kind of ambiguity of intent that goes away if statement locals are made syntactically distinct in addition to being semantically distinct:
.a = (2 as .a) # Syntax error (persistent bindings can't target statement locals) a = (2 as .a) # Binds both ".a" (ephemerally) and "a" (persistently) to "2" .a[b] = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a] = (x as .a) # LHS references .a .a[b].c = (x as .a) # Syntax error (persistent bindings can't target statement locals) b[.a].c = (x as .a) # LHS references .a
-1. Too much grit! And I think trying to put the dots in the right places would be a frequent source of mistakes (albeit mistakes that could usually be corrected quickly).
Okay. I think I have the solution, then. One of two options:
1) SLNBs are not permitted as assignment (incl augassign) targets. Doing so is a SyntaxError. 2) SLNBs are ignored when compiling assignment targets. Doing so will bind to the "real" name.
Using an SLNB to subscript another object is perfectly acceptable, as that's simply referencing. The only case that might slip between the cracks is "a[b].c" which technically is looking up a[b] and only assigning to *that* object (for instance, if 'a' is a tuple and 'b' is zero, it's perfectly legal to write "a[b].c = 1" even though tuples are immutable). Other than that, the intention given in all your examples would be sustained.
Which of the two is preferable? I'm thinking of going with option 2, but there are arguments on both sides.
+1 to one of these two options. +1 to choosing 2). I think it's easier to understand and explain "temporary variables do not apply to the LHS of an assignment statement". (Incidentally, when in an earlier post I suggested that Expression LNBs might be better than Statement LNBs, I didn't realise that a temporary variable created in the first line of a suite ("if", "while" etc.) remained in scope for the rest of that suite. That seems (on balance) like a Good Thing, and a lot of the rationale for SLNBs. But I didn't like a temporary variable applying to the LHS of as assignment. So, with the above change to assignment statements, I am now happy about SLNBs.) Rob Cliffe
On 28/02/2018 05:23, Chris Angelico wrote:
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
Maybe add to last sentence "and of adding (at least conceptually) extra steps: building a 1-element list, then extracting the first element" That's precisely the point that Serhiy's optimization is aiming at, with the intention of making "for x in [expr]" a standard idiom for list comp assignment. If we assume that this does become standard, it won't add the extra steps, but it does still push that expression out to the far end of the comprehension, whereas a named subexpression places it at first use.
I understand that creating the list could be avoided *at runtime*. My point was that in trying to *read and understand* stuff = [[y, y] for x in range(5) for y in [f(x)]] the brain must follow the creation and unpacking of the list. I.e. this is an extra cost of this particular construction.
And here's a thought: What are the semantics of a = (42 as a) # Of course a linter should point this out too At first I thought this was also a laborious synonym for "a=42". But then I re-read your statement (the one I described above as crystal-clear) and realised that its exact wording was even more critical than I had thought: "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement" Note: "until the end of the statement". NOT "until the end of the expression". The distinction matters. If we take this as gospel, all this will do is create a temporary variable "a", assign the value 42 to it twice, then discard it. I.e. it effectively does nothing, slowly. Have I understood correctly? Very likely you have considered this and mean exactly what you say, but I am sure you will understand that I mean no offence by querying it. Actually, that's a very good point, and I had to actually go and do that to confirm. You're correct that the "a =" part is also affected, but there may be more complicated edge cases.
I have read this thread so far - I can't say I have absorbed and understood it all, but I am left with a feeling that Expression-Local-Name-Bindings would be preferable to Statement-Local-Name_Bindings, so that the temporary variable wouldn't apply to the LHS in the above example. I realise that this is a vague statement that needs far more definition, but - hand-waving for now - do you think it would be difficult to change the implementation accordingly? Rob Cliffe
On Thu, Mar 1, 2018 at 5:09 AM, Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:
On 28/02/2018 05:23, Chris Angelico wrote:
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
Maybe add to last sentence "and of adding (at least conceptually) extra steps: building a 1-element list, then extracting the first element"
That's precisely the point that Serhiy's optimization is aiming at, with the intention of making "for x in [expr]" a standard idiom for list comp assignment. If we assume that this does become standard, it won't add the extra steps, but it does still push that expression out to the far end of the comprehension, whereas a named subexpression places it at first use.
I understand that creating the list could be avoided *at runtime*. My point was that in trying to *read and understand* stuff = [[y, y] for x in range(5) for y in [f(x)]] the brain must follow the creation and unpacking of the list. I.e. this is an extra cost of this particular construction.
And here's a thought: What are the semantics of a = (42 as a) # Of course a linter should point this out too At first I thought this was also a laborious synonym for "a=42". But then I re-read your statement (the one I described above as crystal-clear) and realised that its exact wording was even more critical than I had thought: "the new name binding will shadow the other name from the point where it is evaluated until the end of the statement" Note: "until the end of the statement". NOT "until the end of the expression". The distinction matters. If we take this as gospel, all this will do is create a temporary variable "a", assign the value 42 to it twice, then discard it. I.e. it effectively does nothing, slowly. Have I understood correctly? Very likely you have considered this and mean exactly what you say, but I am sure you will understand that I mean no offence by querying it.
Actually, that's a very good point, and I had to actually go and do that to confirm. You're correct that the "a =" part is also affected, but there may be more complicated edge cases.
I have read this thread so far - I can't say I have absorbed and understood it all, but I am left with a feeling that Expression-Local-Name-Bindings would be preferable to Statement-Local-Name_Bindings, so that the temporary variable wouldn't apply to the LHS in the above example. I realise that this is a vague statement that needs far more definition, but - hand-waving for now - do you think it would be difficult to change the implementation accordingly?
Yes it would, but AIUI (I haven't tested it) a competing implementation already exists. So now we just need a competing PEP and we can run with it! ChrisA
On 27/02/2018 22:27, Chris Angelico wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
Hm, apologies. This is in complete contrast to my previous post, where I was pretty enthusiastic about Chris's PEP. But I can't resist sharing these thoughts ... There was some vague uneasiness at the back of my mind, which I think I have finally pinned down. Consider Chris's example: # Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)] I think what bothered me was the *asymmetry* between the two uses of the calculated value of f(x). It is not obvious at first glance that [(f(x) as y), y] defines a 2-element list where the 2 elements are the *same*. Contrast something like (exact syntax bike-sheddable) stuff = [ (with f(x) as y: [y,y]) for x in range(5)] or stuff = [ (y,y] with f(x) as y) for x in range(5)] This also has the advantage (if it is? I think probably it is) that the scope of the temporary variable ("y" here) can be limited to inside the parentheses of the "with" sub-expression. And that it is not dependent on Python's evaluation order. Ir gives the programmer explicit control over the scope, which might conceivably be an advantage in more complicated expressions. Sorry if this is re-hashing a suggestion that has been made before, as it probably is. It just struck me as ... I don't know ... cleaner somehow. Regards Rob Cliffe
On Wed, Feb 28, 2018 at 3:38 PM, Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:
Hm, apologies. This is in complete contrast to my previous post, where I was pretty enthusiastic about Chris's PEP. But I can't resist sharing these thoughts ...
There was some vague uneasiness at the back of my mind, which I think I have finally pinned down. Consider Chris's example: # Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)] I think what bothered me was the asymmetry between the two uses of the calculated value of f(x). It is not obvious at first glance that [(f(x) as y), y] defines a 2-element list where the 2 elements are the same.
Hmm, very good point. In non-toy examples, I suspect this will be somewhat overshadowed by the actual work being done, but this is definitely a bit weird.
Contrast something like (exact syntax bike-sheddable) stuff = [ (with f(x) as y: [y,y]) for x in range(5)] or stuff = [ (y,y] with f(x) as y) for x in range(5)] This also has the advantage (if it is? I think probably it is) that the scope of the temporary variable ("y" here) can be limited to inside the parentheses of the "with" sub-expression.
True, but it's also extremely wordy. Your two proposed syntaxes, if I have this correct, are: 1) '(' 'with' EXPR 'as' NAME ':' EXPR ')' 2) '(' EXPR 'with' EXPR 'as' NAME ')' Of the two, I prefer the first, as the second has the same problem as the if/else expression: execution is middle-first. It also offers only a slight advantage (in a comprehension) over just slapping another 'for' clause at the end of the expression. The first one is most comparable to the lambda helper example, and is (IMO) still very wordy. Perhaps it can be added in a section of "alternative syntax forms"?
And that it is not dependent on Python's evaluation order. Ir gives the programmer explicit control over the scope, which might conceivably be an advantage in more complicated expressions.
That shouldn't normally be an issue (execution order is usually pretty intuitive), but if there are weird enough edge cases found in my current proposal, I'm happy to mention this as a possible solution to them. ChrisA
True, but it's also extremely wordy. Your two proposed syntaxes, if I have this correct, are: 1) '(' 'with' EXPR 'as' NAME ':' EXPR ')' 2) '(' EXPR 'with' EXPR 'as' NAME ')' Of the two, I prefer the first, as the second has the same problem as the if/else expression: execution is middle-first. It also offers only a slight advantage (in a comprehension) over just slapping another 'for' clause at the end of the expression. The first one is most comparable to the lambda helper example, and is (IMO) still very wordy. Perhaps it can be added in a section of "alternative syntax forms"? Considering the 3rd syntax : '(' EXPR 'with' NAME '=' EXPR ')' Wouldn't have the problem of "execution being middle first and would clearly differenciate the "with NAME = CONTEXT" from the "with CONTEXT as NAME:" statement. Considering the PEP : 1) I think I spoke too fast for SqlAlchemy using "where", after looking up, they use "filter" (I was pretty sure I read it somewhere...) 2) talking about the implementation of thektulu in the "where =" part. 3) "C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison." The "=" does variable assignement already, and there is no grammar problem of "=" vs "==" because the "with" keyword is used in the expression, therefore "with a == ..." is a SyntaxError whereas "where a = ..." is alright (See grammar in thektulu implemention of "where"). Remember that the lexer knows the difference between "=" and "==", so those two are clearly different tokens. 4) Would the syntax be allowed after the "for" in a list comprehension ? [[y, y] for x in range(5) with y = x+1] This is exactly the same as "for y in [ x+1 ]", allowing the syntax here would allow adding "if" to filter in the list comp using the new Variable. [[y, y] for x in range(5) with y = x+1 if y % 2 == 0] 5) Any expression vs "post for" only When I say "any expression" I mean: print(y+2 with y = x+1) When I say "post for in list comp" I mean the previous paragraph: [y+2 for x in range(5) with y = x+1] Allowing both cases leads to having two ways in the simple case [(y,y) with y = x+1 for x in range(5)] vs [(y,y) for x in range(5) with y = x+1] (but that's alright) Allowing "any expression" would result in having two ways to have variable assignement : y = x + 1 print(y+2) Vs: print(y+2 with y = x+1) One could argue the first is imperative programming whereas the second is Functional programming. The second case will have to have "y" being a Local variable as the new Variable in list comp are not in the outside scope. 6) with your syntax, how does the simple case work (y+2 with y = x+1) ? Would you write ((x+1 as y) + 2) ? That's very unclear where the variable are defined, in the [(x+1 as y), y] case, the scoping would suggest the "y" Variable is defined between the parenthesis whereas [x+1 as y, y] is not symmetric. The issue is not only about reusing variable. 7) the "lambda example", the "v" variable can be renamed "y" to be consistent with the other examples. 8) there are two ways of using a lamba, either positional args, either keyword arguments, writing (lambda y: [y, y])(x+1) Vs (lambda y: [y, y])(y=x+1) In the second example, the y = x+1 is explicit. 9) the issue is not only about reusing variable, but also readability, otherwise, why would we create Tempory variables we only use once ? 10) Chaining, in the case of the "with =", in thektulu, parenthesis were mandatory: print((z+3 with z = y+2) with y = x+2) What happens when the parenthesis are dropped ? print(z+3 with y = x+2 with z = y+2) Vs print(z+3 with y = x+2 with z = y+2) I prefer the first one be cause it's in the same order as the "post for" [z + 3 for y in [ x+2 ] for z in [ y+2 ]] 11) Scoping, in the case of the "with =" syntax, I think the parenthesis introduce a scope : print(y + (y+1 where y = 2)) Would raise a SyntaxError, it's probably better for the variable beeing local and not in the current function (that would be a mess). Remember that in list comp, the variable is not leaked : x = 5 stuff = [y+2 for y in [x+1] print(y) # SyntaxError Robert
On Wed, Feb 28, 2018 at 8:04 PM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Considering the 3rd syntax : '(' EXPR 'with' NAME '=' EXPR ')'
Wouldn't have the problem of "execution being middle first and would clearly differenciate the "with NAME = CONTEXT" from the "with CONTEXT as NAME:" statement.
It's still right-to-left, which is as bad as middle-outward once you combine it with normal left-to-right evaluation. Python has very little of this, and usually only in contexts where you wouldn't have much code on the left:
def d(): ... print("Getting dictionary") ... return {} ... def k(): ... print("Getting key") ... return "spam" ... def v(): ... print("Getting value") ... return 42 ... d()[k()] = v() Getting value Getting dictionary Getting key
Python executes the RHS of an assignment statement before the LHS, but the LHS is usually going to be so simple that you don't really care (or even notice, usually). By creating a name binding on the right and then evaluating the left, you create a complicated evaluation order that *will* have complex code on the left.
Considering the PEP :
1) I think I spoke too fast for SqlAlchemy using "where", after looking up, they use "filter" (I was pretty sure I read it somewhere...)
There's something with "select where exists" that uses .where(). It may not be as common as filter, but it's certainly out there.
2) talking about the implementation of thektulu in the "where =" part.
?
3) "C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison." The "=" does variable assignement already, and there is no grammar problem of "=" vs "==" because the "with" keyword is used in the expression, therefore "with a == ..." is a SyntaxError whereas "where a = ..." is alright (See grammar in thektulu implemention of "where").
Yes, but in Python, "=" does variable assignment *as a statement*. In C, you can do this: while (ch = getch()) do_something_with(ch) That's an assignment in an arbitrary condition, and that's a bug magnet. You cannot do that in Python. You cannot simply miss out one equals sign and have legal code that does what you don't want. With my proposed syntax, you'll be able to do this: while (getch() as ch): ... There's no way that you could accidentally write this when you really wanted to compare against the character. With yours, I'm not sure whether it handles a 'while' loop at all, but if it does, it would be something like: while (ch with ch = getch()): ... which doesn't read very well, doesn't really save much, but yes, I agree, it isn't going to accidentally assign.
Remember that the lexer knows the difference between "=" and "==", so those two are clearly different tokens.
It's not the lexer I'm worried about :)
4) Would the syntax be allowed after the "for" in a list comprehension ?
[[y, y] for x in range(5) with y = x+1]
This is exactly the same as "for y in [ x+1 ]", allowing the syntax here would allow adding "if" to filter in the list comp using the new Variable.
[[y, y] for x in range(5) with y = x+1 if y % 2 == 0]
I honestly don't know. With my "as" syntax, you would be able to, because it's simply first-use. The (expr as name) unit is itself an expression with a value. The 'with' clause has to bracket the value in some way.
5) Any expression vs "post for" only
When I say "any expression" I mean:
print(y+2 with y = x+1)
When I say "post for in list comp" I mean the previous paragraph:
[y+2 for x in range(5) with y = x+1]
Allowing both cases leads to having two ways in the simple case [(y,y) with y = x+1 for x in range(5)] vs [(y,y) for x in range(5) with y = x+1] (but that's alright)
Allowing "any expression" would result in having two ways to have variable assignement :
y = x + 1 print(y+2)
Vs:
print(y+2 with y = x+1)
One could argue the first is imperative programming whereas the second is Functional programming.
The second case will have to have "y" being a Local variable as the new Variable in list comp are not in the outside scope.
I don't know what the benefit is here, but sure. As long as the grammar is unambiguous, I don't see any particular reason to reject this.
6) with your syntax, how does the simple case work (y+2 with y = x+1) ?
Would you write ((x+1 as y) + 2) ? That's very unclear where the variable are defined, in the [(x+1 as y), y] case, the scoping would suggest the "y" Variable is defined between the parenthesis whereas [x+1 as y, y] is not symmetric.
What simple case? The case where you only use the variable once? I'd write it like this: (x + 1) + 2
The issue is not only about reusing variable.
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
7) the "lambda example", the "v" variable can be renamed "y" to be consistent with the other examples.
Oops, thanks, fixed.
8) there are two ways of using a lamba, either positional args, either keyword arguments, writing
(lambda y: [y, y])(x+1)
Vs
(lambda y: [y, y])(y=x+1)
In the second example, the y = x+1 is explicit.
Ewww. Remind me what the benefit is of writing the variable name that many times? "Explicit" doesn't mean "utterly verbose".
10) Chaining, in the case of the "with =", in thektulu, parenthesis were mandatory:
print((z+3 with z = y+2) with y = x+2)
What happens when the parenthesis are dropped ?
print(z+3 with y = x+2 with z = y+2)
Vs
print(z+3 with y = x+2 with z = y+2)
I prefer the first one be cause it's in the same order as the "post for"
[z + 3 for y in [ x+2 ] for z in [ y+2 ]]
With my proposal, the parens are simply mandatory. Extending this to make them optional can come later.
11) Scoping, in the case of the "with =" syntax, I think the parenthesis introduce a scope :
print(y + (y+1 where y = 2))
Would raise a SyntaxError, it's probably better for the variable beeing local and not in the current function (that would be a mess).
Remember that in list comp, the variable is not leaked :
x = 5 stuff = [y+2 for y in [x+1] print(y) # SyntaxError
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression. ChrisA
On 02/28/2018 02:43 AM, Chris Angelico wrote:
On Wed, Feb 28, 2018 at 8:04 PM, Robert Vanden Eynde wrote:
3) "C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison." The "=" does variable assignement already, and there is no grammar problem of "=" vs "==" because the "with" keyword is used in the expression, therefore "with a == ..." is a SyntaxError whereas "where a = ..." is alright (See grammar in thektulu implemention of "where").
Yes, but in Python, "=" does variable assignment *as a statement*. In C, you can do this:
while (ch = getch()) do_something_with(ch)
That's an assignment in an arbitrary condition, and that's a bug magnet. You cannot do that in Python. You cannot simply miss out one equals sign and have legal code that does what you don't want. With my proposed syntax, you'll be able to do this:
while (getch() as ch): ...
There's no way that you could accidentally write this when you really wanted to compare against the character.
Given the current (posted) proposal, wouldn't 'ch' evaporate before the ':' and be unavailable in the 'while' body? -- ~Ethan~
On Thu, Mar 1, 2018 at 2:25 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 02/28/2018 02:43 AM, Chris Angelico wrote:
On Wed, Feb 28, 2018 at 8:04 PM, Robert Vanden Eynde wrote:
3) "C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison." The "=" does variable assignement already, and there is no grammar problem of "=" vs "==" because the "with" keyword is used in the expression, therefore "with a == ..." is a SyntaxError whereas "where a = ..." is alright (See grammar in thektulu implemention of "where").
Yes, but in Python, "=" does variable assignment *as a statement*. In C, you can do this:
while (ch = getch()) do_something_with(ch)
That's an assignment in an arbitrary condition, and that's a bug magnet. You cannot do that in Python. You cannot simply miss out one equals sign and have legal code that does what you don't want. With my proposed syntax, you'll be able to do this:
while (getch() as ch): ...
There's no way that you could accidentally write this when you really wanted to compare against the character.
Given the current (posted) proposal, wouldn't 'ch' evaporate before the ':' and be unavailable in the 'while' body?
Not my proposal. Others have suggested various forms of *expression* local name bindings, but the definition in the PEP is *statement* local. So it is indeed available. ChrisA
Le 28 févr. 2018 11:43, "Chris Angelico" <rosuav@gmail.com> a écrit :
It's still right-to-left, which is as bad as middle-outward once you combine it with normal left-to-right evaluation. Python has very little of this [..]
I agree [....]
2) talking about the implementation of thektulu in the "where =" part.
?
In the Alternate Syntax, I was talking about adding a link to the thektulu (branch where-expr) <https://github.com/thektulu/cpython/commits/where-expr> implementation as a basis of proof of concept (as you did with the other syntax).
3) "C problem that an equals sign in an expression can now create a name inding, rather than performing a comparison."
As you agreed, with the "ch with ch = getch()" syntax we won't accidentally switch a "==" for a "=". I agree this syntax : ``` while (ch with ch = getch()): ... ``` doesn't read very well, but in the same way as in C or Java while(ch = getch()){} or worse ((ch = getch()) != null) syntax. Your syntax "while (getch() as ch):" may have a less words, but is still not clearer. As we spoke on Github, having this syntax in a while is only useful if the variable does leak.
5) Any expression vs "post for" only
I don't know what the benefit is here, but sure. As long as the grammar is unambiguous, I don't see any particular reason to reject this.
I would like to see a discussion of pros and cons, some might think like me or disagree, that's a strong langage question.
6) with your syntax, how does the simple case work (y+2 with y = x+1) ?
What simple case? The case where you only use the variable once? I'd write it like this: (x + 1) + 2
The issue is not only about reusing variable.
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
Yes, variables are not there "just because we reuse them", but also to include temporary variables to better understand the code. Same for functions, you could inline functions when used only once, but you introduce them for clarity no ? ``` a = v ** 2 / R # the acceleration in a circular motion f = m * a # law of Newton ``` could be written as ``` f = m * (v ** 2 / R) # compute the force, trivial ``` But having temporary variables help a lot to understand the code, otherwise why would we create temporary variables ? I can give you an example where you do a process and each time the variable is used only one.
8) (lambda y: [y, y])(x+1) Vs (lambda y: [y, y])(y=x+1)
Ewww. Remind me what the benefit is of writing the variable name that many times? "Explicit" doesn't mean "utterly verbose". Yep it's verbose, lambdas are verbose, that's why we created this PEP isn't it :)
10) Chaining, in the case of the "with =", in thektulu, parenthesis were mandatory:
print((z+3 with z = y+2) with y = x+2)
What happens when the parenthesis are dropped ?
print(z+3 with y = x+2 with z = y+2)
Vs
print(z+3 with y = x+2 with z = y+2)
I prefer the first one be cause it's in the same order as the "post for"
[z + 3 for y in [ x+2 ] for z in [ y+2 ]]
With my proposal, the parens are simply mandatory. Extending this to make them optional can come later.
Indeed, but that's still questions that can be asked.
11) Scoping, in the case of the "with =" syntax, I think the parenthesis introduce a scope :
print(y + (y+1 where y = 2))
Would raise a SyntaxError, it's probably better for the variable beeing local and not in the current function (that would be a mess).
Remember that in list comp, the variable is not leaked :
x = 5 stuff = [y+2 for y in [x+1] print(y) # SyntaxError
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression.
Confer the discussion on scoping on github ( https://github.com/python/peps/commit/2b4ca20963a24cf5faac054226857ea9705471...) : """ In the current implementation it looks like it is like a regular assignment (function local then). Therefore in the expression usage, the usefulness would be debatable (just assign before). But in a list comprehension *after the for* (as I mentioned in my mail), aka. when used as a replacement for for y in [ x + 1 ] this would make sense. But I think that it would be much better to have a local scope, in the parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when there are no parenthesis like in a = y+2 where y = x+1, it would imply one, giving the same effect as a = (y+2 where y = x+1). Moreover, it would naturally shadow variables in the outermost scope. This would imply while data where data = sock.read(): does not leak data but as a comparison with C and Java, the syntax while((data = sock.read()) != null) is really really ugly and confusing. """
We are currently like a dozen of people talking about multiple sections of a single subject. Isn't it easier to talk on a forum? *Am I the only one* who thinks mailing list isn't easy when lots of people talking about multiple subjects? Of course we would put the link in the mailing list so that everyone can join. A forum (or just few "issues" thread on github) is where we could have different thread in parallel, in my messages I end up with like *10 comments not all related*, in a forum we could talk about everything and it would still be organized by subjects. Also, it's more interactive than email on a global list, people can talk to each other in parallel, if I want to answer about a mail that was 10 mail ago, it gets quickly messy. We could all discuss on a gist or some "Issues" thread on GitHub. 2018-02-28 22:38 GMT+01:00 Robert Vanden Eynde <robertve92@gmail.com>:
Le 28 févr. 2018 11:43, "Chris Angelico" <rosuav@gmail.com> a écrit :
It's still right-to-left, which is as bad as middle-outward once you combine it with normal left-to-right evaluation. Python has very little of this [..]
I agree [....]
2) talking about the implementation of thektulu in the "where =" part.
?
In the Alternate Syntax, I was talking about adding a link to the thektulu (branch where-expr) <https://github.com/thektulu/cpython/commits/where-expr> implementation as a basis of proof of concept (as you did with the other syntax).
3) "C problem that an equals sign in an expression can now create a name inding, rather than performing a comparison."
As you agreed, with the "ch with ch = getch()" syntax we won't accidentally switch a "==" for a "=".
I agree this syntax :
``` while (ch with ch = getch()): ... ```
doesn't read very well, but in the same way as in C or Java while(ch = getch()){} or worse ((ch = getch()) != null) syntax. Your syntax "while (getch() as ch):" may have a less words, but is still not clearer.
As we spoke on Github, having this syntax in a while is only useful if the variable does leak.
5) Any expression vs "post for" only
I don't know what the benefit is here, but sure. As long as the grammar is unambiguous, I don't see any particular reason to reject this.
I would like to see a discussion of pros and cons, some might think like me or disagree, that's a strong langage question.
6) with your syntax, how does the simple case work (y+2 with y = x+1) ?
What simple case? The case where you only use the variable once? I'd write it like this:
(x + 1) + 2
The issue is not only about reusing variable.
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
Yes, variables are not there "just because we reuse them", but also to include temporary variables to better understand the code. Same for functions, you could inline functions when used only once, but you introduce them for clarity no ?
``` a = v ** 2 / R # the acceleration in a circular motion f = m * a # law of Newton ```
could be written as
``` f = m * (v ** 2 / R) # compute the force, trivial ```
But having temporary variables help a lot to understand the code, otherwise why would we create temporary variables ? I can give you an example where you do a process and each time the variable is used only one.
8) (lambda y: [y, y])(x+1) Vs (lambda y: [y, y])(y=x+1)
Ewww. Remind me what the benefit is of writing the variable name that many times? "Explicit" doesn't mean "utterly verbose".
Yep it's verbose, lambdas are verbose, that's why we created this PEP isn't it :)
10) Chaining, in the case of the "with =", in thektulu, parenthesis were mandatory:
print((z+3 with z = y+2) with y = x+2)
What happens when the parenthesis are dropped ?
print(z+3 with y = x+2 with z = y+2)
Vs
print(z+3 with y = x+2 with z = y+2)
I prefer the first one be cause it's in the same order as the "post for"
[z + 3 for y in [ x+2 ] for z in [ y+2 ]]
With my proposal, the parens are simply mandatory. Extending this to make them optional can come later.
Indeed, but that's still questions that can be asked.
11) Scoping, in the case of the "with =" syntax, I think the parenthesis introduce a scope :
print(y + (y+1 where y = 2))
Would raise a SyntaxError, it's probably better for the variable beeing local and not in the current function (that would be a mess).
Remember that in list comp, the variable is not leaked :
x = 5 stuff = [y+2 for y in [x+1] print(y) # SyntaxError
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression.
Confer the discussion on scoping on github (https://github.com/python/ peps/commit/2b4ca20963a24cf5faac054226857ea9705471e5) :
""" In the current implementation it looks like it is like a regular assignment (function local then).
Therefore in the expression usage, the usefulness would be debatable (just assign before).
But in a list comprehension *after the for* (as I mentioned in my mail), aka. when used as a replacement for for y in [ x + 1 ] this would make sense.
But I think that it would be much better to have a local scope, in the parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when there are no parenthesis like in a = y+2 where y = x+1, it would imply one, giving the same effect as a = (y+2 where y = x+1). Moreover, it would naturally shadow variables in the outermost scope.
This would imply while data where data = sock.read(): does not leak data but as a comparison with C and Java, the syntax while((data = sock.read()) != null) is really really ugly and confusing. """
-0 unless archived appropriately. List is the standard for decades. but I guess things change and I get old. On Wed, Feb 28, 2018, 13:49 Robert Vanden Eynde <robertve92@gmail.com> wrote:
We are currently like a dozen of people talking about multiple sections of a single subject.
Isn't it easier to talk on a forum? *Am I the only one* who thinks mailing list isn't easy when lots of people talking about multiple subjects?
Of course we would put the link in the mailing list so that everyone can join.
A forum (or just few "issues" thread on github) is where we could have different thread in parallel, in my messages I end up with like *10 comments not all related*, in a forum we could talk about everything and it would still be organized by subjects.
Also, it's more interactive than email on a global list, people can talk to each other in parallel, if I want to answer about a mail that was 10 mail ago, it gets quickly messy.
We could all discuss on a gist or some "Issues" thread on GitHub.
2018-02-28 22:38 GMT+01:00 Robert Vanden Eynde <robertve92@gmail.com>:
Le 28 févr. 2018 11:43, "Chris Angelico" <rosuav@gmail.com> a écrit :
It's still right-to-left, which is as bad as middle-outward once you combine it with normal left-to-right evaluation. Python has very little of this [..]
I agree [....]
2) talking about the implementation of thektulu in the "where =" part.
?
In the Alternate Syntax, I was talking about adding a link to the thektulu (branch where-expr) <https://github.com/thektulu/cpython/commits/where-expr> implementation as a basis of proof of concept (as you did with the other syntax).
3) "C problem that an equals sign in an expression can now create a name inding, rather than performing a comparison."
As you agreed, with the "ch with ch = getch()" syntax we won't accidentally switch a "==" for a "=".
I agree this syntax :
``` while (ch with ch = getch()): ... ```
doesn't read very well, but in the same way as in C or Java while(ch = getch()){} or worse ((ch = getch()) != null) syntax. Your syntax "while (getch() as ch):" may have a less words, but is still not clearer.
As we spoke on Github, having this syntax in a while is only useful if the variable does leak.
5) Any expression vs "post for" only
I don't know what the benefit is here, but sure. As long as the grammar is unambiguous, I don't see any particular reason to reject this.
I would like to see a discussion of pros and cons, some might think like me or disagree, that's a strong langage question.
6) with your syntax, how does the simple case work (y+2 with y = x+1) ?
What simple case? The case where you only use the variable once? I'd write it like this:
(x + 1) + 2
The issue is not only about reusing variable.
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
Yes, variables are not there "just because we reuse them", but also to include temporary variables to better understand the code. Same for functions, you could inline functions when used only once, but you introduce them for clarity no ?
``` a = v ** 2 / R # the acceleration in a circular motion f = m * a # law of Newton ```
could be written as
``` f = m * (v ** 2 / R) # compute the force, trivial ```
But having temporary variables help a lot to understand the code, otherwise why would we create temporary variables ? I can give you an example where you do a process and each time the variable is used only one.
8) (lambda y: [y, y])(x+1) Vs (lambda y: [y, y])(y=x+1)
Ewww. Remind me what the benefit is of writing the variable name that many times? "Explicit" doesn't mean "utterly verbose".
Yep it's verbose, lambdas are verbose, that's why we created this PEP isn't it :)
10) Chaining, in the case of the "with =", in thektulu, parenthesis were mandatory:
print((z+3 with z = y+2) with y = x+2)
What happens when the parenthesis are dropped ?
print(z+3 with y = x+2 with z = y+2)
Vs
print(z+3 with y = x+2 with z = y+2)
I prefer the first one be cause it's in the same order as the "post for"
[z + 3 for y in [ x+2 ] for z in [ y+2 ]]
With my proposal, the parens are simply mandatory. Extending this to make them optional can come later.
Indeed, but that's still questions that can be asked.
11) Scoping, in the case of the "with =" syntax, I think the parenthesis introduce a scope :
print(y + (y+1 where y = 2))
Would raise a SyntaxError, it's probably better for the variable beeing local and not in the current function (that would be a mess).
Remember that in list comp, the variable is not leaked :
x = 5 stuff = [y+2 for y in [x+1] print(y) # SyntaxError
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression.
Confer the discussion on scoping on github ( https://github.com/python/peps/commit/2b4ca20963a24cf5faac054226857ea9705471...) :
""" In the current implementation it looks like it is like a regular assignment (function local then).
Therefore in the expression usage, the usefulness would be debatable (just assign before).
But in a list comprehension *after the for* (as I mentioned in my mail), aka. when used as a replacement for for y in [ x + 1 ] this would make sense.
But I think that it would be much better to have a local scope, in the parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when there are no parenthesis like in a = y+2 where y = x+1, it would imply one, giving the same effect as a = (y+2 where y = x+1). Moreover, it would naturally shadow variables in the outermost scope.
This would imply while data where data = sock.read(): does not leak data but as a comparison with C and Java, the syntax while((data = sock.read()) != null) is really really ugly and confusing. """
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Mar 1, 2018 at 8:53 AM, Matt Arcidy <marcidy@gmail.com> wrote:
-0 unless archived appropriately. List is the standard for decades. but I guess things change and I get old.
Archived, searchable, and properly threaded AND properly notifying the correct participants. Every few years, a spiffy new thing comes along, and it usually isn't enough of an improvement to survive. Mailing lists continue to be more than adequate, and the alternatives end up being aimed at point-and-click people who dislike mailing lists. I'm much happier sticking to the list. ChrisA
On 02/28/2018 01:48 PM, Robert Vanden Eynde wrote:
We are currently like a dozen of people talking about multiple sections of a single subject.
Isn't it easier to talk on a forum?
No.
*Am I the only one* who thinks mailing list isn't easy when lots of people talking about multiple subjects?
Maybe.
Of course we would put the link in the mailing list so that everyone can join.
Python Ideas is a mailing list. This is where we discuss ideas for future versions of Python. If you're using Google Groups or a lousy mail reader then I sympathize, but it's on you to use the appropriate tools. -- ~Ethan~
On 28 February 2018 at 21:48, Robert Vanden Eynde <robertve92@gmail.com> wrote:
We are currently like a dozen of people talking about multiple sections of a single subject.
Isn't it easier to talk on a forum?
Not for me, certainly. I could probably learn how to effectively work with a forum, but as of right now, if this discussion switched to a forum, I'd be unable to follow the discussion, and would likely not contribute. And whether I'd actually take the time to learn how to work with a forum is debatable (there are many different forms of forum software, and learning a new interface for each discussion group isn't effective for me. Conversely, with mailing lists, I just use the one interface, gmail).
Am I the only one who thinks mailing list isn't easy when lots of people talking about multiple subjects?
Possibly not, but a lot of participants have invested time in learning how to work effectively with mailing lists. IMO, a forum prioritises occasional use and newcomers, while penalising long-term and in-depth use.
Of course we would put the link in the mailing list so that everyone can join.
But the discussion would still be taking place in 2 locations, which would make it even harder to follow. Unless you plan on shutting down the list?
A forum (or just few "issues" thread on github) is where we could have different thread in parallel, in my messages I end up with like 10 comments not all related, in a forum we could talk about everything and it would still be organized by subjects.
Conversely, interested parties would find it harder by default to read all of the discussion on the subject of PEP 572. Rather than flagging *one* thread as important, we'd be relying on having the PEP number in the subject of multiple topics, and searching.
Also, it's more interactive than email on a global list, people can talk to each other in parallel, if I want to answer about a mail that was 10 mail ago, it gets quickly messy.
No, you just reply to that mail. You may lose some of the intermediate comments by doing so, but that happens however you organise the discussion. You're choosing to branch out the discussion from an older point.
We could all discuss on a gist or some "Issues" thread on GitHub.
Not everyone can access gists or github. My work network blocks gists as "personal storage", for example. Paul
On 28 February 2018 at 21:48, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Isn't it easier to talk on a forum?
For me, the fact that all the mailing lists I subscribe to feed into one input queue is a feature, not a bug. It means I can easily keep abreast of developments in many areas at once, without having to laboriously and explicitly go and visit multiple forums and threads. Also, every web forum I've ever used has been slow and klunky to use compared to a mail/news client. -- Greg
On Thu, Mar 1, 2018 at 2:47 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 28 February 2018 at 21:48, Robert Vanden Eynde <robertve92@gmail.com>
wrote:
Isn't it easier to talk on a forum?
For me, the fact that all the mailing lists I subscribe to feed into one input queue is a feature, not a bug. It means I can easily keep abreast of developments in many areas at once, without having to laboriously and explicitly go and visit multiple forums and threads.
Also, every web forum I've ever used has been slow and klunky to use compared to a mail/news client.
We can kill this topic right now. We'll be moving to MM3, which is a mailing list for those who like email, and can also be used as a web forum for those who like that. Personally I'm partial to both, so I think this is the best of both worlds. -- --Guido van Rossum (python.org/~guido)
Reply-To set to *me*. I don't really think this idea has legs at this point. It's been discussed before (it even has a SIG), but that ran out of steam and the move to Mailman 3 (whose archiver provides a more forum-like interface) is more or less imminent anyway, so it's kind of moot. I'm happy to deal with it off-list, but for the moment it's OT IMO.
Robert Vanden Eynde writes:
We are currently like a dozen of people talking about multiple sections of a single subject.
Isn't it easier to talk on a forum?
There's the Overload SIG for this discussion of media (currently inactive for many months). Transparency notice: Mailman dev here, bias acknowledged, some in the SIG were more sympathetic to forums, I don't claim to be representative of SIG opinion.
Am I the only one who thinks mailing list isn't easy when lots of people talking about multiple subjects?
No, you are hardly alone. However, in my experience good forum software is distinctly inferior to good mail software in high-traffic multi-threaded communications. There is very little that forum software can do that a mail client cannot.[1] The main advantages of forum software are better thread management (message ids are reliable and threads are built synchronously, but it's not much better) and server-side storage (but of course that's a SPoF). As the name of the SIG suggests, the problem is not so much that mail is such a bad medium; it's more that the traffic levels are higher than people are used to, so the bad experience of high traffic is conflated with a bad experience of using mail. (We do have one or two on the SIG who say that they do fine with high-traffic multi-threaded forums. I'm not sure how closely the use case matches the main python-* channels, excluding python-list.) The main advantage to using email is that management of multiple high-traffic streams requires smart, highly configurable clients. These have been available for mail since the late 80s. They've only gotten better since. Forums, on the other hand, are based on thin clients (ie web browsers), and so you get only the configuration the forum offers, and you have to tell the forum all about it. Of course we are hackers: all the main browsers are scriptable, the language is javascript, so we could write our own forum clients. But that's exactly what forum advocates want to avoid, at least if you call it "email". The problem for mail advocates is the flip side of that. That is, AFAIK there is no good mail software that runs in a browser (let alone a smartphone), and change of mail clients, like religions and programmer's editors, is advocated only at risk to limb and life. As mentioned above, this whole issue may become moot when the mailing lists are moved to Mailman 3 (we have some experimental Mailman 3 lists; ETA of a general migration is 6--18 months, at a slightly- informed guess), which has a forum-like interface to the archives called "HyperKitty". HyperKitty is not a rival to Discourse, at least not in the near future[2], but it's already pretty good.
A forum (or just few "issues" thread on github) is where we could have different thread in parallel, in my messages I end up with like 10 comments not all related, in a forum we could talk about everything and it would still be organized by subjects.
Also, it's more interactive than email on a global list, people can talk to each other in parallel, if I want to answer about a mail that was 10 mail ago, it gets quickly messy.
I have no idea why any of those things would be unmanageable in email, unless you have disabled threading in your client, or your client doesn't allow you to kill (or collapse) threads of no interest to you.
We could all discuss on a gist or some "Issues" thread on GitHub.
Those are non-starters. *You* can already do that, but very few of the committers will follow. We already have an issue tracker for very focussed discussion. This works because people actually working on an issue are frequently spending hours on it and related work, so a minute or so spent finding the appropriate issue is small overhead. The overhead would be substantially greater if you had to regularly scan all recent issues to see if any are of interest to you. Trackers are not very good at switching threads (issues), so even if you turn on push notification for new issues, there will be substantial overhead in following more than a tiny number of threads. The kinds of things that are discussed on the mailing lists generally need to be broadcast, at least at first, and (assuming a decent mail client) threads of no interest can easily be suppressed in email. That's what the committers do, mostly. (Some just live with the firehose.) Note, that's not to say a *forum* is a non-starter. I guess that good forum software provides similar features. Some senior developers have expressed interest in moving to a forum-based discussion. Just that most of the committers have broad interest in the discussion, so are going to get their news from whatever the main channels are: we need to have a fairly small number of them, and we do *not* want them split across multiple media. Finally, speaking of "split across media", any serious proposal for change of medium will need to deal with the (voluminous) mail archives. This may not be difficult ("just provide a link"), but then again it may not be that easy. I have given no thought to it yet, and I don't think the forum advocates have either. Bottom line: in the end at least one PEP (I would guess two or three) will be needed to make a major change to the workflow like this, unless it's done in the context of a hybrid Mailman + forum system like Mailman 3 including HyperKitty. Once again, I admit that I strongly prefer mailing lists. However, I also believe that it's not bias to suggest that it's volume of communication, not the medium, that is the problem here. At least in principle mail is fully capable of carrying the load. In practice, deficient mail clients (I'm looking at you, GMail and Outlook and anything on a handheld) are all over the place. In the end we'll need to balance the costs of moving (which includes the preferences of those who "just" like email better as well as setup and migration of archives) against the benefits (mostly the preferences of those who have learned to use forums very efficiently). Footnotes: [1] To my knowledge. I'm willing to be educated. [2] And it won't solve the asynchronicity of SMTP messaging. Mailman will still be primarily a mail-based system, at least at first.
On Thu, Mar 1, 2018 at 8:38 AM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Le 28 févr. 2018 11:43, "Chris Angelico" <rosuav@gmail.com> a écrit :
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
Yes, variables are not there "just because we reuse them", but also to include temporary variables to better understand the code. Same for functions, you could inline functions when used only once, but you introduce them for clarity no ?
Sure, but if you're creating temporaries for clarity, you should probably make them regular variables, not statement-local ones. If you're going to use something only once and you don't want to break it out into a separate statement, just use a comment.
``` a = v ** 2 / R # the acceleration in a circular motion f = m * a # law of Newton ```
could be written as
``` f = m * (v ** 2 / R) # compute the force, trivial ```
But having temporary variables help a lot to understand the code, otherwise why would we create temporary variables ? I can give you an example where you do a process and each time the variable is used only one.
Neither of your examples needs SLNBs.
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression.
Confer the discussion on scoping on github (https://github.com/python/peps/commit/2b4ca20963a24cf5faac054226857ea9705471...) :
""" In the current implementation it looks like it is like a regular assignment (function local then).
Therefore in the expression usage, the usefulness would be debatable (just assign before).
But in a list comprehension after the for (as I mentioned in my mail), aka. when used as a replacement for for y in [ x + 1 ] this would make sense.
But I think that it would be much better to have a local scope, in the parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when there are no parenthesis like in a = y+2 where y = x+1, it would imply one, giving the same effect as a = (y+2 where y = x+1). Moreover, it would naturally shadow variables in the outermost scope.
So the question is: what is the benefit of the local name 'y'? In any non-trivial example, it's not going to fit in a single line, so you have to either wrap it as a single expression (one that's been made larger by the "where" clause at the end), or break it out into a real variable as a separate assignment. ChrisA
Sorry, answer to an old post but I just realized I didn't use the correct email address... (See below to see which message I'm answering). That's what I said on github, there are two different use cases : *ListComp* Vs *AnyExpressions* : 1) List comprehension : [y+2 for x in range(5) for y in [ x+1 ]] → [y+2 for x in range(5) with y = x+1] 2) Any expression : print(y + 2 with x = 2) So, IF the syntax is allowed in *both* cases, then yes, the "any expression case" would introduced a "There is not one obvious way to do it". Now introducing *scoping*, the "any expression case" can become handy because it would create local variables in parenthesis : x = 2 print([y, x/y] with y = x + 2) print(y) # NameError It would structure expressions and would be in the same idea as PEP 3150 <https://www.python.org/dev/peps/pep-3150/> but with a less drastic syntax and more simple use case. Maybe both cases are completely different and your we would have 3 proposals : 1) "EXPR as NAME" → to easily reuse an expression used multiple times in an expression. 2) "EXPR with NAME = EXPR" (any expression) → same purpose, different syntax 3) "[EXPR for x in ITERABLE with NAME = EXPR] → to allow reuse of an expression in a list comprehension. Maybe the only use cases are in list comprehensions and would be an improvement to the for y in [ f(x) ] not obvious syntax (that was my original thought weeks ago and in June of last year). Your syntax gives a proof of concept implementation The AnyExpression case had a proof of concept in thektulu (branch where-expr) <https://github.com/thektulu/cpython/tree/where-expr> Should we create "yet another pep" for each of this use cases or is it part of a big same idea ? Cheers, Robert Le 28 févr. 2018 22:53, "Chris Angelico" <rosuav@gmail.com> a écrit : On Thu, Mar 1, 2018 at 8:38 AM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Le 28 févr. 2018 11:43, "Chris Angelico" <rosuav@gmail.com> a écrit :
If you aren't using the variable multiple times, there's no point giving it a name. Unless I'm missing something here?
Yes, variables are not there "just because we reuse them", but also to include temporary variables to better understand the code. Same for functions, you could inline functions when used only once, but you introduce them for clarity no ?
Sure, but if you're creating temporaries for clarity, you should probably make them regular variables, not statement-local ones. If you're going to use something only once and you don't want to break it out into a separate statement, just use a comment.
``` a = v ** 2 / R # the acceleration in a circular motion f = m * a # law of Newton ```
could be written as
``` f = m * (v ** 2 / R) # compute the force, trivial ```
But having temporary variables help a lot to understand the code,
otherwise
why would we create temporary variables ? I can give you an example where you do a process and each time the variable is used only one.
Neither of your examples needs SLNBs.
Scoping is a fundamental part of both my proposal and the others I've seen here. (BTW, that would be a NameError, not a SyntaxError; it's perfectly legal to ask for the name 'y', it just hasn't been given any value.) By my definition, the variable is locked to the statement that created it, even if that's a compound statement. By the definition of a "(expr given var = expr)" proposal, it would be locked to that single expression.
Confer the discussion on scoping on github (https://github.com/python/peps/commit/2b4ca20963a24cf5faac054226857e a9705471e5) :
""" In the current implementation it looks like it is like a regular assignment (function local then).
Therefore in the expression usage, the usefulness would be debatable (just assign before).
But in a list comprehension after the for (as I mentioned in my mail), aka. when used as a replacement for for y in [ x + 1 ] this would make sense.
But I think that it would be much better to have a local scope, in the parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when there are no parenthesis like in a = y+2 where y = x+1, it would imply one, giving the same effect as a = (y+2 where y = x+1). Moreover, it would naturally shadow variables in the outermost scope.
So the question is: what is the benefit of the local name 'y'? In any non-trivial example, it's not going to fit in a single line, so you have to either wrap it as a single expression (one that's been made larger by the "where" clause at the end), or break it out into a real variable as a separate assignment. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Hello Chris and Rob, did you compare your proposal tothe subject called "[Python-ideas] Temporary variables in comprehensions" on this month list ? If you don't want to go through all the mails, I tried to summarize the ideas in this mail : https://mail.python.org/pipermail/python-ideas/2018- February/048987.html In a nutshell one of the proposed syntax was stuff = [(y, y) where y = f(x) for x in range(5)] Or with your choice of keyword, stuff = [(y, y) with y = f(x) for x in range(5)] Your proposal uses the *reverse* syntax : stuff = [(y, y) with f(x) as y for x in range (5)] Your syntax was already proposed in 2008 and you can see in the response to my mail (https://mail.python.org/pipermail/python-ideas/2018- February/049000.html) The first example with the "where y = f(x)" can already be compiled on a CPython branch called "where-expr" on https://github.com/ thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c23f2636 You can see all the conversation on subject "Tempory variables in comprehensions" on https://mail.python.org/pipermail/python-ideas/2018- February/thread.html#start I wish I could write a pep to summarize all the discussions (including yours, my summary, including real world examples, including pro's and con's), or should I do a gist on GitHub so that we can talk in a more "forum like" manner where people can edit their answers and modify the document ? This mailing is driving me a bit crazy. As Rob pointed out, your syntax "(f(x) as y, y)" is really assymmetric and "(y, y) with f(x) as y" or "(y, y) with y = f(x)" is probably prefered. Moreover I agree with you the choice of "with" keyword could be confused with the "with f(x) as x:" statement in context management, so maybe "with x = f(x)" would cleary makes it different ? Or using a new keyword like "where y = f(x)", "let y = f(x)" or probably better "given y = f(x)", "given" isn't used in current librairies like numpy.where or sql alchemy "where". In the conversation, a lot of people wanted real world examples where it's obvious the new syntax is better. Cheers, Robert
On Wed, Feb 28, 2018 at 4:52 PM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Hello Chris and Rob,
did you compare your proposal tothe subject called "[Python-ideas] Temporary variables in comprehensions" on this month list ?
Yes, I did; that's one of many threads on the subject, and is part of why I'm writing this up. One section that I have yet to write is "alternative syntax proposals", which is where I'd collect all of those together.
If you don't want to go through all the mails, I tried to summarize the ideas in this mail : https://mail.python.org/pipermail/python-ideas/2018-February/048987.html
In a nutshell one of the proposed syntax was
stuff = [(y, y) where y = f(x) for x in range(5)]
Or with your choice of keyword,
stuff = [(y, y) with y = f(x) for x in range(5)]
Your proposal uses the *reverse* syntax :
stuff = [(y, y) with f(x) as y for x in range (5)] ... I wish I could write a pep to summarize all the discussions (including yours, my summary, including real world examples, including pro's and con's), or should I do a gist on GitHub so that we can talk in a more "forum like" manner where people can edit their answers and modify the document ? This mailing is driving me a bit crazy.
This is exactly why I am writing up a PEP. Ultimately, it should list every viable proposal (or group of similar proposals), with the arguments for and against. Contributions of actual paragraphs of text are VERY welcome; simply pointing out "hey, don't forget this one" is also welcome, but I then have to go and write something up, so it'll take a bit longer :)
As Rob pointed out, your syntax "(f(x) as y, y)" is really assymmetric and "(y, y) with f(x) as y" or "(y, y) with y = f(x)" is probably prefered. Moreover I agree with you the choice of "with" keyword could be confused with the "with f(x) as x:" statement in context management, so maybe "with x = f(x)" would cleary makes it different ? Or using a new keyword like "where y = f(x)", "let y = f(x)" or probably better "given y = f(x)", "given" isn't used in current librairies like numpy.where or sql alchemy "where".
And also the standard library's tkinter.dnd.Icon, according to a quick 'git grep'; but that might be just an example, rather than actually being covered by backward-compatibility guarantees. I think "given" is the strongest contender of the three, but I'm just mentioning all three together. A new version of the PEP has been pushed, and should be live within a few minutes. https://www.python.org/dev/peps/pep-0572/ Whatever I've missed, do please let me know. This document should end up incorporating, or at least mentioning, all of the proposals you cited. ChrisA
28.02.18 08:52, Chris Angelico пише:
A new version of the PEP has been pushed, and should be live within a few minutes.
https://www.python.org/dev/peps/pep-0572/
Whatever I've missed, do please let me know. This document should end up incorporating, or at least mentioning, all of the proposals you cited.
I have left comments on https://github.com/python/peps/commit/2cd352673896e84c4d30f22d0829fae65e253e.... Not sure that they are visible to you.
On Thu, Mar 1, 2018 at 2:20 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
28.02.18 08:52, Chris Angelico пише:
A new version of the PEP has been pushed, and should be live within a few minutes.
https://www.python.org/dev/peps/pep-0572/
Whatever I've missed, do please let me know. This document should end up incorporating, or at least mentioning, all of the proposals you cited.
I have left comments on https://github.com/python/peps/commit/2cd352673896e84c4d30f22d0829fae65e253e.... Not sure that they are visible to you.
They are, thanks. I've responded to them all, either inline in the GitHub commit notes, and/or by pushing a change that fixes the issue. Appreciated. ChrisA
Thanks for taking the time to write this PEP, Chris, even though I'm -1 on the idea. I'm glad to just have this as a historical document for the idea. On Tue, 27 Feb 2018 at 22:53 Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Feb 28, 2018 at 4:52 PM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
Hello Chris and Rob,
did you compare your proposal tothe subject called "[Python-ideas] Temporary variables in comprehensions" on this month list ?
Yes, I did; that's one of many threads on the subject, and is part of why I'm writing this up.
One section that I have yet to write is "alternative syntax proposals", which is where I'd collect all of those together.
If you don't want to go through all the mails, I tried to summarize the ideas in this mail : https://mail.python.org/pipermail/python-ideas/2018-February/048987.html
In a nutshell one of the proposed syntax was
stuff = [(y, y) where y = f(x) for x in range(5)]
Or with your choice of keyword,
stuff = [(y, y) with y = f(x) for x in range(5)]
Your proposal uses the *reverse* syntax :
stuff = [(y, y) with f(x) as y for x in range (5)] ... I wish I could write a pep to summarize all the discussions (including yours, my summary, including real world examples, including pro's and con's), or should I do a gist on GitHub so that we can talk in a more "forum like" manner where people can edit their answers and modify the document ? This mailing is driving me a bit crazy.
This is exactly why I am writing up a PEP. Ultimately, it should list every viable proposal (or group of similar proposals), with the arguments for and against. Contributions of actual paragraphs of text are VERY welcome; simply pointing out "hey, don't forget this one" is also welcome, but I then have to go and write something up, so it'll take a bit longer :)
As Rob pointed out, your syntax "(f(x) as y, y)" is really assymmetric and "(y, y) with f(x) as y" or "(y, y) with y = f(x)" is probably prefered. Moreover I agree with you the choice of "with" keyword could be confused with the "with f(x) as x:" statement in context management, so maybe "with x = f(x)" would cleary makes it different ? Or using a new keyword like "where y = f(x)", "let y = f(x)" or probably better "given y = f(x)", "given" isn't used in current librairies like numpy.where or sql alchemy "where".
And also the standard library's tkinter.dnd.Icon, according to a quick 'git grep'; but that might be just an example, rather than actually being covered by backward-compatibility guarantees. I think "given" is the strongest contender of the three, but I'm just mentioning all three together.
A new version of the PEP has been pushed, and should be live within a few minutes.
https://www.python.org/dev/peps/pep-0572/
Whatever I've missed, do please let me know. This document should end up incorporating, or at least mentioning, all of the proposals you cited.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Mar 1, 2018 at 6:54 AM, Brett Cannon <brett@python.org> wrote:
Thanks for taking the time to write this PEP, Chris, even though I'm -1 on the idea. I'm glad to just have this as a historical document for the idea.
I'm going to get a reputation for writing up PEPs for dead ideas. PEP 463 (exception-catching expressions) was the same. In this particular case, I'm fairly in favour of it, but only because I think it's cool - not because I have actual need for it - and if the PEP's rejected, so be it. ChrisA
For what its worth, I'm +1 on it. I actually like that it would allow: while (something() as var): something_else(var) ...without being a bug magnet. The bug magnet isn't the assignment of a name in the condition of a while loop, it's the fact that assignment is a simple typo away from comparison. This is not a simple typo away from comparison (the operands are a different order, too). (sorry Chris, I didn't hit reply all the first time.)
-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+tritium- list=sdamon.com@python.org] On Behalf Of Chris Angelico Sent: Wednesday, February 28, 2018 3:01 PM To: python-ideas <python-ideas@python.org> Subject: Re: [Python-ideas] PEP 572: Statement-Local Name Bindings
On Thu, Mar 1, 2018 at 6:54 AM, Brett Cannon <brett@python.org> wrote:
Thanks for taking the time to write this PEP, Chris, even though I'm -1 on the idea. I'm glad to just have this as a historical document for the idea.
I'm going to get a reputation for writing up PEPs for dead ideas. PEP 463 (exception-catching expressions) was the same. In this particular case, I'm fairly in favour of it, but only because I think it's cool - not because I have actual need for it - and if the PEP's rejected, so be it.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 28 February 2018 at 20:16, Alex Walters <tritium-list@sdamon.com> wrote:
For what its worth, I'm +1 on it.
I actually like that it would allow:
while (something() as var): something_else(var)
...without being a bug magnet. The bug magnet isn't the assignment of a name in the condition of a while loop, it's the fact that assignment is a simple typo away from comparison. This is not a simple typo away from comparison (the operands are a different order, too).
For me, the problem isn't so much with the expected use cases (with the ironic exception that I dislike it for the original motivating example of list comprehensions), it's that any "unusual" use of the construct seems to raise more questions about the semantics than it answers. I remain -1, I'm afraid. Paul
On 02/28/2018 12:16 PM, Alex Walters wrote:
For what its worth, I'm +1 on it.
I actually like that it would allow:
while (something() as var): something_else(var)
...without being a bug magnet. The bug magnet isn't the assignment of a name in the condition of a while loop, it's the fact that assignment is a simple typo away from comparison. This is not a simple typo away from comparison (the operands are a different order, too).
I also like the above, but as a more general assignment-in-expression, not as a statement-local -- the difference being that there would be no auto-deletion of the variable, no confusion about when it goes away, no visual name collisions (aside from we have already), etc., etc. Maybe I'll write yet-another-competing-PEP. ;) -- ~Ethan~
On Tue, Feb 27, 2018 at 2:35 PM Chris Angelico <rosuav@gmail.com> wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
https://www.python.org/dev/peps/pep-0572/
ChrisA
PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018
Abstract ========
Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions.
Rationale =========
When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times.
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name.
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement.
Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides.
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
# Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear?
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion.
References ==========
.. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables)
-1 today My first concern for this proposal is that it gives parenthesis a non-intuitive meaning. ()s used to be innocuous. A way to group things, make explicit the desired order of operations, and allow spanning across multiple lines. With this proposal, ()s occasionally take on a new meaning to cause additional action to happen _outside_ of the ()s - an expression length name binding. Does this parse? print(fetch_the_comfy_chair(cardinal) as chair, "==", chair) SyntaxError? My read of the proposal suggests that this would be required: print((fetch_the_comfy_chair(cardinal) as chair), "==", chair) But that sets off my "excess unnecessary parenthesis" radar as this is a new need it isn't trained for. I could retrain the radar... I see people try to cram too much functionality into one liners. By the time you need to refer to a side effect value more than once in a single statement... Just use multiple statements. It is more clear and easier to debug. Anything else warps human minds trying to read, understand, review, and maintain the code. {see_also: "nested list comprehensions"} '''insert favorite Zen of Python quotes here''' We've got a whitespace delimited language. That is a feature. Lets keep it that way rather than adding a form of (non-curly) braces. I do understand the desire. So far I remain unconvinced of a need. Practical examples from existing code that become clearly easier to understand afterwards instead of made the examples with one letter names may help. 2cents, -gps
On Wed, Feb 28, 2018 at 5:53 PM, Gregory P. Smith <greg@krypto.org> wrote:
My first concern for this proposal is that it gives parenthesis a non-intuitive meaning. ()s used to be innocuous. A way to group things, make explicit the desired order of operations, and allow spanning across multiple lines.
With this proposal, ()s occasionally take on a new meaning to cause additional action to happen _outside_ of the ()s - an expression length name binding.
Does this parse?
print(fetch_the_comfy_chair(cardinal) as chair, "==", chair)
SyntaxError? My read of the proposal suggests that this would be required:
print((fetch_the_comfy_chair(cardinal) as chair), "==", chair)
But that sets off my "excess unnecessary parenthesis" radar as this is a new need it isn't trained for. I could retrain the radar...
Hmm, interesting point. The value to be captured DOES need to be grouped, though, and the influence of the assignment is "until end of statement", which is usually fairly clear. Parentheses do already have quite a number of meanings. For instance, a function call does more than simply group a subexpression - there's a lot of difference between x=(1, 2, 3) and x(1, 2, 3) even before you look at all the other syntaxes that can be used inside function calls (eg keyword arguments). The way I see it, this proposal is about the "as" keyword being used for local name binding, and the purpose of the parentheses is to group the name with the value being bound.
I see people try to cram too much functionality into one liners. By the time you need to refer to a side effect value more than once in a single statement... Just use multiple statements. It is more clear and easier to debug.
This concern comes up frequently, and there are definitely times when it's true. But there are also times when the opposite is true, and pushing more syntax into a single logical action makes code MORE readable. For instance, Python allows us to quickly zero out a bunch of variables all at once: a = b = c = d = e = 0 Can this be abused? Sure! But when it's used correctly, it is far MORE readable than laying everything out vertically: a = 0 b = 0 c = 0 d = 0 e = 0 Code is attempting to express abstract ideas. If your idea of "readable code" is "code that makes it easy to see the concrete actions being done", you're missing out on a lot of the power of Python.
We've got a whitespace delimited language. That is a feature. Lets keep it that way rather than adding a form of (non-curly) braces.
I don't fully understand you here. Python has always used symbols as delimiters, and I doubt anyone would want to do things any differently.
Practical examples from existing code that become clearly easier to understand afterwards instead of made the examples with one letter names may help.
Fair enough. I'll try to dig some up. The trouble is that practical examples are seldom as clear and simple as the shorter examples are. ChrisA
I have been struggling to justify the need based on what I have read. I hope this isn't a dupe, I only saw caching mentioned in passing. Also please excuse some of the naive generalizations below for illustrative purposes. Is there a reason memoization doesn't work? If f is truly expensive, using this syntax only makes sense if you have few comprehensions (as opposed to many iterations) and few other calls to f. Calling f in 10 comprehensions would (naively) benefit from memoization more than this. It appears to me to be ad-hoc memoization with limited scope. is this a fair statement?
From readability, the examples put forth have been to explain the advantage, with which I agree. However, i do not believe this scales well.
[(foo(x,y) as g)*(bar(y) as i) + g*foo(x,a) +baz(g,i) for x... for y...] That's 3 functions, 2 iterators, 3 calls saved ('a' is some constant just to trigger a new call on foo). I'm not trying to show ugly statements can be constructed, but show how quickly in _n iterators and _m functions readability declines. it's seems the utility is bounded by f being not complex/costly enough for memoization, and ( _m, _n) being small enough to pass code review. The syntax does not create a readability issue, but by adding symbols, exacerbates an inherent issue with 'complex' comprehensions. If I have understood, does this bounding on f, _m, and _n yield a tool with sufficient applicability for a language change? I know it's already spoken about as an edge case, I'm just not clear on the bounds of that. I am not against it, I just haven't seen this addressed. Thank you for putting the PEP together. Again, I don't want to sound negative on it, I may have misunderstood wildly. I like the idea conceptually, and I don't think it's anymore confusing to me than comprehensions were when I first encountered them, and it will yield a 'cool!' statement much like they did when I learned them. On Tue, Feb 27, 2018, 22:55 Gregory P. Smith <greg@krypto.org> wrote:
On Tue, Feb 27, 2018 at 2:35 PM Chris Angelico <rosuav@gmail.com> wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
https://www.python.org/dev/peps/pep-0572/
ChrisA
PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018
Abstract ========
Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions.
Rationale =========
When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times.
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name.
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement.
Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides.
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
# Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear?
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion.
References ==========
.. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables)
-1 today
My first concern for this proposal is that it gives parenthesis a non-intuitive meaning. ()s used to be innocuous. A way to group things, make explicit the desired order of operations, and allow spanning across multiple lines.
With this proposal, ()s occasionally take on a new meaning to cause additional action to happen _outside_ of the ()s - an expression length name binding.
Does this parse?
print(fetch_the_comfy_chair(cardinal) as chair, "==", chair)
SyntaxError? My read of the proposal suggests that this would be required:
print((fetch_the_comfy_chair(cardinal) as chair), "==", chair)
But that sets off my "excess unnecessary parenthesis" radar as this is a new need it isn't trained for. I could retrain the radar...
I see people try to cram too much functionality into one liners. By the time you need to refer to a side effect value more than once in a single statement... Just use multiple statements. It is more clear and easier to debug.
Anything else warps human minds trying to read, understand, review, and maintain the code. {see_also: "nested list comprehensions"}
'''insert favorite Zen of Python quotes here'''
We've got a whitespace delimited language. That is a feature. Lets keep it that way rather than adding a form of (non-curly) braces.
I do understand the desire. So far I remain unconvinced of a need.
Practical examples from existing code that become clearly easier to understand afterwards instead of made the examples with one letter names may help.
2cents, -gps
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Feb 28, 2018 at 6:46 PM, Matt Arcidy <marcidy@gmail.com> wrote:
I have been struggling to justify the need based on what I have read. I hope this isn't a dupe, I only saw caching mentioned in passing.
Also please excuse some of the naive generalizations below for illustrative purposes.
Is there a reason memoization doesn't work? If f is truly expensive, using this syntax only makes sense if you have few comprehensions (as opposed to many iterations) and few other calls to f. Calling f in 10 comprehensions would (naively) benefit from memoization more than this. It appears to me to be ad-hoc memoization with limited scope. is this a fair statement?
Memoization is only an option if the expression in question is (a) a single cacheable function call, and (b) used twice without any variation. If it's any sort of more complicated expression, that concept doesn't work. ChrisA
I appreciate that point as it is what I must be misunderstanding. I believe the performance speed up of [((f(x) as h), g(h)) for x range(10)] is that there are 10 calls to compute f, not 20. You can do this with a dictionary right now, at least for the example we're talking about: [(d[x], g(d[x])) for x in range(10) if d.update({x:f(x)}) is None] It's ugly but get's the job done. The proposed syntax is leagues better than that, but just to give my point a concrete example. If f is memoized, isn't [( f(x), g(f(x)) ) for range(10)] the same? You compute f 10 times, not 20. You get the second f(x) as cache retrieval instead of recomputing it, precisely because the argument x is the same. Here I can use 'd' later if needed, as opposed to with the proposal. That's really my point about memoization (or cacheing). If 'f' is really expensive, I don't really see the point of using an ad-hoc local caching of the value that lives just for one statement when I could use it where-ever, even persist it if it makes sense. I fully admit I'm at my depth here, so I can comfortably concede it's better than memoization and I just don't understand! On Wed, Feb 28, 2018 at 12:37 AM, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Feb 28, 2018 at 6:46 PM, Matt Arcidy <marcidy@gmail.com> wrote:
I have been struggling to justify the need based on what I have read. I hope this isn't a dupe, I only saw caching mentioned in passing.
Also please excuse some of the naive generalizations below for illustrative purposes.
Is there a reason memoization doesn't work? If f is truly expensive, using this syntax only makes sense if you have few comprehensions (as opposed to many iterations) and few other calls to f. Calling f in 10 comprehensions would (naively) benefit from memoization more than this. It appears to me to be ad-hoc memoization with limited scope. is this a fair statement?
Memoization is only an option if the expression in question is (a) a single cacheable function call, and (b) used twice without any variation. If it's any sort of more complicated expression, that concept doesn't work.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Feb 28, 2018 at 8:55 PM, Matt Arcidy <marcidy@gmail.com> wrote:
I appreciate that point as it is what I must be misunderstanding.
I believe the performance speed up of [((f(x) as h), g(h)) for x range(10)] is that there are 10 calls to compute f, not 20.
You can do this with a dictionary right now, at least for the example we're talking about: [(d[x], g(d[x])) for x in range(10) if d.update({x:f(x)}) is None]
It's ugly but get's the job done. The proposed syntax is leagues better than that, but just to give my point a concrete example.
Definitely ugly... if I saw that in code review, I'd ask "When is that condition going to be false?". It also offers nothing that the "extra 'for' loop" syntax can't do better. Memoization can only be done for pure functions. If the call in question is actually, say, "next(iter)", you can't memoize it to avoid duplicate calls. Possibly not the greatest example (since you could probably zip() to solve all those sorts of cases), but anything else that has side effects could do the same thing. ChrisA
On Tue, Feb 27, 2018 at 11:46 PM, Matt Arcidy <marcidy@gmail.com> wrote:
From readability, the examples put forth have been to explain the advantage, with which I agree. However, i do not believe this scales well.
[(foo(x,y) as g)*(bar(y) as i) + g*foo(x,a) +baz(g,i) for x... for y...]
This definitely looks hard to read. Let's compare it to: lst = [] for x in something: for y in other_thing: g = f(x, y) i = bar(y) lst.append(g*foo(x,a) + baz(g,i)) Obviously the one-liner is shorter, but the full loop looks a heck of a lot more readable to me. I was thinking of an example closer to the PEP like this: [((my_object.calculate_the_quantity(quant1, vect2, arr3) as x), log(x)) for quant1 in quants] Just one "as" clause, but a long enough expression I wouldn't want to repeat it. I still feel this suffers in readability compared to the existing option of (even as a non-unrolled comprehension): [(x, log(x)) for x in (my_object.calculate_the_quantity(quant1, vect2, arr3) for quant1 in quants)] Sure, again we save a couple characters under the PEP, but readability feels harmed not helped. And most likely this is another thing better spelled as a regular loop. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 2018-02-28 02:46, Matt Arcidy wrote:
From readability, the examples put forth have been to explain the advantage, with which I agree. However, i do not believe this scales well.
[(foo(x,y) as g)*(bar(y) as i) + g*foo(x,a) +baz(g,i) for x... for y...]
That's 3 functions, 2 iterators, 3 calls saved ('a' is some constant just to trigger a new call on foo). I'm not trying to show ugly statements can be constructed, but show how quickly in _n iterators and _m functions readability declines.
You could put it on multiple lines
[ (g * i) + g * foo(x, a) + baz(g, i) for x in X for y in Y for g in [foo(x,y)] for i in [bar(y)] ]
and then notice a common factor! :)
[ g * (i + foo(x, a) + baz(g, i)) for x in X for y in Y for g in [foo(x,y)] for i in [bar(y)] ]
On 27 February 2018 at 22:27, Chris Angelico <rosuav@gmail.com> wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
https://www.python.org/dev/peps/pep-0572/
ChrisA
PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018
Thanks for writing this - as you mention, this will be a useful document even if the proposal ultimately gets rejected. FWIW, I'm currently -1 on this, so "rejected" is what I'm expecting, but it's possible that subsequent discussions could refine this into something that people are happy with (or that I'm in the minority, of course :-))
Abstract ========
Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions.
In the context of multi-line statements like with, or while, describing this as a "very small scope" is inaccurate (and maybe even misleading). See later for an example using def.
Rationale =========
When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times.
I agree with Rob Cliffe, this is a bit overstated. How about "there are currently several ways to spell this, none of which is universally accepted as ideal". Specifically, the point to me is that opinions are (strongly) divided, rather than that everyone agrees that it's a problem but we haven't found a good solution yet.
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name.
I agree with requiring parentheses - although I dislike the tendency towards "mandatory parentheses" in recent syntax proposals :-( But "1 as x + 1 as y" is an abomination. Conversely "sqrt((1 as x))" is a little annoying. So it feels like a case of the lesser of two evils to me, rather than an actually good idea...
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement.
While there's basically no justification for doing so, it should be noted that under this proposal, ((((((((1 as x) as y) as z) as w) as v) as u) as t) as s) is valid. Of course, "you can write confusing code using this" isn't an argument against a useful enhancement, but potential for abuse is something to be aware of. There's also (slightly more realistically) something like [(sqrt((b*b as bsq) + (4*a*c as fourac)) as root1), (sqrt(bsq - fourac) as root2)], which I can see someone thinking is a good idea! The question here is whether the readability of "reasonable" uses of the construct is sufficient to outweigh the risk of well-intentioned misuses.
Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides.
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
# Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
Honestly, the asymmetry in [(f(x) as y), y] makes this the *least* readable option to me :-( All of the other options clearly show that the 2 elements of the list are the same, but the statement-local name version requires me to stop and think to confirm that it's a list of 2 copies of the same value.
If calling `f(x)` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra `for` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning.
"retains the simplicity" is subjective. I'd prefer something like "makes it clear that f(x) is called only once". Of course, all of the other options to this too, the main question is whether it's as clear as in the named subexpression version.
Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
IMO, relying on evaluation order is the only viable option, but it's confusing. I would immediately reject something like `(x, (1 as x), x)` as bad style, simply because the meaning of x at the two places it is used is non-obvious. I'm -1 on a warning. I'd prefer an error, but I can't see how you'd implement (or document) it.
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
I'm strongly -1 on "works like the current implementation" as a definition of the behaviour. While having a proof of concept to clarify behaviour is great, my first question would be "how is the behaviour documented to work?" So what is the PEP proposing would happen if I did if ('.'.join((str(x) for x in sys.version_info[:2])) as ver) == '3.6': # Python 3.6 specific code here elif sys.version_info[0] < 3: print(f"Version {ver} is not supported") at the top level of a Python file? To me, that's a perfectly reasonable way of using the new feature to avoid exposing a binding for "ver" in my module...
3. The interaction with locals() is currently[1] slightly buggy. Should statement-local names appear in locals() while they are active (and shadow any other names from the same function), or should they simply not appear?
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
Wait - except (Exception as e): would set e to the type Exception, and not capture the actual exception object? Even though that's unambiguous, it's *incredibly* confusing. But saying we allow "except <expr-but-not-a-name-binding>" is also bad, in the sense that it's an annoying exception (sic) to have to include in the documentation. Maybe it would be viable to say that a (top-level) expression can never be a name binding - after all, there's no point because the name will be immediately discarded. Only allow name bindings in subexpressions. That would disallow this case as part of that general rule. But I've no idea what that rule would do to the grammar (in particular, whether it would still be possible to parse without lookahead). (Actually no, this would prohibit constructs such as `while (f(x) as val) > 0`, which I presume you're trying to support[1], although you don't mention this in the rationale or example usage sections). [1] Based on the fact that you want the name binding to remain active for the enclosing *statement*, not just the enclosing *expression*.
5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion.
This seems to imply that the name in (expr as name) when used as a top level expression will persist after the closing parenthesis. Is that intended? It's not mentioned anywhere in the PEP (that I could see). On re-reading, I see that you say "for the remainder of the current *statement*" and not (as I had misread it) the remainder of the current *expression*. So multi-line statements will give it a larger scope? That strikes me as giving this proposal a much wider applicability than implied by the summary. Consider def f(x, y=(object() as default_value)): if y is default_value: print("You did not supply y") That's an "obvious" use of the new feature, and I could see it very quickly becoming the standard way to define sentinel values. I *think* it's a reasonable idiom, but honestly, I'm not sure. It certainly feels like scope creep from the original use case, which was naming bits of list comprehensions. Overall, it feels like the semantics of having the name bindings persist for the enclosing *statement* rather than the enclosing *expression* is a significant extension of the scope of the proposal, to the extent that the actual use cases that it would allow are mostly not mentioned in the rationale, and using it in comprehensions becomes merely yet another suboptimal solution to the problem of reusing calculated values in comprehensions!!! So IMO, either the semantics should be reduced to exposing the binding to just the enclosing top-level expression, or the rationale, use cases and examples should be significantly beefed up to reflect the much wider applicability of the feature (and then you'd need to address any questions that arise from that wider scope). Paul
On Wed, Feb 28, 2018 at 10:49 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 February 2018 at 22:27, Chris Angelico <rosuav@gmail.com> wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
https://www.python.org/dev/peps/pep-0572/
ChrisA
PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018
Thanks for writing this - as you mention, this will be a useful document even if the proposal ultimately gets rejected.
FWIW, I'm currently -1 on this, so "rejected" is what I'm expecting, but it's possible that subsequent discussions could refine this into something that people are happy with (or that I'm in the minority, of course :-))
TBH, I'm no more than +0.5 on the proposal being accepted, but a very strong +1 on it being all written up in a PEP :)
Abstract ========
Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with very small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against collisions.
In the context of multi-line statements like with, or while, describing this as a "very small scope" is inaccurate (and maybe even misleading). See later for an example using def.
True. When I first wrote that paragraph, I wasn't thinking of compound statements at all. They are, however, a powerful use of the new syntax IMO. I'll remove the word "very" from there; "small scope" is still the intention though, as the point of this is to be a subscope within a (larger) function scope.
Rationale =========
When an expression is used multiple times in a list comprehension, there are currently several suboptimal ways to spell this, and no truly good ways. A statement-local name allows any expression to be temporarily captured and then used multiple times.
I agree with Rob Cliffe, this is a bit overstated.
How about "there are currently several ways to spell this, none of which is universally accepted as ideal". Specifically, the point to me is that opinions are (strongly) divided, rather than that everyone agrees that it's a problem but we haven't found a good solution yet.
I can run with that wording. Thanks.
Syntax and semantics ====================
In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form `(expr as NAME)` where `expr` is any valid Python expression, and `NAME` is a simple name.
I agree with requiring parentheses - although I dislike the tendency towards "mandatory parentheses" in recent syntax proposals :-( But "1 as x + 1 as y" is an abomination. Conversely "sqrt((1 as x))" is a little annoying. So it feels like a case of the lesser of two evils to me, rather than an actually good idea...
The sqrt example could be changed in the future (either before or after the PEP's acceptance). It's like a genexp - parens mandatory but function calls are special-cased.
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement.
While there's basically no justification for doing so, it should be noted that under this proposal, ((((((((1 as x) as y) as z) as w) as v) as u) as t) as s) is valid. Of course, "you can write confusing code using this" isn't an argument against a useful enhancement, but potential for abuse is something to be aware of. There's also (slightly more realistically) something like [(sqrt((b*b as bsq) + (4*a*c as fourac)) as root1), (sqrt(bsq - fourac) as root2)], which I can see someone thinking is a good idea!
Sure! Though I'm not sure what you're representing there; it looks almost, but not quite, like the quadratic formula. If that was the intention, I'd be curious to see the discriminant broken out, with some kind of trap for the case where it's negative.
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
# Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
Honestly, the asymmetry in [(f(x) as y), y] makes this the *least* readable option to me :-( All of the other options clearly show that the 2 elements of the list are the same, but the statement-local name version requires me to stop and think to confirm that it's a list of 2 copies of the same value.
I need some real-world examples where it's not as trivial as [y, y] so people don't get hung up on the symmetry issue.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
IMO, relying on evaluation order is the only viable option, but it's confusing. I would immediately reject something like `(x, (1 as x), x)` as bad style, simply because the meaning of x at the two places it is used is non-obvious.
I'm -1 on a warning. I'd prefer an error, but I can't see how you'd implement (or document) it.
Sure. For now, I'm just going to leave it as a perfectly acceptable use of the feature; it can be rejected as poor style, but permitted by the language.
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
I'm strongly -1 on "works like the current implementation" as a definition of the behaviour. While having a proof of concept to clarify behaviour is great, my first question would be "how is the behaviour documented to work?" So what is the PEP proposing would happen if I did
if ('.'.join((str(x) for x in sys.version_info[:2])) as ver) == '3.6': # Python 3.6 specific code here elif sys.version_info[0] < 3: print(f"Version {ver} is not supported")
at the top level of a Python file? To me, that's a perfectly reasonable way of using the new feature to avoid exposing a binding for "ver" in my module...
I agree, sounds good. I'll reword this to be a limitation of implementation.
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
Wait - except (Exception as e): would set e to the type Exception, and not capture the actual exception object?
Correct. The expression "Exception" evaluates to the type Exception, and you can capture that. It's a WutFace moment but it's a logical consequence of the nature of Python.
Even though that's unambiguous, it's *incredibly* confusing. But saying we allow "except <expr-but-not-a-name-binding>" is also bad, in the sense that it's an annoying exception (sic) to have to include in the documentation.
Agreed. I don't want to special-case it out; this is something for code review to catch. Fortunately, this would give fairly obvious results - you try to do something with the exception, and you don't actually have an exception object, you have <class 'Exception'>. It's a little more problematic in a "with" block, because it'll often do the same thing.
Maybe it would be viable to say that a (top-level) expression can never be a name binding - after all, there's no point because the name will be immediately discarded. Only allow name bindings in subexpressions. That would disallow this case as part of that general rule. But I've no idea what that rule would do to the grammar (in particular, whether it would still be possible to parse without lookahead). (Actually no, this would prohibit constructs such as `while (f(x) as val) > 0`, which I presume you're trying to support[1], although you don't mention this in the rationale or example usage sections).
[1] Based on the fact that you want the name binding to remain active for the enclosing *statement*, not just the enclosing *expression*.
Not sure what you mean by a "top-level expression", if it's going to disallow 'while (f(x) as val) > 0:'. Can you elaborate?
5. Similar confusion in `with` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for `__enter__` methods to return `self`. In many cases, `with expr as name:` will do the same thing as `with (expr as name):`, adding to the confusion.
This seems to imply that the name in (expr as name) when used as a top level expression will persist after the closing parenthesis. Is that intended? It's not mentioned anywhere in the PEP (that I could see). On re-reading, I see that you say "for the remainder of the current *statement*" and not (as I had misread it) the remainder of the current *expression*.
Yep. If you have an expression on its own, it's an "expression statement", and the subscope will end at the newline (or the semicolon, if you have one). Inside something larger, it'll persist.
So multi-line statements will give it a larger scope? That strikes me as giving this proposal a much wider applicability than implied by the summary. Consider
def f(x, y=(object() as default_value)): if y is default_value: print("You did not supply y")
That's an "obvious" use of the new feature, and I could see it very quickly becoming the standard way to define sentinel values. I *think* it's a reasonable idiom, but honestly, I'm not sure. It certainly feels like scope creep from the original use case, which was naming bits of list comprehensions.
Eeeuaghh.... okay. Now I gotta think about this one. The 'def' statement is an executable one, to be sure; but the statement doesn't include *running* the function, only *creating* it. So as you construct the function, default_value has that value. Inside the actual running of it, that name doesn't exist any more. So this won't actually work. But you could use this to create annotations and such, I guess...
def f(): ... def g(x=(object() as default_value)) -> default_value: ... ... ... return g ... f().__annotations__ {'return': <object object at 0x7fe37dab2280>} dis.dis(f) 2 0 LOAD_GLOBAL 0 (object) 2 CALL_FUNCTION 0 4 DUP_TOP 6 STORE_FAST 0 (default_value) 8 BUILD_TUPLE 1 10 LOAD_FAST 0 (default_value) 12 LOAD_CONST 1 (('return',)) 14 BUILD_CONST_KEY_MAP 1 16 LOAD_CONST 2 (<code object g at 0x7fe37d974ae0, file "<stdin>", line 2>) 18 LOAD_CONST 3 ('f.<locals>.g') 20 MAKE_FUNCTION 5 22 STORE_FAST 1 (g) 24 DELETE_FAST 0 (default_value)
4 26 LOAD_FAST 1 (g) 28 RETURN_VALUE Disassembly of <code object g at 0x7fe37d974ae0, file "<stdin>", line 2>: 3 0 LOAD_CONST 0 (None) 2 RETURN_VALUE Not terribly useful.
Overall, it feels like the semantics of having the name bindings persist for the enclosing *statement* rather than the enclosing *expression* is a significant extension of the scope of the proposal, to the extent that the actual use cases that it would allow are mostly not mentioned in the rationale, and using it in comprehensions becomes merely yet another suboptimal solution to the problem of reusing calculated values in comprehensions!!!
So IMO, either the semantics should be reduced to exposing the binding to just the enclosing top-level expression, or the rationale, use cases and examples should be significantly beefed up to reflect the much wider applicability of the feature (and then you'd need to address any questions that arise from that wider scope).
I'll add some more examples. I think the if/while usage is potentially of value. Thanks for the feedback! Keep it coming! :) ChrisA
On 28 February 2018 at 13:45, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Feb 28, 2018 at 10:49 PM, Paul Moore <p.f.moore@gmail.com> wrote:
While there's basically no justification for doing so, it should be noted that under this proposal, ((((((((1 as x) as y) as z) as w) as v) as u) as t) as s) is valid. Of course, "you can write confusing code using this" isn't an argument against a useful enhancement, but potential for abuse is something to be aware of. There's also (slightly more realistically) something like [(sqrt((b*b as bsq) + (4*a*c as fourac)) as root1), (sqrt(bsq - fourac) as root2)], which I can see someone thinking is a good idea!
Sure! Though I'm not sure what you're representing there; it looks almost, but not quite, like the quadratic formula. If that was the intention, I'd be curious to see the discriminant broken out, with some kind of trap for the case where it's negative.
lol, it was meant to be the quadratic roots. If I got it wrong, that probably says something about how hard it is top maintain or write code that over-uses the proposed feature ;-) If I didn't get it wrong, that still makes the same point, I guess!
Honestly, the asymmetry in [(f(x) as y), y] makes this the *least* readable option to me :-( All of the other options clearly show that the 2 elements of the list are the same, but the statement-local name version requires me to stop and think to confirm that it's a list of 2 copies of the same value.
I need some real-world examples where it's not as trivial as [y, y] so people don't get hung up on the symmetry issue.
Well, I agree real world examples would be a lot more compelling here, but I don't necessarily agree that the asymmetry won't remain an issue. In practice, I've only ever wanted a feature like this when hacking at the command line, or writing *extremely* hacky one-off scripts. So any sort of demonstration of code that exists in a maintained, production codebase which would actually benefit from this feature would be a major advantage here.
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
IMO, relying on evaluation order is the only viable option, but it's confusing. I would immediately reject something like `(x, (1 as x), x)` as bad style, simply because the meaning of x at the two places it is used is non-obvious.
I'm -1 on a warning. I'd prefer an error, but I can't see how you'd implement (or document) it.
Sure. For now, I'm just going to leave it as a perfectly acceptable use of the feature; it can be rejected as poor style, but permitted by the language.
"Perfectly acceptable" I disagree with. "Unacceptable but impossible to catch in the compiler" is closer to my view. What I'm concerned with is less dealing with code that is written like that (delete it as soon as you see it is the only practical answer ;-)) but rather clearly documenting the feature without either drowning the reader in special cases and clarifications, or leaving important points unspecified or difficult to find the definition for.
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
I'm strongly -1 on "works like the current implementation" as a definition of the behaviour. While having a proof of concept to clarify behaviour is great, my first question would be "how is the behaviour documented to work?" So what is the PEP proposing would happen if I did
if ('.'.join((str(x) for x in sys.version_info[:2])) as ver) == '3.6': # Python 3.6 specific code here elif sys.version_info[0] < 3: print(f"Version {ver} is not supported")
at the top level of a Python file? To me, that's a perfectly reasonable way of using the new feature to avoid exposing a binding for "ver" in my module...
I agree, sounds good. I'll reword this to be a limitation of implementation.
To put it another way, "Intended to work, but we haven't determined how to implement it yet"? Fair enough, although it needs to be possible to implement it. These names are a weird not-quite-scope construct, and interactions with real scopes are going to be tricky to get right (not just implement, but define). Consider x = 12 if (1 as x) == 1: def foo(): print(x) # Actually, you'd probably get a "Name used before definition" error here. # Would "global x" refer to x=12 or to the statement-local x (1)? # Would "nonlocal x" refer to the statement-local x? x = 13 return x print(foo()) print(x) print(foo()) print(x) What should that return? Not "what does the current implementation return", but what is the intended result according to the proposal, and how would you explain that result in the docs? I think I'd expect 1 13 # But see note about global/nonlocal 12 1 xxxxxxx Not sure? Maybe 1? Can you create a closure over a statement-local variable? 13 # But see note about global/nonlocal 12 The most charitable thing I can say here is that the semantics are currently under-specified in the PEP :-)
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
Wait - except (Exception as e): would set e to the type Exception, and not capture the actual exception object?
Correct. The expression "Exception" evaluates to the type Exception, and you can capture that. It's a WutFace moment but it's a logical consequence of the nature of Python.
"Logical consequence of the rules" isn't enough for a Python language feature, IMO. Intuitive and easy to infer are key. Even if this is a corner case, it counts as a mildly ugly wart to me.
Even though that's unambiguous, it's *incredibly* confusing. But saying we allow "except <expr-but-not-a-name-binding>" is also bad, in the sense that it's an annoying exception (sic) to have to include in the documentation.
Agreed. I don't want to special-case it out; this is something for code review to catch. Fortunately, this would give fairly obvious results - you try to do something with the exception, and you don't actually have an exception object, you have <class 'Exception'>. It's a little more problematic in a "with" block, because it'll often do the same thing.
I value Python for making it easy to write correct code, not easy to spot your errors. Too many hings like this would start me thinking I should ban statement-local names from codebases I maintain, which is not a good start for a feature...
Maybe it would be viable to say that a (top-level) expression can never be a name binding - after all, there's no point because the name will be immediately discarded. Only allow name bindings in subexpressions. That would disallow this case as part of that general rule. But I've no idea what that rule would do to the grammar (in particular, whether it would still be possible to parse without lookahead). (Actually no, this would prohibit constructs such as `while (f(x) as val) > 0`, which I presume you're trying to support[1], although you don't mention this in the rationale or example usage sections).
[1] Based on the fact that you want the name binding to remain active for the enclosing *statement*, not just the enclosing *expression*.
Not sure what you mean by a "top-level expression", if it's going to disallow 'while (f(x) as val) > 0:'. Can you elaborate?
Actually, I was wrong. The top level expression in '(f(x) as val) > 0' is the comparison, so the while usage survives. In 'exception (Exception as e)', the top-level expression is '(Exception as e)', so we can ban name bindings in top-level expressions and kill that. But all this feels pretty non-intuitive. Squint hard and you can work out how the rules mean what you think they mean, but it doesn't feel "obvious" (dare I say "Pythonic"?)
This seems to imply that the name in (expr as name) when used as a top level expression will persist after the closing parenthesis. Is that intended? It's not mentioned anywhere in the PEP (that I could see). On re-reading, I see that you say "for the remainder of the current *statement*" and not (as I had misread it) the remainder of the current *expression*.
Yep. If you have an expression on its own, it's an "expression statement", and the subscope will end at the newline (or the semicolon, if you have one). Inside something larger, it'll persist.
Technically you can have more than one expression in a statement. Consider (from the grammar): for_stmt ::= "for" target_list "in" expression_list ":" suite ["else" ":" suite] expression_list ::= expression ( "," expression )* [","] Would a name binding in the first expression in an expression_list be visible in the second expression? Should it be? It will be, because it's visible to the end of the statement, not to the end of the expression, but there might be subtle technical implications here (I haven't checked the grammar to see what other statements allow multiple expressions - that's your job ;-)) To clarify this sort of question, you should probably document in the PEP precisely how the grammar will be modified.
So multi-line statements will give it a larger scope? That strikes me as giving this proposal a much wider applicability than implied by the summary. Consider
def f(x, y=(object() as default_value)): if y is default_value: print("You did not supply y")
That's an "obvious" use of the new feature, and I could see it very quickly becoming the standard way to define sentinel values. I *think* it's a reasonable idiom, but honestly, I'm not sure. It certainly feels like scope creep from the original use case, which was naming bits of list comprehensions.
Eeeuaghh.... okay. Now I gotta think about this one.
The 'def' statement is an executable one, to be sure; but the statement doesn't include *running* the function, only *creating* it. So as you construct the function, default_value has that value. Inside the actual running of it, that name doesn't exist any more. So this won't actually work. But you could use this to create annotations and such, I guess...
lol, see? Closures rear their ugly heads, as I mentioned above.
def f(): ... def g(x=(object() as default_value)) -> default_value: ... ... ... return g ... f().__annotations__ {'return': <object object at 0x7fe37dab2280>} dis.dis(f) 2 0 LOAD_GLOBAL 0 (object) 2 CALL_FUNCTION 0 4 DUP_TOP 6 STORE_FAST 0 (default_value) 8 BUILD_TUPLE 1 10 LOAD_FAST 0 (default_value) 12 LOAD_CONST 1 (('return',)) 14 BUILD_CONST_KEY_MAP 1 16 LOAD_CONST 2 (<code object g at 0x7fe37d974ae0, file "<stdin>", line 2>) 18 LOAD_CONST 3 ('f.<locals>.g') 20 MAKE_FUNCTION 5 22 STORE_FAST 1 (g) 24 DELETE_FAST 0 (default_value)
4 26 LOAD_FAST 1 (g) 28 RETURN_VALUE
Disassembly of <code object g at 0x7fe37d974ae0, file "<stdin>", line 2>: 3 0 LOAD_CONST 0 (None) 2 RETURN_VALUE
Not terribly useful.
"What the current proof of concept implementation does" isn't useful anyway, but even ignoring that I'd prefer to see what it *does* rather than what it *compiles to*. But what needs to be documented is what the PEP *proposes* it does.
I'll add some more examples. I think the if/while usage is potentially of value.
I think it's an unexpected consequence of an overly-broad solution to the original problem, that accidentally solves another long-running debate. But it means you've opened a much bigger can of worms than it originally appeared, and I'm not sure you don't risk losing the simplicity that *expression* local names might have had. But I can even break expression local names: x = ((lambda: boom()) as boom) x() It's possible that the "boom" is just my head exploding, not the interpreter. But I think I just demonstrated a closure over an expression-local name. For added fun, replace "x" with "boom"...
Thanks for the feedback! Keep it coming! :)
Ask and you shall receive :-) Paul
On Thu, Mar 1, 2018 at 3:30 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 February 2018 at 13:45, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Feb 28, 2018 at 10:49 PM, Paul Moore <p.f.moore@gmail.com> wrote:
While there's basically no justification for doing so, it should be noted that under this proposal, ((((((((1 as x) as y) as z) as w) as v) as u) as t) as s) is valid. Of course, "you can write confusing code using this" isn't an argument against a useful enhancement, but potential for abuse is something to be aware of. There's also (slightly more realistically) something like [(sqrt((b*b as bsq) + (4*a*c as fourac)) as root1), (sqrt(bsq - fourac) as root2)], which I can see someone thinking is a good idea!
Sure! Though I'm not sure what you're representing there; it looks almost, but not quite, like the quadratic formula. If that was the intention, I'd be curious to see the discriminant broken out, with some kind of trap for the case where it's negative.
lol, it was meant to be the quadratic roots. If I got it wrong, that probably says something about how hard it is top maintain or write code that over-uses the proposed feature ;-) If I didn't get it wrong, that still makes the same point, I guess!
Or, more likely, it says something about what happens when a programmer bashes out some code to try to represent a famous formula, but doesn't actually debug it. As is often said, code that isn't tested is buggy :) Here's another equally untested piece of code: [(-b + (sqrt(b*b - 4*a*c) as disc)) / (2*a), (-b - disc) / (2*a)]
Open questions ==============
1. What happens if the name has already been used? `(x, (1 as x), x)` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning?
IMO, relying on evaluation order is the only viable option, but it's confusing. I would immediately reject something like `(x, (1 as x), x)` as bad style, simply because the meaning of x at the two places it is used is non-obvious.
I'm -1 on a warning. I'd prefer an error, but I can't see how you'd implement (or document) it.
Sure. For now, I'm just going to leave it as a perfectly acceptable use of the feature; it can be rejected as poor style, but permitted by the language.
"Perfectly acceptable" I disagree with. "Unacceptable but impossible to catch in the compiler" is closer to my view.
Sorry, what I meant by "acceptable" was "legal". The compiler accepts it, the bytecode exec is fine with it, but a human may very well decide that it's unacceptable.
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
I'm strongly -1 on "works like the current implementation" as a definition of the behaviour. While having a proof of concept to clarify behaviour is great, my first question would be "how is the behaviour documented to work?" So what is the PEP proposing would happen if I did
if ('.'.join((str(x) for x in sys.version_info[:2])) as ver) == '3.6': # Python 3.6 specific code here elif sys.version_info[0] < 3: print(f"Version {ver} is not supported")
at the top level of a Python file? To me, that's a perfectly reasonable way of using the new feature to avoid exposing a binding for "ver" in my module...
I agree, sounds good. I'll reword this to be a limitation of implementation.
To put it another way, "Intended to work, but we haven't determined how to implement it yet"? Fair enough, although it needs to be possible to implement it. These names are a weird not-quite-scope construct, and interactions with real scopes are going to be tricky to get right (not just implement, but define).
Yeah. My ideal is something like this: * The subscope names are actual real variables, with unspellable names. * These name bindings get created and destroyed just like other names do, with the exception that they are automatically destroyed when they "fall out of scope". * While a subscope name is visible, locals() will use that value for that name (shadowing any other). * Once that name is removed, locals() will return to the normal form of the name. And yes, ideally this will still work when locals() is globals(). There are a couple of issues, but that's my planned design. An alternative design that is also viable: These subscope names (easily detected internally) are simply hidden from locals() and globals(). I haven't dug into the implementation consequences of any of this at global scope. I know what parts of the code I need to look at, but my days have this annoying habit of having only 24 hours in them. Anyone got a bugfix for that? :|
Consider
x = 12 if (1 as x) == 1: def foo(): print(x) # Actually, you'd probably get a "Name used before definition" error here. # Would "global x" refer to x=12 or to the statement-local x (1)? # Would "nonlocal x" refer to the statement-local x? x = 13 return x print(foo())
Yeah, that's going to give UnboundLocalError, because inside foo(), x has been flagged as local. That's independent of the global scope changes. I'd like to say that "global x" would catch the 12, but until I actually get around to implementing it, I'm not sure.
print(x) print(foo()) print(x)
Anything that executes after the 'if' exits should see x as 12. The temporary variable is completely gone at that point.
What should that return? Not "what does the current implementation return", but what is the intended result according to the proposal, and how would you explain that result in the docs?
I think I'd expect
1 13 # But see note about global/nonlocal 12 1 xxxxxxx Not sure? Maybe 1? Can you create a closure over a statement-local variable? 13 # But see note about global/nonlocal 12
The most charitable thing I can say here is that the semantics are currently under-specified in the PEP :-)
Hah. This is why I started out by saying that this ONLY applies inside a function. Extending this to global scope (and class scope; my guess is that it'll behave the same as global) is something that I'm only just now looking into.
4. Syntactic confusion in `except` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing `except (Exception as e):` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters?
Wait - except (Exception as e): would set e to the type Exception, and not capture the actual exception object?
Correct. The expression "Exception" evaluates to the type Exception, and you can capture that. It's a WutFace moment but it's a logical consequence of the nature of Python.
"Logical consequence of the rules" isn't enough for a Python language feature, IMO. Intuitive and easy to infer are key. Even if this is a corner case, it counts as a mildly ugly wart to me.
Does it need to be special-cased as an error? I do *not* want to special-case it to capture the exception instance, as that would almost certainly misbehave in more complicated scenarios.
I value Python for making it easy to write correct code, not easy to spot your errors. Too many hings like this would start me thinking I should ban statement-local names from codebases I maintain, which is not a good start for a feature...
Banning them from 'except' clauses isn't a bad thing, though. There's nothing that you need to capture; you're normally going to use a static lookup of a simple name (at best, a qualified name).
This seems to imply that the name in (expr as name) when used as a top level expression will persist after the closing parenthesis. Is that intended? It's not mentioned anywhere in the PEP (that I could see). On re-reading, I see that you say "for the remainder of the current *statement*" and not (as I had misread it) the remainder of the current *expression*.
Yep. If you have an expression on its own, it's an "expression statement", and the subscope will end at the newline (or the semicolon, if you have one). Inside something larger, it'll persist.
Technically you can have more than one expression in a statement. Consider (from the grammar):
for_stmt ::= "for" target_list "in" expression_list ":" suite ["else" ":" suite]
expression_list ::= expression ( "," expression )* [","]
Would a name binding in the first expression in an expression_list be visible in the second expression? Should it be? It will be, because it's visible to the end of the statement, not to the end of the expression, but there might be subtle technical implications here (I haven't checked the grammar to see what other statements allow multiple expressions - that's your job ;-)) To clarify this sort of question, you should probably document in the PEP precisely how the grammar will be modified.
Yes, it will. It's exactly the same in any form of statement: the name binding begins to exist at the point where it's evaluated, and it ceases to exist once that statement finishes executing. If it's an expression statement (by which I specifically mean the syntactic construct of putting a bare expression on a line on its own, called "expr_stmt" in the grammar), that point happens to coincide with the end of the expression, but that's a coincidence. So if, in the "in" expression list, you capture something, that thing will be visible all through the suite. Here's an example: for item in (get_items() as items): print(item) print(items) print(items) What actually happens is kinda this: for item in (get_items() as items_0x142857): print(item) print(items_0x142857) del items_0x142857 print(items) except that, internally, the name "items_0x142857" actually has a dot in it, making it impossible to reference using regular syntax. Once the 'for' loop is completely finished, the unbinding is compiled in, and then the name mangling ceases to happen. So it doesn't actually matter how many expressions are in a statement; it's just "this statement".
lol, see? Closures rear their ugly heads, as I mentioned above.
"What the current proof of concept implementation does" isn't useful anyway, but even ignoring that I'd prefer to see what it *does* rather than what it *compiles to*. But what needs to be documented is what the PEP *proposes* it does.
The current implementation matches my proposed semantics, as long as the code in question is all inside a function.
I'll add some more examples. I think the if/while usage is potentially of value.
I think it's an unexpected consequence of an overly-broad solution to the original problem, that accidentally solves another long-running debate. But it means you've opened a much bigger can of worms than it originally appeared, and I'm not sure you don't risk losing the simplicity that *expression* local names might have had.
But I can even break expression local names:
x = ((lambda: boom()) as boom) x()
It's possible that the "boom" is just my head exploding, not the interpreter. But I think I just demonstrated a closure over an expression-local name. For added fun, replace "x" with "boom"...
And this is why I am not trying for expression-local names. If someone wants to run with a competing proposal for list-comprehension-local names, sure, but I'm not in favour of that either. Expression-local is too narrow to be useful AND it still has the problems that statement-local has.
Thanks for the feedback! Keep it coming! :)
Ask and you shall receive :-)
If I take a razor and cut myself with it, it's called "self-harm" and can get me dinged for a psych evaluation. If I take a mailing list and induce it to send me hundreds of emails and force myself to read them all... there's probably a padded cell with my name on it somewhere. You know, I'd be okay with that actually. Just as long as the cell has wifi. ChrisA
On 28 February 2018 at 19:41, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Mar 1, 2018 at 3:30 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Here's another equally untested piece of code:
[(-b + (sqrt(b*b - 4*a*c) as disc)) / (2*a), (-b - disc) / (2*a)]
Yeah, that's a good real world example. And IMO it hides the symmetry between the two expressions, so for me it's a good example of a case where there's a readability *disadvantage* with the proposed syntax.
I haven't dug into the implementation consequences of any of this at global scope. I know what parts of the code I need to look at, but my days have this annoying habit of having only 24 hours in them. Anyone got a bugfix for that? :|
I can give you daylight saving time and a few leap seconds, but that's more of a workaround than a fix :-)
Consider
x = 12 if (1 as x) == 1: def foo(): print(x) # Actually, you'd probably get a "Name used before definition" error here. # Would "global x" refer to x=12 or to the statement-local x (1)? # Would "nonlocal x" refer to the statement-local x? x = 13 return x print(foo())
Yeah, that's going to give UnboundLocalError, because inside foo(), x has been flagged as local. That's independent of the global scope changes.
I'd like to say that "global x" would catch the 12, but until I actually get around to implementing it, I'm not sure.
print(x) print(foo()) print(x)
Anything that executes after the 'if' exits should see x as 12. The temporary variable is completely gone at that point.
So simplifying x = 12 if (1 as x) == 1: def foo(): return x print(foo()) print(foo()) prints 1 12 Personally, I can't work out how to think about that in a way that isn't confusing :-( I gather from what you've said elsewhere that your current implementation uses a hidden name-mangled variable. IIRC, list comprehensions used something similar in the original implementation, but it resulted in weird consequences, and ultimately the implementation switched to a "proper" scoped semantics and implementation. I've no particular reason to think your implementation might suffer in the same way, but understanding the semantics in terms of an "assignment to a hidden variable" bothers me.
The most charitable thing I can say here is that the semantics are currently under-specified in the PEP :-)
Hah. This is why I started out by saying that this ONLY applies inside a function. Extending this to global scope (and class scope; my guess is that it'll behave the same as global) is something that I'm only just now looking into.
And yet, "only works within a function" is IMO an unacceptable limitation - so we need to look into the implications here.
So if, in the "in" expression list, you capture something, that thing will be visible all through the suite. Here's an example:
for item in (get_items() as items): print(item) print(items) print(items)
What actually happens is kinda this:
for item in (get_items() as items_0x142857): print(item) print(items_0x142857) del items_0x142857 print(items)
OK, so that explains why trying to write a closure over a variable introduced via "as" won't work. But I wouldn't want such a renaming to be mandated by the semantics, and I don't know how I'd explain the (implied) behaviour without insisting on a name mangling implementation, so I'm stuck here.
"What the current proof of concept implementation does" isn't useful anyway, but even ignoring that I'd prefer to see what it *does* rather than what it *compiles to*. But what needs to be documented is what the PEP *proposes* it does.
The current implementation matches my proposed semantics, as long as the code in question is all inside a function.
Understood. But I'd like the PEP to fully explain more of the intended semantics, *without* referring to the specific implementation. Remember, PyPy, Jython, IronPython, Cython, etc, will all have to implement it too.
But I can even break expression local names:
x = ((lambda: boom()) as boom) x()
It's possible that the "boom" is just my head exploding, not the interpreter. But I think I just demonstrated a closure over an expression-local name. For added fun, replace "x" with "boom"...
And this is why I am not trying for expression-local names. If someone wants to run with a competing proposal for list-comprehension-local names, sure, but I'm not in favour of that either. Expression-local is too narrow to be useful AND it still has the problems that statement-local has.
I've no idea how that would work under statement-local names either, though. boom = lambda: boom() boom() is just an infinite recursion. I'm less sure that the as version is. Or the alternative form ((lambda: boom()) as boom)() I know you can tell me what the implementation does - but I can't reason it out from the spec.
Thanks for the feedback! Keep it coming! :)
Ask and you shall receive :-)
If I take a razor and cut myself with it, it's called "self-harm" and can get me dinged for a psych evaluation. If I take a mailing list and induce it to send me hundreds of emails and force myself to read them all... there's probably a padded cell with my name on it somewhere.
You know, I'd be okay with that actually. Just as long as the cell has wifi.
lol. Have fun :-) Paul
On Thu, Mar 1, 2018 at 7:28 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I've no idea how that would work under statement-local names either, though.
boom = lambda: boom() boom()
is just an infinite recursion. I'm less sure that the as version is. Or the alternative form
((lambda: boom()) as boom)()
I know you can tell me what the implementation does - but I can't reason it out from the spec.
The only part that isn't yet properly specified is how this interacts with closures, and that's because I really don't know what makes sense. Honestly, *any* situation where you're closing over a SLNB is going to have readability penalties, no matter what the spec says. ChrisA
On 1 March 2018 at 06:34, Chris Angelico <rosuav@gmail.com> wrote:
I've no idea how that would work under statement-local names either,
On Thu, Mar 1, 2018 at 7:28 AM, Paul Moore <p.f.moore@gmail.com> wrote: though.
boom = lambda: boom() boom()
is just an infinite recursion. I'm less sure that the as version is. Or the alternative form
((lambda: boom()) as boom)()
I know you can tell me what the implementation does - but I can't reason it out from the spec.
The only part that isn't yet properly specified is how this interacts with closures, and that's because I really don't know what makes sense. Honestly, *any* situation where you're closing over a SLNB is going to have readability penalties, no matter what the spec says.
My suggestion is to just flat out prohibit it - if closures can capture the reference, then it isn't a statement local name binding any more, and it's reasonable to ask folks to promote it to a regular function local variable. This is another area where syntactic disambiguation on the reference side would help # prints 12 twice x = 12 if (1 as .x) == 1: def foo(): return x print(foo()) print(foo()) # prints 1 twice x = 12 if (1 as .x) == 1: x = .x def foo(): return x print(foo()) print(foo()) # Raises UnboundLocalError x = 12 if (1 as .x) == 1: def foo(): return .x print(foo()) print(foo()) That last example could potentially even raise a compile time error during the symbol table analysis pass, since the compiler would *know* there's no statement local by that name in the current lexical scope, and statement local references wouldn't have a runtime fallback to module globals or the builtins the way regular variable references do. Another perk of using the ".NAME" syntax is that we could extend it to allow statement local name bindings in for loops as well: for .i in range(10): print(.i) # This is fine print(.i) # This is an error (unless an outer statement also sets .i) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
28.02.18 00:27, Chris Angelico пише:
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
The simplest equivalent of [f(x), f(x)] is [f(x)]*2. It would be worth to use less trivial example, e.g. f(x) + x/f(x).
# Helper function def pair(value): return [value, value] stuff = [pair(f(x)) for x in range(5)]
# Inline helper function stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
# Extra 'for' loop - see also Serhiy's optimization stuff = [[y, y] for x in range(5) for y in [f(x)]]
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
# Using a statement-local name stuff = [[(f(x) as y), y] for x in range(5)]
Other options: stuff = [[y, y] for y in (f(x) for x in range(5))] g = (f(x) for x in range(5)) stuff = [[y, y] for y in g] def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g) Seems the two last options are generally considered the most Pythonic. map() and itertools can be helpful in particular cases.
On Thu, Mar 1, 2018 at 12:49 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
28.02.18 00:27, Chris Angelico пише:
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
The simplest equivalent of [f(x), f(x)] is [f(x)]*2. It would be worth to use less trivial example, e.g. f(x) + x/f(x).
Sure, I'll go with that.
# Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, y])
Other options:
g = (f(x) for x in range(5)) stuff = [[y, y] for y in g]
That's the same as the one-liner, but with the genexp broken out. Not sure it helps much as examples go?
def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g)
You're not the first to mention this, but I thought it basically equivalent to the "expand into a loop" form. Is it really beneficial to expand it, not just into a loop, but into a generator function that contains a loop? ChrisA
2018-02-28 16:49 GMT+03:00 Serhiy Storchaka <storchaka@gmail.com>:
28.02.18 00:27, Chris Angelico пише:
Example usage =============
These list comprehensions are all approximately equivalent::
# Calling the function twice stuff = [[f(x), f(x)] for x in range(5)]
Other options:
stuff = [[y, y] for y in (f(x) for x in range(5))]
Why not `stuff = [[y, y] for y in map(f, range(5))]`? It is currently the fastest and most readable version IMHO. With kind regards, -gdg
28.02.18 16:14, Kirill Balunov пише:
2018-02-28 16:49 GMT+03:00 Serhiy Storchaka <storchaka@gmail.com <mailto:storchaka@gmail.com>>: Other options:
stuff = [[y, y] for y in (f(x) for x in range(5))]
Why not `stuff = [[y, y] for y in map(f, range(5))]`? It is currently the fastest and most readable version IMHO.
Only in this particular case. This doesn't work if the expression for items depends on x or if the subexpression is not a function call.
28.02.18 16:06, Chris Angelico пише:
On Thu, Mar 1, 2018 at 12:49 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
Other options:
g = (f(x) for x in range(5)) stuff = [[y, y] for y in g]
That's the same as the one-liner, but with the genexp broken out. Not sure it helps much as examples go?
It is more readable. But can't be used as an expression.
def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g)
You're not the first to mention this, but I thought it basically equivalent to the "expand into a loop" form. Is it really beneficial to expand it, not just into a loop, but into a generator function that contains a loop?
It is slightly faster (if the list is not too small). It doesn't leak a temporary variable after loop. And in many cases you don't need a list, an iterator would work as well. In these cases it is easy to just drop calling list().
On Thu, Mar 1, 2018 at 1:51 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
28.02.18 16:06, Chris Angelico пише:
On Thu, Mar 1, 2018 at 12:49 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
Other options:
g = (f(x) for x in range(5)) stuff = [[y, y] for y in g]
That's the same as the one-liner, but with the genexp broken out. Not sure it helps much as examples go?
It is more readable. But can't be used as an expression.
def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g)
You're not the first to mention this, but I thought it basically equivalent to the "expand into a loop" form. Is it really beneficial to expand it, not just into a loop, but into a generator function that contains a loop?
It is slightly faster (if the list is not too small). It doesn't leak a temporary variable after loop. And in many cases you don't need a list, an iterator would work as well. In these cases it is easy to just drop calling list().
Doesn't leak a temporary? In Python 3, the list comp won't leak anything, but the function is itself a temporary variable with permanent scope. You're right about the generator being sufficient at times, but honestly, if we're going to say "maybe you don't need the same result", then all syntax questions go out the window :D ChrisA
28.02.18 16:56, Chris Angelico пише:
def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g)
You're not the first to mention this, but I thought it basically equivalent to the "expand into a loop" form. Is it really beneficial to expand it, not just into a loop, but into a generator function that contains a loop?
It is slightly faster (if the list is not too small). It doesn't leak a temporary variable after loop. And in many cases you don't need a list, an iterator would work as well. In these cases it is easy to just drop calling list().
Doesn't leak a temporary? In Python 3, the list comp won't leak anything, but the function is itself a temporary variable with permanent scope. You're right about the generator being sufficient at times, but honestly, if we're going to say "maybe you don't need the same result", then all syntax questions go out the window :D
Explicit for loop leaks variables x and y after the loop. They can hold references to large objects. The generator function itself doesn't hold references to the proceeded data.
On Thu, Mar 1, 2018 at 2:16 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
28.02.18 16:56, Chris Angelico пише:
def g(): for x in range(5): y = f(x) yield [y, y] stuff = list(g)
You're not the first to mention this, but I thought it basically equivalent to the "expand into a loop" form. Is it really beneficial to expand it, not just into a loop, but into a generator function that contains a loop?
It is slightly faster (if the list is not too small). It doesn't leak a temporary variable after loop. And in many cases you don't need a list, an iterator would work as well. In these cases it is easy to just drop calling list().
Doesn't leak a temporary? In Python 3, the list comp won't leak anything, but the function is itself a temporary variable with permanent scope. You're right about the generator being sufficient at times, but honestly, if we're going to say "maybe you don't need the same result", then all syntax questions go out the window :D
Explicit for loop leaks variables x and y after the loop. They can hold references to large objects. The generator function itself doesn't hold references to the proceeded data.
Oh, gotcha. Yeah. Will add that as another example. ChrisA
On 02/27/2018 02:27 PM, Chris Angelico wrote:
PEP: 572
Because: a = (2 as a) does not evaluate to 2 and because some statements, such as 'for' and 'while' can cover many, many lines (leaving plenty of potential for confusion over which variable disappear at the end and which persist): -1 I would love to see an expression assignment syntax, but Statement-Local-Name-Binding is confusing and non-intuitive. Is there already a PEP about expression assignment? If not, that would be a good one to write. -- ~Ethan~
The PEP says:
Omitting the parentheses from this PEP's proposed syntax introduces many syntactic ambiguities.
and: As the name's scope extends to the full current statement, even a block
statement, this can be used to good effect in the header of an if or while statement
Will the `from ... import ... as ... statement be a special case, because currently the following form is valid: from math import (tau as two_pi) With kind regards, -gdg
On Thu, Mar 1, 2018 at 5:03 AM, Kirill Balunov <kirillbalunov@gmail.com> wrote:
The PEP says:
Omitting the parentheses from this PEP's proposed syntax introduces many syntactic ambiguities.
and:
As the name's scope extends to the full current statement, even a block statement, this can be used to good effect in the header of an if or while statement
Will the `from ... import ... as ... statement be a special case, because currently the following form is valid:
from math import (tau as two_pi)
No, because that statement doesn't have any expressions in it - it's a series of names. The "tau" in that line is not looked up in the current scope; you can't write a function that returns the symbol "tau" and then use that in the import. So the grammatical hook that enables "(... as ...)" doesn't apply here. ChrisA
On 28 February 2018 at 08:27, Chris Angelico <rosuav@gmail.com> wrote:
This is a suggestion that comes up periodically here or on python-dev. This proposal introduces a way to bind a temporary name to the value of an expression, which can then be used elsewhere in the current statement.
The nicely-rendered version will be visible here shortly:
Thanks for putting this together!
Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as `lambda` should be avoided when `def` is an option.
[snip]
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
It isn't clear to me from the current PEP what the intended lifecycle of the bound names actually is, especially for compound statements. Using list comprehensions as your example currently hides that confusion, since they create an implicit nested scope anyway, so the references will go away when the list comprehension terminates no matter what. So I think it would be useful to have some other examples in the PEP to more explicitly answer that question: x = (expr as y) assert x == y # Does this pass? Or raise NameError for 'y'? if (condition as c): assert c # Does this pass? Or raise NameError for 'c'? else: assert not c # Does this pass? Or raise NameError for 'c'? assert c or not c # Does this pass? Or raise NameError for 'c'? class C: x = (True as y) assert C.y # Does this pass? Or raise AttributeError for 'y'? I think it would also be worth explicitly considering a syntactic variant that requires statement local references to be explicitly disambiguated from regular variable names by way of a leading dot: result = [[(f(x) as .y), .y] for x in range(5)] (.x, (1 as .x), .x) # UnboundLocalError on first subexpression (x, (1 as .x), .x) # Valid if 'x' is a visible name x = (expr as .y) assert x == .y # UnboundLocalError for '.y' if (condition as .c): assert .c # Passes else: assert not .c # Passes assert .c or not .c # UnboundLocalError for '.c' class C: x = (True as .y) assert C..y # SyntaxError on the second dot Since ".NAME" is illegal for both variable and attribute names, this makes the fact statement locals are a distinct namespace visible to readers as well as to the compiler, and also reduces the syntactic ambiguity in with statements and exception handlers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 February 2018 at 08:27, Chris Angelico <rosuav@gmail.com> wrote:
2. The current implementation [1] implements statement-local names using a special (and mostly-invisible) name mangling. This works perfectly inside functions (including list comprehensions), but not at top level. Is this a serious limitation? Is it confusing?
It isn't clear to me from the current PEP what the intended lifecycle of the bound names actually is, especially for compound statements.
I think you're looking at an old version of the PEP, but that's kinda gonna happen on the first day of a hot topic :) But to remove all confusion, I've now added a section clarifying execution order, using several of your examples.
x = (expr as y) assert x == y # Does this pass? Or raise NameError for 'y'?
NameError. The SLNB is gone at end of line.
if (condition as c): assert c # Does this pass? Or raise NameError for 'c'? else: assert not c # Does this pass? Or raise NameError for 'c'? assert c or not c # Does this pass? Or raise NameError for 'c'?
c is available in the indented blocks, and is gone once the entire 'if/else' block is done.
class C: x = (True as y) assert C.y # Does this pass? Or raise AttributeError for 'y'?
That'll raise. (At least, according to the specification and intent. The reference implementation may be lagging a bit.)
I think it would also be worth explicitly considering a syntactic variant that requires statement local references to be explicitly disambiguated from regular variable names by way of a leading dot:
result = [[(f(x) as .y), .y] for x in range(5)]
[chomp]
Since ".NAME" is illegal for both variable and attribute names, this makes the fact statement locals are a distinct namespace visible to readers as well as to the compiler, and also reduces the syntactic ambiguity in with statements and exception handlers.
I've mentioned this in the alternate syntaxes, but I don't like having to state a variable's scope in its name. Python doesn't force us to adorn all global names with a character, and the difference between function-local and statement-local is generally less important than the difference between global and function-local. But it IS a viable syntax, and I'm saying so in the PEP. Thanks for the feedback! Much appreciated. ChrisA
On 1 March 2018 at 06:40, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Since ".NAME" is illegal for both variable and attribute names, this makes the fact statement locals are a distinct namespace visible to readers as well as to the compiler, and also reduces the syntactic ambiguity in with statements and exception handlers.
I've mentioned this in the alternate syntaxes, but I don't like having to state a variable's scope in its name. Python doesn't force us to adorn all global names with a character, and the difference between function-local and statement-local is generally less important than the difference between global and function-local. But it IS a viable syntax, and I'm saying so in the PEP.
Agreed. This feels far to much like Perl's "sigils" that attach to a name ($var is a scalar, @var is a list, etc). Strong -1 from me. Although I *do* agree that such decoration gets rid of a lot of the worst ambiguities in the proposal. But if we're being asked to choose the lesser of 2 evils, my response is "neither" :-) Paul
On 1 March 2018 at 19:30, Paul Moore <p.f.moore@gmail.com> wrote:
On 1 March 2018 at 06:40, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Since ".NAME" is illegal for both variable and attribute names, this makes the fact statement locals are a distinct namespace visible to readers as well as to the compiler, and also reduces the syntactic ambiguity in with statements and exception handlers.
I've mentioned this in the alternate syntaxes, but I don't like having to state a variable's scope in its name. Python doesn't force us to adorn all global names with a character, and the difference between function-local and statement-local is generally less important than the difference between global and function-local. But it IS a viable syntax, and I'm saying so in the PEP.
Agreed. This feels far to much like Perl's "sigils" that attach to a name ($var is a scalar, @var is a list, etc). Strong -1 from me.
While that's a fair criticism, one of the current challenges with Python's variable naming is that given a NAME reference, there are already several potential places for that name to be resolved: * the current local scope * an enclosing function scope * the module globals * the builtin namespace This means the compiler has to be conservative and assume that if a name isn't visible as a local namespace entry or as a closure reference, then it will still be available at runtime (somehow). Structural linters and static analysers like pylint or mypy can do better and say "We can't figure out how you're expecting this name reference to be resolved at runtime", but it's still a very complex system to try to reason about when reading a code snippet. Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do). Thus the intent behind the ".NAME" suggestion is to ask whether or not it's possible to allow for name bindings that are strictly local to a compilation unit (i.e. without allowing dynamic runtime access to outer scopes or from contained scopes), *without* incurring the cost of making ordinary NAME references even more complicated to understand. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/01/2018 09:08 PM, Nick Coghlan wrote:
On 1 March 2018 at 19:30, Paul Moore <p.f.moore@gmail.com wrote:
On 1 March 2018 at 06:40, Chris Angelico wrote:
On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan wrote:
Since ".NAME" is illegal for both variable and attribute names, this makes the fact statement locals are a distinct namespace visible to readers as well as to the compiler, and also reduces the syntactic ambiguity in with statements and exception handlers.
I've mentioned this in the alternate syntaxes, but I don't like having to state a variable's scope in its name. Python doesn't force us to adorn all global names with a character, and the difference between function-local and statement-local is generally less important than the difference between global and function-local. But it IS a viable syntax, and I'm saying so in the PEP.
Agreed. This feels far to much like Perl's "sigils" that attach to a name ($var is a scalar, @var is a list, etc). Strong -1 from me.
While that's a fair criticism, one of the current challenges with Python's variable naming is that given a NAME reference, there are already several potential places for that name to be resolved:
* the current local scope * an enclosing function scope * the module globals * the builtin namespace
This means the compiler has to be conservative and assume that if a name isn't visible as a local namespace entry or as a closure reference, then it will still be available at runtime (somehow). Structural linters and static analysers like pylint or mypy can do better and say "We can't figure out how you're expecting this name reference to be resolved at runtime", but it's still a very complex system to try to reason about when reading a code snippet.
Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such). No grammatical grit on anyone's monitor, no confusion about which variable is being accessed, and no confusion about the lifetime of that variable (okay, no /extra/ confusion ;) . Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to be convinced (not happy to, just willing ;) . -- ~Ethan~
On Fri, Mar 2, 2018 at 5:39 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such).
No grammatical grit on anyone's monitor, no confusion about which variable is being accessed, and no confusion about the lifetime of that variable (okay, no /extra/ confusion ;) .
Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to be convinced (not happy to, just willing ;) .
Sounds like what you're proposing could be done with the exact syntax that I'm using, and just remove subscopes from the discussion. (It'd still need parenthesization, I believe, to prevent syntactic ambiguities.) As a competing proposal, it's plausible; basically, it gives Python a way to assign to any name at any time. I'm honestly not sure which variant would see more backlash :) That's worthy of a mention in the alternates, at any rate. ChrisA
On 2 March 2018 at 16:39, Ethan Furman <ethan@stoneleaf.us> wrote:
On 03/01/2018 09:08 PM, Nick Coghlan wrote:
Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such).
Because that would put us back in the exact same problematic situation we had when "[x*x for x in sequence]" leaked the iteration variable (only worse): no function locals would be safe, since arbitrary expressions could clobber them, not just name binding operations (assignment, import statements, for loops, with statements, exception handlers, class and function definitions).
No grammatical grit on anyone's monitor, no confusion about which variable is being accessed, and no confusion about the lifetime of that variable (okay, no /extra/ confusion ;) .
Unfortunately, it would mean a lot more "All I did was name a repeated subexpression and now my function is behaving weirdly".
Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to be convinced (not happy to, just willing ;) .
It breaks the expectation that only a well defined subset of statement can make changes to local name bindings. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/02/2018 02:47 AM, Nick Coghlan wrote:
On 2 March 2018 at 16:39, Ethan Furman wrote:
On 03/01/2018 09:08 PM, Nick Coghlan wrote:
Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such).
Because that would put us back in the exact same problematic situation we had when "[x*x for x in sequence]" leaked the iteration variable (only worse): no function locals would be safe, since arbitrary expressions could clobber them, not just name binding operations (assignment, import statements, for loops, with statements, exception handlers, class and function definitions).
Ah, right. Since the PEP primarily covers comprehensions, but then went on to discuss multi-line statements, I had forgotten the comprehension part. The answer is easy: assignment expressions in comprehensions stay inside comprehensions, just like other inner comprehension variables already do (function sub-scope, after all). Thank you for pointing that out.
Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to be convinced (not happy to, just willing ;) .
It breaks the expectation that only a well defined subset of statement can make changes to local name bindings.
We already use the keyword "as" to assign names, so there is nothing extra there -- the only change is that we can now use it in one more place. Are those valid counter-arguments, or is there still something I am missing? -- ~Ethan~
On 3 March 2018 at 03:51, Ethan Furman <ethan@stoneleaf.us> wrote:
On 03/02/2018 02:47 AM, Nick Coghlan wrote:
On 2 March 2018 at 16:39, Ethan Furman wrote:
On 03/01/2018 09:08 PM, Nick Coghlan wrote:
Adding statement local variables into that mix *without* some form of
syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such).
Because that would put us back in the exact same problematic situation we had when "[x*x for x in sequence]" leaked the iteration variable (only worse): no function locals would be safe, since arbitrary expressions could clobber them, not just name binding operations (assignment, import statements, for loops, with statements, exception handlers, class and function definitions).
Ah, right. Since the PEP primarily covers comprehensions, but then went on to discuss multi-line statements, I had forgotten the comprehension part. The answer is easy: assignment expressions in comprehensions stay inside comprehensions, just like other inner comprehension variables already do (function sub-scope, after all). Thank you for pointing that out.
That wasn't the point I was try to make: my attempted point was that I see allowing an expression like "print((f() as x), x^2, x^3)" to overwrite the regular function local "x" as being just as unacceptable as "data = [x^2 for x in sequence]" overwriting it, and we already decided that the latter was sufficiently undesirable that we were willing to break backwards compatibility in order to change it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/02/2018 11:11 PM, Nick Coghlan wrote:
On 3 March 2018 at 03:51, Ethan Furman wrote:
Ah, right. Since the PEP primarily covers comprehensions, but then went on to discuss multi-line statements, I had forgotten the comprehension part. The answer is easy: assignment expressions in comprehensions stay inside comprehensions, just like other inner comprehension variables already do (function sub-scope, after all). Thank you for pointing that out.
That wasn't the point I was try to make: my attempted point was that I see allowing an expression like "print((f() as x), x^2, x^3)" to overwrite the regular function local "x" as being just as unacceptable as "data = [x^2 for x in sequence]" overwriting it, and we already decided that the latter was sufficiently undesirable that we were willing to break backwards compatibility in order to change it.
I think I explained myself poorly. I'm agreeing with you, and pointing out that the (var as expr) syntax /inside/ a comprehension would stay inside a comprehension, i.e. not leak out to locals(), just like your "x" above. -- ~Ethan~
On 2 March 2018 at 05:08, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 1 March 2018 at 19:30, Paul Moore <p.f.moore@gmail.com> wrote:
Agreed. This feels far to much like Perl's "sigils" that attach to a name ($var is a scalar, @var is a list, etc). Strong -1 from me.
While that's a fair criticism, one of the current challenges with Python's variable naming is that given a NAME reference, there are already several potential places for that name to be resolved:
* the current local scope * an enclosing function scope * the module globals * the builtin namespace
While that is true, the current scenario is (conceptually, at least) a relatively traditional set of 4 nested namespaces, with name resolution working from innermost to outermost, and with the extent of the scopes being pretty clearly defined. So although the *compiler* may have to defer a chunk of that resolution to runtime, the human reader can work with a relatively conventional model and not hit problems in the majority of cases. The problem with statement local variables is that the extent over which the name is in scope is not as clear to the human reader (the rules the *compiler* follows may be precise, but they aren't obvious to the human reader - that's the root of the debate I'm having with Chris over "what the reference implementation does isn't a sufficient spec"). In particular, assignment statements are non-obvious, as shown by the examples that triggered your suggestion of a "." prefix. [...]
Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Well, an alternative to a syntactic marker would be an easy-to-determine extent. That's where proposals like PEP 3150 (the "given" clause) work better, because they provide a clearer indication of the extent of the new scope. IMO, lack of a well-defined extent is a flaw of this proposal, and syntactic markers are essentially a (ugly) workaround for that flaw.
Thus the intent behind the ".NAME" suggestion is to ask whether or not it's possible to allow for name bindings that are strictly local to a compilation unit (i.e. without allowing dynamic runtime access to outer scopes or from contained scopes), *without* incurring the cost of making ordinary NAME references even more complicated to understand.
... or the cost of imposing a more user-visible indication of the extent of the scope into the proposal. Paul
On 2 March 2018 at 19:05, Paul Moore <p.f.moore@gmail.com> wrote:
The problem with statement local variables is that the extent over which the name is in scope is not as clear to the human reader (the rules the *compiler* follows may be precise, but they aren't obvious to the human reader - that's the root of the debate I'm having with Chris over "what the reference implementation does isn't a sufficient spec"). In particular, assignment statements are non-obvious, as shown by the examples that triggered your suggestion of a "." prefix.
Those examples didn't trigger the suggestion: the suggestion was borne from the fact that I don't think it should be possible to close over statement locals. If you can't close over statement locals, then it isn't acceptable to allow this scenario: x = 12 if (True as x): def f(): return x print(x, f()) # "True True"? Or "True 12"? print(x, f()) # "12 12", but it's not obvious why it isn't "True True" By contrast, if the two kinds of local namespace are visibly different, then the nested scope *can't* give the appearance of referencing the statement local: x = 12 if (True as .x): def f(): return x print(.x, x, f()) # Clearly "True, 12, 12", since x never gets rebound print(x, f()) # Clearly "12, 12", since x never gets rebound
[...]
Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do).
Well, an alternative to a syntactic marker would be an easy-to-determine extent. That's where proposals like PEP 3150 (the "given" clause) work better, because they provide a clearer indication of the extent of the new scope. IMO, lack of a well-defined extent is a flaw of this proposal, and syntactic markers are essentially a (ugly) workaround for that flaw.
PEP 3150 ended up needing syntactic markers as well, to handle the forward references to names set in the `given` clause while staying within the LL(1) parsing design constraint imposed on Python's grammar. Currently it proposes `?.name` as that marker, with `?` referring to the entire given namespace, but it could equally well use `.name` instead (and if you wanted a reference to a namespace instead, you'd need to define one inside the given clause). One of the key *problems* with PEP 3150 though is that it doesn't compose nicely with other compound statements, whereas PEP 572 does (by treating each statement as its own extent - PEP 3150 then just provides a way to add a suite to statements that don't already have one of their own).
Thus the intent behind the ".NAME" suggestion is to ask whether or not it's possible to allow for name bindings that are strictly local to a compilation unit (i.e. without allowing dynamic runtime access to outer scopes or from contained scopes), *without* incurring the cost of making ordinary NAME references even more complicated to understand.
... or the cost of imposing a more user-visible indication of the extent of the scope into the proposal.
Right, but that extra notation *does* convey useful information to a reader that better enables local reasoning about a piece of code. Currently, if you're looking at an unfamiliar function and see a name you don't recognise, then you need to search the whole module for that name to see whether or not it's defined anywhere. Even if it's missing, you may still need to check for dynamic injection of module level names via globals(). Seeing ".name" would be different (both for the compiler and for the human reader): if such a reference can't be resolved explicitly within the scope of the current statement, then *it's a bug* (and the compiler would be able to flag it as such at compile time). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 March 2018 at 11:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 March 2018 at 19:05, Paul Moore <p.f.moore@gmail.com> wrote:
The problem with statement local variables is that the extent over which the name is in scope is not as clear to the human reader (the rules the *compiler* follows may be precise, but they aren't obvious to the human reader - that's the root of the debate I'm having with Chris over "what the reference implementation does isn't a sufficient spec"). In particular, assignment statements are non-obvious, as shown by the examples that triggered your suggestion of a "." prefix.
Those examples didn't trigger the suggestion: the suggestion was borne from the fact that I don't think it should be possible to close over statement locals.
Ah, OK. If closing over statement locals isn't allowed, then yes, they are a different type of name, and you may need to distinguish them. On the other hand, I'm not sure I agree with you that it shouldn't be possible to close over statement locals. I can see that there are a lot of *difficulties* with allowing it, but that's not the same. What's your logic for saying you shouldn't be able to close over a statement local name? What is fundamentally different about them that makes them unsuitable to work like all other names in Python?
PEP 3150 ended up needing syntactic markers as well, to handle the forward references to names set in the `given` clause while staying within the LL(1) parsing design constraint imposed on Python's grammar.
Is this basically about forward references then? Certainly the "a = (1 as a)" examples are a problem because of forward references. And the problem here is that we can't use the normal solution of simply prohibiting forward references ("Name assigned before declaration" errors) because - well, I'm not quite sure why, actually. Surely if you're naming a subexpression that's repeated, you can always just choose to name the *first* occurrence and use that name for the rest? I guess the exception is statements that introduce or bind names (assignments, for, etc). I'd still be inclined to just say prohibit such cases (if we can't, then we're back into the territory of implementation difficulties driving the design). This all still feels to me like an attempt to rescue the proposal from the issues that arise from not treating statement-local names exactly like any other name.
Right, but that extra notation *does* convey useful information to a reader that better enables local reasoning about a piece of code. Currently, if you're looking at an unfamiliar function and see a name you don't recognise, then you need to search the whole module for that name to see whether or not it's defined anywhere. Even if it's missing, you may still need to check for dynamic injection of module level names via globals().
Hang on, that's how all existing names in Python work (and in pretty much any language that doesn't require explicit declarations). Surely no-one is trying to suggest that this is a fundamental flaw?
Seeing ".name" would be different (both for the compiler and for the human reader): if such a reference can't be resolved explicitly within the scope of the current statement, then *it's a bug* (and the compiler would be able to flag it as such at compile time).
Sorry, but you could use exactly that argument to propose that function local variables should be prefixed with "$". I don't buy it. I guess I remain -1 on the proposal, and nothing that's getting said about how we can make it work is doing anything to persuade me otherwise (quite the opposite). Paul
On 2 March 2018 at 21:50, Paul Moore <p.f.moore@gmail.com> wrote:
On 2 March 2018 at 11:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 March 2018 at 19:05, Paul Moore <p.f.moore@gmail.com> wrote:
The problem with statement local variables is that the extent over which the name is in scope is not as clear to the human reader (the rules the *compiler* follows may be precise, but they aren't obvious to the human reader - that's the root of the debate I'm having with Chris over "what the reference implementation does isn't a sufficient spec"). In particular, assignment statements are non-obvious, as shown by the examples that triggered your suggestion of a "." prefix.
Those examples didn't trigger the suggestion: the suggestion was borne from the fact that I don't think it should be possible to close over statement locals.
Ah, OK. If closing over statement locals isn't allowed, then yes, they are a different type of name, and you may need to distinguish them. On the other hand, I'm not sure I agree with you that it shouldn't be possible to close over statement locals. I can see that there are a lot of *difficulties* with allowing it, but that's not the same.
What's your logic for saying you shouldn't be able to close over a statement local name? What is fundamentally different about them that makes them unsuitable to work like all other names in Python?
I have two reasons, one based on design intent, one based on practicality of implementation. Starting with the practical motivation first: based on my experience preventing the implementation variable from leaking in comprehensions, I don't think it's going to be practical to allow closing over statement locals while still scoping them to the statement that sets them in any meaningful way. The difficulty of implementing that correctly is why I ended up going for the implicit-nested-scope implementation for Python 3 comprehensions in the first place. While I do assume it would technically be possible to go that way (it's only software after all), it would require some *significant* changes to the way at least CPython's compiler handles symbol analysis, and likely to the way symtable analysis is handled in other compilers as well. By contrast, if you drop the "must support closures" requirement, then the required name mangling to only nested statements to re-use the same statement local names without interfering with each other or with matching function local names gets *much* simpler (especially with a syntactic marker to distinguish statement local references from normal variable names), since all you need to track is how many levels deep you are in statement nesting within the current compilation unit. The design intent related rationale stems from the fact that closed over references can live for an arbitrarily long time (as can regular function locals in a generator or coroutine), and I think statement locals should be as reliably ephemeral as we can reasonably make them if they're going to be different enough from function locals to be worth the hassle of adding them. So that means asking a bunch of questions and deliberately giving the *opposite* answer for statement locals than I would for function locals: * Visible in locals()? Yes for function locals, no for statement locals * Visible in frame.f_locals? Yes for function locals, no for statement locals * Prevents access to the same name in outer scopes? Yes for function locals, no for statement locals * Can be closed over? Yes for function locals, no for statement locals Answer "no" to all those questions is only reasonable if statement local references *look* different from regular variable names. But I also think we need to proposing answering "No" to all of them to make statement locals potentially interesting enough to be worth considering. (Debuggers would need to access these in order to display them, so they'd have to be available on the frame object together with metadata on the code object to correctly populate them at runtime, but that's a solvable problem) While this would require non-trivial compiler changes as well, they'd be much safer updates with a lower risk of unintended consequences, since they wouldn't need to touch the existing name resolution code - they'd be their own thing, operating in parallel with the established dynamic name lookup support.
PEP 3150 ended up needing syntactic markers as well, to handle the forward references to names set in the `given` clause while staying within the LL(1) parsing design constraint imposed on Python's grammar.
Is this basically about forward references then? Certainly the "a = (1 as a)" examples are a problem because of forward references. And the problem here is that we can't use the normal solution of simply prohibiting forward references ("Name assigned before declaration" errors) because - well, I'm not quite sure why, actually. Surely if you're naming a subexpression that's repeated, you can always just choose to name the *first* occurrence and use that name for the rest? I guess the exception is statements that introduce or bind names (assignments, for, etc). I'd still be inclined to just say prohibit such cases (if we can't, then we're back into the territory of implementation difficulties driving the design).
Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can figure out what is going on because it does the symbol table analysis pass before the code generation pass, and hence can tag names appropriately based on what it finds. My concern is for people reading the code, where omitting a syntactic marker also forces *humans* to read things in two passes to make sure they understand them correctly. Consider this example: x = 10 data = [x*x for i in range(10)] This will give you a 10 item list, where each item is 100. But now consider a world with statement locals, where statement locals use the same reference syntax as regular locals: x = 10 data = [x*x for i in range(10) if (12 as x)] That's a deliberately pathological example (and you can already write something similarly misleading using the "for x in [12]" trick), but the fact remains that we allow out of order evaluation in expressions in a way that we don't permit for statements. With a syntactic marker though, there's typically going to be less ambiguity about where a name comes from: x = 10 data = [.x*.x for i in range(10) if (12 as .x)] It's not a panacea (since you may still be shadowing a name from an outer statement), and adapting it for PEP 3150 isn't without it's problems (specifically, you need to allow ".x = 12" in the given clause to make the names match up), but it should be less confusing than allowing a new subexpression to interfere with name resolution semantics folks have been relying on for years. This all still feels to me like an attempt to rescue the proposal from
the issues that arise from not treating statement-local names exactly like any other name.
If statement locals behave just like function locals, then there's no reason to add them to the language in the first place - "(expr as name)" would just become a confusing second way to spell "name = expr". It's only potentially worthwhile if statement locals offer us something that function locals don't, and the clearest potential candidate for that differentiation is to provide a greater level of assurance regarding locality of use than regular name binding operations do.
Right, but that extra notation *does* convey useful information to a reader
that better enables local reasoning about a piece of code. Currently, if you're looking at an unfamiliar function and see a name you don't recognise, then you need to search the whole module for that name to see whether or not it's defined anywhere. Even if it's missing, you may still need to check for dynamic injection of module level names via globals().
Hang on, that's how all existing names in Python work (and in pretty much any language that doesn't require explicit declarations). Surely no-one is trying to suggest that this is a fundamental flaw?
It's good when that's what you want (which is most of the time once you've made the decision to use Python in the first place). It's not good when you're just wanting to name a subexpression to avoid evaluating it twice, and want to minimise the risk of introducing unintended side effects when doing so (hence the decision to hide comprehension iteration variables in Py3).
Seeing ".name" would be different (both for the compiler and for the human reader): if such a reference can't be resolved explicitly within the scope of the current statement, then *it's a bug* (and the compiler would be able to flag it as such at compile time).
Sorry, but you could use exactly that argument to propose that function local variables should be prefixed with "$". I don't buy it.
Function locals have been part of the language from the beginning though, rather than only being added after folks have already had years or decades to develop their intuitions about how variable name resolution works in Python. I'll also note that we *do* offer declarations to override what the compiler would infer by default based on assignment statements: global and nonlocal.
I guess I remain -1 on the proposal, and nothing that's getting said about how we can make it work is doing anything to persuade me otherwise (quite the opposite).
Yep, that's fair, as there are *lots* of viable alternatives to inline naming of subexpressions (each with various trade-offs). The question at hand is whether we can come up with semantics and syntax that folks actually like well enough to put to python-dev for a yes/no decision, with the pros and cons consolidated in one place. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 March 2018 at 12:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
The design intent related rationale stems from the fact that closed over references can live for an arbitrarily long time (as can regular function locals in a generator or coroutine), and I think statement locals should be as reliably ephemeral as we can reasonably make them if they're going to be different enough from function locals to be worth the hassle of adding them.
So that means asking a bunch of questions and deliberately giving the *opposite* answer for statement locals than I would for function locals:
* Visible in locals()? Yes for function locals, no for statement locals * Visible in frame.f_locals? Yes for function locals, no for statement locals * Prevents access to the same name in outer scopes? Yes for function locals, no for statement locals * Can be closed over? Yes for function locals, no for statement locals
Answer "no" to all those questions is only reasonable if statement local references *look* different from regular variable names. But I also think we need to proposing answering "No" to all of them to make statement locals potentially interesting enough to be worth considering.
OK, that argument makes sense to me. At least in the sense that it makes a good case for how statement-local names should work *if* we adopt them. I reserve the right to change my mind, as I haven't thought this through fully, but at least superficially I'm with you this far. I will note that I don't see this as a compelling reason for *having* them, just a good analysis of how they might behave if we do.
Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can figure out what is going on because it does the symbol table analysis pass before the code generation pass, and hence can tag names appropriately based on what it finds.
My concern is for people reading the code, where omitting a syntactic marker also forces *humans* to read things in two passes to make sure they understand them correctly. Consider this example:
x = 10 data = [x*x for i in range(10)]
This will give you a 10 item list, where each item is 100.
But now consider a world with statement locals, where statement locals use the same reference syntax as regular locals:
x = 10 data = [x*x for i in range(10) if (12 as x)]
That's a deliberately pathological example (and you can already write something similarly misleading using the "for x in [12]" trick), but the fact remains that we allow out of order evaluation in expressions in a way that we don't permit for statements.
OK. That's my concern as well. But my conclusion is different - I view this as a pretty strong argument that the complexity cost is too high to justify the feature, rather than as a difficulty we need to mitigate in order to make the feature usable.
With a syntactic marker though, there's typically going to be less ambiguity about where a name comes from:
x = 10 data = [.x*.x for i in range(10) if (12 as .x)]
It's not a panacea (since you may still be shadowing a name from an outer statement), and adapting it for PEP 3150 isn't without it's problems (specifically, you need to allow ".x = 12" in the given clause to make the names match up), but it should be less confusing than allowing a new subexpression to interfere with name resolution semantics folks have been relying on for years.
On my screen right now, I can barely see the dots. That's in email with a proportional font, so not "real coding", but it's surprising (and somewhat depressing :-() to think that probably the majority of code I read these days is in emails. So I don't think that this is a good enough fix to warrant making Tim's screen look gritty.
If statement locals behave just like function locals, then there's no reason to add them to the language in the first place - "(expr as name)" would just become a confusing second way to spell "name = expr".
Well, the whole reason this debate keeps coming up is because people keep finding places they want to type "name = expr" but they can't, because their context isn't a statement. We're basically trying to design a replacement for "name = expr" that can be used in an expression. And the status quo is "refactor your expression into multiple statements". Doing so isn't always ideal (and particularly grates on people coming from languages where assignments are expressions, like C/C++) but it is always possible. Adding "mustn't just be another way to spell name = expr" to the requirements is an extra requirement - and arguably one that contradicts the original requirement which was "find a way to allow name = expr in expressions". Designing a broader feature which solves the original problem is a good goal, but we need to take care not to introduce scope creep or over generalisation. In this case, IMO, we're not doing that (yet!) but we are hitting design problems which are orthogonal to the original requirement.
It's only potentially worthwhile if statement locals offer us something that function locals don't, and the clearest potential candidate for that differentiation is to provide a greater level of assurance regarding locality of use than regular name binding operations do.
I disagree. It's *still* potentially worthwhile if it just offers a clean resolution for people who want to name complex sub-parts of a comprehension. But the bar for "clean" in that case is higher, because the benefit is pretty limited. I don't think it reaches that higher bar, but I also am not sure that the benefits of the wider proposal are sufficient to warrant the amount of lowering of the bar that is needed. (That metaphor got way too strained. I hope it still made sense).
I guess I remain -1 on the proposal, and nothing that's getting said about how we can make it work is doing anything to persuade me otherwise (quite the opposite).
Yep, that's fair, as there are *lots* of viable alternatives to inline naming of subexpressions (each with various trade-offs). The question at hand is whether we can come up with semantics and syntax that folks actually like well enough to put to python-dev for a yes/no decision, with the pros and cons consolidated in one place.
Precisely. And the usual proviso that "status quo wins" applies strongly here, where (as noted in Chris' PEP) there are already *many* existing solutions. Paul
On 2 March 2018 at 23:26, Paul Moore <p.f.moore@gmail.com> wrote:
Adding "mustn't just be another way to spell name = expr" to the requirements is an extra requirement - and arguably one that contradicts the original requirement which was "find a way to allow name = expr in expressions".
I consider allowing the semantic equivalent of "name = expr" as part of arbitrary expressions to be an anti-requirement, not a requirement, for three reasons: 1. It means that arbitrary statements may accidentally clobber local variables that we care about. We used to have this for comprehension iteration variables, and it was so widely deplored that we broke compatibility with the variable overwriting behaviour in Python 3.0. 2. It means that naming a subexpression may significantly increase the longevity of that reference (especially in generators and coroutines). 3. It means that naming a subexpression will define a new class or module attribute when used in a class or module scope Folks do sometimes *ask* for these semantics, as it's the most obvious behaviour to ask for when it comes to naming subexpressions, but it would be a really bad idea to say "yes" to those requests. Chris's PEP addresses all those potential problems: 1. Statement locals explicitly avoid overwriting regular function locals (although they may shadow them in the current PEP - the '.name' syntax avoids even that) 2. Statement local bindings are cleared after each statement, so they only significantly extend the reference lifespan when used in compound statement headers or directly in the same statement as a yield or await. Even in comprehensions they'll typically be cleared on the next loop iteration. 3. Statement locals aren't added to locals() or globals(), so they never show up as module or class attributes either The eventual conclusion may still be "The proposal adds too much additional complexity without an adequate corresponding gain in expressiveness", but that complexity does have a solid rationale behind it. (Although it may be worth explicitly answering "Why not use the exact same semantics as 'name = expr'? in the PEP itself, since it's a reasonable question to ask) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 March 2018 at 14:23, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 March 2018 at 23:26, Paul Moore <p.f.moore@gmail.com> wrote:
Adding "mustn't just be another way to spell name = expr" to the requirements is an extra requirement - and arguably one that contradicts the original requirement which was "find a way to allow name = expr in expressions".
I consider allowing the semantic equivalent of "name = expr" as part of arbitrary expressions to be an anti-requirement, not a requirement, for three reasons:
1. It means that arbitrary statements may accidentally clobber local variables that we care about. We used to have this for comprehension iteration variables, and it was so widely deplored that we broke compatibility with the variable overwriting behaviour in Python 3.0.
2. It means that naming a subexpression may significantly increase the longevity of that reference (especially in generators and coroutines).
3. It means that naming a subexpression will define a new class or module attribute when used in a class or module scope
Folks do sometimes *ask* for these semantics, as it's the most obvious behaviour to ask for when it comes to naming subexpressions, but it would be a really bad idea to say "yes" to those requests.
Chris's PEP addresses all those potential problems:
1. Statement locals explicitly avoid overwriting regular function locals (although they may shadow them in the current PEP - the '.name' syntax avoids even that) 2. Statement local bindings are cleared after each statement, so they only significantly extend the reference lifespan when used in compound statement headers or directly in the same statement as a yield or await. Even in comprehensions they'll typically be cleared on the next loop iteration. 3. Statement locals aren't added to locals() or globals(), so they never show up as module or class attributes either
The eventual conclusion may still be "The proposal adds too much additional complexity without an adequate corresponding gain in expressiveness", but that complexity does have a solid rationale behind it. (Although it may be worth explicitly answering "Why not use the exact same semantics as 'name = expr'? in the PEP itself, since it's a reasonable question to ask)
OK, understood. Yes, I think it would be worth adding that question to the PEP. I also think it would be worth being more explicit in the body of the PEP over how[1] the binding of statement-local names is fundamentally different from assignment, as that certainly wasn't obvious to me, and may not be to others (particularly people reading it with the point of view "This satisfies my request to allow assignment in expressions"). I don't really have much more to add in that case. I don't personally think the added complexity is justified by the benefits, but I've already said that and it's been noted - so thanks for helping me understand the details of the proposal better, but it hasn't changed my view much in the end. Paul [1] The "why" is covered by the question you suggested.
Hi all, I would like to observe that there is already a way to bind names in an expression: a = (lambda b: [b, b])(b=f()) Currently this is not so performant, but there is nothing stopping the compiler from special-casing such an IIFE (Immediate Invoked Function Expression), to use the Javascript terminology. It would then be fast in, say, Python 3.8, but still work in earlier versions. Stephan 2018-03-02 14:26 GMT+01:00 Paul Moore <p.f.moore@gmail.com>:
The design intent related rationale stems from the fact that closed over references can live for an arbitrarily long time (as can regular function locals in a generator or coroutine), and I think statement locals should be as reliably ephemeral as we can reasonably make them if they're going to be different enough from function locals to be worth the hassle of adding
On 2 March 2018 at 12:48, Nick Coghlan <ncoghlan@gmail.com> wrote: them.
So that means asking a bunch of questions and deliberately giving the *opposite* answer for statement locals than I would for function locals:
* Visible in locals()? Yes for function locals, no for statement locals * Visible in frame.f_locals? Yes for function locals, no for statement locals * Prevents access to the same name in outer scopes? Yes for function
no for statement locals * Can be closed over? Yes for function locals, no for statement locals
Answer "no" to all those questions is only reasonable if statement local references *look* different from regular variable names. But I also
locals, think we
need to proposing answering "No" to all of them to make statement locals potentially interesting enough to be worth considering.
OK, that argument makes sense to me. At least in the sense that it makes a good case for how statement-local names should work *if* we adopt them. I reserve the right to change my mind, as I haven't thought this through fully, but at least superficially I'm with you this far.
I will note that I don't see this as a compelling reason for *having* them, just a good analysis of how they might behave if we do.
Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can figure out what is going on because it does the symbol table analysis pass before the code generation pass, and hence can tag names appropriately based on what it finds.
My concern is for people reading the code, where omitting a syntactic marker also forces *humans* to read things in two passes to make sure they understand them correctly. Consider this example:
x = 10 data = [x*x for i in range(10)]
This will give you a 10 item list, where each item is 100.
But now consider a world with statement locals, where statement locals use the same reference syntax as regular locals:
x = 10 data = [x*x for i in range(10) if (12 as x)]
That's a deliberately pathological example (and you can already write something similarly misleading using the "for x in [12]" trick), but the fact remains that we allow out of order evaluation in expressions in a way that we don't permit for statements.
OK. That's my concern as well. But my conclusion is different - I view this as a pretty strong argument that the complexity cost is too high to justify the feature, rather than as a difficulty we need to mitigate in order to make the feature usable.
With a syntactic marker though, there's typically going to be less ambiguity about where a name comes from:
x = 10 data = [.x*.x for i in range(10) if (12 as .x)]
It's not a panacea (since you may still be shadowing a name from an outer statement), and adapting it for PEP 3150 isn't without it's problems (specifically, you need to allow ".x = 12" in the given clause to make the names match up), but it should be less confusing than allowing a new subexpression to interfere with name resolution semantics folks have been relying on for years.
On my screen right now, I can barely see the dots. That's in email with a proportional font, so not "real coding", but it's surprising (and somewhat depressing :-() to think that probably the majority of code I read these days is in emails.
So I don't think that this is a good enough fix to warrant making Tim's screen look gritty.
If statement locals behave just like function locals, then there's no reason to add them to the language in the first place - "(expr as name)" would just become a confusing second way to spell "name = expr".
Well, the whole reason this debate keeps coming up is because people keep finding places they want to type "name = expr" but they can't, because their context isn't a statement. We're basically trying to design a replacement for "name = expr" that can be used in an expression. And the status quo is "refactor your expression into multiple statements". Doing so isn't always ideal (and particularly grates on people coming from languages where assignments are expressions, like C/C++) but it is always possible.
Adding "mustn't just be another way to spell name = expr" to the requirements is an extra requirement - and arguably one that contradicts the original requirement which was "find a way to allow name = expr in expressions".
Designing a broader feature which solves the original problem is a good goal, but we need to take care not to introduce scope creep or over generalisation. In this case, IMO, we're not doing that (yet!) but we are hitting design problems which are orthogonal to the original requirement.
It's only potentially worthwhile if statement locals offer us something that function locals don't, and the clearest potential candidate for that differentiation is to provide a greater level of assurance regarding locality of use than regular name binding operations do.
I disagree. It's *still* potentially worthwhile if it just offers a clean resolution for people who want to name complex sub-parts of a comprehension. But the bar for "clean" in that case is higher, because the benefit is pretty limited. I don't think it reaches that higher bar, but I also am not sure that the benefits of the wider proposal are sufficient to warrant the amount of lowering of the bar that is needed. (That metaphor got way too strained. I hope it still made sense).
I guess I remain -1 on the proposal, and nothing that's getting said about how we can make it work is doing anything to persuade me otherwise (quite the opposite).
Yep, that's fair, as there are *lots* of viable alternatives to inline naming of subexpressions (each with various trade-offs). The question at hand is whether we can come up with semantics and syntax that folks actually like well enough to put to python-dev for a yes/no decision, with the pros and cons consolidated in one place.
Precisely.
And the usual proviso that "status quo wins" applies strongly here, where (as noted in Chris' PEP) there are already *many* existing solutions.
Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 3 March 2018 at 10:09, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
I don't think it should be possible to close over statement locals.
Why not?
I gave a more detailed answer to that in https://mail.python.org/pipermail/python-ideas/2018-March/049138.html, but the short version is: * prohibiting closing over them opens up a lot of opportunities to simplify the implementation * prohibiting closing over them increase their semantic distance from regular function locals, making it easier for people to answer the question "Should I use a function local or a statement local?" in the future Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (19)
-
Alex Walters -
Brendan Barnwell -
Brett Cannon -
Chris Angelico -
David Mertz -
Ethan Furman -
Greg Ewing -
Gregory P. Smith -
Guido van Rossum -
Kirill Balunov -
Kyle Lahnakoski -
Matt Arcidy -
Nick Coghlan -
Paul Moore -
Rob Cliffe -
Robert Vanden Eynde -
Serhiy Storchaka -
Stephan Houben -
Stephen J. Turnbull