A "local" pseudo-function
A brain dump, inspired by various use cases that came up during the binding expression discussions. Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope. As an expression, it's "local" "(" arguments ")" - Because it "looks like" a function call, nobody will expect the targets of named arguments to be fancier than plain names. - `a=12` in the "argument" list will (& helpfully so) mean pretty much the same as "a=12" in a "def" statement. - In a "local call" on its own, the scope of a named argument begins at the start of the next (if any) argument, and ends at the closing ")". For the duration, any variable of the same name in an enclosing scope is shadowed. - The parentheses allow for extending over multiple lines without needing to teach editors (etc) any new tricks (they already know how to format function calls with arglists spilling over multiple lines). - The _value_ of a local "call" is the value of its last "argument". In part, this is a way to sneak in C's comma operator without adding cryptic new line noise syntax. Time for an example. First a useless one: a = 1 b = 2 c = local(a=3) * local(b=4) Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end: c = local(a=3, b=4, a*b) And just to be obscure, also the same: c = local(a=3, b=local(a=2, a*a), a*b) There the inner `a=2` temporarily shadows the outer `a=3` just long enough to compute `a*a` (4). This is one that little else really handled nicely: r1, r2 = local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), twoa = 2*a, ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa)) Everyone's favorite: if local(m = re.match(regexp, line)): print(m.group(0)) Here's where it's truly essential that the compiler know everything about "local", because in _that_ context it's required that the new scope extend through the end of the entire block construct (exactly what that means TBD - certainly through the end of the `if` block, but possibly also through the end of its associated (if any) `elif` and `else` blocks - and similarly for while/else constructs). Of course that example could also be written as: if local(m = re.match(regexp, line), m): print(m.group(0)) or more specifically: if local(m = re.match(regexp, line), m is not None): print(m.group(0)) or even: if local(m = re.match(regexp, line)) is not None: print(m.group(0)) A listcomp example, building the squares of integers from an iterable but only when the square is a multiple of 18: squares18 = [i2 for i in iterable if local(i2=i*i) % 18 == 0] That's a bit mind-bending, but becomes clear if you picture the kinda-equivalent nest: for i in iterable: if local(i2=i*i) % 18 == 0: append i2 to the output list That should also make clear that if `iterable` or `i` had been named `i2` instead, no problem. The `i2` created by `local()` is in a wholly enclosed scope. Drawbacks: since this is just a brain dump, absolutely none ;-) Q: Some of those would be clearer if it were the more Haskell-like local(...) "in" expression A: Yup, but for some of the others needing to add "in m" would be annoyingly redundant noise. Making an "in" clause optional doesn't really fly either, because then local(a='z') in 'xyz' would be ambiguous. Is it meant to return `'xyz'`, or evaluate `'z' in 'xyz'`? And any connector other than "in" would make the loose resemblance to Haskell purely imaginary ;-) Q: Didn't you drone on about how assignment expressions with complex targets seemed essentially useless without also introducing a "comma operator" - and now you're sneaking the latter in but _still_ avoiding complex targets?! A. Yes, and yes :-) The syntactic complexity of the fully general assignment statement is just too crushing to _sanely_ shoehorn into any "expression-like" context. Q: What's the value of this? local(a=7, local(a=a+1, a*2)) A: 16. Obviously. Q: Wow - that _is_ obvious! OK, what about this, where there is no `a` in any enclosing scope: local(a) A: I think it should raise NameError, just like a function call would. There is no _intent_ here to allow merely declaring a local variable without supplying an initial value. Q: What about local(2, 4, 5)? A: It should return 5, and introduce no names. I don't see a point to trying to outlaw stupidity ;-) Then again, it would be consistent with the _intent_ to require that all but the last "argument" be of the `name=expression` form. Q: Isn't changing the meaning of scope depending on context waaaay magical? A: Yup! But in a language with such a strong distinction between statements and expressions, without a bit of deep magic there's no single syntax I can dream up that could work well for both that didn't require _some_ deep magic. The gimmick here is something I expect will be surprising the first time it's seen, less so the second, and then you're never confused about it again. Q: Are you trying to kill PEP 572? A: Nope! But since this largely subsumes the functionality of binding expressions, I did want to put this out there before 572's fate is history. Binding expressions are certainly easier to implement, and I like them just fine :-) Note: the thing I'm most interested in isn't debates, but in whether this would be of real use in real code.
Hi Tim, This is interesting. Even "as is" I prefer this to PEP 572. Below are some comments and a slightly different idea inspired by yours (sorry!) On Fri, Apr 27, 2018 at 10:41 PM Tim Peters <tim.peters@gmail.com> wrote: [..]
As an expression, it's
"local" "(" arguments ")"
if local(m = re.match(regexp, line)): print(m.group(0))
It does look like a function call, although it has a slightly different syntax. In regular calls we don't allow positional arguments to go after keyword arguments. Hence the compiler/parser will have to know what 'local(..)' is *regardless* of where it appears. If you don't want to make 'local' a new keyword, we would need to make the compiler/parser to trace the "local()" name to check if it was imported or is otherwise "local". This would add some extra complexity to already complex code. Another problematic case is when one has a big file and someone adds their own "def local()" function to it at some point, which would break things. Therefore, "local" should probably be a keyword. Perhaps added to Python with a corresponding "from __future__" import. The other way would be to depart from the function call syntax by dropping the parens. (And maybe rename "local" to "let" ;)) In this case, the syntax will become less like a function call but still distinct enough. We will be able to unambiguously parse & compile it. The cherry on top is that we can make it work even without a "__future__" import! When we implemented PEP 492 in Python 3.5 we did a little trick in tokenizer to treat "async def" in a special way. Tokenizer would switch to an "async" mode and yield ASYNC and AWAIT tokens instead of NAME tokens. This resulted in async/await syntax available without a __future__ import, while having full backwards compatibility. We can do a similar trick for "local" / "let" syntax, allowing the following: "let" NAME "=" expr ("," NAME = expr)* ["," expr] * "if local(m = re.match(...), m):" becomes "if let m = re.match(...), m:" * "c = local(a=3) * local(b=4)" becomes "c = let a=3, b=4, a*b" or "c = (let a=3, b=4, a*b)" * for i in iterable: if let i2=i*i, i2 % 18 == 0: append i2 to the output list etc. Note that I don't propose this new "let" or "local" to return their last assignment. That should be done explicitly (as in your "local(..)" idea): `let a = 'spam', a`. Potentially we could reuse our function return annotation syntax, changing the last example to `let a = "spam" -> a` but I think it makes the whole thing to look unnecessarily complex. One obvious downside is that "=" would have a different precedence compared to a regular assignment statement. But it already has a different precedent in function calls, so maybe this isn't a big deal, considered that we'll have a keyword before it. I think that "let" was discussed a couple of times recently, but it's really hard to find a definitive reason of why it was rejected (or was it?) in the ocean of emails about assignment expressions. Yury
[Yury Selivanov <yselivanov.ml@gmail.com>]
This is interesting. Even "as is" I prefer this to PEP 572. Below are some comments and a slightly different idea inspired by yours (sorry!)
That's fine :-)
Not only for that reason, but because the semantics have almost nothing in common with function calls. For example, in local(a=1, b=a+1) the new binding of `a` needs to be used to establish the binding of `b`. Not to mention that a new scope may need to be established, and torn down later. To the compiler, it's approximately nothing like "a function call". "Looking like" a function call nevertheless has compelling benefits: - People already know the syntax for specifying keyword arguments. - The precedence of "=" in a function call is already exactly right for this purpose. So nothing new to learn there either. - The explicit parentheses make it impossible to misunderstand where the expression begins or ends. - Even if someone knows nothing about "local", they _will_ correctly assume that, at runtime, it will evaluate to an object. In that key respect it is exactly like the function call it "looks like". I do want to leverage what people "already know".
I believe it absolutely needs to become a reserved word. While binding expressions suffice to capture values in conditionals, they're not all that pleasant for use in expressions (that really wants a comma operator too), and it's a fool's errand to imagine that _any_ currently-unused sequence of gibberish symbols and/or abused keywords can plausibly suggest "and here we're also creating a new scope, and here I'm declaring some names to live in that scope". If all that is actually wanted, a new reserved word seems pragmatically necessary. That's a high bar. Speaking of which, "sublocal" would be a more accurate name, and less likely to conflict with existing code names, but after people got over the shock to their sense of purity, they'd appreciate typing the shorter "local" instead ;-) For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
Therefore, "local" should probably be a keyword. Perhaps added to Python with a corresponding "from __future__" import.
Yes.
I'm far less concerned about pain for the compiler than pain for the human reader. There's almost no precedent in Python's expression grammar that allows two names separated only by whitespace. The screams "statement" instead, whether "class NAME" or "async for" or "import modulename" or "global foo" or "for i in" or ... I'm afraid it would prove just too jarring to see that inside what's supposed to be "an expression". That's why I settled on the (admittedly unexpected) "pseudo-function" syntax. The exceptions involve expressions that are really test-&-branch structures in disguise ("and", "or", ternary "if"). In those we can find NAME NAME snippets, but the keyword is never at the start of those. So, curiously enough, I'd be fonder of result_expression "where" name=expression, ... than of "let" name=expression, ... if I hadn't already resigned myself to believing function-like syntax is overwhelmingly less jarring regardless.
Which was clever, and effective, but - as far as I know - limited to _statements_, where KEYWORD NAME thingies were already common as mud.
See the bullet list near the top for all the questions that _raises_ that don't even need to be asked (by users) when using a function-like syntax instead.
I assume that was meant to be c = (let a=3, a) * (let b=4, b) instead. In _an expression_, I naturally group the a = 3, a part as the unintended a = (3, a) When I'm thinking of function calls, though, I naturally use the intended (a=3), a grouping. I really don't want to fight with what "everyone already knows", but build on that to the extent possible.
In the second line there too, I did like that in if local(i2=i*i) % 18 == 0: the mandatory parentheses made it wholly explicit where "the binding part" ended.
Why not? A great many objects in Python are _designed_ so that their __bool__ method does a useful thing in a plain if object: test. In these common cases, needing to type if let name=object, object: instead of if let name=object: is an instance of what my pseudo-FAQ called "annoyingly redundant noise". Deciding to return the value of the last "argument" expression was a "practicality beats purity" thing.
Me too - but then I thought it was _already_ too wordy to require `let a="spam", a` ;-)
I think my response to that is too predictable by now to annoy you by giving it ;-)
I don't know either. There were a number of halfheartedly presented ideas, though, that floundered (to my eyes) on the rocks of trying to ignore that syntax ideas borrowed from "everything's an expression" functional languages don't carry over well to a language where many constructs aren't expressions at all. Or, conversely, that syntax unique to the opening line of a statement-oriented language's block doesn't carry over at all to expressions. I'm trying to do both at once, because I'm not a wimp - and neither are you ;-)
On Sat, Apr 28, 2018 at 2:07 AM Tim Peters <tim.peters@gmail.com> wrote: [...]
For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
Great! :) [...]
So, curiously enough, I'd be fonder of
result_expression "where" name=expression, ...
than of
"let" name=expression, ...
My gripe about the "where"-like syntax family (including "as", proposed by many) is that the whole expression needs to be read backwards to make sense of `name*. It's one of the things that annoy me in languages like SQL, where the relevant "AS" keyword can be buried a couple of screens below/above the area of interest. That's why I find the "let" form superior. [..]
instead. In _an expression_, I naturally group the
a = 3, a
part as the unintended
a = (3, a)
When I'm thinking of function calls, though, I naturally use the intended
(a=3), a
grouping. I really don't want to fight with what "everyone already knows", but build on that to the extent possible.
Looking at all of these and other examples in your email I have to agree that the version with parenthesis reads way more clearly. A different (and correct in this case) precedence of "," between parens is pretty much hardwired in our brains. [..]
Why not? A great many objects in Python are _designed_ so that their __bool__ method does a useful thing in a plain
if object:
test. In these common cases, needing to type
if let name=object, object:
Alright. And yes, I agree, surrounding parens are badly needed. Maybe we can replace parens with {}? Although my immediate reaction to that is "it's too ugly to be taken seriously". In any case, the similarity of this new syntax to a function call still bothers me. If I just looked at some Python 3.xx code and saw the new "let(..)" syntax, I would assume that the names it declares are only visible *within* the parens. Parens imply some locality of whatever is happening between them. Even if I googled the docs first to learn some basics about "let", when I saw "if let(name...)" it wouldn't be immediately apparent to me that `name` is set only for its "if" block (contrary to "if let a = 1: print(a)"). Yury
: On 28 April 2018 at 07:07, Tim Peters <tim.peters@gmail.com> wrote:
[...] For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
If you really wanted to overcome objections that this looks too much like a function, you could spell it "$". I'm not sure what I think about $(a=7, $(a=a+1, a*2)) yet, but it doesn't make me want to run screaming in the way that := does. I think I finally worked out why I have such a violent reaction to := in the end, by the way: it's because it reminds me of Javascript's "===" (not the meaning, but the fact that it exists). -[]z.
$(a=7, $(a=a+1, a*2))
I suspect you ALREADY have bash installed on your computer, you don't need Python to emulate it. On Sat, Apr 28, 2018 at 6:22 AM, Zero Piraeus <schesis@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Tim Peters wrote:
To the compiler, it's approximately nothing like "a function call".
It's nothing like a function call to the user, either, except in the most superficial of ways.
- The explicit parentheses make it impossible to misunderstand where the expression begins or ends.
Except that you then go and break that rule by saying that it doesn't apply when you're in the condition of an "if" statement.
I do want to leverage what people "already know".
Seems to me this proposal does that in the worst possible way, by deceiving the user into thinking it's something familiar when it's not. -- Greg
To all of the following, I was talking about the syntax. Which includes that all existing Python-aware editors and IDEs already know how to format it intelligibly. It would be nice if they also colored "local" (or "let") as a keyword, though. For the rest, of course I'm already aware of the _semantic_ trickery. Indeed, that may be too much to bear. But I'm already sympathetic to that too :-) On Sat, Apr 28, 2018 at 9:03 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 28/04/2018 04:34, Yury Selivanov wrote:
If things were to go the keyword direction I personally think that a great deal of clarity could be achieved by borrowing somewhat from Pascal with either use <exp> as <local_name>[, <exp> as <local_name> ...]: # Local names only exist in this scope, # existing names are in scope unless overridden # new names introduced here go out to nesting scope OR (for those who prefer name first, a function like look, and slightly less typing): using(<local_name>=<exp>[ ,<local_name>=<exp>...]): # Local names only exist in this scope, # existing names are in scope unless overridden # new names introduced here go out to nesting scope Lastly how about: <name> = <calculation> using(<local_name>=<exp>[ ,<local_name>=<exp>...]) # in this case where might be better than using Presumably, in the latter cases, the using function would return a sub_local dictionary that would only exist to the end of the scope be that the current line or indented block which should minimise the required magic. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com
On 28 April 2018 at 03:37, Tim Peters <tim.peters@gmail.com> wrote:
Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope.
This looks disturbingly good to me. I say "disturbingly" because the amount of magic involved in that "function call" is pretty high, and the more you look at it, the more weird its behaviour seems. I have essentially no real world use cases for this, though (other than ones that have been brought up already in the binding expression debate), so my comments are purely subjective opinion. [...]
This is where, in my mind, the magic behaviour goes too far. I can see why it's essential that this happens, but I can't come up with a justification for it other than pure expediency. And while I know "practicality beats purity" (look at me, daring to quote the Zen at Tim!!! :-)) this just feels like a step too far to me. Paul
On Fri, Apr 27, 2018 at 09:37:53PM -0500, Tim Peters wrote:
Chris' PEP 572 started off with the concept that binding expressions would create a "sub-local" scope, below function locals. After some debate on Python-Ideas, Chris, Nick and Guido took the discussion off list and decided to drop the sub-local scope idea as confusing and hard to implement. But the biggest problem is that this re-introduces exactly the same awful C mistake that := was chosen to avoid. Which of the following two contains the typo? local(spam=expression, eggs=expression, cheese = spam+eggs) local(spam=expression, eggs=expression, cheese == spam+eggs) I have other objections, but I'll leave them for now, since I think these two alone are fatal. Once you drop those two flaws, you're basically left with PEP 572 :-) -- Steve
[Steven D'Aprano <steve@pearwood.info>]
Enormously harder to implement than binding expressions, and the latter (to my eyes) capture many high-value use cases "good enough". I'm not concerned about "confusing". "Sub-local" scopes are ubiquitous in modern languages, and they're aimed at experienced programmers. It was also the case that nesting scopes _at all_ was very controversial in Python's earliest years, and Guido resisted it mightily (with my full support). The only scopes at first were function-local, module-global, and builtin, and while functions could _textually_ nest, they had no access to enclosing local scopes. Cute: to write a recursive nested function, you needed to pass it its own name (because its own name isn't in its _own_ local scope, but in the local scope of the function that contains it). Adding nested local scopes was also "confusing" at the time, and indeed made the scoping rules far harder to explain to newbies, and complicated the implementation. Then again, experienced programmers overwhelmingly (unanimously?) welcomed the change after it was done. Since then, Python has gone down a pretty bizarre path, inventing sublocal scopes on an ad hoc basis by "pure magic" when their absence in some specific context seemed just too unbearable to live with (e.g., in comprehensions). So we already have sublocal scopes, but in no explicit form that can be either exploited or explained.
Neither :-) I don't expect that to be a real problem. In C I'm _thinking_ "if a equals b" and type "if (a=b)" by mistake in haste. In a "local" I'm _ thinking_ "I want to create these names with these values" in the former case, and in the latter case also "and I want to to test whether cheese equals spam + eggs". But having already typed "=" to mean "binding" twice in the same line, "but the third time I type it it will mean equality instead" just doesn't seem likely. The original C mistake is exceedingly unlikely on the face of it: if what I'm thinking is "if a equals b", or "while a equals b", I'm not going to use "local()" _at all_. Forcing the programmer to be explicit about that they're trying to create a new scope limits the only possible confusions to cases where they _are_ going out of their way to use a "local()" construct, in which case binding behavior is very much at the top of their mind. Plain old "if a=b" remains a SyntaxError regardless. Still, if people are scared of that, a variation of Yury's alternative avoids it: the last "argument" must be an expression (not a binding). In that case your first line above is a compile-time error. I didn't like that because I really dislike the textual redundancy in the common if local(matchobject=re.match(regexp, line), matchobject): compared to if local(matchobject=re.match(regexp, line)): But I could compromise ;-) - There must be at least one argument. - The first argument must be a binding. - All but the last argument must also be bindings. - If there's more than one argument, the last argument must be an expression. Then your first line above is a compile-time error, the common "just name a result and test its truthiness" doesn't require repetition, and "local(a==b)" is also a compile-time error.
I have other objections, but I'll leave them for now, since I think these two alone are fatal.
I don't.
Once you drop those two flaws, you're basically left with PEP 572 :-)
Which is fine by me, but do realize that since PEP 572 dropped any notion of sublocal scopes, that recurring issue remains wholly unaddressed regardless.
On 2018-04-28 18:16, Tim Peters wrote:
How about if you just can't have an expression in a local()? There are a few obvious alternative possibilities: 1. No special case at all: local(x=1, y=2, _=x+y) 2. Co-opt a keyword after local(): local(x=1, y=2) in x+y 3. Co-opt a keyword inside local(): local(x=1, y=2, return x+y) I hate the first and wish the _ pattern would die in all its forms, but it's worth mentioning. I don't think there's much to choose between the other two, but 2 uses syntax that might have been valid and meant something else, so 3 is probably less confusing.
[Ed Kellett <e+python-ideas@kellett.im>]
How about if you just can't have an expression in a local()?
See the quadratic equation example in the original post. When working with expressions, the entire point of the construct is to define (sub)local names for use in a result expression.
As above.
2. Co-opt a keyword after local():
local(x=1, y=2) in x+y
Requiring "in" requires annoying syntactic repetition in the common if local(match=re.match(regexp, line)) in match: kinds of cases. Making "in" optional instead was discussed near the end of the original post. I agree that your spelling just above is more obvious than `local(x=1, y=2, x+y)` which is why the original post discussed making an "in clause" optional. But, overwhelmingly, it appears that people are more interested in establishing sublocal scopes in `if` and `while` constructs than in standalone expressions, so I picked a spelling that's _most_ convenient for the latter's common "just name a result and test its truthiness" uses.
3. Co-opt a keyword inside local():
local(x=1, y=2, return x+y)
Why would that be better than local(x=1, y=2, x+y)? That no binding is intended for `x+y` is already obvious in the latter.
Indeed, 3 is the only one I'd consider, but I don't see that it's a real improvement. It seems to require extra typing every time just to avoid learning "and it returns the value of the last expression" once.
On 2018-04-28 21:40, Tim Peters wrote:
I don't mean "in" should be required, but rather that the last thing is always an assignment, and the local() yields the assigned value. So that'd remain: if local(match=re.match(regexp, line)): whereas your quadratic equation might do the "in" thing: r1, r2 = local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), twoa = 2*a) in ( (-b + sqrtD)/twoa, (-b - sqrtD)/twoa) ... which doesn't look particularly wonderful (I think it's nicer with my option 3), but then I've never thought the quadratic equation was a good motivating case for any version of an assignment expression. In most cases I can imagine, a plain local() would be sufficient, e.g.: if local(match=re.match(regexp, line)) and match[1] == 'frog':
I agree with pretty much all of this--I was trying to attack the "but you could make the terrible C mistake" problem from another angle. I don't think I mind the last "argument" being *allowed* to be an expression, but if the consensus is that it makes '='/'==' confusion too easy, I think it'd be more straightforward to say they're all assignments (with optional, explicit syntax to yield an expression) than to say "they're all assignments except the last thing" (argh) "except in the special case of one argument" (double argh).
On Sat, Apr 28, 2018 at 12:16:16PM -0500, Tim Peters wrote:
And yet you're suggesting an alternative which is harder and more confusing. What's the motivation here for re-introducing sublocal scopes, if they're hard to do and locals are "good enough"? That's not a rhetorical question: why have you suggested this sublocal scoping idea? PEP 572 stopped talking about sublocals back in revision 2 or so, and as far as I can see, *not a single objection* since has been that the variables weren't sublocal. For what it is worth, if we ever did introduce a sublocal scope, I don't hate Nick's "given" block statement: https://www.python.org/dev/peps/pep-3150/ [...]
While I started off with Python 1.5, I wasn't part of the discussions about nested scopes. But I'm astonished that you say that nested scopes were controversial. *Closures* I would completely believe, but mere lexical scoping? Astonishing. Even when I started, as a novice programmer who wouldn't have recognised the term "lexical scoping" if it fell on my head from a great height, I thought it was strange that inner functions couldn't see their surrounding function's variables. Nested scopes just seemed intuitively obvious: if a function sees the variables in the module surrounding it, then it should also see the variables in any function surrounding it. This behaviour in Python 1.5 made functions MUCH less useful:
I think it is fair to say that inner functions in Python 1.5 were crippled to the point of uselessness.
I agree with the above regarding closures, which are harder to explain, welcomed by experienced programmers, and often a source of confusion for newbies and experts alike: https://stackoverflow.com/questions/7546285/creating-lambda-inside-a-loop http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/comment-page-1/ but I disagree that lexical scoping alone is or ever was confusing. Neither did Niklaus Wirth, who included it in Pascal, a language intended to be friendly for beginners *wink*
I'm not entirely sure that comprehensions (including generator expressions) alone counts as "a path" :-) but I agree with this. I'm not a fan of comprehensions being their own scope. As far as I am concerned, leakage of comprehension variables was never a problem that needed to be solved, and was (very occasionally) a useful feature. Mostly for introspection and debugging. But the decision was made for generator comprehensions to be in their own scope, and from there I guess it was inevitable that list comprehensions would have to match.
I'm sure the C designers didn't either. You miss the point that looking at the above, it is impossible to tell whether I meant assignment or an equality test. Typos of = for == do happen, even in Python, for whatever reason typos occur. Regardless of whether this makes them more likely or not (I didn't make that claim) once made, it is a bug that can fail silently in a way that is hard to see and debug. Most = for == typos in Python give an instant SyntaxError, but there are two places where they don't: - a statement like "spam == eggs" called for its side-effects only; - in a function call, func(spam==eggs, spam=eggs) are both legal. The first is so vanishingly rare that we can forget it. If you see spam = eggs as a statement, we can safely assume it means exactly what it says. Inside function calls, it's a bit less cut and dried: func(foo=bar) *could* be a typoed positional argument (foo == bar) but in practice a couple of factors mitigate that risk: - PEP 8 style conventions: we expect to see func(foo=bar) for the keyword argument case and func(foo == bar) for the positional argument case; - if we mess it up, unless there happens to be a parameter called foo we'll get a TypeError, not a silent bug. But with your suggested local() pseudo-function, neither mitigating factor applies and we can't tell or even guess which meaning is intended just by sight.
Of course people won't *consciously* think that the operator for equality testing is = but they'll be primed to hit the key once, not twice, and they'll be less likely to notice their mistake. I never make more = instead of == typos than after I've just spent a lot of time working on maths problems, even though I am still consciously aware that I should be using == I simply don't notice the error.
Given that while ... is one of the major motivating use-cases for binding expressions, I think you are mistaken to say that people won't use this local() pseudo-function in while statements. [...]
Indeed. That sort of "Repeat Yourself To Satisfy The Compiler" will be an ugly anti-pattern.
That's not really a complete specification of the pseudo-function though, since sometimes the sublocals it introduces extend past the final parenthesis and into the subsequent block. What will, for example, this function return? spam = eggs = "global" def func(arg=local(spam="sublocal", eggs="sublocal", 1)): eggs = "local" return (spam, eggs) Even if you think nobody will be tempted to write such "clever" (dumb?) code, the behaviour still has to be specified. [...]
I don't think that sublocal scopes is a recurring issue, nor that we need address it now. -- Steve
On Sun, Apr 29, 2018 at 12:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
No objections, per se, but I do know a number of people were saddened at their loss. And it's not like eliminating sublocals solved problems without creating more; it's just a different set of trade-offs. Sublocals have their value.
I'm not sure how you can distinguish them:
What you expect here is lexical scope, yes. But if you have lexical scope with no closures, the inner function can ONLY be used while its calling function is still running. What would happen if you returned 'inner' uncalled, and then called the result? How would it resolve the name 'x'? I can't even begin to imagine what lexical scope would do in the absence of closures. At least, not with first-class functions. If functions aren't first-class objects, it's much easier, and a nested function serves as a refactored block of code. But then you can't even pass a nested function to a higher order function. ChrisA
On Sun, Apr 29, 2018 at 01:36:31PM +1000, Chris Angelico wrote: [...]
Easily. Pascal, for example, had lexical scoping back in the 1970s, but no closures. I expect Algol probably did also, even earlier. So they are certainly distinct concepts. (And yes, Pascal functions were *not* first class values.)
Failing to resolve 'x' is an option. It would simply raise NameError, the same as any other name lookup that doesn't find the name. Without closures, we could say that names are looked up in the following scopes: # inner function, called from inside the creating function (1) Local to inner. (2) Local to outer (nonlocal). (3) Global (module). (4) Builtins. If you returned the inner function and called it from the outside of the factory function which created it, we could use the exact same name resolution order, except that (2) the nonlocals would be either absent or empty. Obviously that would limit the usefulness of factory functions, but since Python 1.5 didn't have closures anyway, that would have been no worse that what we had. Whether you have a strict Local-Global-Builtins scoping, or lexical scoping without closures, the effect *outside* of the factory function is the same. But at least with the lexical scoping option, inner functions can call each other while still *inside* the factory. (Another alternative would be dynamic scoping, where nonlocals becomes the environment of the caller.)
I can't even begin to imagine what lexical scope would do in the absence of closures. At least, not with first-class functions.
What they would likely do is raise NameError, of course :-) An inner function that didn't rely on its surrounding nonlocal scope wouldn't be affected. Or if you had globals that happened to match the names it was relying on, the function could still work. (Whether it would work as you expected is another question.) I expect that given the lack of closures, the best approach is to simply make sure that any attempt to refer to a nonlocal from the surrounding function outside of that function would raise NameError. All in all, closures are much better :-) -- Steve
Steven D'Aprano wrote:
Pascal, for example, had lexical scoping back in the 1970s, but no closures.
Well, it kind of had a limited form of closure. You could pass a procedure or function *in* to another procedure or function as a parameter, but there was no way to return one or pass it out in any way. This ensured that the passed-in procedure or function couldn't outlive its lexical environment. -- Greg
Steven D'Aprano wrote:
So what was the closure? If the surrounding function was still running, there was no need to capture the running environment in a closure?
You seem to be interpreting the word "closure" a bit differently from most people. It doesn't imply anything about whether a surrounding function is still running or not. A closure is just a piece of code together with a runtime environment. In typical Pascal implementations, a closure is represented by a (code_address, frame_pointer) pair, where the frame_pointer points to a chain of lexically enclosing stack frames. The language rules make it possible to manage the frames strictly stack-wise, which simplifies the memory management, but that doesn't make the closure any less of a closure. Contrast this with Modula-2, where only top-level functions can be passed as parameters. When you pass a function in Modula-2, only the code address is passed, with no environment pointer. -- Greg
On Tue, May 01, 2018 at 09:26:09PM +1200, Greg Ewing wrote:
That's not *all* it is, because obviously *every* function has both a piece of code and a runtime environment. y = 1 def func(): x = 2 return x+y Here, there's a local environment as well as an implicit global one. Surely we don't want to call this a closure? If we do, then all functions are closures and the term is just a synonym for function. Wikipedia gives a definition: a closure (also lexical closure or function closure) is a technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment. https://en.wikipedia.org/wiki/Closure_%28computer_programming%29 but also warns As different languages do not always have a common definition of the lexical environment, their definitions of closure may vary also and explicitly says that Pascal (at least standard Pascal) didn't do closures: Traditional imperative languages such as Algol, C and Pascal either do not support nested functions (C) or do not support calling nested functions after the enclosing function has exited (GNU C, Pascal), thus avoiding the need to use closures. Neal Gafter gives what I believe is *the* standard definition of closures, at least in academic and functional programming circles, probably taken from Guy Steele and Scheme: A closure is a function that captures the bindings of free variables in its lexical context. http://gafter.blogspot.com.au/2007/01/definition-of-closures.html Since Steele basically wrote the book on closures, I think we ought to take his definition seriously :-) (That's also the definition I was thinking of.) By this definition, if there are no free variables, or no captured bindings, then its not a closure. func() above isn't a closure, since "x" isn't a free variable, and while "y" is, the value of it isn't captured. Unfortunately, the terminology has become a real mess, especially since the technique has moved into languages like Ruby and Javascript. Martin Fowler declares that closures, lambdas and blocks are the same thing: https://martinfowler.com/bliki/Lambda.html although he does admit that technically you can have lambdas that aren't closures. I think Fowler is abusing the terminology since all three of his terms are orthogonal: - closures can be created with def or lambda and are not necessarily anonymous; - lambda comes from the lambda calculus, where it refers to anonymous function expressions, not specifically whether they capture the value of free variables in their environment; - "block" typically refers to the structure of the source code; in Ruby it also refers to a kind of first-class anonymous callable object. Such blocks may, or may not, contain free variables. At least I don't use this definition of closure: "a closure is a callback that Just Works." https://reprog.wordpress.com/2010/02/27/closures-finally-explained/ *wink*
Well... it's a pretty feeble closure, since Pascal doesn't support functions as first-class values. More like second-class. The c2 wiki describes Pascal: However, when a function is passed to another function and later called, it will execute in the lexical context it was defined in, so it is, IN SOME SENSE [emphasis added], "closed over" that context. http://wiki.c2.com/?LexicalClosure but I think that sense is the same sense that func() above is "closed over" the free value y. The loosest possible sense. If we accept that "Pascal has closures", they're pretty crap closures, since they don't let you do any of the interesting and useful things you can do in languages with "real" closures like Scheme and Python. And because it is based on an implementation detail (as you say, it is only *typical*, not mandatory, for Pascal inner functions to be implemented this way), we have to consider what happens if we come across a Pascal implementation which *doesn't* use a (code_address, frame_pointer) pair. Is that no longer Pascal? This is the interesting, and frustrating, thing about definitions: deciding where the boundaries lie, and we could argue the boundaries for days without getting anywhere :-) -- Steve
Steven D'Aprano wrote:
Python probably isn't the best choice of language for the point you're making, because even top-level functions in Python *do* carry a reference to their environment (the dict of the module they were defined in). C would be a better choice: int y = 1; int func(void) { int x = 2; return x + y; } int (*fp)() = func; Even here, I don't think there's anything wrong with calling fp a closure. It's just that it's a particularly simple form of closure -- since there's only one global environment, we don't need to bother explicitly carrying it around. However, that doesn't mean all functions are closures. An example of a non-closure might be a C++ member function pointer. It points to a piece of code, but you can't use it on its own, because it's missing a piece of environmental information (a value for "this") that you need to supply by hand when you call it.
In anything that implements Pascal semantics, a function parameter is going to have to be represented by *something* that specifies both code and environment. Code alone is not enough.
Not all of them, but it can do some of them. They're more useful than in Modula-2, for example, where nested functions can't even be passed as parameters. Anyway, that's why I said "a limited form of closure". You could read that two ways: (1) It's totally a closure, but there are limitations on what you can do with it (i.e. it's not first class). Or (2) it has some of the characteristics of a closure, but not all of them. In my mind I tend towards (1), but it doesn't really matter. -- Greg
[Tim]
Enormously harder to implement than binding expressions, and the latter (to my eyes) capture many high-value use cases "good enough".
[Steven D'Aprano <steve@pearwood.info>]
And yet you're suggesting an alternative which is harder and more confusing.
I am? I said at the start that it was a "brain dump". It was meant to be a point of discussion for anyone interested. I also said I was more interested in real use cases from real code than in debating, and I wasn't lying about that ;-) Since no real new use cases (let alone compelling ones) have turned up yet, I'm ready to drop it for now.
What's the motivation here for re-introducing sublocal scopes, if they're hard to do and locals are "good enough"?
That's why I wanted to see if there were significant unaddressed use cases. That _my_ particular itches would be scratched "good enough" if the PEP is accepted doesn't imply everyone's will be. And my particular itches will continue to annoy if the PEP is rejected.
That's not a rhetorical question: why have you suggested this sublocal scoping idea?
Putting an idea out for discussion isn't suggesting it be adopted. The list is named "python-ideas", not "python-advocacy-death-match" ;-)
Meh. Chris didn't seem all that thrilled about dropping them, and I saw a number of messages more-than-less supporting the idea _before_ they were dropped. When it became clear that the PEP didn't stand a chance _unless_ they were dropped, nobody was willing to die for it, because they weren't the PEP's _primary_ point.
And Chris just tried introducing it again. That's in reference to the last sentence of your reply: I don't think that sublocal scopes is a recurring issue How many times does it have to come up before "recurring" applies? ;-) I've seen it come up many times over ... well, literally decades by now.
But true. Guido agonized over it for a long time. Limiting to 3 scopes was a wholly deliberate design decision at the start, not just, .e.g, due to lack of time to implement lexical scoping at the start. And that shouldn't be surprising given Python's start as "somewhere between a scripting language and C", and the many influences carried over from Guido's time working on ABC's implementation team (ABC had no lexical scoping either - nor, if I recall correctly, even textual nesting of its flavor of functions). I'm glad he tried it! Everyone learned something from it.
I don't think that's fair to say. A great many functions are in fact ... functions ;-) That is, they compute a result from the arguments passed to them. They don't need more than that, although being able to access globals and builtins and import whatever they want from the standard library made them perfectly capable of doing a whole lot more than just staring at their arguments. To this day, _most_ of the nested functions I write would have worked fine under the original scoping rules, because that's all they need. Many of the rest are recursive, but would also work fine if I passed their names into them and rewrote the bits of code to do recursive calls via the passed-in name. But, yes, I am relieved I don't need to do the latter anymore ;-) ... [snip similar things about closures] ...
I didn't mind comprehensions "leaking" either. But I expect the need became acute when generator expressions were introduced, because the body of those can execute in an environment entirely unrelated to the definition site: def process(g): i = 12 for x in g: pass print(i) # was i clobbered? nope! process(i for i in range(3)) ... [snip more "head arguments" about "=" vs "=="] ...
It was talking about compile-time-checkable syntactic requirements, nothing about semantics.
It would return (spam, "local"), for whatever value `spam` happened to be bound to at the time of the call. `local()` had magically extended scope only in `if`/`elif` and `while` openers. In this example, it would the same if the `def` had been written def func(arg=1): Remember that default argument values are computed at the time `def` is executed, and never again. At that time, local(spam="sublocal", eggs="sublocal", 1) created two local names that are never referenced, threw the new scope away, and returned 1, which was saved away as the default value for `arg`.
Even if you think nobody will be tempted to write such "clever" (dumb?) code, the behaviour still has to be specified.
Of course, but that specific case wasn't even slightly subtle ;-)
Tim Peters wrote:
If Python hadn't allowed textual nesting either, folks might have been content to leave it that way. But having textual nesting without lexical scoping was just weird and confusing!
Yes, but they often make use of other functions to do that, and not being able to call other local functions in the same scope seemed like a perverse restriction. -- Greg
On Sat, Apr 28, 2018 at 11:20:52PM -0500, Tim Peters wrote:
Ah, my mistake... I thought you were advocating sublocal scopes as well as just brain dumping the idea. [...]
Sure, but not having access to the surrounding function scope means that inner functions couldn't call other inner functions. Given: def outer(): def f(): ... def g(): ... f cannot call g, or vice versa. I think it was a noble experiment in how minimal you could make scoping rules and still be usable, but I don't think that particular aspect was a success. -- Steve
On 04/28/2018 10:16 AM, Tim Peters wrote:
If we need a sublocal scope, I think the most Pythonic* route to have it would be: with sublocal(): blah blah which would act just like local/global does now: - any assignment creates a new variable - unless that variable has been declared global/nonlocal - plain reads (no assignment ever happens) refer to nonlocal/global/built-in names This has the advantages of: - no confusion about which variables are sublocal (acts like a new function scope) - no extra parens, assignments, expressions on "with sublocal():" line Possible enhancements: - give sublocal block a name "with sublocal() as blahblah:" and then reuse that block again later ("with blahblah:") or maybe pass it to other functions... Of course, as previously stated, this is orthogonal to PEP 572. -- ~Ethan~
[Ethan Furman <ethan@stoneleaf.us>]
As covered most recently in an exchange with Tim Delaney, best I can tell absolutely nobody has wanted that. By "sublocal scope" they don't mean a full-fledged new scope at all, but a kind of limited "shadowing" of a handful of specific, explicitly given names. It acts like a context manager, if there were a way to clearly spell save the current state of these specific identifiers at the start (& I couldn't care less whether they're local, nonlocal, or global - I don't know & don't care) then execute the code exactly as if this gimmick had never been used then, at the end, restore the specific identifier states we saved at the start It's the same kind of shadowing Python already does by magic for, e.g., `i`, in [i for i in range(3)] So, e.g., """ a = 42 def showa(): print(a) def run(): global a local a: # assuming this existed a = 43 showa() showa() """ would print 43 and then 42. Which makes "local a:" sound senseless on the face of it ;-) "shadow" would be a more descriptive name for what it actually does.
...
This doesn't address the fact no one actually needs it. But if we WANTED a sublocal() context manager, we could spell it something like this: In [42]: @contextmanager ...: def sublocal(**kws): ...: _locals = locals().copy() ...: _globals = globals().copy() ...: for k, v in kws.items(): ...: if k in locals(): ...: exec(f"locals()['{k}'] = {v}") ...: elif k in globals(): ...: exec(f"globals()['{k}'] = {v}") ...: yield ...: locals().update(_locals) ...: globals().update(_globals) ...: In [43]: a = 42 In [44]: with sublocal(a=43): ...: showa() ...: 43 In [45]: showa() 42 In [46]: with sublocal(): ...: a = 41 ...: showa() ...: 41 In [47]: showa() 42 On Sun, Apr 29, 2018 at 4:20 PM, Tim Peters <tim.peters@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 29 April 2018 at 21:20, Tim Peters <tim.peters@gmail.com> wrote:
So maybe adding such a primitive (maybe something live states = sys.get_variable_state('a', 'b', 'c') and sys.set_variable_state(states)) would be useful? Of course, we've moved away from real use cases and back to theoretical arguments now, so it's entirely possible that doing so would only solve problems that no-one actually has... David Mertz' sublocal context manager would be a good prototype of such a thing - at least good enough to demonstrate that it's of no benefit in practice <wink> Paul
Ooops. My proof on anti-concept has a flaw. It only "shadows" names that already exist. Presumably that's the wrong idea, but it's easy enough to change if desired. On Sun, Apr 29, 2018 at 5:24 PM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
[David Mertz <mertz@gnosis.cx>]
Even in the very early days when Python's runtime was more relentlessly simple-minded than it is now, these kinds of things were always hard to get right in all cases. But it's heartening to see _someone_ still has courage enough to leap into the void ;-) I see that the docs for `locals()` now say: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. I thought that explained something I was seeing, but the problem turned out to be more obvious: the "locals()" inside your "sublocal()" context manager refer to _sublocal_'s own locals, not to the locals of the code invoking `sublocal()`.. So, e.g., run this: def run(): a = 1 b = 2 def g(tag): print(f"{tag}: a {a} b {b}") with sublocal(a=6): g("first in block") a = 5 g("set a to 5") b = 19 g("set b to 19") g("after") Here's output: first in block: a 1 b 2 # the `a=6` had no visible effect set a to 5: a 5 b 2 # golden set b to 19: a 5 b 19 # also golden after: a 5 b 19 # `a` wasn't restored to 1 To be very clear, the output is the same as if the `with` statement were replaced with `if True:`. But even if we crawled up the call stack to get the right locals() dict, looks like `exec` is no longer enough (in Python 3) to badger the runtime into making it work anyway: https://bugs.python.org/issue4831 """
Specifically, what is the approved way to have exec() modify the local environment of a function?
There is none. To modify the locals of a function on the fly is not possible without several consequences: normally, function locals are not stored in a dictionary, but an array, whose indices are determined at compile time from the known locales. This collides at least with new locals added by exec. The old exec statement circumvented this, because the compiler knew that if an exec without globals/locals args occurred in a function, that namespace would be "unoptimized", i.e. not using the locals array. Since exec() is now a normal function, the compiler does not know what "exec" may be bound to, and therefore can not treat is specially. """ Worm around that (offhand I don't know how), and there are nonlocal names too. I don't know whether Python's current introspection features are enough to even find out which nonlocals have been declared, let alone to _which_ scope each nonlocal belongs. Worm around that too, then going back to the example at the top, if the manager's locals().update(_locals) had the intended effect, it would end up restoring `b` to 2 too, yes? The only names that "should be" restored are the names in the `kws` dict. So, in all, this may be a case where it's easier to implement in the compiler than to get working at runtime via ever-more-tortured Python code. And when that's all fixed, "a" can appear in both locals() and globals() (not to mention also in enclosing scopes), in which case what to do is unclear regardless how this is implemented. Which "a"(s) did the user _intend_ to shadow? The fun never ends ;-)
On Sun, Apr 29, 2018 at 9:28 PM, Tim Peters <tim.peters@gmail.com> wrote:
Actually, that wasn't my intention. As I imagined the semantics, I wanted a context manager that restored the "outside" context for anything defined "inside" the context. Allowing keyword arguments was just an extra "convenience" that was meant to be equivalent to defining/overwriting variables inside the body. So these would be equivalent: ## 1 with sublocal(): a = 1 b = 2 x = a + b # a, b now have their old values again ## 2 with sublocal(a=1, b=2): x = a + b # a, b now have their old values again ## 3 with sublocal(a=1): b = 2 x = a + b # a, b now have their old values again I knew I was ignoring nonlocals and nested function scopes. But just trying something really simple that looks like the un-proposal in some cases. Maybe there's no way in pure-Python to deal with the edge cases though.
[Tim]
[David]
What about x? I assume that's also restored.
Or even just locals - there doesn't appear to be any supported way in Python 3 for even `exec` to reliably change locals anymore. If the intent is to create an "honest-to-Guido new Python scope", then I expect changing BLOCK_OF_CODE to def unique_name(): BLOCK_OF_CODE unique_name() del unique_name gets pretty close? Any name "created" inside BLOCK_OF_CODE would be tagged by the compiler as function-local, and vanish when the function returned. Anything merely referenced would inherit the enclosing function's meaning. I can think of some warts. E.g., after global g if BLOCK_OF_CODE were g += 1 then wrapping that line alone in a function would lead to an UnboundLocalError when the function was called (because the `global g` isn't "inherited"). Anyway, since I've never done that in my life, never saw anyone else do it, and never even thought of it before today, I'm pretty confident there isn't a hitherto undetected groundswell of demand for a quick way to create an honest-to-Guido new Python scope ;-)
On 2018-04-29 10:20 PM, Ethan Furman wrote:
That ain't shadow. That is dynamic scoping. Shadowing is something different: def f(): a = 42 def g(): print(a) local a: a = 43 g() g() should print "42" both times, *if it's lexically scoped*. If it's lexically scoped, this is just adding another scope: blocks. (instead of the smallest possible scope being function scope)
I really liked the syntax that mimicked lambda even if I find it verbose : a = local x=1, y=2: x + y + 3 Even if I still prefer the postfix syntax : a = x + 3 where x = 2 About scheme "let" vs "let*", the paralel in Python is : a, b, c = 5, a+1, 2 # let syntax a = 5; b = a+1; c = 2 # let* syntax Which makes be wonder, we could use the semicolon in the syntax ? a = local x = 1; y = x+1: x + y + 3 Or with the postfix syntax : a = x + y + 3 where x = 1; y = x+1 Chaining where would be is syntax error : a = x + y + 3 where x = 1 where y = x+1 Parenthesis could be mandatory if one wants to use tuple assignment : a = local (x, y) = 1, 2: x + y + 3 When I see that, I really want to call it "def" a = def (x, y) = 1, 2: x + y + 3 a = def x = 1; y = 2: x + y + 3 Which is read define x = 1 then y = 2 in x + y + 3 Using def would be obvious this is not a function call.
[Soni L. <fakedme+py@gmail.com>]
That ain't shadow. That is dynamic scoping.
I don't believe either term is technically accurate, but don't really care.
Why? The `local` statement, despite its name, and casual talk about it, isn't _intended_ to create a new scope in any technically accurate sense. I think what it is intended to do has been adequately explained several times already. The `a` in `a = 42` is intended to be exactly the same as the `a` in `a = 43`, changing nothing at all about Python's lexical scoping rules. It is _the_ `a` local to `f`. If lexical scoping hasn't changed one whit (and it hasn't), the code _must_ print 43 first. Same as if `local a:` were replaced by `if True:`. `local` has no effect on a's value until the new "scope" _ends_, and never any effect at all on a's visibility (a's "scope"). "local" or not, after `a = 43` there is no scope anywhere, neither lexical nor dynamic, in which `a` is still bound to 42. _The_ value of `a` is 43 then, `local` or not. The _only_ twist is that `local` wants to save/restore `a`'s binding before/after the suite of code it controls. That's not really about "scope" at all with its technical meaning. I don't much care if people casually _think_ about it as being "about.scope", though. Yes, the effect is similar to what you might see in a language with dynamic scoping _if_ it pushed a _copy_ of a's current <name, object> binding on an evaluation stack at the start, and popped that (possibly mutated) copy later, but actual dynamic binding doesn't push copies, and Python isn't using any sort of dynamic evaluation stack regardless. That both restore previous bindings at times is a shallow coincidence. Neither is it really shadowing, which is lexical hiding of names. Neither is an accurate model for what it actually does, but, yes, it bears more _resemblance_ to dynamic scoping if you ignore that it's not dynamic scoping ;-)
If it's lexically scoped, this is just adding another scope: blocks. (instead of the smallest possible scope being function scope)
I expect that thinking about "scope" at all just confuses people here, unless they don't think too much about it ;-) Nothing about Python's scoping rules changes one whit. Exactly the same in the above could be achieved by replacing the `local a:` construct above by, e.g,, __save_a_with_a_unique_name = a a = 43 g() a = __save_a_with_a_unique_name Indeed, that's one way the compiler could _implement_ it. Nothing about a's scope is altered; it's merely a block-structured save-value/restore-value gimmick.
On 2018-04-30 03:49, Tim Peters wrote:
[snip] I think it should be lexically scoped. The purpose of 'local' would be to allow you to use a name that _might_ be used elsewhere. The problem with a dynamic scope is that you might call some global function from within the local scope, but find that it's "not working correctly" because you've inadvertently shadowed a name that the function refers to. Imagine, in a local scope, that you call a global function that calls 'len', but you've shadowed 'len'...
[MRAB <python@mrabarnett.plus.com>]
I think it should be lexically scoped.
That's certainly arguable, but that's why I like real-code driven design: abstract arguments never end, and often yield a dubious in-real-life outcome after one side is worn out and the other side "wins" by attrition ;-)
Already explained at excessive length that there's nothing akin to "dynamic scopes" here, except that both happen to restore a previous binding at times. That's a shallow coincidence. It's no more "dynamic scope" than that savea = a try: a += 1 f(a) finally: a = savea is "dynamic scoping". It's merely saving/restoring a binding across a block of code.
Imagine, in a local scope, that you call a global function that calls 'len', but you've shadowed 'len'...
I'm not clear on whether you picked the name of a builtin to make a subtle point not spelled out, but I don't think it matters. Regardless of whether `len` refers to a builtin or a module global inside your global function now, the _current_ def f(): len = 12 global_function() has no effect at all on the binding of `len` seen inside `global_function`. Because my understanding of "local:" changes absolutely nothing about Python's current scope rules, it's necessarily the case that the same would be true in: def f(): local len: len = 12 call_something() The only difference from current semantics is that if print(len) were added after the `local:` block, UnboundLocalError would be raised (restoring the state of the function-local-with-or-without-'local:' `len` to what it was before the block). To have "local:" mean "new nested lexical scope" instead requires specifying a world of semantics that haven't even been mentioned yet. In Python today, in the absence of `global` and `nonlocal` declarations, the names local to a given lexical scope are determined entirely by analyzing binding sites. If you intend something other than that, then it needs to be spelled out. But if you intend to keep "and names appearing in binding sites are also local to the new lexical scope", I expect that's pretty much useless. For example, def f(): ... local x. y: x = a*b y = a/b r1, r2 = x+y, x-y That is, the programmer surely doesn't _intend_ to throw away r1 and r2 when the block ends. If they have to add a nonlocal r1, r2 declaration at the top of the block, maybe it would work as intended. But it still wouldn't work unless `r1` and `r2` _also_ appeared in binding sites in an enclosing lexical scope. If they don't, you'd get a compile-time error like SyntaxError: no binding for nonlocal 'r1' found To be more accurate, the message should really say "sorry, but I have no idea in which scope you _intend_ 'r1' to live, because the only way I could know that is to find a binding site for 'r1', and I can't find any except inside _this_ scope containing the 'nonlocal'". But that's kind of wordy ;-) If you agree that makes the feature probably unusable, you don't get off the hook by saying "no, unlike current Python scopes, binding sites have nothing to do with what's local to a new lexical scope introduced by 'local:'". The same question raised in the example above doesn't go away: in which scope(s) are 'r1' and 'r2' to be bound? There's more than one plausible answer to that, but in the absence of real use cases how can they be judged? Under "'local:' changes nothing at all about Python's scopes", the answer is obvious: `r1` and `r2` are function locals (exactly the same as if "local:" hadn't been used). There's nothing new about scope to learn, and the code works as intended on the first try ;-) Of course "local:" would be a misleading name for the construct, though. Going back to your original example, where a global (not builtin) "len" was intended: def f(): global len # LINE ADDED HERE local len: len = 12 global_function() yes, in _that_ case the global-or-builtin "len" seen inside `global_function` would change under my "nothing about scoping changes" reading, but would not under your reading. That's worth _something_ ;-) But without fleshing out the rules for all the other stuff (like which scope(s) own r1 and r2 in the example above) I can't judge whether it's worth enough to care. All the plausibly realistic use cases I've considered don't _really_ want a full-blown new scope (just robust save/restore for a handful of explicitly given names), and the example just above is contrived in comparison. Nobody types "global len" unless they _intend_ to rebind the global `len`, in which case i'm happy to let them shoot both feet off ;-) In any case, nothing can change the binding of the builtin "len" short of mucking directly with the mapping object implementing builtin lookups. Note: most of this doesn't come up in most other languages because they require explicitly declaring in which scope a name lives. Python's "infer that in almost all cases instead from examining binding sites" has consequences.
On 2018-04-30 21:41, Tim Peters wrote:
After all, what's the point of specifying names after the 'local' if _any_ binding in the local scope was local?
local b: local c: c = 0 # Bound in the "local c" scope. b = 0 # Bound in the "local b" scope. a = 0 # Bound in the main scope (function, global, whatever)
Would/should it be possible to inject a name into a local scope? You can't inject into a function scope, and names in a function scope can be determined statically (they are allocated slots), so could the same kind of thing be done for names in a local scope?
[MRAB <python@mrabarnett.plus.com>]
Don't look at me ;-) In the absence of use cases, I don't know which problem(s) you're trying to solve. All the use cases I've looked at are adequately addressed by having some spelling of "local:" change nothing at all about Python's current scope rules. If you have uses in mind that require more than just that, I'd need to see them.
Any binding that's not specified as local is bound in the parent scope:
Reverse-engineering the example following, is this a fair way of making that more precise? Given a binding-target name N in scope S, N is bound in scope T, where T is the closest-containing scope (which may be S itself) for which T is either 1. established by a "local:" block that declares name N or 2. not established by a "local: block
By clause #1 above, "c" is declared in the starting "local:" scope.
b = 0 # Bound in the "local b" scope.
By clause #1 above, after searching one scope up to find `b` declared in a "local:" scope
a = 0 # Bound in the main scope (function, global, whatever)
By clause #2 above, after searching two scopes up and not finding any "local:" scope declaring name "a". By your original "the parent scope", I would have expected this be bound in the "local b:" scope (which is the parent scope of the "local c:" scope). So that's _a_ possible answer. It's not like the scoping rules in any other language I'm aware of, but then Python's current scoping rules are unique too. Are those useful rules? Optimal? The first thing that popped into your head? The third? Again I'd need to see use cases to even begin to guess. I agree it's well defined, though, and so miles ahead of most ideas ;-) ...
Sorry, I'm unclear on what "inject a name into a local scope" means. Do you mean at runtime? In Python's very early days, all scope namespaces were implemented as Python dicts, and you could mutate those at runtime any way you liked. Name lookup first tried the "local" dict ("the" because local scopes didn't nest), then the "global" dict, then the "builtin" dict. Names could be added or removed from any of those at will. People had a lot of fun playing with that, but nobody seriously complained as that extreme flexibility was incrementally traded away for faster runtime. So now take it as given that the full set of names in a local scope must be determinable at compile-time (modulo whatever hacks may still exist to keep old "from module import *" code working - if any still do exist). I don't believe CPython has grown any optimizations preventing free runtime mutation of global (module-level) or builtin namespace mappings, but I may be wrong about that.
[MRAB]
Any binding that's not specified as local is bound in the parent scope:
[Tim]
Here's an example where I don't know what the consequences of "the rules" should be: def f(): a = 10 local a: def showa(): print("a is", a) showa() # 10 a = 20 showa() # 20 a = 30 showa() # 10 The comments show what the output would be under the "nothing about scope rules change" meaning. They're all obvious (since there is is no new scope then - it's all function-local). But under the other meaning ...? The twist here is that `def` is an executable statement in Python, and is a "binding site" for the name of the function being defined. So despite that `showa` appears to be defined in a new nested lexical scope, it's _actually_ bound as a function-local name. That's bound to be surprising to people from other languages: "I defined it in a nested lexical scope, but the name is still visible after that scope ends?". I don't know what the first `showa()` is intended to do. Presumably `a` is unbound at the start of the new nested scope? So raises NameError? If so, comment that line out so we can make progress ;-) It seems clear that the second `showa()` will display 20 under any reading. But the third? Now we're out of the `local a:` scope, but call a function whose textual definition was inside that scope. What does `showa()` do now to find a's value? f's local `a` had nothing to do with the `a` in the nested scope, so presumably it shouldn't display 10 now. What should it do? Does the final state of the nested scope's locals need to preserved so that showa() can display 30 instead? Or ...? Not necessarily complaining - just fleshing out a bit my earlier claim that a world of semantics need to be defined if anything akin to a "real scope" is desired.
[Tim]
Just noting that a sane way to answer such questions is to model the intent with nested functions in current Python. It's annoying because you have to explicitly declare the _non_-local names instead, and also make dummy assignments to those names to tell Python in which scope they _should_ be bound. So, e.g., for the above you can write this: def f(): a = 10 # Have to give `showa` _some_ binding in its intended scope, # else the "nonlocal showa" below is a compile-time error showa = None def local_a(): nonlocal showa def showa(): print("a", a) #showa() # raises NameError a = 20 showa() # 20 a = 30 local_a() del local_a showa() # 30 print("but f's a is", a) # 10 The comments show what happens when you call `f()`, matching the guesses in the original message. One thing to note: if this does model the intent, it's essentially proof that it's implementable without intractable effort; e.g., the machinery needed to implement funky closures already exists. But another: since I've never seen code anything like this before, that casts doubt on whether the semantics are actually useful in real life. That's why I always ask for real use cases ;-)
On 2018-05-01 04:40, Tim Peters wrote: the local scope's "a", not the function scope's "a". Imagine renaming the specified names that are declared 'local' throughout the nested portion: def f(): a = 10 local local_a: def showa(): print("a is", local_a) showa() # Raises NameError in showa because nothing bound to local_a yet. local_a = 20 showa() # 20 local_a = 30 showa() # Raises NameError in f because local_a not defined.
[MRAB]
In a later message I showed executable-today Python code that I believe models your intent. That code does indeed display "30" for the last call, and while I myself don't see that it's _useful_ yet, the model is incomprehensible if it doesn't. It doesn't matter that local_a isn't defined at function scope, because showa() refers to the locals _in_ the "local a:" scope. The last value bound to local_a was 30, so that's what showa() needs to show. That the name `showa` is _bound_ in f's locals has nothing to do with whose locals showa() _uses_. Other current messages about "closures" go on more about that.
[MRAB]
[Tim]
[MRAB]
Under my meaning, the same answer as always: exactly the same as if "local a:' were replaced by "if True:" ;-) Under yours, you're not going to like the answer. It seems obvious that since globals() refers to the module dict, nothing about that would change. You can add prints to my workalike-code-under-current-Python snippet to see what locals() _does_ return in various places. It's not what you'd expect (that locals() inside the `local a:` block is a dict with only key "local_a"), because of what I believe are undocumented implementation details. This is much easier to see in a bare-bones example: def f(): b = 12 def g(): nonlocal b b = 17 a = 42 print("inner", locals()) g() print("outer", locals()) f() The output: inner {'a': 42, 'b': 17} outer {'g': <function f.<locals>.g at 0x0000024E28C44AE8>, 'b': 17} That is, despite that `b` is explicitly declared as `nonlocal` in the inner function, and does in fact belong to the outer scope, it nevertheless shows up inside the inner function's locals(). The compiler isn't confused, but `locals()` existed long before nested scopes were added, and, e.g., no "nonlocals()` builtin was added later. PEP 227 (which introduced nested scopes) explicitly declined to add one, and said "it will not be possible to gain dictionary-style access to all visible scopes". I was quite surprised to see "b" as a key in the inner locals() above! But so long as it is, any new gimmick building on the current implementation would inherit that behavior.
[MRAB]
There's another question that hasn't been asked yet: what should locals() and globals() return?
[Tim, "globals()" is obvious, "locals()" can be surprising now]
...
And here recording the results of some code spelunking Dicts don't really have anything to do with how locals are implemented anymore; calling "locals()" now inside a function _constructs_ a dict on the fly out of lower-level implementation gimmicks, to be kinda-compatible with what locals() returned before nested scopes were introduced. The real distinctions of interest are recorded in the code object now, under these attributes, where I'm giving a fuller explanation than can be found in the docs, and where "referenced" doesn't distinguish between "binding merely retrieved" and "binding is changed": 1. co_varnames: tuple of names of locals not referenced in an enclosed local scope 2. co_cellvars: tuple of names of locals that are referenced in an enclosed local scope 3 .co_freevars: tuple of names local to an enclosing local scope and referenced in this enclosed code "locals()" now generally builds a dict out of all three of those name sources. #1 is the only one that made sense before nested scopes were introduced. The union of #1 and #2 is the set of names local to the code's scope; CPython implements their bindings in different ways, so needs to distinguish them. The names in #3 aren't local to the code block at all (they are local to some enclosing local scope), but for whatever reason are included in the code's "locals()" dict anyway. I would not have included them. For concreteness: def disp(func): print("for", func.__name__) code = func.__code__ for attr in "co_varnames", "co_cellvars", "co_freevars": print(" ", attr, getattr(code, attr)) def outer(): outer_only = 1 outer_used_inner = 2 outer_bound_by_inner = 3 def inner(): nonlocal outer_bound_by_inner inner_only = 4 outer_bound_by_inner = 5 inner_only = outer_used_inner inner() disp(outer) disp(inner) outer() And its output: for outer co_varnames ('outer_only', 'inner') co_cellvars ('outer_bound_by_inner', 'outer_used_inner') co_freevars () for inner co_varnames ('inner_only',) co_cellvars () co_freevars ('outer_bound_by_inner', 'outer_used_inner')
On Mon, Apr 30, 2018 at 08:52:13PM -0500, Tim Peters wrote:
I don't know what MRAB means by "inject", but I know what *I* mean, and I have a real use-case for it. There is a long-running micro-optimization, often championed by Raymond, for avoiding slow global lookups (and even slower builtin lookups, since they require a global lookup to fail first) by turning them in local lookups at function-definition time. E.g. some methods from the random.Random class: def randrange(self, start, stop=None, step=1, _int=int): ... def _randbelow(self, n, int=int, maxsize=1<<BPF, type=type, Method=_MethodType, BuiltinMethod=_BuiltinMethodType): ... (copied from 3.5, I don't know what the most recent version looks like) That's a nice way to make the binding: _int = int occur once only, instead of putting it inside the function body which then needs to be executed on ever call. Effectively it's a static variable for the method, one which persists from one call to the next without requiring re-initialisation. But it's ugly :-( The function signature is full of *implementation details* instead of the parameters needed for the method's interface. Look at _randbelow which takes one actual parameter, n, plus FIVE fake parameters, int, maxsize, type, Method and BuiltinMethod, none of which should ever be passed arguments. So when I talk about injecting values into a function, that is the sort of thing I'm referring to: at function definition time, push or inject a reference to a known value (say, the builtin int) into something which behaves as a static variable. It would be nice if we could do that without polluting the function signature. I'll admit that the name "inject" came to me when I was thinking of some hypothetical decorator: @inject(int=int, maxsize=1<<BPF, type=type, ...) def _randbelow(self, n): ... that somehow pushed, or *injected*, those bindings into the function, turning them into locals, but in an alternative universe where Guido loved making new keywords, I'd use a static initialisation block and stick it inside the def: def _randbelow(self, n): static: # this gets executed once only, at function definition time int=int maxsize=1<<BPF type=type # body of _randbelow ... -- Steve
[MRAB]
So you do mean at runtime, I think. Then as before, you can do that with module and the builtin namespaces now, but all function locals need to be identifiable at compile time now. People often get confused about that when they read that "it's a binding" that makes a name local to a scope, but it's not "_doing_ a binding" that makes it local, merely that the name _syntactically_ appears as a target in a binding construct. That analysis is entirely done at compile time. In Python's very early days, all namespaces could be dynamically altered at any time in any way. As time went on, more & more restrictions were placed on runtime alteration of local scopes. I don't believe that, in current Python 3, dynamic alteration of local scopes is supported in any case anymore. Perhaps the last to go was runtime alteration of locals via `exec`. This still works in Python 2:
`exec` was a statement type in Python 2, so the compiler could recognize that and generate radically different name-lookup code in a scope containing an `exec` statement. But in Python 3, `exec` is just another builtin function, and the compiler knows nothing about what it does. Similar code run in Python 3 has no effect on f's locals.
[MRAB]
... [Steven D'Aprano <steve@pearwood.info>]
Before nested scopes, it was also used to help nested functions call each other; e.g., def f(): def g(): ... # h needs to call g, but can't see f's locals def h(g=g): # but default arg expressions can see f's locals g() # works great :-)
Doesn't matter; code like that has been in the std lib since the start, although Raymond is inordinately fond of it ;-)
Error-prone too, because absolutely nothing stops a user from calling these things with more positional arguments than are intended. I don't want to bother looking now, but there was _some_ "bug report" complaining that one of these "default arguments" went away somewhere, which a user was specifying intentionally to override some accidental implementation detail. However, while it's dead obviously "error prone" in theory, it's remarkable how few times I've heard of anyone getting burned by it in practice.
I'm surprised that hasn't already been done.
Blame computer scientists. I started my paying career working on Cray Research's Fortran compiler, a language in which NO words were reserved. The syntax was such that it was always possible to determine whether a language keyword or a user-supplied name was intended. That worked out great. But it seemed to require ad hoc parsing tricks. Computer scientists invented "general" parser generators, and then every new language started declaring that some words were reserved for the language's use. That was just to make things easier for parser-generator authors, not for users ;-) Maybe we could steal a trick from the way the async statements ("async def", "async for", "async with") were added without making "async" a reserved word? Even if user code already uses the name "static", just as in Fortran the only way to parse static: that makes sense is that it's introducing a block; USER_NAME: as a statement has no meaning. In any other context, the block-opening meaning of "static" makes no sense, so it must mean a user-defined name then. We could also add, e.g., MRAB local a, b, c: and Uncle Timmy not really local at all a, b, c: without introducing any actual ambiguity in code already using any of the leading words on those lines. It would be fun to deprive Guido of the dead easy "but it's a new reserved word!" way to veto new features ;-)
I have to say, this idea feels really nice to me. It's far easier to read than := and separates the assignments and the result expression nicely. Others have brought up the same problem of = vs ==. IMO a solution could be to make a requirement that the last argument is NOT an assignment. In other words, this would be illegal: local(a=1) and you would have to do this: local(a=1, a) Now if the user mixes up = and ==, it'd be a "compile-time error". -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ On April 27, 2018 9:41:57 PM Tim Peters <tim.peters@gmail.com> wrote:
On 2018-04-27 11:37 PM, Tim Peters wrote:
Why not non-lexical variables? Basically, make this work (and print 3): def test(): i = 3 def test_inner(): print(i) hide i i = 4 test_inner() # 3 print(i) # 4 `hide`, unlike `del`, only applies to the current scope, and only forward. it does what it says on the tin: makes the variable disappear/be hidden. ofc, python doesn't support shadowing of variables, so this is kinda useless, but eh I thought it was a cool idea anyway :/ (See also this thread on the Lua mailing list: https://marc.info/?l=lua-l&m=152149915527486&w=2 )
On Sat, Apr 28, 2018 at 12:37 PM, Tim Peters <tim.peters@gmail.com> wrote:
I'm concerned that there are, in effect, two quite different uses of the exact same syntax. 1) In an arbitrary expression, local() creates a scope that is defined entirely by the parentheses. 2) In an 'if' header, the exact same local() call creates a scope that extends to the corresponding suite. For instance: a = 1; b = 2 x = a + local(a = 3, b = 4, a + b) + b if x == 10: # Prints "x is 10: 1 2" print("x is 10: ", a, b) This makes reasonable sense. The parentheses completely enclose the local scope. It's compiler magic, and you cannot explain it as a function call, but it makes intuitive sense. But the same thing inside the if header itself would be much weirder. I'm actually not even sure what it would do. And you've clearly shown that the local() call can be anywhere inside the condition, based on these examples:
At what point does the name 'm' stop referring to the local? More generally: if local(m = ...) is not m: print("Will I ever happen?") Perhaps it would be better to make this special case *extremely* special. For instance: if_local: 'if' 'local' '(' local_item (',' local_item)* ')' ':' suite as the ONLY way to have the local names persist. In other words, if you tack "is not None" onto the outside of the local() call, it becomes a regular expression-local, and its names die at the close parentheses. It'd still be a special case, but it'd be a bit saner to try to think about. ChrisA
On 2018-04-28 18:36, Chris Angelico wrote:
What if the names persist until the end of the statement? That covers if (where the statement lasts until the end of the if...elif...else block) and regular expressions, though it does introduce a potentially annoying shadowing thing:
On Sun, Apr 29, 2018 at 6:04 AM, Ed Kellett <e+python-ideas@kellett.im> wrote:
Oh, you mean exactly like PEP 572 used to advocate for? :) Can't say I'd be against that, although I'm not enamoured of the function-like syntax. But if you remove the function-like syntax and change the semantics, it isn't exactly Tim's proposal any more. :) ChrisA
[Chris Angelico <rosuav@gmail.com>]
I'm concerned that there are, in effect, two quite different uses of the exact same syntax.
Yes, the construct implements a profoundly different meaning of "scope" depending on the context it appears in.
1) In an arbitrary expression, local() creates a scope that is defined entirely by the parentheses.
Yes.
2) In an 'if' header, the exact same local() call creates a scope that extends to the corresponding suite.
And in a 'while' header, and also possibly (likely) including associated suites (elif/else). So it goes ;-) There is nothing "obvious" you can say inside an "if" or "while" expression that says "and, oh ya, this name also shadows anything of the same name from here on, except when it stops doing so". Even in C, e.g., it's not *obvious* what the scope of `i` is in: for (int i = 0; ...) { } It needs to be learned. Indeed, it's so non-obvious that C and C++ give different answers. The {...} part by itself introduces a new scope in both languages. In C++ the `int i` is viewed as being _part_ of that scope, despite that it's outside the braces. But in C the `int i` is really viewed as being part of a Yet Another new scope _enclosing_ the scope introduced by {...}, but nevertheless ending when the {...} scope ends. Not that it matters much. The practical effect is that, e.g., double i = 3.0; is legal as the first line of the block in C (shadows the `int i`), but illegal in C++ (a conflicting declaration for `i` in a single scope). In either case, it's only "obvious" if you learned it and then stopped thinking too much about it ;-)
Yup, it's effectively a function-like spelling of any number of binding constructs widely used in functional languages. I had mostly in mind Haskell's "let" pile-of-bindings "in" expression spelled as "local(" pile-of-bindings "," expression ")" The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there", since the syntax for specifying keyword arguments is already understood, and already groups as intended).
But the same thing inside the if header itself would be much weirder. I'm actually not even sure what it would do.
You think I am? ;-) I don't know that it matters, because intended use cases are far simpler than all the goofy things people _can_ dream up just for the hell of it. They need to be defined, but exactly how isn't of much interest to me. For example, let's put your example in an `if`: a = 1; b = 2 if a + local(a = 3, b = 4, a + b) + b: The rules I sketched pretty clearly imply that would be evaluated as: if 1 + (3+4) + 4: It's the final "4" that's of interest. In your original example the original `b` was restored because ")" ended the new scope, leaving the final "+b" to resolve to "+2". But because it's in an "if" expression here, the new scope doesn't end at ")" anymore.
And you've clearly shown that the local() call can be anywhere inside the condition, based on these examples:
And/or used multiple times, and/or used in nested ways. None of which anyone will actually do ;-)
Probably at the end of the final (if any) `elif` or `else` suite associated with the `if`/`while`, but possibly at the end of the suite associated with the `if`/`while`. Time to note another subtlety: people don't _really_ want "a new scope" in Python. If they did, then _every_ name appearing in a binding context (assignment statement target, `for` target, ...) for the duration would vanish when the new scope ended. What they really want is a new scope with an implied "nonlocal" declaration for every name appearing in a binding context _except_ for the specific names they're effectively trying to declare as being "sublocal" instead. So It's somewhat of a conceptual mess no mater how it's spelled ;) In most other languages this doesn't come up because the existence of a variable in a scope is established by an explicit declaration rather than inferred from examining binding sites.
if local(m = ...) is not m: print("Will I ever happen?")
No, that `print` can't be reached.
That's an interesting twist ... but, to me, if "local()" inside an if/while expression _can_ be deeply magical, then I'd be less surprised over time it it were _always_ deeply magical in those contexts. I could change my mind if use cases derived from real code suggest it would be a real problem. Or if there's no real-life interest in catering to sublocal scopes in expressions anyway ... then there's no reason to even try to use something that makes clean sense for an expression.
Tim Peters wrote:
The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there",
The trouble is that one usually expects "nothing syntactically new" to imply "nothing semantically new" as well, which is very far from the case here. So I think that not using *any* new syntax would actually be hurting users rather than helping them. If you really want to leverage existing knowledge, I'd suggest something based on lambda: let a = 3, b = 4: a + b This can be easily explained as a shorthand for (lambda a = 3, b = 4: a + b)() except, of course, for the magic needed to make it DWIM in an if or while statement. I'm still pretty uncomfortable about that. -- Greg
Tim] Peters wrote:
The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there",
[Greg Ewing] [> The trouble is that one usually expects "nothing syntactically
I expect anyone who has programmed for over a year has proved beyond doubt that they can run a marathon even if forced to wear a backpack containing a ton of lead, with barbed wire wrapped around their feet ;-) They're indestructible. If they came from Perl, they'd even have a hearty laugh when they learned the syntax had utterly fooled them :-)
In that case, yes, but not in all. There's more than one kind of magic here, even outside of block constructs. See, e.g., the quadratic equation example in the original post local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), ... Try that with a lambda, and you get a NameError when computing sqrt(D) - or, worse, pick up an irrelevant value of D left over from earlier code. In Scheme terminology, what Python does with default args (keyword args too) is "let" (as if you evaluate all the expressions before doing any of the bindings), but what's wanted is "let*" (bindings are established left-to-right, one at a time, and each binding already established is visible in the expression part of each later binding). But even with `let*`, I'm not sure whether this would (or should, or shouldn't) work: local(even = (lambda n: n == 0 or odd(n-1)), odd = (lambda n: False if n == 0 else even(n-1)), odd(7)) Well, OK, I'm pretty sure it would work. But by design or by accident? ;-)
except, of course, for the magic needed to make it DWIM in an if or while statement. I'm still pretty uncomfortable about that.
That's because it's horrid. Honest, I'm not even convinced it's _worth_ "solving" , even if it didn't seem to require deep magic.
On Sat, 28 Apr 2018 at 12:41, Tim Peters <tim.peters@gmail.com> wrote: My big concern here involves the: if local(m = re.match(regexp, line)): print(m.group(0)) example. The entire block needs to be implicitly local for that to work - what happens if I assign a new name in that block? Also, what happens with: if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2, line) ): print(m.group(0)) Would the special-casing of local still apply to the block? Or would you need to do: if local(m = re.match(regexp1, line) or re.match(regexp2, line)): print(m.group(0)) This might just be lack of coffee and sleep talking, but maybe new "scoping delimiters" could be introduced. Yes - I'm suggesting introducing curly braces for blocks, but with a limited scope (pun intended). Within a local {} block statements and expressions are evaluated exactly like they currently are, including branching statements, optional semi-colons, etc. The value returned from the block is from an explicit return, or the last evalauted expression. a = 1
c = local { a=3 } * local { b=4 } c =
c = local { a=3 ; b=4 ; a*b } c = local { a = 3 b = 4 a * b } c = local(a=3,
b=local(a=2, a*a), a*b)
c = local { a = 3 b = local(a=2, a*a) return a * b }
r1, r2 = local { D = b**2 - 4*a*c sqrtD = math.sqrt(D) twoa = 2*a return ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa) }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0)) And a further implication: a = lambda a, b: local(c=4, a*b*c) a = lambda a, b: local { c = 4 return a * b * c } Tim Delaney
[Tim Delaney <timothy.c.delaney@gmail.com>]
I really don't know what you're asking there. Can you make it concrete? If, e.g., you're asking what happens if this appeared after the `print`: x = 3.14 then the answer is "the same as what would happen if `local` had not been used". We can't know what that is without context, though. Maybe x is global. Maybe x was declared nonlocal earlier. Maybe it's function-local. While it may be irrelevant to what you're asking, I noted just before: """ Time to note another subtlety: people don't _really_ want "a new scope" in Python. If they did, then _every_ name appearing in a binding context (assignment statement target, `for` target, ...) for the duration would vanish when the new scope ended. What they really want is a new scope with an implied "nonlocal" declaration for every name appearing in a binding context _except_ for the specific names they're effectively trying to declare as being "sublocal" instead. """ If by "new name" you mean that `x` didn't appear in any earlier line, then Python's current analysis would classify `x` as local to the current function (or as global if this is module-level code ...). That wouldn't change.
As is, if `local()` appears in an `if` or `while` expression, the scope extends to the end of the block construct. In that specific case, I'd expect to get a compile-time error, for attempting to initialize the same name more than once in the new scope. If the scope also encompasses associated `elif` statements, I'd instead expect local(m=...) in one of those to be treated as a re-initialization ;-) instead.
Would the special-casing of local still apply to the block?
As is, `local()` in an `if` or `while` expressions triggers deeply magical behavior, period.
Yes, not to trigger magical behavior, but to avoid the compile-time error.
c = local { a=3 } * local { b=4 }
c = local(a=3 , b=4, a*b)
c = local(a=3, b=local(a=2, a*a), a*b)
I expect you wanted b = local{a=2; a*a} there instead (braces instead of parens, and semicolon instead of comma).
return a * b }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0))
OK, this is the only case in which you used it in an `if` or `while` expression. All the questions you asked of me at the start can be asked of this spelling too. You seemed to imply at the start that the right curly brace would always mark the end of the new scope. But if that's so, the `m` in `m.group()` has nothing to do with the `m` assigned to in the `local` block - _that_ scope ended before `print` was reached. So if you're not just trying to increase the level of complexity of what can appear in a local block, a fundamental problem still needs solving ;-) I suppose you could solve it like so: local { m = re.match(regexp, line) if m: print(m.group(0)) } but, besides losing the "shortcut", it would also mean something radically different if x = 3.14 appeared after the "print". Right? If a "local block" is taken seriously, then _all_ names bound inside it vanish when the block ends.
If people do want a for-real "new scope" in Python, I certainly agree `local {...}` is far better suited for that purpose.
On Sun, 29 Apr 2018 at 10:30, Tim Peters <tim.peters@gmail.com> wrote:
That's exactly what I was asking, and as I understand what you're saying, we would have a local name m available in the indented block which went away when the block ended, but any names modified in the block are not local to the block. That seems likely to be a source of errors. To clarify my understanding, if the names 'x' and 'm' did not exist prior to the following code, what would x and m refer to after the block completed? if local(m = re.match(regexp, line)): x = 1 m = 2
Yes - I think this is exactly the same issue as with your proposed syntax.
Indeed, and I don't have a proposal - just concerns it wold be very difficult to explain and understand exactly what would happen in the case of something like: if local(m = re.match(regexp, line)): x = 1 m = 2 Regarding the syntax, I didn't want to really change your proposal, but just thought the functionality was different enough from the function call it appears to be that it probably merits different syntax. Tim Delaney
[Tim Delaney <timothy.c.delaney@gmail.com>]
[Tim Peters]
[Tim D]
If you what you _want_ is a genuinely new scope, yes. But no actual use cases so far wanted that at all. This is the kind of code about which there have been background complaints "forever": m1 = regexp1.match(line) m2 = regexp2.match(iine) if m1 and m2: do all sorts of stuff with m1 and/or m2, including perhaps modifying local variables and/or global variables and/or nonlocal variables The complaints are of two distinct kinds: 1. "I want to compute m1 and m2 _in_ the `if` test". 2. "I don't want these temp names (m1 and m2) accidentally conflicting with local names already in scope - if these names already exist, I want the temp names to shadow their current bindings until the `if` structure is done". So, if local(m1=regexp1.match(line), m2 = regexp2.match(iine), m1 and m2): intends to address both complaints via means embarrassingly obvious to the most casual observer ;-) This is, e.g., the same kind of name-specific "shadowing" magically done by list and dict comprehensions now, and by generator expressions. For example, [i**2 for i in range(10)] has no effect on whatever `i` meant before the listcomp was executed.
I hope the explanation above made that clear. What's wanted is exactly what the current m = re.match(regexp, line): if m: x =1 m = 2 _would_ do if only there were a sane way to spell "save m's current status before that all started and restore it after that all ends". So they want `x == 1` after it's over, and `m` to raise NameError.
if local { m = re.match(regexp, line) }: print(m.group(0))
Yes - I think this is exactly the same issue as with your proposed syntax.
Wholly agreed :-)
Only names appearing as targets _in_ the `local(...)` are affected in any way. The states of those names are captured, then those names are bound to the values of the associated expressions in the `local(...)`, and when the scope of the `local` construct ends (which _is_ hard to explain!) those names' original states are restored. So the effects on names are actually pretty easy to explain: all and only the names appearing inside the `local(...)` are affected.
Probably so!
On 2018-04-29 07:57, Tim Peters wrote:
How about these: local m1, m2: m1 = regexp1.match(line) m2 = regexp2.match(line): if m1 and m2: ... local m1, m2: if (m1 := regexp1.match(line)) and (m2 := regexp2.match(line)): ... local m1=regexp1.match(line), m2=regexp2.match(line): if m1 and m2: ... ? [snip]
[Tim]
[MRAB <python@mrabarnett.plus.com>]
They address complaint #2 in what seems to me a thoroughly Pythonic (direct, transparent, no more magical than necessary, easy to read) way. They don't address complaint #1 at all, but as you've shown (in the 2nd spelling) that isn't _inherently_ tied to complaint #2 (complaint #1 is what PEP 572 addresses). So _if_ PEP 572 is accepted, adding this form of a compound `local` statement too would address both of the listed complaints, at the "cost" of a bit more typing and adding a level of indentation. Neither of which bother me ;-) `local()` itself was also intended to address the even-more-in-the-background recurring desires for an expression (as opposed to statement) oriented way to use throwaway bindings; e.g., instead of temp = x + y - z + 1 r = temp**2 - 1/temp this instead: r = local(t=x + y - z + 1, t**2 - 1/t) It's not an accident that the shorter `t` is used in the latter than the former's `temp`: when people are wary of clobbering names by accident, they tend to use longer names that say "I'm just a temp - please don't _expect_ my binding to persist beyond the immediate uses on the next few lines":. Anyway, that kind of thing is common n functional languages, where "let" pile-of-bindings "in" expression kinds of constructs are widely used _as_ (sub)expressions themselves. local t = x + y - z + 1: r = t**2 - 1/t would be the same semantically, but they'd still complain about the "extra" typing and the visual "heaviness" of introducing a block for what they _think_ of as being "just another kind of expression". The `local()` I brought up was, I think, far too biased _toward_ that use. It didn't "play nice" with block-oriented uses short of excruciatingly deep magic. Your `local` statement is biased in the other direction, but that's a Good Thing :-)
On 2018-04-29 18:01, Tim Peters wrote:
As well as: local t = x + y - z + 1: r = t**2 - 1/t I wonder if it could be rewritten as: r = local t = x + y - z + 1: t**2 - 1/t Would parentheses be needed? r = (local t = x + y - z + 1: t**2 - 1/t) It kind of resembles the use of default parameters with lambda! The names would be local to the suite if used as a statement or the following expression if used in an expression, either way, the bit after the colon.
On Sun, Apr 29, 2018 at 3:30 AM, Tim Peters <tim.peters@gmail.com> wrote:
I have hard time understanding what is the demand here actually. (it's been too many posts and ideas to absorb)
def d(): global a x = 1; y = 2 a = x + y d() print (a) And this will do the thing: here if the "a" variable is "new" then it will be initialized and pushed to the outer scope, right? If there is demand for this, how about just introducing a derived syntax for the "auto-called" def block, say, just "def" without a name: def : global a x = 1; y = 2 a = x + y print (a) Which would act just like a scope block without any new rules introduced. And for inline usage directly inside single-line expression - I don't think it is plausible to come up with very nice syntax anyway, and I bet at best you'll end up with something looking C-ish, e.g.: if (def {x=1; y=2; &a = x + y } ) : ... As a short-cut for the above multi-line scope block. Mikhail
On Sun, Apr 29, 2018 at 7:22 PM, Mikhail V <mikhailwas@gmail.com> wrote:
Or even better, it would be better to avoid overloading "global" or "return", and use dedicated prefix for variables that are pushed to outer scope. I think it would look way better for cases with multiple variables: def func(): state = 0 def: localstate1 = state + 1 localstate2 = state + 2 & localstate1 & localstate2 print (localstate1) print (localstate2) Prefix in assignment would allow more expressive dispatching: def func(): state = 0 def: localstate1 = state + 1 localstate2 = state + 2 & M = state + 3 & L1, & L2 = localstate1, localstate2 print (L1, L2, M) Imo such syntax is closest do "def" block and should be so, because functions have very strong association with new scope definition, and same rules should work here. Not only it is more readable than "with local()", but also makes it easier to edit, comment/uncomment lines.
let[x](EXPR) x == EXPR let[x](a=1) x == 1 let[x](a=1, EXPR) x == EXPR let[x, y](a=1, EXPR) x == 1 y == EXPR let[x, y](a=1, b=2, EXPR) x == 2 y == EXPR z = let[x, y](a=1, EXPR) x == 1 y == EXPR z == (1, EXPR) Anybody seeing how the above might be useful, and address some of the concerns I've read? I don't recall seeing this suggested prior. I like the idea behind pseudo-function let/local, especially when paired with the explanation of equal sign precedence changes within paren, but I'm having a really hard time getting over the name binding leaking out of the paren. I like this item-ish style because it puts the name itself outside the parentheses while still retaining the benefits in readability. It also allows capturing the entire resultset, or individual parts. On Sat, Apr 28, 2018, 6:00 PM Tim Delaney <timothy.c.delaney@gmail.com> wrote:
On Apr 28, 2018 6:00 PM, "Tim Delaney" <timothy.c.delaney@gmail.com> wrote: On Sat, 28 Apr 2018 at 12:41, Tim Peters <tim.peters@gmail.com> wrote: My big concern here involves the: if local(m = re.match(regexp, line)): print(m.group(0)) example. The entire block needs to be implicitly local for that to work - what happens if I assign a new name in that block? Also, what happens with: if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2, line) ): print(m.group(0)) Would the special-casing of local still apply to the block? Or would you need to do: if local(m = re.match(regexp1, line) or re.match(regexp2, line)): print(m.group(0)) This might just be lack of coffee and sleep talking, but maybe new "scoping delimiters" could be introduced. Yes - I'm suggesting introducing curly braces for blocks, but with a limited scope (pun intended). Within a local {} block statements and expressions are evaluated exactly like they currently are, including branching statements, optional semi-colons, etc. The value returned from the block is from an explicit return, or the last evalauted expression. a = 1
c = local { a=3 } * local { b=4 } c =
c = local { a=3 ; b=4 ; a*b } c = local { a = 3 b = 4 a * b } c = local(a=3,
b=local(a=2, a*a), a*b)
c = local { a = 3 b = local(a=2, a*a) return a * b }
r1, r2 = local { D = b**2 - 4*a*c sqrtD = math.sqrt(D) twoa = 2*a return ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa) }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0)) And a further implication: a = lambda a, b: local(c=4, a*b*c) a = lambda a, b: local { c = 4 return a * b * c } Tim Delaney _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 04/27/2018 07:37 PM, Tim Peters wrote:
Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope.
Note: the thing I'm most interested in isn't debates, but in whether this would be of real use in real code.
I keep going back and forth on the ":=" syntax as on the one hand I find the functionality very useful but on the other hand it's ugly and doesn't really read well. However, I can say I am solidly -1 on local, let, etc.: - local() vs locals(): very similar words with disparate meanings - more parens -- ugh - sublocal: one more scope for extra complexity -- ~Ethan~
On Apr 27 2018, Tim Peters <tim.peters-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
I think this can be done already with slighly different syntax: c = (lambda a=3, b=4: a*b)() The trailing () is a little ugly, but the semantics are much more obvious. So maybe go with a variation that makes function evaluation implicit? c = lambda! a=3, b=4: a*b (reads terrible, but maybe someone has a better idea).
if local(m = re.match(regexp, line)): print(m.group(0))
Of course, that wouldn't (and shouldn't) work anymore. But that's a good thing, IMO :-). Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
[Tim]
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
[Nikolaus Rath <Nikolaus@rath.org>]
But also broken, in a way that can't be sanely fixed. Covered before in other messages. Short course:
This context really demands (3, 4) instead. In Scheme terms, Python's lambda default arguments do "let" binding ("all at once"), but "let*" binding is what's needed ("one at a time, left to right, with bindings already done visible to later bindings"). Of course in Scheme you explicitly type either "let" or "let*" (or "letrec", or ...) depending on what you want at the time, but "let*" is overwhelmingly what's wanted when it makes a difference (in the example at the top, it makes no difference at all). Otherwise you can't build up a complex result from little named pieces that may depend on pieces already defined. See,.e.g, the quadratic equation example in the original post, where this was implicit.
...
[Tim]
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
[Nikolaus Rath <Nikolaus@rath.org>]
[Tim]
[Chris Angelico <rosuav@gmail.com>]
Almost, but by that point the idea that this is already "easily spelled" via lambdas has become ludicrously difficult to argue with a straight face ;-) By "almost", I mean there are other cases where even nesting Python lambdas doesn't capture the intent. In these cases, not only does the expression defining b refer to a, but _also_ the expression defining a refers to b. You can play, if you like, with trying to define the `iseven` lambda here in one line by nesting lambdas to define `even` and `odd` as default arguments: even = (lambda n: n == 0 or odd(n-1)) odd = (lambda n: False if n == 0 else even(n-1)) iseven = lambda n: even(n) Scheme supplies `letrec` for when "mutually recursive" bindings are needed. In Python that distinction isn't nearly as evidently needed, because Python's idea of closures doesn't capture all the bindings currently in effect,. For example, when `odd` above is defined, Python has no idea at all what the then-current binding for `even` is - it doesn't even look for "even" until the lambda is _executed_. But, to be fair, I'm not sure: iseven =- local( even = (lambda n: n == 0 or odd(n-1)), odd = (lambda n: False if n == 0 else even(n-1)). lambda n: even(n)) would have worked either. At the moment I'm certain it wouldn't. Last night I was pretty sure it would ;-)
[Tim, on differences among Scheme-ish `let`, `let*`, `letrec` binding]
Just FYI, I still haven't managed to do it as 1-liner (well, one statement). I expected the following would work, but it doesn't :-) iseven = lambda n: ( lambda n=n, \ even = (lambda n: n == 0 or odd(n-1)), \ odd = (lambda n: False if n == 0 else even(n-1)): even(n))() Ugly and obscure, but why not? In the inner lambda, `n`, `even`, and `odd` are all defined in its namespace, so why does it fail anyway?
Because while Python indeed doesn't capture the current binding for `odd` when the `even` lambda is compiled, it _does_ recognize that the name `odd` is not local to the lambda at compile-time, so generates a LOAD_GLOBAL opcode to retrieve `odd`'s binding at runtime. But there is no global `odd` (well, unless there is - and then there's no guessing what the code would do). For `even` to know at compile-time that `odd` will show up later in its enclosing lambda's arglist requires that Python do `letrec`-style binding instead. For a start ;-)
[Tim, still trying to define `iseven` in one statement]
...
[and the last attempt failed because a LOAD_GLOBAL was generated instead of a more-general runtime lookup]
So if I want LOAD_FAST instead, that has to be forced, leading to a one-statement definition that's wonderfully clear: iseven = lambda n: ( lambda n=n, even = (lambda n, e, o: n == 0 or o(n-1, e, o)), odd = (lambda n, e, o: False if n == 0 else e(n-1, e, o)): even(n, even, odd) )() Meaning "wonderfully clear" to the compiler, not necessarily to you ;-) Amusingly enough, that's a lot like the tricks we sometimes did in Python's early days, before nested scopes were added, to get recursive - or mutually referring - non-global functions to work at all. That is, since their names weren't in any scope they could access, their names had to be passed as arguments (or, sure, stuffed in globals - but that would have been ugly ;-) ).
Tim Peters wrote:
But 'even' and 'odd' not defined in the environment of the lambdas assigned to them, because default values of a function's arguments are evaluated outside of that function.
One could envisage adding a letrec-like construct, but making the argument list of an ordinary lambda behave like a letrec would be warping things rather too much, IMO. Personally I'd rather just add a "where" clause, and backwards compatibility be damned. :-) -- Greg
Hi Tim, This is interesting. Even "as is" I prefer this to PEP 572. Below are some comments and a slightly different idea inspired by yours (sorry!) On Fri, Apr 27, 2018 at 10:41 PM Tim Peters <tim.peters@gmail.com> wrote: [..]
As an expression, it's
"local" "(" arguments ")"
if local(m = re.match(regexp, line)): print(m.group(0))
It does look like a function call, although it has a slightly different syntax. In regular calls we don't allow positional arguments to go after keyword arguments. Hence the compiler/parser will have to know what 'local(..)' is *regardless* of where it appears. If you don't want to make 'local' a new keyword, we would need to make the compiler/parser to trace the "local()" name to check if it was imported or is otherwise "local". This would add some extra complexity to already complex code. Another problematic case is when one has a big file and someone adds their own "def local()" function to it at some point, which would break things. Therefore, "local" should probably be a keyword. Perhaps added to Python with a corresponding "from __future__" import. The other way would be to depart from the function call syntax by dropping the parens. (And maybe rename "local" to "let" ;)) In this case, the syntax will become less like a function call but still distinct enough. We will be able to unambiguously parse & compile it. The cherry on top is that we can make it work even without a "__future__" import! When we implemented PEP 492 in Python 3.5 we did a little trick in tokenizer to treat "async def" in a special way. Tokenizer would switch to an "async" mode and yield ASYNC and AWAIT tokens instead of NAME tokens. This resulted in async/await syntax available without a __future__ import, while having full backwards compatibility. We can do a similar trick for "local" / "let" syntax, allowing the following: "let" NAME "=" expr ("," NAME = expr)* ["," expr] * "if local(m = re.match(...), m):" becomes "if let m = re.match(...), m:" * "c = local(a=3) * local(b=4)" becomes "c = let a=3, b=4, a*b" or "c = (let a=3, b=4, a*b)" * for i in iterable: if let i2=i*i, i2 % 18 == 0: append i2 to the output list etc. Note that I don't propose this new "let" or "local" to return their last assignment. That should be done explicitly (as in your "local(..)" idea): `let a = 'spam', a`. Potentially we could reuse our function return annotation syntax, changing the last example to `let a = "spam" -> a` but I think it makes the whole thing to look unnecessarily complex. One obvious downside is that "=" would have a different precedence compared to a regular assignment statement. But it already has a different precedent in function calls, so maybe this isn't a big deal, considered that we'll have a keyword before it. I think that "let" was discussed a couple of times recently, but it's really hard to find a definitive reason of why it was rejected (or was it?) in the ocean of emails about assignment expressions. Yury
[Yury Selivanov <yselivanov.ml@gmail.com>]
This is interesting. Even "as is" I prefer this to PEP 572. Below are some comments and a slightly different idea inspired by yours (sorry!)
That's fine :-)
Not only for that reason, but because the semantics have almost nothing in common with function calls. For example, in local(a=1, b=a+1) the new binding of `a` needs to be used to establish the binding of `b`. Not to mention that a new scope may need to be established, and torn down later. To the compiler, it's approximately nothing like "a function call". "Looking like" a function call nevertheless has compelling benefits: - People already know the syntax for specifying keyword arguments. - The precedence of "=" in a function call is already exactly right for this purpose. So nothing new to learn there either. - The explicit parentheses make it impossible to misunderstand where the expression begins or ends. - Even if someone knows nothing about "local", they _will_ correctly assume that, at runtime, it will evaluate to an object. In that key respect it is exactly like the function call it "looks like". I do want to leverage what people "already know".
I believe it absolutely needs to become a reserved word. While binding expressions suffice to capture values in conditionals, they're not all that pleasant for use in expressions (that really wants a comma operator too), and it's a fool's errand to imagine that _any_ currently-unused sequence of gibberish symbols and/or abused keywords can plausibly suggest "and here we're also creating a new scope, and here I'm declaring some names to live in that scope". If all that is actually wanted, a new reserved word seems pragmatically necessary. That's a high bar. Speaking of which, "sublocal" would be a more accurate name, and less likely to conflict with existing code names, but after people got over the shock to their sense of purity, they'd appreciate typing the shorter "local" instead ;-) For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
Therefore, "local" should probably be a keyword. Perhaps added to Python with a corresponding "from __future__" import.
Yes.
I'm far less concerned about pain for the compiler than pain for the human reader. There's almost no precedent in Python's expression grammar that allows two names separated only by whitespace. The screams "statement" instead, whether "class NAME" or "async for" or "import modulename" or "global foo" or "for i in" or ... I'm afraid it would prove just too jarring to see that inside what's supposed to be "an expression". That's why I settled on the (admittedly unexpected) "pseudo-function" syntax. The exceptions involve expressions that are really test-&-branch structures in disguise ("and", "or", ternary "if"). In those we can find NAME NAME snippets, but the keyword is never at the start of those. So, curiously enough, I'd be fonder of result_expression "where" name=expression, ... than of "let" name=expression, ... if I hadn't already resigned myself to believing function-like syntax is overwhelmingly less jarring regardless.
Which was clever, and effective, but - as far as I know - limited to _statements_, where KEYWORD NAME thingies were already common as mud.
See the bullet list near the top for all the questions that _raises_ that don't even need to be asked (by users) when using a function-like syntax instead.
I assume that was meant to be c = (let a=3, a) * (let b=4, b) instead. In _an expression_, I naturally group the a = 3, a part as the unintended a = (3, a) When I'm thinking of function calls, though, I naturally use the intended (a=3), a grouping. I really don't want to fight with what "everyone already knows", but build on that to the extent possible.
In the second line there too, I did like that in if local(i2=i*i) % 18 == 0: the mandatory parentheses made it wholly explicit where "the binding part" ended.
Why not? A great many objects in Python are _designed_ so that their __bool__ method does a useful thing in a plain if object: test. In these common cases, needing to type if let name=object, object: instead of if let name=object: is an instance of what my pseudo-FAQ called "annoyingly redundant noise". Deciding to return the value of the last "argument" expression was a "practicality beats purity" thing.
Me too - but then I thought it was _already_ too wordy to require `let a="spam", a` ;-)
I think my response to that is too predictable by now to annoy you by giving it ;-)
I don't know either. There were a number of halfheartedly presented ideas, though, that floundered (to my eyes) on the rocks of trying to ignore that syntax ideas borrowed from "everything's an expression" functional languages don't carry over well to a language where many constructs aren't expressions at all. Or, conversely, that syntax unique to the opening line of a statement-oriented language's block doesn't carry over at all to expressions. I'm trying to do both at once, because I'm not a wimp - and neither are you ;-)
On Sat, Apr 28, 2018 at 2:07 AM Tim Peters <tim.peters@gmail.com> wrote: [...]
For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
Great! :) [...]
So, curiously enough, I'd be fonder of
result_expression "where" name=expression, ...
than of
"let" name=expression, ...
My gripe about the "where"-like syntax family (including "as", proposed by many) is that the whole expression needs to be read backwards to make sense of `name*. It's one of the things that annoy me in languages like SQL, where the relevant "AS" keyword can be buried a couple of screens below/above the area of interest. That's why I find the "let" form superior. [..]
instead. In _an expression_, I naturally group the
a = 3, a
part as the unintended
a = (3, a)
When I'm thinking of function calls, though, I naturally use the intended
(a=3), a
grouping. I really don't want to fight with what "everyone already knows", but build on that to the extent possible.
Looking at all of these and other examples in your email I have to agree that the version with parenthesis reads way more clearly. A different (and correct in this case) precedence of "," between parens is pretty much hardwired in our brains. [..]
Why not? A great many objects in Python are _designed_ so that their __bool__ method does a useful thing in a plain
if object:
test. In these common cases, needing to type
if let name=object, object:
Alright. And yes, I agree, surrounding parens are badly needed. Maybe we can replace parens with {}? Although my immediate reaction to that is "it's too ugly to be taken seriously". In any case, the similarity of this new syntax to a function call still bothers me. If I just looked at some Python 3.xx code and saw the new "let(..)" syntax, I would assume that the names it declares are only visible *within* the parens. Parens imply some locality of whatever is happening between them. Even if I googled the docs first to learn some basics about "let", when I saw "if let(name...)" it wouldn't be immediately apparent to me that `name` is set only for its "if" block (contrary to "if let a = 1: print(a)"). Yury
: On 28 April 2018 at 07:07, Tim Peters <tim.peters@gmail.com> wrote:
[...] For that matter, I'd be fine too with shortening it to "let". In fact, I prefer that! Thanks :-)
If you really wanted to overcome objections that this looks too much like a function, you could spell it "$". I'm not sure what I think about $(a=7, $(a=a+1, a*2)) yet, but it doesn't make me want to run screaming in the way that := does. I think I finally worked out why I have such a violent reaction to := in the end, by the way: it's because it reminds me of Javascript's "===" (not the meaning, but the fact that it exists). -[]z.
$(a=7, $(a=a+1, a*2))
I suspect you ALREADY have bash installed on your computer, you don't need Python to emulate it. On Sat, Apr 28, 2018 at 6:22 AM, Zero Piraeus <schesis@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Tim Peters wrote:
To the compiler, it's approximately nothing like "a function call".
It's nothing like a function call to the user, either, except in the most superficial of ways.
- The explicit parentheses make it impossible to misunderstand where the expression begins or ends.
Except that you then go and break that rule by saying that it doesn't apply when you're in the condition of an "if" statement.
I do want to leverage what people "already know".
Seems to me this proposal does that in the worst possible way, by deceiving the user into thinking it's something familiar when it's not. -- Greg
To all of the following, I was talking about the syntax. Which includes that all existing Python-aware editors and IDEs already know how to format it intelligibly. It would be nice if they also colored "local" (or "let") as a keyword, though. For the rest, of course I'm already aware of the _semantic_ trickery. Indeed, that may be too much to bear. But I'm already sympathetic to that too :-) On Sat, Apr 28, 2018 at 9:03 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 28/04/2018 04:34, Yury Selivanov wrote:
If things were to go the keyword direction I personally think that a great deal of clarity could be achieved by borrowing somewhat from Pascal with either use <exp> as <local_name>[, <exp> as <local_name> ...]: # Local names only exist in this scope, # existing names are in scope unless overridden # new names introduced here go out to nesting scope OR (for those who prefer name first, a function like look, and slightly less typing): using(<local_name>=<exp>[ ,<local_name>=<exp>...]): # Local names only exist in this scope, # existing names are in scope unless overridden # new names introduced here go out to nesting scope Lastly how about: <name> = <calculation> using(<local_name>=<exp>[ ,<local_name>=<exp>...]) # in this case where might be better than using Presumably, in the latter cases, the using function would return a sub_local dictionary that would only exist to the end of the scope be that the current line or indented block which should minimise the required magic. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com
On 28 April 2018 at 03:37, Tim Peters <tim.peters@gmail.com> wrote:
Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope.
This looks disturbingly good to me. I say "disturbingly" because the amount of magic involved in that "function call" is pretty high, and the more you look at it, the more weird its behaviour seems. I have essentially no real world use cases for this, though (other than ones that have been brought up already in the binding expression debate), so my comments are purely subjective opinion. [...]
This is where, in my mind, the magic behaviour goes too far. I can see why it's essential that this happens, but I can't come up with a justification for it other than pure expediency. And while I know "practicality beats purity" (look at me, daring to quote the Zen at Tim!!! :-)) this just feels like a step too far to me. Paul
On Fri, Apr 27, 2018 at 09:37:53PM -0500, Tim Peters wrote:
Chris' PEP 572 started off with the concept that binding expressions would create a "sub-local" scope, below function locals. After some debate on Python-Ideas, Chris, Nick and Guido took the discussion off list and decided to drop the sub-local scope idea as confusing and hard to implement. But the biggest problem is that this re-introduces exactly the same awful C mistake that := was chosen to avoid. Which of the following two contains the typo? local(spam=expression, eggs=expression, cheese = spam+eggs) local(spam=expression, eggs=expression, cheese == spam+eggs) I have other objections, but I'll leave them for now, since I think these two alone are fatal. Once you drop those two flaws, you're basically left with PEP 572 :-) -- Steve
[Steven D'Aprano <steve@pearwood.info>]
Enormously harder to implement than binding expressions, and the latter (to my eyes) capture many high-value use cases "good enough". I'm not concerned about "confusing". "Sub-local" scopes are ubiquitous in modern languages, and they're aimed at experienced programmers. It was also the case that nesting scopes _at all_ was very controversial in Python's earliest years, and Guido resisted it mightily (with my full support). The only scopes at first were function-local, module-global, and builtin, and while functions could _textually_ nest, they had no access to enclosing local scopes. Cute: to write a recursive nested function, you needed to pass it its own name (because its own name isn't in its _own_ local scope, but in the local scope of the function that contains it). Adding nested local scopes was also "confusing" at the time, and indeed made the scoping rules far harder to explain to newbies, and complicated the implementation. Then again, experienced programmers overwhelmingly (unanimously?) welcomed the change after it was done. Since then, Python has gone down a pretty bizarre path, inventing sublocal scopes on an ad hoc basis by "pure magic" when their absence in some specific context seemed just too unbearable to live with (e.g., in comprehensions). So we already have sublocal scopes, but in no explicit form that can be either exploited or explained.
Neither :-) I don't expect that to be a real problem. In C I'm _thinking_ "if a equals b" and type "if (a=b)" by mistake in haste. In a "local" I'm _ thinking_ "I want to create these names with these values" in the former case, and in the latter case also "and I want to to test whether cheese equals spam + eggs". But having already typed "=" to mean "binding" twice in the same line, "but the third time I type it it will mean equality instead" just doesn't seem likely. The original C mistake is exceedingly unlikely on the face of it: if what I'm thinking is "if a equals b", or "while a equals b", I'm not going to use "local()" _at all_. Forcing the programmer to be explicit about that they're trying to create a new scope limits the only possible confusions to cases where they _are_ going out of their way to use a "local()" construct, in which case binding behavior is very much at the top of their mind. Plain old "if a=b" remains a SyntaxError regardless. Still, if people are scared of that, a variation of Yury's alternative avoids it: the last "argument" must be an expression (not a binding). In that case your first line above is a compile-time error. I didn't like that because I really dislike the textual redundancy in the common if local(matchobject=re.match(regexp, line), matchobject): compared to if local(matchobject=re.match(regexp, line)): But I could compromise ;-) - There must be at least one argument. - The first argument must be a binding. - All but the last argument must also be bindings. - If there's more than one argument, the last argument must be an expression. Then your first line above is a compile-time error, the common "just name a result and test its truthiness" doesn't require repetition, and "local(a==b)" is also a compile-time error.
I have other objections, but I'll leave them for now, since I think these two alone are fatal.
I don't.
Once you drop those two flaws, you're basically left with PEP 572 :-)
Which is fine by me, but do realize that since PEP 572 dropped any notion of sublocal scopes, that recurring issue remains wholly unaddressed regardless.
On 2018-04-28 18:16, Tim Peters wrote:
How about if you just can't have an expression in a local()? There are a few obvious alternative possibilities: 1. No special case at all: local(x=1, y=2, _=x+y) 2. Co-opt a keyword after local(): local(x=1, y=2) in x+y 3. Co-opt a keyword inside local(): local(x=1, y=2, return x+y) I hate the first and wish the _ pattern would die in all its forms, but it's worth mentioning. I don't think there's much to choose between the other two, but 2 uses syntax that might have been valid and meant something else, so 3 is probably less confusing.
[Ed Kellett <e+python-ideas@kellett.im>]
How about if you just can't have an expression in a local()?
See the quadratic equation example in the original post. When working with expressions, the entire point of the construct is to define (sub)local names for use in a result expression.
As above.
2. Co-opt a keyword after local():
local(x=1, y=2) in x+y
Requiring "in" requires annoying syntactic repetition in the common if local(match=re.match(regexp, line)) in match: kinds of cases. Making "in" optional instead was discussed near the end of the original post. I agree that your spelling just above is more obvious than `local(x=1, y=2, x+y)` which is why the original post discussed making an "in clause" optional. But, overwhelmingly, it appears that people are more interested in establishing sublocal scopes in `if` and `while` constructs than in standalone expressions, so I picked a spelling that's _most_ convenient for the latter's common "just name a result and test its truthiness" uses.
3. Co-opt a keyword inside local():
local(x=1, y=2, return x+y)
Why would that be better than local(x=1, y=2, x+y)? That no binding is intended for `x+y` is already obvious in the latter.
Indeed, 3 is the only one I'd consider, but I don't see that it's a real improvement. It seems to require extra typing every time just to avoid learning "and it returns the value of the last expression" once.
On 2018-04-28 21:40, Tim Peters wrote:
I don't mean "in" should be required, but rather that the last thing is always an assignment, and the local() yields the assigned value. So that'd remain: if local(match=re.match(regexp, line)): whereas your quadratic equation might do the "in" thing: r1, r2 = local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), twoa = 2*a) in ( (-b + sqrtD)/twoa, (-b - sqrtD)/twoa) ... which doesn't look particularly wonderful (I think it's nicer with my option 3), but then I've never thought the quadratic equation was a good motivating case for any version of an assignment expression. In most cases I can imagine, a plain local() would be sufficient, e.g.: if local(match=re.match(regexp, line)) and match[1] == 'frog':
I agree with pretty much all of this--I was trying to attack the "but you could make the terrible C mistake" problem from another angle. I don't think I mind the last "argument" being *allowed* to be an expression, but if the consensus is that it makes '='/'==' confusion too easy, I think it'd be more straightforward to say they're all assignments (with optional, explicit syntax to yield an expression) than to say "they're all assignments except the last thing" (argh) "except in the special case of one argument" (double argh).
On Sat, Apr 28, 2018 at 12:16:16PM -0500, Tim Peters wrote:
And yet you're suggesting an alternative which is harder and more confusing. What's the motivation here for re-introducing sublocal scopes, if they're hard to do and locals are "good enough"? That's not a rhetorical question: why have you suggested this sublocal scoping idea? PEP 572 stopped talking about sublocals back in revision 2 or so, and as far as I can see, *not a single objection* since has been that the variables weren't sublocal. For what it is worth, if we ever did introduce a sublocal scope, I don't hate Nick's "given" block statement: https://www.python.org/dev/peps/pep-3150/ [...]
While I started off with Python 1.5, I wasn't part of the discussions about nested scopes. But I'm astonished that you say that nested scopes were controversial. *Closures* I would completely believe, but mere lexical scoping? Astonishing. Even when I started, as a novice programmer who wouldn't have recognised the term "lexical scoping" if it fell on my head from a great height, I thought it was strange that inner functions couldn't see their surrounding function's variables. Nested scopes just seemed intuitively obvious: if a function sees the variables in the module surrounding it, then it should also see the variables in any function surrounding it. This behaviour in Python 1.5 made functions MUCH less useful:
I think it is fair to say that inner functions in Python 1.5 were crippled to the point of uselessness.
I agree with the above regarding closures, which are harder to explain, welcomed by experienced programmers, and often a source of confusion for newbies and experts alike: https://stackoverflow.com/questions/7546285/creating-lambda-inside-a-loop http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/comment-page-1/ but I disagree that lexical scoping alone is or ever was confusing. Neither did Niklaus Wirth, who included it in Pascal, a language intended to be friendly for beginners *wink*
I'm not entirely sure that comprehensions (including generator expressions) alone counts as "a path" :-) but I agree with this. I'm not a fan of comprehensions being their own scope. As far as I am concerned, leakage of comprehension variables was never a problem that needed to be solved, and was (very occasionally) a useful feature. Mostly for introspection and debugging. But the decision was made for generator comprehensions to be in their own scope, and from there I guess it was inevitable that list comprehensions would have to match.
I'm sure the C designers didn't either. You miss the point that looking at the above, it is impossible to tell whether I meant assignment or an equality test. Typos of = for == do happen, even in Python, for whatever reason typos occur. Regardless of whether this makes them more likely or not (I didn't make that claim) once made, it is a bug that can fail silently in a way that is hard to see and debug. Most = for == typos in Python give an instant SyntaxError, but there are two places where they don't: - a statement like "spam == eggs" called for its side-effects only; - in a function call, func(spam==eggs, spam=eggs) are both legal. The first is so vanishingly rare that we can forget it. If you see spam = eggs as a statement, we can safely assume it means exactly what it says. Inside function calls, it's a bit less cut and dried: func(foo=bar) *could* be a typoed positional argument (foo == bar) but in practice a couple of factors mitigate that risk: - PEP 8 style conventions: we expect to see func(foo=bar) for the keyword argument case and func(foo == bar) for the positional argument case; - if we mess it up, unless there happens to be a parameter called foo we'll get a TypeError, not a silent bug. But with your suggested local() pseudo-function, neither mitigating factor applies and we can't tell or even guess which meaning is intended just by sight.
Of course people won't *consciously* think that the operator for equality testing is = but they'll be primed to hit the key once, not twice, and they'll be less likely to notice their mistake. I never make more = instead of == typos than after I've just spent a lot of time working on maths problems, even though I am still consciously aware that I should be using == I simply don't notice the error.
Given that while ... is one of the major motivating use-cases for binding expressions, I think you are mistaken to say that people won't use this local() pseudo-function in while statements. [...]
Indeed. That sort of "Repeat Yourself To Satisfy The Compiler" will be an ugly anti-pattern.
That's not really a complete specification of the pseudo-function though, since sometimes the sublocals it introduces extend past the final parenthesis and into the subsequent block. What will, for example, this function return? spam = eggs = "global" def func(arg=local(spam="sublocal", eggs="sublocal", 1)): eggs = "local" return (spam, eggs) Even if you think nobody will be tempted to write such "clever" (dumb?) code, the behaviour still has to be specified. [...]
I don't think that sublocal scopes is a recurring issue, nor that we need address it now. -- Steve
On Sun, Apr 29, 2018 at 12:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
No objections, per se, but I do know a number of people were saddened at their loss. And it's not like eliminating sublocals solved problems without creating more; it's just a different set of trade-offs. Sublocals have their value.
I'm not sure how you can distinguish them:
What you expect here is lexical scope, yes. But if you have lexical scope with no closures, the inner function can ONLY be used while its calling function is still running. What would happen if you returned 'inner' uncalled, and then called the result? How would it resolve the name 'x'? I can't even begin to imagine what lexical scope would do in the absence of closures. At least, not with first-class functions. If functions aren't first-class objects, it's much easier, and a nested function serves as a refactored block of code. But then you can't even pass a nested function to a higher order function. ChrisA
On Sun, Apr 29, 2018 at 01:36:31PM +1000, Chris Angelico wrote: [...]
Easily. Pascal, for example, had lexical scoping back in the 1970s, but no closures. I expect Algol probably did also, even earlier. So they are certainly distinct concepts. (And yes, Pascal functions were *not* first class values.)
Failing to resolve 'x' is an option. It would simply raise NameError, the same as any other name lookup that doesn't find the name. Without closures, we could say that names are looked up in the following scopes: # inner function, called from inside the creating function (1) Local to inner. (2) Local to outer (nonlocal). (3) Global (module). (4) Builtins. If you returned the inner function and called it from the outside of the factory function which created it, we could use the exact same name resolution order, except that (2) the nonlocals would be either absent or empty. Obviously that would limit the usefulness of factory functions, but since Python 1.5 didn't have closures anyway, that would have been no worse that what we had. Whether you have a strict Local-Global-Builtins scoping, or lexical scoping without closures, the effect *outside* of the factory function is the same. But at least with the lexical scoping option, inner functions can call each other while still *inside* the factory. (Another alternative would be dynamic scoping, where nonlocals becomes the environment of the caller.)
I can't even begin to imagine what lexical scope would do in the absence of closures. At least, not with first-class functions.
What they would likely do is raise NameError, of course :-) An inner function that didn't rely on its surrounding nonlocal scope wouldn't be affected. Or if you had globals that happened to match the names it was relying on, the function could still work. (Whether it would work as you expected is another question.) I expect that given the lack of closures, the best approach is to simply make sure that any attempt to refer to a nonlocal from the surrounding function outside of that function would raise NameError. All in all, closures are much better :-) -- Steve
Steven D'Aprano wrote:
Pascal, for example, had lexical scoping back in the 1970s, but no closures.
Well, it kind of had a limited form of closure. You could pass a procedure or function *in* to another procedure or function as a parameter, but there was no way to return one or pass it out in any way. This ensured that the passed-in procedure or function couldn't outlive its lexical environment. -- Greg
Steven D'Aprano wrote:
So what was the closure? If the surrounding function was still running, there was no need to capture the running environment in a closure?
You seem to be interpreting the word "closure" a bit differently from most people. It doesn't imply anything about whether a surrounding function is still running or not. A closure is just a piece of code together with a runtime environment. In typical Pascal implementations, a closure is represented by a (code_address, frame_pointer) pair, where the frame_pointer points to a chain of lexically enclosing stack frames. The language rules make it possible to manage the frames strictly stack-wise, which simplifies the memory management, but that doesn't make the closure any less of a closure. Contrast this with Modula-2, where only top-level functions can be passed as parameters. When you pass a function in Modula-2, only the code address is passed, with no environment pointer. -- Greg
On Tue, May 01, 2018 at 09:26:09PM +1200, Greg Ewing wrote:
That's not *all* it is, because obviously *every* function has both a piece of code and a runtime environment. y = 1 def func(): x = 2 return x+y Here, there's a local environment as well as an implicit global one. Surely we don't want to call this a closure? If we do, then all functions are closures and the term is just a synonym for function. Wikipedia gives a definition: a closure (also lexical closure or function closure) is a technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment. https://en.wikipedia.org/wiki/Closure_%28computer_programming%29 but also warns As different languages do not always have a common definition of the lexical environment, their definitions of closure may vary also and explicitly says that Pascal (at least standard Pascal) didn't do closures: Traditional imperative languages such as Algol, C and Pascal either do not support nested functions (C) or do not support calling nested functions after the enclosing function has exited (GNU C, Pascal), thus avoiding the need to use closures. Neal Gafter gives what I believe is *the* standard definition of closures, at least in academic and functional programming circles, probably taken from Guy Steele and Scheme: A closure is a function that captures the bindings of free variables in its lexical context. http://gafter.blogspot.com.au/2007/01/definition-of-closures.html Since Steele basically wrote the book on closures, I think we ought to take his definition seriously :-) (That's also the definition I was thinking of.) By this definition, if there are no free variables, or no captured bindings, then its not a closure. func() above isn't a closure, since "x" isn't a free variable, and while "y" is, the value of it isn't captured. Unfortunately, the terminology has become a real mess, especially since the technique has moved into languages like Ruby and Javascript. Martin Fowler declares that closures, lambdas and blocks are the same thing: https://martinfowler.com/bliki/Lambda.html although he does admit that technically you can have lambdas that aren't closures. I think Fowler is abusing the terminology since all three of his terms are orthogonal: - closures can be created with def or lambda and are not necessarily anonymous; - lambda comes from the lambda calculus, where it refers to anonymous function expressions, not specifically whether they capture the value of free variables in their environment; - "block" typically refers to the structure of the source code; in Ruby it also refers to a kind of first-class anonymous callable object. Such blocks may, or may not, contain free variables. At least I don't use this definition of closure: "a closure is a callback that Just Works." https://reprog.wordpress.com/2010/02/27/closures-finally-explained/ *wink*
Well... it's a pretty feeble closure, since Pascal doesn't support functions as first-class values. More like second-class. The c2 wiki describes Pascal: However, when a function is passed to another function and later called, it will execute in the lexical context it was defined in, so it is, IN SOME SENSE [emphasis added], "closed over" that context. http://wiki.c2.com/?LexicalClosure but I think that sense is the same sense that func() above is "closed over" the free value y. The loosest possible sense. If we accept that "Pascal has closures", they're pretty crap closures, since they don't let you do any of the interesting and useful things you can do in languages with "real" closures like Scheme and Python. And because it is based on an implementation detail (as you say, it is only *typical*, not mandatory, for Pascal inner functions to be implemented this way), we have to consider what happens if we come across a Pascal implementation which *doesn't* use a (code_address, frame_pointer) pair. Is that no longer Pascal? This is the interesting, and frustrating, thing about definitions: deciding where the boundaries lie, and we could argue the boundaries for days without getting anywhere :-) -- Steve
Steven D'Aprano wrote:
Python probably isn't the best choice of language for the point you're making, because even top-level functions in Python *do* carry a reference to their environment (the dict of the module they were defined in). C would be a better choice: int y = 1; int func(void) { int x = 2; return x + y; } int (*fp)() = func; Even here, I don't think there's anything wrong with calling fp a closure. It's just that it's a particularly simple form of closure -- since there's only one global environment, we don't need to bother explicitly carrying it around. However, that doesn't mean all functions are closures. An example of a non-closure might be a C++ member function pointer. It points to a piece of code, but you can't use it on its own, because it's missing a piece of environmental information (a value for "this") that you need to supply by hand when you call it.
In anything that implements Pascal semantics, a function parameter is going to have to be represented by *something* that specifies both code and environment. Code alone is not enough.
Not all of them, but it can do some of them. They're more useful than in Modula-2, for example, where nested functions can't even be passed as parameters. Anyway, that's why I said "a limited form of closure". You could read that two ways: (1) It's totally a closure, but there are limitations on what you can do with it (i.e. it's not first class). Or (2) it has some of the characteristics of a closure, but not all of them. In my mind I tend towards (1), but it doesn't really matter. -- Greg
[Tim]
Enormously harder to implement than binding expressions, and the latter (to my eyes) capture many high-value use cases "good enough".
[Steven D'Aprano <steve@pearwood.info>]
And yet you're suggesting an alternative which is harder and more confusing.
I am? I said at the start that it was a "brain dump". It was meant to be a point of discussion for anyone interested. I also said I was more interested in real use cases from real code than in debating, and I wasn't lying about that ;-) Since no real new use cases (let alone compelling ones) have turned up yet, I'm ready to drop it for now.
What's the motivation here for re-introducing sublocal scopes, if they're hard to do and locals are "good enough"?
That's why I wanted to see if there were significant unaddressed use cases. That _my_ particular itches would be scratched "good enough" if the PEP is accepted doesn't imply everyone's will be. And my particular itches will continue to annoy if the PEP is rejected.
That's not a rhetorical question: why have you suggested this sublocal scoping idea?
Putting an idea out for discussion isn't suggesting it be adopted. The list is named "python-ideas", not "python-advocacy-death-match" ;-)
Meh. Chris didn't seem all that thrilled about dropping them, and I saw a number of messages more-than-less supporting the idea _before_ they were dropped. When it became clear that the PEP didn't stand a chance _unless_ they were dropped, nobody was willing to die for it, because they weren't the PEP's _primary_ point.
And Chris just tried introducing it again. That's in reference to the last sentence of your reply: I don't think that sublocal scopes is a recurring issue How many times does it have to come up before "recurring" applies? ;-) I've seen it come up many times over ... well, literally decades by now.
But true. Guido agonized over it for a long time. Limiting to 3 scopes was a wholly deliberate design decision at the start, not just, .e.g, due to lack of time to implement lexical scoping at the start. And that shouldn't be surprising given Python's start as "somewhere between a scripting language and C", and the many influences carried over from Guido's time working on ABC's implementation team (ABC had no lexical scoping either - nor, if I recall correctly, even textual nesting of its flavor of functions). I'm glad he tried it! Everyone learned something from it.
I don't think that's fair to say. A great many functions are in fact ... functions ;-) That is, they compute a result from the arguments passed to them. They don't need more than that, although being able to access globals and builtins and import whatever they want from the standard library made them perfectly capable of doing a whole lot more than just staring at their arguments. To this day, _most_ of the nested functions I write would have worked fine under the original scoping rules, because that's all they need. Many of the rest are recursive, but would also work fine if I passed their names into them and rewrote the bits of code to do recursive calls via the passed-in name. But, yes, I am relieved I don't need to do the latter anymore ;-) ... [snip similar things about closures] ...
I didn't mind comprehensions "leaking" either. But I expect the need became acute when generator expressions were introduced, because the body of those can execute in an environment entirely unrelated to the definition site: def process(g): i = 12 for x in g: pass print(i) # was i clobbered? nope! process(i for i in range(3)) ... [snip more "head arguments" about "=" vs "=="] ...
It was talking about compile-time-checkable syntactic requirements, nothing about semantics.
It would return (spam, "local"), for whatever value `spam` happened to be bound to at the time of the call. `local()` had magically extended scope only in `if`/`elif` and `while` openers. In this example, it would the same if the `def` had been written def func(arg=1): Remember that default argument values are computed at the time `def` is executed, and never again. At that time, local(spam="sublocal", eggs="sublocal", 1) created two local names that are never referenced, threw the new scope away, and returned 1, which was saved away as the default value for `arg`.
Even if you think nobody will be tempted to write such "clever" (dumb?) code, the behaviour still has to be specified.
Of course, but that specific case wasn't even slightly subtle ;-)
Tim Peters wrote:
If Python hadn't allowed textual nesting either, folks might have been content to leave it that way. But having textual nesting without lexical scoping was just weird and confusing!
Yes, but they often make use of other functions to do that, and not being able to call other local functions in the same scope seemed like a perverse restriction. -- Greg
On Sat, Apr 28, 2018 at 11:20:52PM -0500, Tim Peters wrote:
Ah, my mistake... I thought you were advocating sublocal scopes as well as just brain dumping the idea. [...]
Sure, but not having access to the surrounding function scope means that inner functions couldn't call other inner functions. Given: def outer(): def f(): ... def g(): ... f cannot call g, or vice versa. I think it was a noble experiment in how minimal you could make scoping rules and still be usable, but I don't think that particular aspect was a success. -- Steve
On 04/28/2018 10:16 AM, Tim Peters wrote:
If we need a sublocal scope, I think the most Pythonic* route to have it would be: with sublocal(): blah blah which would act just like local/global does now: - any assignment creates a new variable - unless that variable has been declared global/nonlocal - plain reads (no assignment ever happens) refer to nonlocal/global/built-in names This has the advantages of: - no confusion about which variables are sublocal (acts like a new function scope) - no extra parens, assignments, expressions on "with sublocal():" line Possible enhancements: - give sublocal block a name "with sublocal() as blahblah:" and then reuse that block again later ("with blahblah:") or maybe pass it to other functions... Of course, as previously stated, this is orthogonal to PEP 572. -- ~Ethan~
[Ethan Furman <ethan@stoneleaf.us>]
As covered most recently in an exchange with Tim Delaney, best I can tell absolutely nobody has wanted that. By "sublocal scope" they don't mean a full-fledged new scope at all, but a kind of limited "shadowing" of a handful of specific, explicitly given names. It acts like a context manager, if there were a way to clearly spell save the current state of these specific identifiers at the start (& I couldn't care less whether they're local, nonlocal, or global - I don't know & don't care) then execute the code exactly as if this gimmick had never been used then, at the end, restore the specific identifier states we saved at the start It's the same kind of shadowing Python already does by magic for, e.g., `i`, in [i for i in range(3)] So, e.g., """ a = 42 def showa(): print(a) def run(): global a local a: # assuming this existed a = 43 showa() showa() """ would print 43 and then 42. Which makes "local a:" sound senseless on the face of it ;-) "shadow" would be a more descriptive name for what it actually does.
...
This doesn't address the fact no one actually needs it. But if we WANTED a sublocal() context manager, we could spell it something like this: In [42]: @contextmanager ...: def sublocal(**kws): ...: _locals = locals().copy() ...: _globals = globals().copy() ...: for k, v in kws.items(): ...: if k in locals(): ...: exec(f"locals()['{k}'] = {v}") ...: elif k in globals(): ...: exec(f"globals()['{k}'] = {v}") ...: yield ...: locals().update(_locals) ...: globals().update(_globals) ...: In [43]: a = 42 In [44]: with sublocal(a=43): ...: showa() ...: 43 In [45]: showa() 42 In [46]: with sublocal(): ...: a = 41 ...: showa() ...: 41 In [47]: showa() 42 On Sun, Apr 29, 2018 at 4:20 PM, Tim Peters <tim.peters@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 29 April 2018 at 21:20, Tim Peters <tim.peters@gmail.com> wrote:
So maybe adding such a primitive (maybe something live states = sys.get_variable_state('a', 'b', 'c') and sys.set_variable_state(states)) would be useful? Of course, we've moved away from real use cases and back to theoretical arguments now, so it's entirely possible that doing so would only solve problems that no-one actually has... David Mertz' sublocal context manager would be a good prototype of such a thing - at least good enough to demonstrate that it's of no benefit in practice <wink> Paul
Ooops. My proof on anti-concept has a flaw. It only "shadows" names that already exist. Presumably that's the wrong idea, but it's easy enough to change if desired. On Sun, Apr 29, 2018 at 5:24 PM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
[David Mertz <mertz@gnosis.cx>]
Even in the very early days when Python's runtime was more relentlessly simple-minded than it is now, these kinds of things were always hard to get right in all cases. But it's heartening to see _someone_ still has courage enough to leap into the void ;-) I see that the docs for `locals()` now say: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. I thought that explained something I was seeing, but the problem turned out to be more obvious: the "locals()" inside your "sublocal()" context manager refer to _sublocal_'s own locals, not to the locals of the code invoking `sublocal()`.. So, e.g., run this: def run(): a = 1 b = 2 def g(tag): print(f"{tag}: a {a} b {b}") with sublocal(a=6): g("first in block") a = 5 g("set a to 5") b = 19 g("set b to 19") g("after") Here's output: first in block: a 1 b 2 # the `a=6` had no visible effect set a to 5: a 5 b 2 # golden set b to 19: a 5 b 19 # also golden after: a 5 b 19 # `a` wasn't restored to 1 To be very clear, the output is the same as if the `with` statement were replaced with `if True:`. But even if we crawled up the call stack to get the right locals() dict, looks like `exec` is no longer enough (in Python 3) to badger the runtime into making it work anyway: https://bugs.python.org/issue4831 """
Specifically, what is the approved way to have exec() modify the local environment of a function?
There is none. To modify the locals of a function on the fly is not possible without several consequences: normally, function locals are not stored in a dictionary, but an array, whose indices are determined at compile time from the known locales. This collides at least with new locals added by exec. The old exec statement circumvented this, because the compiler knew that if an exec without globals/locals args occurred in a function, that namespace would be "unoptimized", i.e. not using the locals array. Since exec() is now a normal function, the compiler does not know what "exec" may be bound to, and therefore can not treat is specially. """ Worm around that (offhand I don't know how), and there are nonlocal names too. I don't know whether Python's current introspection features are enough to even find out which nonlocals have been declared, let alone to _which_ scope each nonlocal belongs. Worm around that too, then going back to the example at the top, if the manager's locals().update(_locals) had the intended effect, it would end up restoring `b` to 2 too, yes? The only names that "should be" restored are the names in the `kws` dict. So, in all, this may be a case where it's easier to implement in the compiler than to get working at runtime via ever-more-tortured Python code. And when that's all fixed, "a" can appear in both locals() and globals() (not to mention also in enclosing scopes), in which case what to do is unclear regardless how this is implemented. Which "a"(s) did the user _intend_ to shadow? The fun never ends ;-)
On Sun, Apr 29, 2018 at 9:28 PM, Tim Peters <tim.peters@gmail.com> wrote:
Actually, that wasn't my intention. As I imagined the semantics, I wanted a context manager that restored the "outside" context for anything defined "inside" the context. Allowing keyword arguments was just an extra "convenience" that was meant to be equivalent to defining/overwriting variables inside the body. So these would be equivalent: ## 1 with sublocal(): a = 1 b = 2 x = a + b # a, b now have their old values again ## 2 with sublocal(a=1, b=2): x = a + b # a, b now have their old values again ## 3 with sublocal(a=1): b = 2 x = a + b # a, b now have their old values again I knew I was ignoring nonlocals and nested function scopes. But just trying something really simple that looks like the un-proposal in some cases. Maybe there's no way in pure-Python to deal with the edge cases though.
[Tim]
[David]
What about x? I assume that's also restored.
Or even just locals - there doesn't appear to be any supported way in Python 3 for even `exec` to reliably change locals anymore. If the intent is to create an "honest-to-Guido new Python scope", then I expect changing BLOCK_OF_CODE to def unique_name(): BLOCK_OF_CODE unique_name() del unique_name gets pretty close? Any name "created" inside BLOCK_OF_CODE would be tagged by the compiler as function-local, and vanish when the function returned. Anything merely referenced would inherit the enclosing function's meaning. I can think of some warts. E.g., after global g if BLOCK_OF_CODE were g += 1 then wrapping that line alone in a function would lead to an UnboundLocalError when the function was called (because the `global g` isn't "inherited"). Anyway, since I've never done that in my life, never saw anyone else do it, and never even thought of it before today, I'm pretty confident there isn't a hitherto undetected groundswell of demand for a quick way to create an honest-to-Guido new Python scope ;-)
On 2018-04-29 10:20 PM, Ethan Furman wrote:
That ain't shadow. That is dynamic scoping. Shadowing is something different: def f(): a = 42 def g(): print(a) local a: a = 43 g() g() should print "42" both times, *if it's lexically scoped*. If it's lexically scoped, this is just adding another scope: blocks. (instead of the smallest possible scope being function scope)
I really liked the syntax that mimicked lambda even if I find it verbose : a = local x=1, y=2: x + y + 3 Even if I still prefer the postfix syntax : a = x + 3 where x = 2 About scheme "let" vs "let*", the paralel in Python is : a, b, c = 5, a+1, 2 # let syntax a = 5; b = a+1; c = 2 # let* syntax Which makes be wonder, we could use the semicolon in the syntax ? a = local x = 1; y = x+1: x + y + 3 Or with the postfix syntax : a = x + y + 3 where x = 1; y = x+1 Chaining where would be is syntax error : a = x + y + 3 where x = 1 where y = x+1 Parenthesis could be mandatory if one wants to use tuple assignment : a = local (x, y) = 1, 2: x + y + 3 When I see that, I really want to call it "def" a = def (x, y) = 1, 2: x + y + 3 a = def x = 1; y = 2: x + y + 3 Which is read define x = 1 then y = 2 in x + y + 3 Using def would be obvious this is not a function call.
[Soni L. <fakedme+py@gmail.com>]
That ain't shadow. That is dynamic scoping.
I don't believe either term is technically accurate, but don't really care.
Why? The `local` statement, despite its name, and casual talk about it, isn't _intended_ to create a new scope in any technically accurate sense. I think what it is intended to do has been adequately explained several times already. The `a` in `a = 42` is intended to be exactly the same as the `a` in `a = 43`, changing nothing at all about Python's lexical scoping rules. It is _the_ `a` local to `f`. If lexical scoping hasn't changed one whit (and it hasn't), the code _must_ print 43 first. Same as if `local a:` were replaced by `if True:`. `local` has no effect on a's value until the new "scope" _ends_, and never any effect at all on a's visibility (a's "scope"). "local" or not, after `a = 43` there is no scope anywhere, neither lexical nor dynamic, in which `a` is still bound to 42. _The_ value of `a` is 43 then, `local` or not. The _only_ twist is that `local` wants to save/restore `a`'s binding before/after the suite of code it controls. That's not really about "scope" at all with its technical meaning. I don't much care if people casually _think_ about it as being "about.scope", though. Yes, the effect is similar to what you might see in a language with dynamic scoping _if_ it pushed a _copy_ of a's current <name, object> binding on an evaluation stack at the start, and popped that (possibly mutated) copy later, but actual dynamic binding doesn't push copies, and Python isn't using any sort of dynamic evaluation stack regardless. That both restore previous bindings at times is a shallow coincidence. Neither is it really shadowing, which is lexical hiding of names. Neither is an accurate model for what it actually does, but, yes, it bears more _resemblance_ to dynamic scoping if you ignore that it's not dynamic scoping ;-)
If it's lexically scoped, this is just adding another scope: blocks. (instead of the smallest possible scope being function scope)
I expect that thinking about "scope" at all just confuses people here, unless they don't think too much about it ;-) Nothing about Python's scoping rules changes one whit. Exactly the same in the above could be achieved by replacing the `local a:` construct above by, e.g,, __save_a_with_a_unique_name = a a = 43 g() a = __save_a_with_a_unique_name Indeed, that's one way the compiler could _implement_ it. Nothing about a's scope is altered; it's merely a block-structured save-value/restore-value gimmick.
On 2018-04-30 03:49, Tim Peters wrote:
[snip] I think it should be lexically scoped. The purpose of 'local' would be to allow you to use a name that _might_ be used elsewhere. The problem with a dynamic scope is that you might call some global function from within the local scope, but find that it's "not working correctly" because you've inadvertently shadowed a name that the function refers to. Imagine, in a local scope, that you call a global function that calls 'len', but you've shadowed 'len'...
[MRAB <python@mrabarnett.plus.com>]
I think it should be lexically scoped.
That's certainly arguable, but that's why I like real-code driven design: abstract arguments never end, and often yield a dubious in-real-life outcome after one side is worn out and the other side "wins" by attrition ;-)
Already explained at excessive length that there's nothing akin to "dynamic scopes" here, except that both happen to restore a previous binding at times. That's a shallow coincidence. It's no more "dynamic scope" than that savea = a try: a += 1 f(a) finally: a = savea is "dynamic scoping". It's merely saving/restoring a binding across a block of code.
Imagine, in a local scope, that you call a global function that calls 'len', but you've shadowed 'len'...
I'm not clear on whether you picked the name of a builtin to make a subtle point not spelled out, but I don't think it matters. Regardless of whether `len` refers to a builtin or a module global inside your global function now, the _current_ def f(): len = 12 global_function() has no effect at all on the binding of `len` seen inside `global_function`. Because my understanding of "local:" changes absolutely nothing about Python's current scope rules, it's necessarily the case that the same would be true in: def f(): local len: len = 12 call_something() The only difference from current semantics is that if print(len) were added after the `local:` block, UnboundLocalError would be raised (restoring the state of the function-local-with-or-without-'local:' `len` to what it was before the block). To have "local:" mean "new nested lexical scope" instead requires specifying a world of semantics that haven't even been mentioned yet. In Python today, in the absence of `global` and `nonlocal` declarations, the names local to a given lexical scope are determined entirely by analyzing binding sites. If you intend something other than that, then it needs to be spelled out. But if you intend to keep "and names appearing in binding sites are also local to the new lexical scope", I expect that's pretty much useless. For example, def f(): ... local x. y: x = a*b y = a/b r1, r2 = x+y, x-y That is, the programmer surely doesn't _intend_ to throw away r1 and r2 when the block ends. If they have to add a nonlocal r1, r2 declaration at the top of the block, maybe it would work as intended. But it still wouldn't work unless `r1` and `r2` _also_ appeared in binding sites in an enclosing lexical scope. If they don't, you'd get a compile-time error like SyntaxError: no binding for nonlocal 'r1' found To be more accurate, the message should really say "sorry, but I have no idea in which scope you _intend_ 'r1' to live, because the only way I could know that is to find a binding site for 'r1', and I can't find any except inside _this_ scope containing the 'nonlocal'". But that's kind of wordy ;-) If you agree that makes the feature probably unusable, you don't get off the hook by saying "no, unlike current Python scopes, binding sites have nothing to do with what's local to a new lexical scope introduced by 'local:'". The same question raised in the example above doesn't go away: in which scope(s) are 'r1' and 'r2' to be bound? There's more than one plausible answer to that, but in the absence of real use cases how can they be judged? Under "'local:' changes nothing at all about Python's scopes", the answer is obvious: `r1` and `r2` are function locals (exactly the same as if "local:" hadn't been used). There's nothing new about scope to learn, and the code works as intended on the first try ;-) Of course "local:" would be a misleading name for the construct, though. Going back to your original example, where a global (not builtin) "len" was intended: def f(): global len # LINE ADDED HERE local len: len = 12 global_function() yes, in _that_ case the global-or-builtin "len" seen inside `global_function` would change under my "nothing about scoping changes" reading, but would not under your reading. That's worth _something_ ;-) But without fleshing out the rules for all the other stuff (like which scope(s) own r1 and r2 in the example above) I can't judge whether it's worth enough to care. All the plausibly realistic use cases I've considered don't _really_ want a full-blown new scope (just robust save/restore for a handful of explicitly given names), and the example just above is contrived in comparison. Nobody types "global len" unless they _intend_ to rebind the global `len`, in which case i'm happy to let them shoot both feet off ;-) In any case, nothing can change the binding of the builtin "len" short of mucking directly with the mapping object implementing builtin lookups. Note: most of this doesn't come up in most other languages because they require explicitly declaring in which scope a name lives. Python's "infer that in almost all cases instead from examining binding sites" has consequences.
On 2018-04-30 21:41, Tim Peters wrote:
After all, what's the point of specifying names after the 'local' if _any_ binding in the local scope was local?
local b: local c: c = 0 # Bound in the "local c" scope. b = 0 # Bound in the "local b" scope. a = 0 # Bound in the main scope (function, global, whatever)
Would/should it be possible to inject a name into a local scope? You can't inject into a function scope, and names in a function scope can be determined statically (they are allocated slots), so could the same kind of thing be done for names in a local scope?
[MRAB <python@mrabarnett.plus.com>]
Don't look at me ;-) In the absence of use cases, I don't know which problem(s) you're trying to solve. All the use cases I've looked at are adequately addressed by having some spelling of "local:" change nothing at all about Python's current scope rules. If you have uses in mind that require more than just that, I'd need to see them.
Any binding that's not specified as local is bound in the parent scope:
Reverse-engineering the example following, is this a fair way of making that more precise? Given a binding-target name N in scope S, N is bound in scope T, where T is the closest-containing scope (which may be S itself) for which T is either 1. established by a "local:" block that declares name N or 2. not established by a "local: block
By clause #1 above, "c" is declared in the starting "local:" scope.
b = 0 # Bound in the "local b" scope.
By clause #1 above, after searching one scope up to find `b` declared in a "local:" scope
a = 0 # Bound in the main scope (function, global, whatever)
By clause #2 above, after searching two scopes up and not finding any "local:" scope declaring name "a". By your original "the parent scope", I would have expected this be bound in the "local b:" scope (which is the parent scope of the "local c:" scope). So that's _a_ possible answer. It's not like the scoping rules in any other language I'm aware of, but then Python's current scoping rules are unique too. Are those useful rules? Optimal? The first thing that popped into your head? The third? Again I'd need to see use cases to even begin to guess. I agree it's well defined, though, and so miles ahead of most ideas ;-) ...
Sorry, I'm unclear on what "inject a name into a local scope" means. Do you mean at runtime? In Python's very early days, all scope namespaces were implemented as Python dicts, and you could mutate those at runtime any way you liked. Name lookup first tried the "local" dict ("the" because local scopes didn't nest), then the "global" dict, then the "builtin" dict. Names could be added or removed from any of those at will. People had a lot of fun playing with that, but nobody seriously complained as that extreme flexibility was incrementally traded away for faster runtime. So now take it as given that the full set of names in a local scope must be determinable at compile-time (modulo whatever hacks may still exist to keep old "from module import *" code working - if any still do exist). I don't believe CPython has grown any optimizations preventing free runtime mutation of global (module-level) or builtin namespace mappings, but I may be wrong about that.
[MRAB]
Any binding that's not specified as local is bound in the parent scope:
[Tim]
Here's an example where I don't know what the consequences of "the rules" should be: def f(): a = 10 local a: def showa(): print("a is", a) showa() # 10 a = 20 showa() # 20 a = 30 showa() # 10 The comments show what the output would be under the "nothing about scope rules change" meaning. They're all obvious (since there is is no new scope then - it's all function-local). But under the other meaning ...? The twist here is that `def` is an executable statement in Python, and is a "binding site" for the name of the function being defined. So despite that `showa` appears to be defined in a new nested lexical scope, it's _actually_ bound as a function-local name. That's bound to be surprising to people from other languages: "I defined it in a nested lexical scope, but the name is still visible after that scope ends?". I don't know what the first `showa()` is intended to do. Presumably `a` is unbound at the start of the new nested scope? So raises NameError? If so, comment that line out so we can make progress ;-) It seems clear that the second `showa()` will display 20 under any reading. But the third? Now we're out of the `local a:` scope, but call a function whose textual definition was inside that scope. What does `showa()` do now to find a's value? f's local `a` had nothing to do with the `a` in the nested scope, so presumably it shouldn't display 10 now. What should it do? Does the final state of the nested scope's locals need to preserved so that showa() can display 30 instead? Or ...? Not necessarily complaining - just fleshing out a bit my earlier claim that a world of semantics need to be defined if anything akin to a "real scope" is desired.
[Tim]
Just noting that a sane way to answer such questions is to model the intent with nested functions in current Python. It's annoying because you have to explicitly declare the _non_-local names instead, and also make dummy assignments to those names to tell Python in which scope they _should_ be bound. So, e.g., for the above you can write this: def f(): a = 10 # Have to give `showa` _some_ binding in its intended scope, # else the "nonlocal showa" below is a compile-time error showa = None def local_a(): nonlocal showa def showa(): print("a", a) #showa() # raises NameError a = 20 showa() # 20 a = 30 local_a() del local_a showa() # 30 print("but f's a is", a) # 10 The comments show what happens when you call `f()`, matching the guesses in the original message. One thing to note: if this does model the intent, it's essentially proof that it's implementable without intractable effort; e.g., the machinery needed to implement funky closures already exists. But another: since I've never seen code anything like this before, that casts doubt on whether the semantics are actually useful in real life. That's why I always ask for real use cases ;-)
On 2018-05-01 04:40, Tim Peters wrote: the local scope's "a", not the function scope's "a". Imagine renaming the specified names that are declared 'local' throughout the nested portion: def f(): a = 10 local local_a: def showa(): print("a is", local_a) showa() # Raises NameError in showa because nothing bound to local_a yet. local_a = 20 showa() # 20 local_a = 30 showa() # Raises NameError in f because local_a not defined.
[MRAB]
In a later message I showed executable-today Python code that I believe models your intent. That code does indeed display "30" for the last call, and while I myself don't see that it's _useful_ yet, the model is incomprehensible if it doesn't. It doesn't matter that local_a isn't defined at function scope, because showa() refers to the locals _in_ the "local a:" scope. The last value bound to local_a was 30, so that's what showa() needs to show. That the name `showa` is _bound_ in f's locals has nothing to do with whose locals showa() _uses_. Other current messages about "closures" go on more about that.
[MRAB]
[Tim]
[MRAB]
Under my meaning, the same answer as always: exactly the same as if "local a:' were replaced by "if True:" ;-) Under yours, you're not going to like the answer. It seems obvious that since globals() refers to the module dict, nothing about that would change. You can add prints to my workalike-code-under-current-Python snippet to see what locals() _does_ return in various places. It's not what you'd expect (that locals() inside the `local a:` block is a dict with only key "local_a"), because of what I believe are undocumented implementation details. This is much easier to see in a bare-bones example: def f(): b = 12 def g(): nonlocal b b = 17 a = 42 print("inner", locals()) g() print("outer", locals()) f() The output: inner {'a': 42, 'b': 17} outer {'g': <function f.<locals>.g at 0x0000024E28C44AE8>, 'b': 17} That is, despite that `b` is explicitly declared as `nonlocal` in the inner function, and does in fact belong to the outer scope, it nevertheless shows up inside the inner function's locals(). The compiler isn't confused, but `locals()` existed long before nested scopes were added, and, e.g., no "nonlocals()` builtin was added later. PEP 227 (which introduced nested scopes) explicitly declined to add one, and said "it will not be possible to gain dictionary-style access to all visible scopes". I was quite surprised to see "b" as a key in the inner locals() above! But so long as it is, any new gimmick building on the current implementation would inherit that behavior.
[MRAB]
There's another question that hasn't been asked yet: what should locals() and globals() return?
[Tim, "globals()" is obvious, "locals()" can be surprising now]
...
And here recording the results of some code spelunking Dicts don't really have anything to do with how locals are implemented anymore; calling "locals()" now inside a function _constructs_ a dict on the fly out of lower-level implementation gimmicks, to be kinda-compatible with what locals() returned before nested scopes were introduced. The real distinctions of interest are recorded in the code object now, under these attributes, where I'm giving a fuller explanation than can be found in the docs, and where "referenced" doesn't distinguish between "binding merely retrieved" and "binding is changed": 1. co_varnames: tuple of names of locals not referenced in an enclosed local scope 2. co_cellvars: tuple of names of locals that are referenced in an enclosed local scope 3 .co_freevars: tuple of names local to an enclosing local scope and referenced in this enclosed code "locals()" now generally builds a dict out of all three of those name sources. #1 is the only one that made sense before nested scopes were introduced. The union of #1 and #2 is the set of names local to the code's scope; CPython implements their bindings in different ways, so needs to distinguish them. The names in #3 aren't local to the code block at all (they are local to some enclosing local scope), but for whatever reason are included in the code's "locals()" dict anyway. I would not have included them. For concreteness: def disp(func): print("for", func.__name__) code = func.__code__ for attr in "co_varnames", "co_cellvars", "co_freevars": print(" ", attr, getattr(code, attr)) def outer(): outer_only = 1 outer_used_inner = 2 outer_bound_by_inner = 3 def inner(): nonlocal outer_bound_by_inner inner_only = 4 outer_bound_by_inner = 5 inner_only = outer_used_inner inner() disp(outer) disp(inner) outer() And its output: for outer co_varnames ('outer_only', 'inner') co_cellvars ('outer_bound_by_inner', 'outer_used_inner') co_freevars () for inner co_varnames ('inner_only',) co_cellvars () co_freevars ('outer_bound_by_inner', 'outer_used_inner')
On Mon, Apr 30, 2018 at 08:52:13PM -0500, Tim Peters wrote:
I don't know what MRAB means by "inject", but I know what *I* mean, and I have a real use-case for it. There is a long-running micro-optimization, often championed by Raymond, for avoiding slow global lookups (and even slower builtin lookups, since they require a global lookup to fail first) by turning them in local lookups at function-definition time. E.g. some methods from the random.Random class: def randrange(self, start, stop=None, step=1, _int=int): ... def _randbelow(self, n, int=int, maxsize=1<<BPF, type=type, Method=_MethodType, BuiltinMethod=_BuiltinMethodType): ... (copied from 3.5, I don't know what the most recent version looks like) That's a nice way to make the binding: _int = int occur once only, instead of putting it inside the function body which then needs to be executed on ever call. Effectively it's a static variable for the method, one which persists from one call to the next without requiring re-initialisation. But it's ugly :-( The function signature is full of *implementation details* instead of the parameters needed for the method's interface. Look at _randbelow which takes one actual parameter, n, plus FIVE fake parameters, int, maxsize, type, Method and BuiltinMethod, none of which should ever be passed arguments. So when I talk about injecting values into a function, that is the sort of thing I'm referring to: at function definition time, push or inject a reference to a known value (say, the builtin int) into something which behaves as a static variable. It would be nice if we could do that without polluting the function signature. I'll admit that the name "inject" came to me when I was thinking of some hypothetical decorator: @inject(int=int, maxsize=1<<BPF, type=type, ...) def _randbelow(self, n): ... that somehow pushed, or *injected*, those bindings into the function, turning them into locals, but in an alternative universe where Guido loved making new keywords, I'd use a static initialisation block and stick it inside the def: def _randbelow(self, n): static: # this gets executed once only, at function definition time int=int maxsize=1<<BPF type=type # body of _randbelow ... -- Steve
[MRAB]
So you do mean at runtime, I think. Then as before, you can do that with module and the builtin namespaces now, but all function locals need to be identifiable at compile time now. People often get confused about that when they read that "it's a binding" that makes a name local to a scope, but it's not "_doing_ a binding" that makes it local, merely that the name _syntactically_ appears as a target in a binding construct. That analysis is entirely done at compile time. In Python's very early days, all namespaces could be dynamically altered at any time in any way. As time went on, more & more restrictions were placed on runtime alteration of local scopes. I don't believe that, in current Python 3, dynamic alteration of local scopes is supported in any case anymore. Perhaps the last to go was runtime alteration of locals via `exec`. This still works in Python 2:
`exec` was a statement type in Python 2, so the compiler could recognize that and generate radically different name-lookup code in a scope containing an `exec` statement. But in Python 3, `exec` is just another builtin function, and the compiler knows nothing about what it does. Similar code run in Python 3 has no effect on f's locals.
[MRAB]
... [Steven D'Aprano <steve@pearwood.info>]
Before nested scopes, it was also used to help nested functions call each other; e.g., def f(): def g(): ... # h needs to call g, but can't see f's locals def h(g=g): # but default arg expressions can see f's locals g() # works great :-)
Doesn't matter; code like that has been in the std lib since the start, although Raymond is inordinately fond of it ;-)
Error-prone too, because absolutely nothing stops a user from calling these things with more positional arguments than are intended. I don't want to bother looking now, but there was _some_ "bug report" complaining that one of these "default arguments" went away somewhere, which a user was specifying intentionally to override some accidental implementation detail. However, while it's dead obviously "error prone" in theory, it's remarkable how few times I've heard of anyone getting burned by it in practice.
I'm surprised that hasn't already been done.
Blame computer scientists. I started my paying career working on Cray Research's Fortran compiler, a language in which NO words were reserved. The syntax was such that it was always possible to determine whether a language keyword or a user-supplied name was intended. That worked out great. But it seemed to require ad hoc parsing tricks. Computer scientists invented "general" parser generators, and then every new language started declaring that some words were reserved for the language's use. That was just to make things easier for parser-generator authors, not for users ;-) Maybe we could steal a trick from the way the async statements ("async def", "async for", "async with") were added without making "async" a reserved word? Even if user code already uses the name "static", just as in Fortran the only way to parse static: that makes sense is that it's introducing a block; USER_NAME: as a statement has no meaning. In any other context, the block-opening meaning of "static" makes no sense, so it must mean a user-defined name then. We could also add, e.g., MRAB local a, b, c: and Uncle Timmy not really local at all a, b, c: without introducing any actual ambiguity in code already using any of the leading words on those lines. It would be fun to deprive Guido of the dead easy "but it's a new reserved word!" way to veto new features ;-)
I have to say, this idea feels really nice to me. It's far easier to read than := and separates the assignments and the result expression nicely. Others have brought up the same problem of = vs ==. IMO a solution could be to make a requirement that the last argument is NOT an assignment. In other words, this would be illegal: local(a=1) and you would have to do this: local(a=1, a) Now if the user mixes up = and ==, it'd be a "compile-time error". -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ On April 27, 2018 9:41:57 PM Tim Peters <tim.peters@gmail.com> wrote:
On 2018-04-27 11:37 PM, Tim Peters wrote:
Why not non-lexical variables? Basically, make this work (and print 3): def test(): i = 3 def test_inner(): print(i) hide i i = 4 test_inner() # 3 print(i) # 4 `hide`, unlike `del`, only applies to the current scope, and only forward. it does what it says on the tin: makes the variable disappear/be hidden. ofc, python doesn't support shadowing of variables, so this is kinda useless, but eh I thought it was a cool idea anyway :/ (See also this thread on the Lua mailing list: https://marc.info/?l=lua-l&m=152149915527486&w=2 )
On Sat, Apr 28, 2018 at 12:37 PM, Tim Peters <tim.peters@gmail.com> wrote:
I'm concerned that there are, in effect, two quite different uses of the exact same syntax. 1) In an arbitrary expression, local() creates a scope that is defined entirely by the parentheses. 2) In an 'if' header, the exact same local() call creates a scope that extends to the corresponding suite. For instance: a = 1; b = 2 x = a + local(a = 3, b = 4, a + b) + b if x == 10: # Prints "x is 10: 1 2" print("x is 10: ", a, b) This makes reasonable sense. The parentheses completely enclose the local scope. It's compiler magic, and you cannot explain it as a function call, but it makes intuitive sense. But the same thing inside the if header itself would be much weirder. I'm actually not even sure what it would do. And you've clearly shown that the local() call can be anywhere inside the condition, based on these examples:
At what point does the name 'm' stop referring to the local? More generally: if local(m = ...) is not m: print("Will I ever happen?") Perhaps it would be better to make this special case *extremely* special. For instance: if_local: 'if' 'local' '(' local_item (',' local_item)* ')' ':' suite as the ONLY way to have the local names persist. In other words, if you tack "is not None" onto the outside of the local() call, it becomes a regular expression-local, and its names die at the close parentheses. It'd still be a special case, but it'd be a bit saner to try to think about. ChrisA
On 2018-04-28 18:36, Chris Angelico wrote:
What if the names persist until the end of the statement? That covers if (where the statement lasts until the end of the if...elif...else block) and regular expressions, though it does introduce a potentially annoying shadowing thing:
On Sun, Apr 29, 2018 at 6:04 AM, Ed Kellett <e+python-ideas@kellett.im> wrote:
Oh, you mean exactly like PEP 572 used to advocate for? :) Can't say I'd be against that, although I'm not enamoured of the function-like syntax. But if you remove the function-like syntax and change the semantics, it isn't exactly Tim's proposal any more. :) ChrisA
[Chris Angelico <rosuav@gmail.com>]
I'm concerned that there are, in effect, two quite different uses of the exact same syntax.
Yes, the construct implements a profoundly different meaning of "scope" depending on the context it appears in.
1) In an arbitrary expression, local() creates a scope that is defined entirely by the parentheses.
Yes.
2) In an 'if' header, the exact same local() call creates a scope that extends to the corresponding suite.
And in a 'while' header, and also possibly (likely) including associated suites (elif/else). So it goes ;-) There is nothing "obvious" you can say inside an "if" or "while" expression that says "and, oh ya, this name also shadows anything of the same name from here on, except when it stops doing so". Even in C, e.g., it's not *obvious* what the scope of `i` is in: for (int i = 0; ...) { } It needs to be learned. Indeed, it's so non-obvious that C and C++ give different answers. The {...} part by itself introduces a new scope in both languages. In C++ the `int i` is viewed as being _part_ of that scope, despite that it's outside the braces. But in C the `int i` is really viewed as being part of a Yet Another new scope _enclosing_ the scope introduced by {...}, but nevertheless ending when the {...} scope ends. Not that it matters much. The practical effect is that, e.g., double i = 3.0; is legal as the first line of the block in C (shadows the `int i`), but illegal in C++ (a conflicting declaration for `i` in a single scope). In either case, it's only "obvious" if you learned it and then stopped thinking too much about it ;-)
Yup, it's effectively a function-like spelling of any number of binding constructs widely used in functional languages. I had mostly in mind Haskell's "let" pile-of-bindings "in" expression spelled as "local(" pile-of-bindings "," expression ")" The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there", since the syntax for specifying keyword arguments is already understood, and already groups as intended).
But the same thing inside the if header itself would be much weirder. I'm actually not even sure what it would do.
You think I am? ;-) I don't know that it matters, because intended use cases are far simpler than all the goofy things people _can_ dream up just for the hell of it. They need to be defined, but exactly how isn't of much interest to me. For example, let's put your example in an `if`: a = 1; b = 2 if a + local(a = 3, b = 4, a + b) + b: The rules I sketched pretty clearly imply that would be evaluated as: if 1 + (3+4) + 4: It's the final "4" that's of interest. In your original example the original `b` was restored because ")" ended the new scope, leaving the final "+b" to resolve to "+2". But because it's in an "if" expression here, the new scope doesn't end at ")" anymore.
And you've clearly shown that the local() call can be anywhere inside the condition, based on these examples:
And/or used multiple times, and/or used in nested ways. None of which anyone will actually do ;-)
Probably at the end of the final (if any) `elif` or `else` suite associated with the `if`/`while`, but possibly at the end of the suite associated with the `if`/`while`. Time to note another subtlety: people don't _really_ want "a new scope" in Python. If they did, then _every_ name appearing in a binding context (assignment statement target, `for` target, ...) for the duration would vanish when the new scope ended. What they really want is a new scope with an implied "nonlocal" declaration for every name appearing in a binding context _except_ for the specific names they're effectively trying to declare as being "sublocal" instead. So It's somewhat of a conceptual mess no mater how it's spelled ;) In most other languages this doesn't come up because the existence of a variable in a scope is established by an explicit declaration rather than inferred from examining binding sites.
if local(m = ...) is not m: print("Will I ever happen?")
No, that `print` can't be reached.
That's an interesting twist ... but, to me, if "local()" inside an if/while expression _can_ be deeply magical, then I'd be less surprised over time it it were _always_ deeply magical in those contexts. I could change my mind if use cases derived from real code suggest it would be a real problem. Or if there's no real-life interest in catering to sublocal scopes in expressions anyway ... then there's no reason to even try to use something that makes clean sense for an expression.
Tim Peters wrote:
The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there",
The trouble is that one usually expects "nothing syntactically new" to imply "nothing semantically new" as well, which is very far from the case here. So I think that not using *any* new syntax would actually be hurting users rather than helping them. If you really want to leverage existing knowledge, I'd suggest something based on lambda: let a = 3, b = 4: a + b This can be easily explained as a shorthand for (lambda a = 3, b = 4: a + b)() except, of course, for the magic needed to make it DWIM in an if or while statement. I'm still pretty uncomfortable about that. -- Greg
Tim] Peters wrote:
The points to using function-call-like syntax were already covered ("nothing syntactically new to learn there",
[Greg Ewing] [> The trouble is that one usually expects "nothing syntactically
I expect anyone who has programmed for over a year has proved beyond doubt that they can run a marathon even if forced to wear a backpack containing a ton of lead, with barbed wire wrapped around their feet ;-) They're indestructible. If they came from Perl, they'd even have a hearty laugh when they learned the syntax had utterly fooled them :-)
In that case, yes, but not in all. There's more than one kind of magic here, even outside of block constructs. See, e.g., the quadratic equation example in the original post local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), ... Try that with a lambda, and you get a NameError when computing sqrt(D) - or, worse, pick up an irrelevant value of D left over from earlier code. In Scheme terminology, what Python does with default args (keyword args too) is "let" (as if you evaluate all the expressions before doing any of the bindings), but what's wanted is "let*" (bindings are established left-to-right, one at a time, and each binding already established is visible in the expression part of each later binding). But even with `let*`, I'm not sure whether this would (or should, or shouldn't) work: local(even = (lambda n: n == 0 or odd(n-1)), odd = (lambda n: False if n == 0 else even(n-1)), odd(7)) Well, OK, I'm pretty sure it would work. But by design or by accident? ;-)
except, of course, for the magic needed to make it DWIM in an if or while statement. I'm still pretty uncomfortable about that.
That's because it's horrid. Honest, I'm not even convinced it's _worth_ "solving" , even if it didn't seem to require deep magic.
On Sat, 28 Apr 2018 at 12:41, Tim Peters <tim.peters@gmail.com> wrote: My big concern here involves the: if local(m = re.match(regexp, line)): print(m.group(0)) example. The entire block needs to be implicitly local for that to work - what happens if I assign a new name in that block? Also, what happens with: if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2, line) ): print(m.group(0)) Would the special-casing of local still apply to the block? Or would you need to do: if local(m = re.match(regexp1, line) or re.match(regexp2, line)): print(m.group(0)) This might just be lack of coffee and sleep talking, but maybe new "scoping delimiters" could be introduced. Yes - I'm suggesting introducing curly braces for blocks, but with a limited scope (pun intended). Within a local {} block statements and expressions are evaluated exactly like they currently are, including branching statements, optional semi-colons, etc. The value returned from the block is from an explicit return, or the last evalauted expression. a = 1
c = local { a=3 } * local { b=4 } c =
c = local { a=3 ; b=4 ; a*b } c = local { a = 3 b = 4 a * b } c = local(a=3,
b=local(a=2, a*a), a*b)
c = local { a = 3 b = local(a=2, a*a) return a * b }
r1, r2 = local { D = b**2 - 4*a*c sqrtD = math.sqrt(D) twoa = 2*a return ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa) }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0)) And a further implication: a = lambda a, b: local(c=4, a*b*c) a = lambda a, b: local { c = 4 return a * b * c } Tim Delaney
[Tim Delaney <timothy.c.delaney@gmail.com>]
I really don't know what you're asking there. Can you make it concrete? If, e.g., you're asking what happens if this appeared after the `print`: x = 3.14 then the answer is "the same as what would happen if `local` had not been used". We can't know what that is without context, though. Maybe x is global. Maybe x was declared nonlocal earlier. Maybe it's function-local. While it may be irrelevant to what you're asking, I noted just before: """ Time to note another subtlety: people don't _really_ want "a new scope" in Python. If they did, then _every_ name appearing in a binding context (assignment statement target, `for` target, ...) for the duration would vanish when the new scope ended. What they really want is a new scope with an implied "nonlocal" declaration for every name appearing in a binding context _except_ for the specific names they're effectively trying to declare as being "sublocal" instead. """ If by "new name" you mean that `x` didn't appear in any earlier line, then Python's current analysis would classify `x` as local to the current function (or as global if this is module-level code ...). That wouldn't change.
As is, if `local()` appears in an `if` or `while` expression, the scope extends to the end of the block construct. In that specific case, I'd expect to get a compile-time error, for attempting to initialize the same name more than once in the new scope. If the scope also encompasses associated `elif` statements, I'd instead expect local(m=...) in one of those to be treated as a re-initialization ;-) instead.
Would the special-casing of local still apply to the block?
As is, `local()` in an `if` or `while` expressions triggers deeply magical behavior, period.
Yes, not to trigger magical behavior, but to avoid the compile-time error.
c = local { a=3 } * local { b=4 }
c = local(a=3 , b=4, a*b)
c = local(a=3, b=local(a=2, a*a), a*b)
I expect you wanted b = local{a=2; a*a} there instead (braces instead of parens, and semicolon instead of comma).
return a * b }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0))
OK, this is the only case in which you used it in an `if` or `while` expression. All the questions you asked of me at the start can be asked of this spelling too. You seemed to imply at the start that the right curly brace would always mark the end of the new scope. But if that's so, the `m` in `m.group()` has nothing to do with the `m` assigned to in the `local` block - _that_ scope ended before `print` was reached. So if you're not just trying to increase the level of complexity of what can appear in a local block, a fundamental problem still needs solving ;-) I suppose you could solve it like so: local { m = re.match(regexp, line) if m: print(m.group(0)) } but, besides losing the "shortcut", it would also mean something radically different if x = 3.14 appeared after the "print". Right? If a "local block" is taken seriously, then _all_ names bound inside it vanish when the block ends.
If people do want a for-real "new scope" in Python, I certainly agree `local {...}` is far better suited for that purpose.
On Sun, 29 Apr 2018 at 10:30, Tim Peters <tim.peters@gmail.com> wrote:
That's exactly what I was asking, and as I understand what you're saying, we would have a local name m available in the indented block which went away when the block ended, but any names modified in the block are not local to the block. That seems likely to be a source of errors. To clarify my understanding, if the names 'x' and 'm' did not exist prior to the following code, what would x and m refer to after the block completed? if local(m = re.match(regexp, line)): x = 1 m = 2
Yes - I think this is exactly the same issue as with your proposed syntax.
Indeed, and I don't have a proposal - just concerns it wold be very difficult to explain and understand exactly what would happen in the case of something like: if local(m = re.match(regexp, line)): x = 1 m = 2 Regarding the syntax, I didn't want to really change your proposal, but just thought the functionality was different enough from the function call it appears to be that it probably merits different syntax. Tim Delaney
[Tim Delaney <timothy.c.delaney@gmail.com>]
[Tim Peters]
[Tim D]
If you what you _want_ is a genuinely new scope, yes. But no actual use cases so far wanted that at all. This is the kind of code about which there have been background complaints "forever": m1 = regexp1.match(line) m2 = regexp2.match(iine) if m1 and m2: do all sorts of stuff with m1 and/or m2, including perhaps modifying local variables and/or global variables and/or nonlocal variables The complaints are of two distinct kinds: 1. "I want to compute m1 and m2 _in_ the `if` test". 2. "I don't want these temp names (m1 and m2) accidentally conflicting with local names already in scope - if these names already exist, I want the temp names to shadow their current bindings until the `if` structure is done". So, if local(m1=regexp1.match(line), m2 = regexp2.match(iine), m1 and m2): intends to address both complaints via means embarrassingly obvious to the most casual observer ;-) This is, e.g., the same kind of name-specific "shadowing" magically done by list and dict comprehensions now, and by generator expressions. For example, [i**2 for i in range(10)] has no effect on whatever `i` meant before the listcomp was executed.
I hope the explanation above made that clear. What's wanted is exactly what the current m = re.match(regexp, line): if m: x =1 m = 2 _would_ do if only there were a sane way to spell "save m's current status before that all started and restore it after that all ends". So they want `x == 1` after it's over, and `m` to raise NameError.
if local { m = re.match(regexp, line) }: print(m.group(0))
Yes - I think this is exactly the same issue as with your proposed syntax.
Wholly agreed :-)
Only names appearing as targets _in_ the `local(...)` are affected in any way. The states of those names are captured, then those names are bound to the values of the associated expressions in the `local(...)`, and when the scope of the `local` construct ends (which _is_ hard to explain!) those names' original states are restored. So the effects on names are actually pretty easy to explain: all and only the names appearing inside the `local(...)` are affected.
Probably so!
On 2018-04-29 07:57, Tim Peters wrote:
How about these: local m1, m2: m1 = regexp1.match(line) m2 = regexp2.match(line): if m1 and m2: ... local m1, m2: if (m1 := regexp1.match(line)) and (m2 := regexp2.match(line)): ... local m1=regexp1.match(line), m2=regexp2.match(line): if m1 and m2: ... ? [snip]
[Tim]
[MRAB <python@mrabarnett.plus.com>]
They address complaint #2 in what seems to me a thoroughly Pythonic (direct, transparent, no more magical than necessary, easy to read) way. They don't address complaint #1 at all, but as you've shown (in the 2nd spelling) that isn't _inherently_ tied to complaint #2 (complaint #1 is what PEP 572 addresses). So _if_ PEP 572 is accepted, adding this form of a compound `local` statement too would address both of the listed complaints, at the "cost" of a bit more typing and adding a level of indentation. Neither of which bother me ;-) `local()` itself was also intended to address the even-more-in-the-background recurring desires for an expression (as opposed to statement) oriented way to use throwaway bindings; e.g., instead of temp = x + y - z + 1 r = temp**2 - 1/temp this instead: r = local(t=x + y - z + 1, t**2 - 1/t) It's not an accident that the shorter `t` is used in the latter than the former's `temp`: when people are wary of clobbering names by accident, they tend to use longer names that say "I'm just a temp - please don't _expect_ my binding to persist beyond the immediate uses on the next few lines":. Anyway, that kind of thing is common n functional languages, where "let" pile-of-bindings "in" expression kinds of constructs are widely used _as_ (sub)expressions themselves. local t = x + y - z + 1: r = t**2 - 1/t would be the same semantically, but they'd still complain about the "extra" typing and the visual "heaviness" of introducing a block for what they _think_ of as being "just another kind of expression". The `local()` I brought up was, I think, far too biased _toward_ that use. It didn't "play nice" with block-oriented uses short of excruciatingly deep magic. Your `local` statement is biased in the other direction, but that's a Good Thing :-)
On 2018-04-29 18:01, Tim Peters wrote:
As well as: local t = x + y - z + 1: r = t**2 - 1/t I wonder if it could be rewritten as: r = local t = x + y - z + 1: t**2 - 1/t Would parentheses be needed? r = (local t = x + y - z + 1: t**2 - 1/t) It kind of resembles the use of default parameters with lambda! The names would be local to the suite if used as a statement or the following expression if used in an expression, either way, the bit after the colon.
On Sun, Apr 29, 2018 at 3:30 AM, Tim Peters <tim.peters@gmail.com> wrote:
I have hard time understanding what is the demand here actually. (it's been too many posts and ideas to absorb)
def d(): global a x = 1; y = 2 a = x + y d() print (a) And this will do the thing: here if the "a" variable is "new" then it will be initialized and pushed to the outer scope, right? If there is demand for this, how about just introducing a derived syntax for the "auto-called" def block, say, just "def" without a name: def : global a x = 1; y = 2 a = x + y print (a) Which would act just like a scope block without any new rules introduced. And for inline usage directly inside single-line expression - I don't think it is plausible to come up with very nice syntax anyway, and I bet at best you'll end up with something looking C-ish, e.g.: if (def {x=1; y=2; &a = x + y } ) : ... As a short-cut for the above multi-line scope block. Mikhail
On Sun, Apr 29, 2018 at 7:22 PM, Mikhail V <mikhailwas@gmail.com> wrote:
Or even better, it would be better to avoid overloading "global" or "return", and use dedicated prefix for variables that are pushed to outer scope. I think it would look way better for cases with multiple variables: def func(): state = 0 def: localstate1 = state + 1 localstate2 = state + 2 & localstate1 & localstate2 print (localstate1) print (localstate2) Prefix in assignment would allow more expressive dispatching: def func(): state = 0 def: localstate1 = state + 1 localstate2 = state + 2 & M = state + 3 & L1, & L2 = localstate1, localstate2 print (L1, L2, M) Imo such syntax is closest do "def" block and should be so, because functions have very strong association with new scope definition, and same rules should work here. Not only it is more readable than "with local()", but also makes it easier to edit, comment/uncomment lines.
let[x](EXPR) x == EXPR let[x](a=1) x == 1 let[x](a=1, EXPR) x == EXPR let[x, y](a=1, EXPR) x == 1 y == EXPR let[x, y](a=1, b=2, EXPR) x == 2 y == EXPR z = let[x, y](a=1, EXPR) x == 1 y == EXPR z == (1, EXPR) Anybody seeing how the above might be useful, and address some of the concerns I've read? I don't recall seeing this suggested prior. I like the idea behind pseudo-function let/local, especially when paired with the explanation of equal sign precedence changes within paren, but I'm having a really hard time getting over the name binding leaking out of the paren. I like this item-ish style because it puts the name itself outside the parentheses while still retaining the benefits in readability. It also allows capturing the entire resultset, or individual parts. On Sat, Apr 28, 2018, 6:00 PM Tim Delaney <timothy.c.delaney@gmail.com> wrote:
On Apr 28, 2018 6:00 PM, "Tim Delaney" <timothy.c.delaney@gmail.com> wrote: On Sat, 28 Apr 2018 at 12:41, Tim Peters <tim.peters@gmail.com> wrote: My big concern here involves the: if local(m = re.match(regexp, line)): print(m.group(0)) example. The entire block needs to be implicitly local for that to work - what happens if I assign a new name in that block? Also, what happens with: if local(m = re.match(regexp1, line)) or local(m = re.match(regexp2, line) ): print(m.group(0)) Would the special-casing of local still apply to the block? Or would you need to do: if local(m = re.match(regexp1, line) or re.match(regexp2, line)): print(m.group(0)) This might just be lack of coffee and sleep talking, but maybe new "scoping delimiters" could be introduced. Yes - I'm suggesting introducing curly braces for blocks, but with a limited scope (pun intended). Within a local {} block statements and expressions are evaluated exactly like they currently are, including branching statements, optional semi-colons, etc. The value returned from the block is from an explicit return, or the last evalauted expression. a = 1
c = local { a=3 } * local { b=4 } c =
c = local { a=3 ; b=4 ; a*b } c = local { a = 3 b = 4 a * b } c = local(a=3,
b=local(a=2, a*a), a*b)
c = local { a = 3 b = local(a=2, a*a) return a * b }
r1, r2 = local { D = b**2 - 4*a*c sqrtD = math.sqrt(D) twoa = 2*a return ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa) }
if local(m = re.match(regexp, line)): print(m.group(0))
if local { m = re.match(regexp, line) }: print(m.group(0)) And a further implication: a = lambda a, b: local(c=4, a*b*c) a = lambda a, b: local { c = 4 return a * b * c } Tim Delaney _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 04/27/2018 07:37 PM, Tim Peters wrote:
Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope.
Note: the thing I'm most interested in isn't debates, but in whether this would be of real use in real code.
I keep going back and forth on the ":=" syntax as on the one hand I find the functionality very useful but on the other hand it's ugly and doesn't really read well. However, I can say I am solidly -1 on local, let, etc.: - local() vs locals(): very similar words with disparate meanings - more parens -- ugh - sublocal: one more scope for extra complexity -- ~Ethan~
On Apr 27 2018, Tim Peters <tim.peters-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
I think this can be done already with slighly different syntax: c = (lambda a=3, b=4: a*b)() The trailing () is a little ugly, but the semantics are much more obvious. So maybe go with a variation that makes function evaluation implicit? c = lambda! a=3, b=4: a*b (reads terrible, but maybe someone has a better idea).
if local(m = re.match(regexp, line)): print(m.group(0))
Of course, that wouldn't (and shouldn't) work anymore. But that's a good thing, IMO :-). Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
[Tim]
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
[Nikolaus Rath <Nikolaus@rath.org>]
But also broken, in a way that can't be sanely fixed. Covered before in other messages. Short course:
This context really demands (3, 4) instead. In Scheme terms, Python's lambda default arguments do "let" binding ("all at once"), but "let*" binding is what's needed ("one at a time, left to right, with bindings already done visible to later bindings"). Of course in Scheme you explicitly type either "let" or "let*" (or "letrec", or ...) depending on what you want at the time, but "let*" is overwhelmingly what's wanted when it makes a difference (in the example at the top, it makes no difference at all). Otherwise you can't build up a complex result from little named pieces that may depend on pieces already defined. See,.e.g, the quadratic equation example in the original post, where this was implicit.
...
[Tim]
Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end:
c = local(a=3, b=4, a*b)
[Nikolaus Rath <Nikolaus@rath.org>]
[Tim]
[Chris Angelico <rosuav@gmail.com>]
Almost, but by that point the idea that this is already "easily spelled" via lambdas has become ludicrously difficult to argue with a straight face ;-) By "almost", I mean there are other cases where even nesting Python lambdas doesn't capture the intent. In these cases, not only does the expression defining b refer to a, but _also_ the expression defining a refers to b. You can play, if you like, with trying to define the `iseven` lambda here in one line by nesting lambdas to define `even` and `odd` as default arguments: even = (lambda n: n == 0 or odd(n-1)) odd = (lambda n: False if n == 0 else even(n-1)) iseven = lambda n: even(n) Scheme supplies `letrec` for when "mutually recursive" bindings are needed. In Python that distinction isn't nearly as evidently needed, because Python's idea of closures doesn't capture all the bindings currently in effect,. For example, when `odd` above is defined, Python has no idea at all what the then-current binding for `even` is - it doesn't even look for "even" until the lambda is _executed_. But, to be fair, I'm not sure: iseven =- local( even = (lambda n: n == 0 or odd(n-1)), odd = (lambda n: False if n == 0 else even(n-1)). lambda n: even(n)) would have worked either. At the moment I'm certain it wouldn't. Last night I was pretty sure it would ;-)
[Tim, on differences among Scheme-ish `let`, `let*`, `letrec` binding]
Just FYI, I still haven't managed to do it as 1-liner (well, one statement). I expected the following would work, but it doesn't :-) iseven = lambda n: ( lambda n=n, \ even = (lambda n: n == 0 or odd(n-1)), \ odd = (lambda n: False if n == 0 else even(n-1)): even(n))() Ugly and obscure, but why not? In the inner lambda, `n`, `even`, and `odd` are all defined in its namespace, so why does it fail anyway?
Because while Python indeed doesn't capture the current binding for `odd` when the `even` lambda is compiled, it _does_ recognize that the name `odd` is not local to the lambda at compile-time, so generates a LOAD_GLOBAL opcode to retrieve `odd`'s binding at runtime. But there is no global `odd` (well, unless there is - and then there's no guessing what the code would do). For `even` to know at compile-time that `odd` will show up later in its enclosing lambda's arglist requires that Python do `letrec`-style binding instead. For a start ;-)
[Tim, still trying to define `iseven` in one statement]
...
[and the last attempt failed because a LOAD_GLOBAL was generated instead of a more-general runtime lookup]
So if I want LOAD_FAST instead, that has to be forced, leading to a one-statement definition that's wonderfully clear: iseven = lambda n: ( lambda n=n, even = (lambda n, e, o: n == 0 or o(n-1, e, o)), odd = (lambda n, e, o: False if n == 0 else e(n-1, e, o)): even(n, even, odd) )() Meaning "wonderfully clear" to the compiler, not necessarily to you ;-) Amusingly enough, that's a lot like the tricks we sometimes did in Python's early days, before nested scopes were added, to get recursive - or mutually referring - non-global functions to work at all. That is, since their names weren't in any scope they could access, their names had to be passed as arguments (or, sure, stuffed in globals - but that would have been ugly ;-) ).
Tim Peters wrote:
But 'even' and 'odd' not defined in the environment of the lambdas assigned to them, because default values of a function's arguments are evaluated outside of that function.
One could envisage adding a letrec-like construct, but making the argument list of an ordinary lambda behave like a letrec would be warping things rather too much, IMO. Personally I'd rather just add a "where" clause, and backwards compatibility be damned. :-) -- Greg
participants (21)
-
C Anthony Risinger -
Chris Angelico -
David Mertz -
Ed Kellett -
Ethan Furman -
Greg Ewing -
Kirill Balunov -
Mikhail V -
MRAB -
Nikolaus Rath -
Paul Moore -
Robert Vanden Eynde -
Ryan Gonzalez -
Soni L. -
Stephen J. Turnbull -
Steve Barnes -
Steven D'Aprano -
Tim Delaney -
Tim Peters -
Yury Selivanov -
Zero Piraeus