For-loop variable scope: simultaneous possession and ingestion of cake
There's been another discussion on c.l.py about the problem of lst = [] for i in range(10): lst.append(lambda: i) for f in lst: print f() printing 9 ten times instead of 0 to 9. The usual response is to do lst.append(lambda i=i: i) but this is not a very satisfying solution. For one thing, it's still abusing default arguments, something that lexical scoping was supposed to have removed the need for, and it won't work in some situations, such as if the function needs to take a variable number of arguments. Also, most other languages which have lexical scoping and first-class functions don't seem to suffer from problems like this. To someone familiar with one of those languages (e.g. Scheme, Haskell) it looks as if there's something broken about the way scoping of nested functions works in Python. However, it's not lambda that's broken, it's the for loop. In Scheme, for example, the way you normally write the equivalent of a Python for-loop results in a new scope being created for each value of the loop variable. Previous proposals to make for-loop variables local to the loop have stumbled on the problem of existing code that relies on the loop variable keeping its value after exiting the loop, and it seems that this is regarded as a desirable feature. So I'd like to propose something that would satisfy both requirements: 0. There is no change if the loop variable is not referenced by a nested function defined in the loop body. The vast majority of loop code will therefore be completely unaffected. 1. If the loop variable is referenced by such a nested function, a new local scope is effectively created to hold each successive value of the loop variable. 2. Upon exiting the loop, the final value of the loop variable is copied into the surrounding scope, for use by code outside the loop body. Rules 0 and 1 would also apply to list comprehensions and generator expressions. There is a very simple and efficient way to implement this in current CPython: If the loop variable is referenced by a nested function, it will be in a cell. Instead of rebinding the existing cell, each time around the loop a new cell is created, replacing the previous cell. Immediately before exiting the loop, one more new cell is created and the final value of the loop variable copied into it. Implementations other than CPython that aren't using cells may need to do something more traditional, such as compiling the loop body as a separate function. I think this arrangement would allow almost all existing code to continue working, and new code to be written that takes advantage of final values of loop variables. There would be a few obscure situations where the results would be different, for example if a nested function modifies the loop variable and expects the result to be reflected in the value seen from outside the loop. But I can't imagine such cases being anything other than extremely rare. The benefit would be that almost all code involving loops and nested functions would behave intuitively, Python would free itself from any remaining perception of having broken scope rules, and we would finally be able to consign the default-argument hack to the garbage collector of history. -- Greg
On Fri, Oct 03, 2008 at 10:37:39PM +1200, Greg Ewing wrote:
I lost count how many times I've stumbled upon that wart.
+1 Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
On Fri, Oct 03, 2008, Greg Ewing wrote:
Very yes, I use this constantly.
You'll need to make sure that exceptions don't break this. I'm not sure to what extent the current test suite covers the current behavior, I think that beefing it up is a necessary precondition to trying this approach. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "...if I were on life-support, I'd rather have it run by a Gameboy than a Windows box." --Cliff Wells, comp.lang.python, 3/13/2002
Aahz wrote:
You'll need to make sure that exceptions don't break this.
Good point. I think that can be addressed by wrapping the final value copying in a finally block. That will ensure that the final value is always decoupled from anything captured by a nested function. -- Greg
Greg Ewing wrote:
Why only the loop variable? What about: >>> for i in range(3): ... j= str(i) ... funs.append(lambda: j) >>> funs[0]() '2' It seems an odd sort of scope that lets rebindings inside it fall through outwards.
That's not always because of anything to do with introducing scopes though. Some of those languages can bind the variable value early, so if you were to write the equivalent of: >>> i= 3 >>> f= lambda: i >>> i= 4 f() would give you 3. I would love to see an early-value-binding language with similar syntax to Python, but I can't see how to get there from here. There are other languages with lexical scope and late value binding, such as JavaScript; their for loops behave the same as Python.
You mean: >>> i= 0 >>> geti= lambda: i >>> for i in [1]: ... print i is geti() True >>> for i in [1]: ... dummy= lambda: i ... print i is geti() False This seems unconscionably spooky to me. Explicit is better than yadda yadda. How about explicitly requesting to be given a new scope each time around the loop? This would clear up the compatibility problems. >>> for local i in range(3): ... funs.append(lambda: i) ... q= 3 >>> funs[0]() 0 >>> q NameError Or similar syntax as preferred. -- And Clover mailto:and@doxdesk.com http://www.doxdesk.com/
Andrew Clover wrote:
Spanking good point. To hack this "properly" all cell variables closed over within the loop would have to go into the per-iteration scope.
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
True, but a lot of Python programs would depend on this - even new ones because of social inertia.
Python could close over the values rather than a cell. That's *almost always* what people really care about anyway! It'd break almost no code. Of course, to satisfy Paul Graham, there'd have to be a way to go back to cells. A "cell" keyword? Other languages make this explicit - they're often called "boxes". I wonder if "nonlocal" could go if there were a "cell" keyword... It wouldn't mean the variable carries pass-by-reference semantics, though. That would be bad. Neil
Neil Toronto wrote:
Python could close over the values rather than a cell. That's *almost always* what people really care about anyway! It'd break almost no code.
No, it would break huge amounts of code. Consider def f(): g() def g(): print "Gee" Would you really like to get a NameError or UnboundLocalError on g here? -- Greg
Andrew Clover wrote:
My next step would be to propose a 'let' statement for dealing with things like that: for i in range(3): let j = str(i): funs.append(lambda: j) The 'let' statement would do the same thing for its variable as the for-loop, but in a one-off fashion.
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
I think this is an unavoidable consequence of not having variable declarations. Otherwise it would be impossible for an assignment inside a for-loop to perform an ordinary rebinding of some local variable outside it.
Can you give me an example of an imperative language that behaves that way? I don't think I've ever seen one. (Note that the above would be illegal in any functional (i.e. side-effect-free) language, since the syntax doesn't allow you to express rebinding an existing variable.)
There are other languages with lexical scope and late value binding, such as JavaScript; their for loops behave the same as Python.
Yes, and I would say that their for-loops are broken (or perhaps I should say suboptimally designed) in the same way.
Hm, there's something I left out of the specification. My intention was that the current value of the loop variable should *also* be seen from outside the loop while the loop is executing, so both of the above would print True. The implementation I suggested using cells has this property.
That's a possibility, too. I actually proposed something like it once before, using 'new': for new i in range(3): ... However, it's quite an unexpected thing to have to do, so it would do nothing to reduce the frequency of questions on c.l.py about why people's lambdas are broken. It would provide a slightly more elegant answer to give them, though. One advantage would be that it could be extended to be usable on any assignment, not just for loops, so there wouldn't be a need for a separate 'let' statement. There would be some details to sort out, e.g. if you're unpacking into multiple variables, do you use just one 'new' for them all, or do you have to put 'new' in front of each one? I.e. for new a, b, c in stuff: ... or for new a, new b, new c in stuff: ... -- Greg
On Friday 03 October 2008, Greg Ewing wrote:
Well now, that seems more than a little ridiculous. If we're going to be creating a keyword that rescopes variables, why not just use that instead of messing with the for loop. Seems like that would be a generally more useful solution anyway. Perhaps instead: for i in range(10): j = str(i) scope i, j: funs.append(lambda: (j,i))
How about C? int i; int get(void) {return i;} int main() { i=3; get() i=4; } I think the confusion is the fact that the a local scope is effectively a _global_ scope in locally defined functions. It's really a rather conceptually elegant setup, though a tad confusing. Perhaps that's because we're used to seeing globals as "the global scope" rather than "all encompassing scopes".
On Fri, Oct 3, 2008 at 6:00 PM, Dillon Collins <dillonco@comcast.net> wrote:
I don't see how this is messing with the for loop at all.
The difference between my or Greg's proposal is that the scope of the variable is the inner block while your proposal the variable has two scopes. That is, to annotate the code: for i in range(10): # creates a new variable i in function scope (if it didn't already exist) local k: # creates a new scope for k k = i**2 # creates a new variable k (every time we execute this block) a.append(lambda: k) # closure references the variable in the enclosing scope (as it normally does) print(k) # error -- there is no k in this scope for i in range(10): # creates a new variable i in function scope (if it didn't already exist) k = i**2 # creates a new variable k in function scope (if it didn't already exist) scope k: # creates new variables k and copies the values from the outer scope a.append(lambda: k) # closure references the variable in the enclosing scope (as it normally does) print(k) # prints 81, the last k from the loop I don't care for the fact that there are really two k variables here (or more properly N+1). Also, the implicit copying sort of obscures some details. The fact that my alternative requires explicit setting of the value of the scoped variable is a good thing. For example, consider this: a = [0,1,2] for i in range(3): scope a: a.append(i) ... lambda: ... a ... Yes, a is in a different scope, but references the same list so the scope is useless. At least in my or Greg's version, since you have to assign to the local variable, there's a clear place where you can see the copying is missing. a = [0,1,2] for i in range(3): local b: b = a # easy to see that this should be b = copy.copy(a) b.append(i) ... lambda: ... b ... I've used my suggested syntax because I like it a bit better although the above would also apply to Greg's suggestion. Comparing our suggestions, I think local a,b: is a bit more clear than let a = value, b = value: and mine does not force you to initialize the variables. They could be combined: 'local' var [ '=' expr ] ( ',' var [ '=' expr ] )* ':' with the ',' in the local statement taking priority over ',' in the expressions just as it does for function calls. --- Bruce
On Friday 03 October 2008, Bruce Leban wrote:
The original proposal is that the for loop create a local scope for it's variable and those created within it. That's what I was referring to; the let statement itself obviously doesn't. The point being that if we're going to make a scoping keyword, why add implicit scoping to for loops as well.
<snip examples>
If you look at the original proposal for this thread: "Upon exiting the loop, the final value of the loop variable is copied into the surrounding scope" So we are already dealing with at least one implicit copy. This is why I think that the loop scoping suggestion should basically be dropped in favor of some sort of scope block.
I consider that an advantage. Throughout all the rest of Python we deal with the concepts of variable assignments and mutable objects. Things like default arguments, global variables, and scopes (particularly prior to nonlocal). Why should scoping be any different?
'local' var [ '=' expr ] ( ',' var [ '=' expr ] )* ':'
While I like it conceptually, as I said above I don't really see the value in requiring assignment. If you did, you'd see "local a=a:" most of the time. Then, in a few months, someone will say that it's unnecessarily verbose/kind of confusing and post on this list a suggestion that "local a:" be equivalent to "local a=a:"
Bruce Leban wrote:
I tend to favour 'let' because it has a long history of use in other languages for a very similar purpose, and will be instantly recognisable to anyone familiar with those languages. But it's really only a choice of keyword. If allowing for non-initialization is desirable, it could be permitted to say let a: ... -- Greg
On Fri, Oct 3, 2008 at 11:25 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If I might suggest that the legacy way is not necessarily the right way, 'let' feels kind of like a variable declaration, while 'scope' or something else (I think I saw 'local' somewhere, which looked okay) would hopefully give more of a notion that there is something deeper going on. I am mostly wary of new users discovering 'let' and using it everywhere because they don't understand what it means. -- Cheers, Leif
Let me widen the scope of the discussion. I think it's a bit strange that the with statement doesn't have a scope. That is: with f() as x: body # x is still defined here Is this useful? To my thought it would make more sense if it introduced a scope. --- Bruce
Leif Walsh wrote:
If I might suggest that the legacy way is not necessarily the right way, 'let' feels kind of like a variable declaration
It *is* a variable declaration. If you're saying it's not *only* a variable declaration, that's true, but it's also true in other languages that have 'let'.
I am mostly wary of new users discovering 'let' and using it everywhere because they don't understand what it means.
It wouldn't do much harm if they did -- there's no semantic or performance difference if the variable isn't referenced from a nested function. It could even be considered helpful if it makes it clear that a particular temporary variable is only used in a certain region of the code. -- Greg
Dillon Collins wrote:
Well now, that seems more than a little ridiculous.
I don't think someone coming from Lisp, Scheme or Haskell would think it ridiculous. The 'let' statement will be instantly recognisable to them -- unlike your 'scope' statement, which will be familiar to nobody. It's true that with a 'let' statement or equivalent, there's no strict need for a change to the for-loop, since you can always say for i in range(10): let i = i: funcs.append(lambda: i) But it's an annoying and odd-looking piece of boilerplate to have to use, and in that respect is similar to the existing solutions of inserting another lambda or using a default argument value. So as a *convenience*, I'm suggesting that the for-loop be given automatic let-like behaviour.
No, that's not the same thing at all. You're not creating a closure when i==3 and then calling it after i=4; you're calling the function while i is still 3. The claim was that there exist side-effectful languages with closures that close over values instead of variables. C can't be one of those, because it doesn't even have closures. -- Greg
Greg Ewing wrote: tement, which will be familiar to nobody.
Whereas I consider the proposed automaticity to be a grave inconvenience and confusion factor. What if I *want* a closure to be over variables, as normal, instead of values. It seems to me that what you want is fine-grained control over scoping, or something like that. I would prefer that you overtly propose and argue for some syntax to do that explicitly, instead of sneaking something implicit into for-loops. Or perhaps a different proposal: Def statements close over variables, as currently. Lambda expression close over values, as some people seem to expect them to. This expectation seems to be the crux of the purported 'problem'. This change would also deal with Guido's example. Terry Jan Reedy
On Sat, Oct 4, 2008 at 10:18 AM, Terry Reedy <tjreedy@udel.edu> wrote: Greg Ewing wrote:
Why would you want that for a loop variable ? Can you give an example where this would be the desired behavior ? George
Terry Reedy wrote:
I don't think it would be a good idea to make defs and lambdas have different behaviour, because they're not interchangeable otherwise. If you're forced to use a def because what you want can't be done in a lambda, you'd be forced to accept different binding behaviour as well. Also, I'm not convinced that people think of defs and lambdas as differing like that. I suspect that they would have the same expectation if a def were written inside the loop instead of a lambda. -- Greg
Terry Reedy wrote:
What if I *want* a closure to be over variables, as normal, instead of values.
You can always assign the loop value to another variable inside the loop. Or use some other approach, such as a while loop. -- Greg
On Friday 03 October 2008, Greg Ewing wrote:
For what percentage of your for loops will this matter? For me, 0, though I did have one IIRC before I rewrote it. I imagine most people are no different. My point is that it's a lot of added complexity, and possible bugs, gotchas, and performance penalties for an extremely basic part of the language. If we need to have another keyword anyway, why do we need to deal with all that just to save a line in a handful of loops?
Ah, my apologies; I seem to have totally spaced reading the relevant part of the GP.
+1 from me too. Neil Toronto wrote:
Spanking good point. To hack this "properly" all cell variables closed over within the loop would have to go into the per-iteration scope.
Agreed. And to preserve current semantics these values would need to be copied to the new scope of every next iteration (if it's closed over).
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
True, but a lot of Python programs would depend on this - even new ones because of social inertia.
It'll unfortunately have to wait til python 4000 :). I like finally fixing this problem, which I've also run into. But I don't like the idea of introducing a new keyword to create new scopes. I think all variable that are assigned to in a loop body and closed over should be put into a cell as Greg proposes, not just the index variable. (In both for and while loops.) At the end of each loop iteration all such variables would need to be copied to the 'next' scope, which could be the parent scope or the next iteration. I'm trying really hard to think about cases that would break if this new behaviour was introduced, but I can't think about anything. The only thing that would 'break' is if you would want the standard example of lst = [] for i in range(10): lst.append(lambda: i) for f in lst: print f() to actually print 9 times 10. But if you want your bunch of lambda functions that you create in the loop body to all refer to the last value of i, why on earth would you even attempt to create a whole bunch of lambdas in this way?? (except to show that for loops are broken) Dillon Collins wrote:
Are you suggesting doing something with the scope of all block constructs? If the variables of the block are copied into their outer scope at the end, there really is allmost no difference between, say, an if block with it's own scope and the if blocks we have now. The only time it matters is if the block is executed multiple times and if a variable is closed over in it. So that's only in loops that create functions/lambdas in their bodies. If you are suggesting to only introduce a new 'scope' keyword or something like that and leave loops alone, I'd say I would prefer to fix the loops without introducing new grammar. Greg wrote:
The (IMO) natural approach that many functional-minded languages take is to have each block-like structure create a new scope. I think your proposal with cells for loop variables would fix it for loops, if it is applied to all variables closed over in a loop. But in other block structures that aren't executed repeatedly, this problem doesn't come up. Are there any other problem cases that aren't taken care of by this proposal or the 'nonlocal' keyword? Andrew Clover wrote:
This does point at an implementation gotcha. The global i and the i in the 'current' loop scope have to be kept in sync. But actually having two variables and keeping them synchronized so they appear to be one is near impossible due to pythons dynamic nature and multithreading. So this would require the variable i to be both a global variable and a cell variable. The module namespace dict would need to point to the cell containing i. This would require the python interpreter to check on every global lookup if it is looking at the value itself or at a cell containing the value. Perhaps this could be done with an extra tag bit on the value pointers in the module namespace dictionaries. So for some annotated code:
Whether a global variable contains a value or a cell is something the interpreter needs to check at runtime, but I think it would not have to be very expensive.
2008/10/6 Jan Kanis <jan.kanis@phil.uu.nl>:
How do you want this to behave? lst = [] a = [0] for i in range(10): a[0] = i lst.append(lambda: a[0]) for f in lst: print(f()) How about this? for a[0] in range(10): lst.append(lambda: a[0]) for f in lst: print(f()) ATM, I think this proposal will only make things more complicated from every point of view. -- Arnaud
On 06/10/2008, Arnaud Delobelle <arnodel@googlemail.com> wrote:
In my previous post I argued not to distinguish between the loop index variable and other 'loop-scope' variables, so both pieces of code should show the same behaviour. In what that behaviour is, there are two possibilities: 1) Treat arbitrary location specifications like a[0] the same as normal local/global variables. That would imply that both examples print 0, 1, ... 9. It would also imply that lists and other arbitrary python data structures need to be able to hold cells (over which the python interpreter transparently indirects, so they would not be directly visible to python code). So at the end of the second loop, the python memory layout looks like this: +---+ a ---> |[0]| -----------\ +---+ | lst | | | v | +---+ v |[9]| ---> lambda ---> cell ---> 9 |[ ]| |[8]| ---> lambda ---> cell ---> 8 |[ ]| |[7]| ---> lambda ---> cell ---> 7 |[ ]| ... (Note in case ascii art breaks: a[0] is pointing to the cell that holds the 9, the same one lst[9]'s lambda is pointing at.) But I think this would be a better solution: 2) Treat location specifiers other than local and global variables (variables you can write down without using dots or square brackets) the same as they are treated today. In that case, both loops would print ten times 9. I would want to argue this is the better approach, because when you write down a[0] you are specifically thinking of a and a[0] in terms of objects, while when you use a local variable you just need some place to store a temporary result, and not be bothered with it any more than absolutely necessary. However, I don't think anyone in their right mind would write loops using a[0] as the index variable. Have you ever had the need to do this?
ATM, I think this proposal will only make things more complicated from every point of view.
Partly true, more (variants of) proposals makes decisions harder. But you should read my proposal as merely extending Gregs original proposal of using cells in loop indexes to all variables that are used in loops, obviating the need for a separate scope/let construct. So I think it has a place in this discussion. Jan
On Wed, Oct 8, 2008 at 6:08 AM, Jan Kanis <jan.kanis@phil.uu.nl> wrote:
I don't think this is better. Not that I'm proposing we add macros to the language but if we did then macro find(a, x, i): for i in range(len(x)): if x[i]: return lambda: x[i] would operate differently for find(a, x, i) and find(a, x, i[0]). I think that's a bad idea. --- Bruce
On 08/10/2008, Bruce Leban <bruce@leapyear.org> wrote:
No, it wouldn't. The loop stops iterating as soon as one lambda is created, so the whole situation which started this thread, with multiple lambdas being created in a loop and each next iteration 'overwriting' the variable the previous lambdas refer to, does not occur. Your example would behave the same under current semantics, under my proposal, and under my proposal with complex variables also being 'celled' (the variant I don't prefer). Also, as you say, Python doesn't actually have macros and isn't going to get them any time soon. Saying that my proposal doesn't interact nicely with feature X which python doesn't have and is not going to get in the forseeable future is not really a convincing argument. You are more likely to convince me if you can show an actual example in which a[0] or an other non-simple variable is used as index variable. Like I said before, I don't think such a situation occurs, and if it does you'd want the behaviour I defended previously. Jan
On Fri, Oct 3, 2008 at 3:37 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
However, it's not lambda that's broken, it's the for loop.
I disagree. If you propose to change the for-loop to create new cells, you would also need to introduce new syntax for introducing new cells in other contexts. While it is common (especially when demonstrating the problem) to use a for loop variable in the lambda, the same problem exists when the variable referenced is constructed via other means. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Like this? >>> i = 0 >>> f = lambda: i >>> i = 1 >>> f() 1 Whether the for loop is broken really depends on what's meant by "broken". Some would say the very idea of cells is broken because *values* ought to be closed over. That's bunk, of course. ;) But I think most people are comfortable with the above example. If you unroll the loop, current behavior makes perfect sense: >>> f = [] >>> for i in [0, 1]: f.append(lambda: i) ... >>> [g() for g in f] [1, 1] >>> f = [] >>> i = 0 >>> f.append(lambda: i) >>> i = 1 >>> f.append(lambda: i) >>> [g() for g in f] [1, 1] But the deal with loops is that we don't usually unroll them in our heads when we reason about their behavior. We generally consider each iteration in isolation, or as an abstract single iteration. (After all, it only appears once in the program text.) If iterations need to depend on each other, it's almost always done via something that already exists outside the loop. The semantics that best matches that kind of reasoning is fully scoped loops. Python 4000 for scoped blocks? Neil
On Fri, Oct 3, 2008 at 3:51 PM, Neil Toronto <ntoronto@cs.byu.edu> wrote:
No, I was thinking of examples like this:
This leads me to reject claims that "the for-loop is broken" and in particular clamoring for fixing the for-loop without allowing us to fix this example.
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
I don't think the for loop is broken. If you want scopes other than global and function, then you should add that explicitly, maybe something like: a = [] for i in range(10): local k: k = i**2 a.append(lambda: k) # k is not accessible here a[3]() => 9 which is roughly equivalent to: a = [] for i in range(10): def local_k(): k = i**2 a.append(lambda: k) local_k() --- Bruce
Guido van Rossum wrote:
Yeah, I never said the for-loop was the *only* thing that's broken. :-) Perhaps "broken" is too strong a word. What I really mean is that it's designed in a way that interacts badly with nested functions. More generally, Python's inability to distinguish clearly between creating new bindings and changing existing bindings interacts badly with nested functions. I agree that the wider problem needs to be addressed somehow, and perhaps that should be the starting point. Depending on the solution adopted, we can then look at whether a change to the for-loop is still needed. -- Greg
On Fri, Oct 3, 2008 at 10:56 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
Since this idea didn't get much steam, a more modest proposal would be to relax the restriction on cells: allow the creation of new cells and the rebinding of func_closure in pure Python. Then one could explicitly create a new scope without any other change in the language through a 'localize' decorator that would create a new cell for every free variable (i.e. global or value from an enclosing scope) of the function: lst = [] for i in range(10): @localize def f(): print i lst.append(f) lst.append(localize(lambda: i**2)) I'd love to be proven wrong but I don't think localize() can be implemented in current Python. George
I think you probably can in CPython, but that would involve bytecode introspection and using ctypes.pythonapi.PyCell_New, and it would be terribly inefficient. I wrote a similar decorator that takes a function and bind some of its variables to some values. e.g @bind(x=40, y=2) def foo(): return x+y
foo() 42
It's useless of course. -- Arnaud
On 13 Oct 2008, at 15:22, George Sakkis wrote:
When I was saying it was useless, I was talking about my bind decorator of course! It's useless because the above can be written def foo(x=40, y=2): return x+y It's inefficient because it works by deconstructing and reconstructing the function bytecode. If I have the time I will post an implementation of your localize decorator in CPython later (I think it would be easy, if one ignores nonlocal variables in nested functions). -- Arnaud
On Mon, Oct 13, 2008 at 2:05 PM, Arnaud Delobelle <arnodel@googlemail.com>wrote:
But the whole point of this thread is that this is an abuse of default arguments since it changes f's signature; f(5,6), f(1), f(y=10) should all raise TypeError. It's inefficient because it works by deconstructing and reconstructing the
function bytecode.
But this happens only once at decoration time, not every time f is called, right ? George
On Monday 13 October 2008, Arnaud Delobelle wrote:
It's inefficient because it works by deconstructing and reconstructing the function bytecode.
That's not necessary. Just make a new closure for it. Here's some code (I was bored/curious). The function reclose_kwds is provided for fun. Enjoy: def cell(v): """Create a cell containing the arg via a dummy function""" def noop(): return v return noop.func_closure[0] def reclose(func, closure): """copy func, but use the given closure""" return function(func.func_code, func.func_globals, func.func_name, func.func_defaults, closure) def reclose_kwds(func, **kwds): """update func's closure using the names/values given as keywords""" cinfo = zip(func.func_code.co_freevars, func.func_closure) closure = tuple(cell(kwds[v]) if v in kwds else c for v,c in cinfo) return reclose(func, closure) def close(*names): """lock the given (non-global) variable names to their current values for function""" def _close(func): cinfo = zip(func.func_code.co_freevars, func.func_closure) closure = tuple(cell(c.cell_contents) if v in names else c for v,c in cinfo) return reclose(func, closure) return _close def close_all(func): """lock all non-global variables in function to their current values""" closure = tuple(cell(c.cell_contents) for c in func.func_closure) return reclose(func, closure) def g(): j=1 def f(): ret = [] for i in range(3): #q=lambda x:x*i*j def q(x): return x*i*j ret.append(q) return ret return f() def g2(): j=1 def f(): ret = [] for i in range(3): #q=close('i')(lambda x:x*i*j) @close('i') def q(x): return x*i*j ret.append(q) return ret return f() q1, q2, q3 = g() p1, p2, p3 = g2() print q1, q1(2) print q2, q2(2) print q3, q3(2) print print p1, p1(2) print p2, p2(2) print p3, p3(2) print
On Tuesday 14 October 2008, Arnaud Delobelle wrote:
Nope. However, I expect that globals would be even easier. If you want to freeze all the variables, you can just replace func_globals with func_globals.copy(). Otherwise, you can replace it with some proxy object that would read from your dict and fall back to the real globals if necessary. (Probably subclass dict with the __missing__ method.) Or at least I think that would work. I don't have time to try it now...
2008/10/14 Dillon Collins <dillonco@comcast.net>:
I guess you're right. My version was just an adaptation of the 'bind' decorator that I mentioned above, where only *some* non local variables were 'frozen', so I had to change the bytecode for that. I tried just to adapt it but it was the wrong approach! Anyway here is another version, without using bytecode introspection or ctypes: def new_closure(vals): args = ','.join('x%i' % i for i in range(len(vals))) f = eval("lambda %s:lambda:(%s)" % (args, args)) return f(*vals).func_closure def localize(f): f_globals = dict((n, f.func_globals[n]) for n in f.func_code.co_names) f_closure = ( f.func_closure and new_closure([c.cell_contents for c in f.func_closure]) ) return type(f)(f.func_code, f_globals, f.func_name, f.func_defaults, f_closure) -- Arnaud
On 13 Oct 2008, at 01:24, George Sakkis wrote:
Here is a (very) quick and dirty implementation in CPython (requires ctypes). I'm sure it breaks in all sorts of ways but I don't have more time to test it :) Tests follow the implementation. ------------------------- localize.py --------------------- from itertools import * import sys import ctypes from array import array from opcode import opmap, HAVE_ARGUMENT new_cell = ctypes.pythonapi.PyCell_New new_cell.restype = ctypes.py_object new_cell.argtypes = [ctypes.py_object] from types import CodeType, FunctionType LOAD_GLOBAL = opmap['LOAD_GLOBAL'] LOAD_DEREF = opmap['LOAD_DEREF'] LOAD_FAST = opmap['LOAD_FAST'] STORE_GLOBAL = opmap['STORE_GLOBAL'] STORE_DEREF = opmap['STORE_DEREF'] STORE_FAST = opmap['STORE_FAST'] code_args = ( 'argcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names', 'varnames', 'filename', 'name', 'firstlineno', 'lnotab', 'freevars', 'cellvars' ) def copy_code(code_obj, **kwargs): "Make a copy of a code object, maybe changing some attributes" for arg in code_args: if not kwargs.has_key(arg): kwargs[arg] = getattr(code_obj, 'co_%s' % arg) return CodeType(*map(kwargs.__getitem__, code_args)) def code_walker(code): l = len(code) code = array('B', code) i = 0 while i < l: op = code[i] if op >= HAVE_ARGUMENT: yield op, code[i+1] + (code[i+2] << 8) i += 3 else: yield op, None i += 1 class CodeMaker(object): def __init__(self): self.code = array('B') def append(self, opcode, arg=None): app = self.code.append app(opcode) if arg is not None: app(arg & 0xFF) app(arg >> 8) def getcode(self): return self.code.tostring() def localize(f): if not isinstance(f, FunctionType): return f nonlocal_vars = [] new_cells = [] frame = sys._getframe(1) values = dict(frame.f_globals) values.update(frame.f_locals) co = f.func_code deref = co.co_cellvars + co.co_freevars names = co.co_names varnames = co.co_varnames offset = len(deref) varindex = {} new_code = CodeMaker() # Disable CO_NOFREE in the code object's flags flags = co.co_flags & (0xFFFF - 0x40) # Count the number of arguments of f, including *args & **kwargs argcount = co.co_argcount if flags & 0x04: argcount += 1 if flags & 0x08: argcount += 1 # Change the code object so that the non local variables are # bound to new cells which are initialised to the current value # of the variable with that name in the surrounding frame. for opcode, arg in code_walker(co.co_code): vname = None if opcode in (LOAD_GLOBAL, STORE_GLOBAL): vname = names[arg] elif opcode in (LOAD_DEREF, STORE_DEREF): vname = deref[arg] else: new_code.append(opcode, arg) continue try: vi = varindex[vname] except KeyError: nonlocal_vars.append(vname) new_cells.append(new_cell(values[vname])) vi = varindex[vname] = offset offset += 1 if opcode in (LOAD_GLOBAL, LOAD_DEREF): new_code.append(LOAD_DEREF, vi) else: new_code.append(STORE_DEREF, vi) co = copy_code(co, code=new_code.getcode(), freevars=co.co_freevars + tuple(nonlocal_vars), flags=flags) return FunctionType(co, f.func_globals, f.func_name, f.func_defaults, (f.func_closure or ()) + tuple(new_cells)) ------------------------ /localize.py --------------------- Some examples:
-- Arnaud
Greg Ewing wrote:
There's been another discussion on c.l.py about the problem of
The behavior is a fact. Calling it a 'problem' is an opinion, one that I disagree with. So to me, your 'solution' is a solution to a non-problem.
If one understands that 'lambda: i' is essentially 'def f(): return i' and that code bodies are only executed when called, the behavior is obvious.
The usual response is to do
lst.append(lambda i=i: i)
Here are 5 more alternatives that have the same effect: (The 3rd is new since my c.l.p response) lst = [] for i in range(10): lst.append(eval("lambda: %d" %i)) lst = [] def f(i): return lambda: i for i in range(10): lst.append(f(i)) lst = [] def f(i): lst.append(lambda:i) for i in range(10): f(i) def populate(n): n -= 1 if n >= 0: return populate(n)+[lambda:n] else: return [] lst = populate(10) def populate(i,n,lst): if i < n: return populate(i+1,n,lst+[lambda:i]) else: return lst lst = populate(0,10,[])
but this is not a very satisfying solution.
To you.
For one thing, it's still abusing default arguments,
Use is a fact, abuse is an opinion.
something that lexical scoping was supposed to have removed the need for,
Lexical scoping allows reading of variables that vary (get rebound). In 2.6/3.0, one can also write this from within the closure. This addition was anticipated from the beginning; it just took a while for Guido to decide on the syntax from among the 10-20 proposals. Modifying func.__defaults__ is much more awkward (I somehow thought it to be read-only, but in 3.0 it is not.)
So use another method. Are not 5 others enough?
A respondant on c.l.p pointed out that Python works the same as C and Common Lisp. There are quite a few differences between Scheme/Haskell and Python. I believe neither is as widely known and used as Python.
As I understand this, you are proposing that for i in it: body be rewritten as def _(i): body for i in it: _(i) which is my third alternative above and only takes about 15 additional keystrokes, and only those are needed by an anti-default purist. Someone who wants this semantic should write it explicitly. I believe this sort of automagic would make Python even harder to learn and understand. One should be able to learn and use loops and simple functions before learning about nested functions and closures. If the loop is inside a function, as is typical for real code, and the loop body rebinds names outside the loop, then automagic addition of nonlocal declarations would be needed.
My rewrite above does not require this. [snip]
The benefit would be that almost all code involving loops and nested functions would behave intuitively,
To you, perhaps, but not to all.
Python would free itself from any remaining perception of having broken scope rules,
That is a very idiosyncratic perception.
By ruining the language? Just to save a few keystrokes? No thanks. -1000 (Overblown rhetoric meets overblown rhetoric ;-) Terry Jan Reedy
On Friday 03 October 2008, Terry Reedy wrote:
Or better yet: for i in range(10): lst.append((lambda i: lambda:i)(i)) But I'm probably not helping ;). I'd have to say, though, that if this was a problem, it'd have to be with lambda. Most people don't expect control blocks to have their own context, they only expect functions to have them, and then a 'global' one. Nested functions are awkward because they have their own context but can fall back to the parent if need be and people don't really see the sort of local-global aspect of closures. Also, how awful would the 'nonlocal' boilerplate be: count = 0 for i in lst: nonlocal count if i is not None: count += i And, unless I'm mistaken, this would make for loops incompatible with comprehensions:
That's not good.
Dillon Collins wrote:
And, unless I'm mistaken, this would make for loops incompatible with comprehensions:
No, the same thing would be done in list comprehensions as well (except for preserving the final value, since LC variables no longer leak). -- Greg
Terry Reedy wrote:
It's not just about lack of understanding -- even when you fully understand what's going on, you have to do something about it, and the available solutions are, to me, less than satisfying. Yes, that's an opinion. Most things in programming language design are. I'm discussing this to find whether anyone shares my opinion.
Here are 5 more alternatives that have the same effect:
All of which are even worse, to my eyes.
For one thing, it's still abusing default arguments,
Use is a fact, abuse is an opinion.
The reason I call it "abuse" is that the intended use of default argument values is as just that, a default to use if no other value is passed in. In this case there's no intention of passing a value in, so you're using the feature for something other than its intended purpose. What's more, if a value does happen to get passed in, it *breaks* what you're trying to do. So it only works in favourable circumstances, and isn't a general solution.
as if the function needs to take a variable number of arguments.
Usually it doesn't, but it could, if the API you're passing the function to requires it to.
A respondant on c.l.p pointed out that Python works the same as C and Common Lisp.
Yes, but... in Lisp or its derivatives, you *don't* normally write the equivalent of a for-loop by rebinding an existing control variable. You use a mapping function of some sort, in which the whole loop body is a lambda, and therefore receives a new binding for each loop value. This is the natural way to code in such languages. Most of the time you create new bindings rather than change existing ones, and this interacts well with nested functions. Python's assignment rules and lack of variable declarations, on the other hand, interact rather badly with nested functions. The most natural way of writing code often ends up rebinding where a new binding would be more appropriate. I'm suggesting a change to the for-loop because it's a place where, if it matters at all, a new binding is almost certainly what you want. To address the rest of the cases, there would be a 'let' statement or some such to introduce new bindings.
Only conceptually. It would be unacceptably inefficient to actually implement it that way in current CPython. This translation isn't quite equivalent, because if the body assigns to i, in your version the change won't be seen from outside the loop during that iteration. In my version, it will.
Since loops without any nested functions would be completely unaffected, either conceptually or implementation-wise, I don't see how this would interfere with learning about loops before closures. -- Greg
Greg Ewing wrote:
If you mean worse than the simple default arg hack, I could be persuaded. But the default arg issue is a red herring. There is no need for what you call abuse, as I amply demonstrated, and there is no way for you to stop others from (mis)using them without removing them. If you mean worse than turning for-loops into an esoteric CS monster (my view), we disagree. My deeper objection is this. Your intended-to-be-motivating example, similar to what others have occasionally posted, is a toy snippet that illustrates some points of Python behavior, but which I see no use for in real application code. Given def f(i): return i; your lst[i]() is equivalent to f(i). So just write and use the function. OK, to be persnickety, we need more code, of about the same length as needed to generate lst: def f(i): if not isinstance(i,int): raise TypeError("requires int i") if not -10 <=i <10: raise ValueError("requires -10 <= i < 10") return i So, as near as I can see, your list of identical functions with variant closure cells simulates type and range checking. What is the point of that? Perhaps you have an undisclosed real use case with much more complicated closures. It would still be true that the array index could instead to fed to the function as an arg. If there were really a reason not to do that, the 15 keystroke overhead would be relatively much smaller for more complicated (and realistic) closures. Terry Jan Reedy
Terry Reedy wrote:
My example wasn't intended to prove the existence of the problem, only refer to an already-acknowledged one. Its existence is attested by the fact that people regularly get tripped up by it. Here's a more realistic example: menu_items = [ ("New Game", 'new'), ("Resume", 'resume'), ("Quit", 'quit') ] buttons = [] for title, action in menu_items: buttons.append(Button(title, lambda: getattr(game, action)())) which gives you three buttons that all execute the 'quit' action. -- Greg Given def f(i): return i; your lst[i]() is
On 4 Oct 2008, at 05:12, Greg Ewing wrote:
Isn't this better as: buttons.append(Button(title, getattr(game, action))) Unless you want late binding of 'game', but that would be confusing.
which gives you three buttons that all execute the 'quit' action.
-- Arnaud
Arnaud Delobelle wrote:
Well, you might, for example if you implement restoring a saved game by unpickling a Game object and assigning it to game. There's always some way to rearrange it so that it works, but the point is that it's easy to write things like this that don't work, unless you really keep your wits about you. -- Greg
Greg Ewing wrote:
To me, this example and comment proves my point. If you want 'action' interpolated immediately, to not actually be a variable of each function, while you want 'game' left to be a true free variable, then overtly say so specifically in one way or another without magic. In my opinion, the 'evil default arg hack' does this nicely (though not completely), especially if a new name is used for the lambda local. lambda a=action: getattr(game,a) This is only 4 extra keystrokes. If the function is left anonymous and only called by clicking a button, there is no danger of a 'user' accidentally calling the function with an extra arg that overrides the default. This fact to me eliminates the main objection to the usage. If one still does not like that for whatever reason, complete value interpolation is nicely done by eval("lambda: getattr(game,%s)" % action) That is 13 extra spaces, including 2 spaces for easier reading.
There's always some way to rearrange it so that it works,
Also my point.
It is also easy to write things that don't work, unless you really keep your wits about you, with lists and other mutables, and with floats ;-). Terry Jan Reedy
On Fri, Oct 03, 2008 at 10:37:39PM +1200, Greg Ewing wrote:
I lost count how many times I've stumbled upon that wart.
+1 Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
On Fri, Oct 03, 2008, Greg Ewing wrote:
Very yes, I use this constantly.
You'll need to make sure that exceptions don't break this. I'm not sure to what extent the current test suite covers the current behavior, I think that beefing it up is a necessary precondition to trying this approach. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "...if I were on life-support, I'd rather have it run by a Gameboy than a Windows box." --Cliff Wells, comp.lang.python, 3/13/2002
Aahz wrote:
You'll need to make sure that exceptions don't break this.
Good point. I think that can be addressed by wrapping the final value copying in a finally block. That will ensure that the final value is always decoupled from anything captured by a nested function. -- Greg
Greg Ewing wrote:
Why only the loop variable? What about: >>> for i in range(3): ... j= str(i) ... funs.append(lambda: j) >>> funs[0]() '2' It seems an odd sort of scope that lets rebindings inside it fall through outwards.
That's not always because of anything to do with introducing scopes though. Some of those languages can bind the variable value early, so if you were to write the equivalent of: >>> i= 3 >>> f= lambda: i >>> i= 4 f() would give you 3. I would love to see an early-value-binding language with similar syntax to Python, but I can't see how to get there from here. There are other languages with lexical scope and late value binding, such as JavaScript; their for loops behave the same as Python.
You mean: >>> i= 0 >>> geti= lambda: i >>> for i in [1]: ... print i is geti() True >>> for i in [1]: ... dummy= lambda: i ... print i is geti() False This seems unconscionably spooky to me. Explicit is better than yadda yadda. How about explicitly requesting to be given a new scope each time around the loop? This would clear up the compatibility problems. >>> for local i in range(3): ... funs.append(lambda: i) ... q= 3 >>> funs[0]() 0 >>> q NameError Or similar syntax as preferred. -- And Clover mailto:and@doxdesk.com http://www.doxdesk.com/
Andrew Clover wrote:
Spanking good point. To hack this "properly" all cell variables closed over within the loop would have to go into the per-iteration scope.
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
True, but a lot of Python programs would depend on this - even new ones because of social inertia.
Python could close over the values rather than a cell. That's *almost always* what people really care about anyway! It'd break almost no code. Of course, to satisfy Paul Graham, there'd have to be a way to go back to cells. A "cell" keyword? Other languages make this explicit - they're often called "boxes". I wonder if "nonlocal" could go if there were a "cell" keyword... It wouldn't mean the variable carries pass-by-reference semantics, though. That would be bad. Neil
Neil Toronto wrote:
Python could close over the values rather than a cell. That's *almost always* what people really care about anyway! It'd break almost no code.
No, it would break huge amounts of code. Consider def f(): g() def g(): print "Gee" Would you really like to get a NameError or UnboundLocalError on g here? -- Greg
Andrew Clover wrote:
My next step would be to propose a 'let' statement for dealing with things like that: for i in range(3): let j = str(i): funs.append(lambda: j) The 'let' statement would do the same thing for its variable as the for-loop, but in a one-off fashion.
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
I think this is an unavoidable consequence of not having variable declarations. Otherwise it would be impossible for an assignment inside a for-loop to perform an ordinary rebinding of some local variable outside it.
Can you give me an example of an imperative language that behaves that way? I don't think I've ever seen one. (Note that the above would be illegal in any functional (i.e. side-effect-free) language, since the syntax doesn't allow you to express rebinding an existing variable.)
There are other languages with lexical scope and late value binding, such as JavaScript; their for loops behave the same as Python.
Yes, and I would say that their for-loops are broken (or perhaps I should say suboptimally designed) in the same way.
Hm, there's something I left out of the specification. My intention was that the current value of the loop variable should *also* be seen from outside the loop while the loop is executing, so both of the above would print True. The implementation I suggested using cells has this property.
That's a possibility, too. I actually proposed something like it once before, using 'new': for new i in range(3): ... However, it's quite an unexpected thing to have to do, so it would do nothing to reduce the frequency of questions on c.l.py about why people's lambdas are broken. It would provide a slightly more elegant answer to give them, though. One advantage would be that it could be extended to be usable on any assignment, not just for loops, so there wouldn't be a need for a separate 'let' statement. There would be some details to sort out, e.g. if you're unpacking into multiple variables, do you use just one 'new' for them all, or do you have to put 'new' in front of each one? I.e. for new a, b, c in stuff: ... or for new a, new b, new c in stuff: ... -- Greg
On Friday 03 October 2008, Greg Ewing wrote:
Well now, that seems more than a little ridiculous. If we're going to be creating a keyword that rescopes variables, why not just use that instead of messing with the for loop. Seems like that would be a generally more useful solution anyway. Perhaps instead: for i in range(10): j = str(i) scope i, j: funs.append(lambda: (j,i))
How about C? int i; int get(void) {return i;} int main() { i=3; get() i=4; } I think the confusion is the fact that the a local scope is effectively a _global_ scope in locally defined functions. It's really a rather conceptually elegant setup, though a tad confusing. Perhaps that's because we're used to seeing globals as "the global scope" rather than "all encompassing scopes".
On Fri, Oct 3, 2008 at 6:00 PM, Dillon Collins <dillonco@comcast.net> wrote:
I don't see how this is messing with the for loop at all.
The difference between my or Greg's proposal is that the scope of the variable is the inner block while your proposal the variable has two scopes. That is, to annotate the code: for i in range(10): # creates a new variable i in function scope (if it didn't already exist) local k: # creates a new scope for k k = i**2 # creates a new variable k (every time we execute this block) a.append(lambda: k) # closure references the variable in the enclosing scope (as it normally does) print(k) # error -- there is no k in this scope for i in range(10): # creates a new variable i in function scope (if it didn't already exist) k = i**2 # creates a new variable k in function scope (if it didn't already exist) scope k: # creates new variables k and copies the values from the outer scope a.append(lambda: k) # closure references the variable in the enclosing scope (as it normally does) print(k) # prints 81, the last k from the loop I don't care for the fact that there are really two k variables here (or more properly N+1). Also, the implicit copying sort of obscures some details. The fact that my alternative requires explicit setting of the value of the scoped variable is a good thing. For example, consider this: a = [0,1,2] for i in range(3): scope a: a.append(i) ... lambda: ... a ... Yes, a is in a different scope, but references the same list so the scope is useless. At least in my or Greg's version, since you have to assign to the local variable, there's a clear place where you can see the copying is missing. a = [0,1,2] for i in range(3): local b: b = a # easy to see that this should be b = copy.copy(a) b.append(i) ... lambda: ... b ... I've used my suggested syntax because I like it a bit better although the above would also apply to Greg's suggestion. Comparing our suggestions, I think local a,b: is a bit more clear than let a = value, b = value: and mine does not force you to initialize the variables. They could be combined: 'local' var [ '=' expr ] ( ',' var [ '=' expr ] )* ':' with the ',' in the local statement taking priority over ',' in the expressions just as it does for function calls. --- Bruce
On Friday 03 October 2008, Bruce Leban wrote:
The original proposal is that the for loop create a local scope for it's variable and those created within it. That's what I was referring to; the let statement itself obviously doesn't. The point being that if we're going to make a scoping keyword, why add implicit scoping to for loops as well.
<snip examples>
If you look at the original proposal for this thread: "Upon exiting the loop, the final value of the loop variable is copied into the surrounding scope" So we are already dealing with at least one implicit copy. This is why I think that the loop scoping suggestion should basically be dropped in favor of some sort of scope block.
I consider that an advantage. Throughout all the rest of Python we deal with the concepts of variable assignments and mutable objects. Things like default arguments, global variables, and scopes (particularly prior to nonlocal). Why should scoping be any different?
'local' var [ '=' expr ] ( ',' var [ '=' expr ] )* ':'
While I like it conceptually, as I said above I don't really see the value in requiring assignment. If you did, you'd see "local a=a:" most of the time. Then, in a few months, someone will say that it's unnecessarily verbose/kind of confusing and post on this list a suggestion that "local a:" be equivalent to "local a=a:"
Bruce Leban wrote:
I tend to favour 'let' because it has a long history of use in other languages for a very similar purpose, and will be instantly recognisable to anyone familiar with those languages. But it's really only a choice of keyword. If allowing for non-initialization is desirable, it could be permitted to say let a: ... -- Greg
On Fri, Oct 3, 2008 at 11:25 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If I might suggest that the legacy way is not necessarily the right way, 'let' feels kind of like a variable declaration, while 'scope' or something else (I think I saw 'local' somewhere, which looked okay) would hopefully give more of a notion that there is something deeper going on. I am mostly wary of new users discovering 'let' and using it everywhere because they don't understand what it means. -- Cheers, Leif
Let me widen the scope of the discussion. I think it's a bit strange that the with statement doesn't have a scope. That is: with f() as x: body # x is still defined here Is this useful? To my thought it would make more sense if it introduced a scope. --- Bruce
Leif Walsh wrote:
If I might suggest that the legacy way is not necessarily the right way, 'let' feels kind of like a variable declaration
It *is* a variable declaration. If you're saying it's not *only* a variable declaration, that's true, but it's also true in other languages that have 'let'.
I am mostly wary of new users discovering 'let' and using it everywhere because they don't understand what it means.
It wouldn't do much harm if they did -- there's no semantic or performance difference if the variable isn't referenced from a nested function. It could even be considered helpful if it makes it clear that a particular temporary variable is only used in a certain region of the code. -- Greg
Dillon Collins wrote:
Well now, that seems more than a little ridiculous.
I don't think someone coming from Lisp, Scheme or Haskell would think it ridiculous. The 'let' statement will be instantly recognisable to them -- unlike your 'scope' statement, which will be familiar to nobody. It's true that with a 'let' statement or equivalent, there's no strict need for a change to the for-loop, since you can always say for i in range(10): let i = i: funcs.append(lambda: i) But it's an annoying and odd-looking piece of boilerplate to have to use, and in that respect is similar to the existing solutions of inserting another lambda or using a default argument value. So as a *convenience*, I'm suggesting that the for-loop be given automatic let-like behaviour.
No, that's not the same thing at all. You're not creating a closure when i==3 and then calling it after i=4; you're calling the function while i is still 3. The claim was that there exist side-effectful languages with closures that close over values instead of variables. C can't be one of those, because it doesn't even have closures. -- Greg
Greg Ewing wrote: tement, which will be familiar to nobody.
Whereas I consider the proposed automaticity to be a grave inconvenience and confusion factor. What if I *want* a closure to be over variables, as normal, instead of values. It seems to me that what you want is fine-grained control over scoping, or something like that. I would prefer that you overtly propose and argue for some syntax to do that explicitly, instead of sneaking something implicit into for-loops. Or perhaps a different proposal: Def statements close over variables, as currently. Lambda expression close over values, as some people seem to expect them to. This expectation seems to be the crux of the purported 'problem'. This change would also deal with Guido's example. Terry Jan Reedy
On Sat, Oct 4, 2008 at 10:18 AM, Terry Reedy <tjreedy@udel.edu> wrote: Greg Ewing wrote:
Why would you want that for a loop variable ? Can you give an example where this would be the desired behavior ? George
Terry Reedy wrote:
I don't think it would be a good idea to make defs and lambdas have different behaviour, because they're not interchangeable otherwise. If you're forced to use a def because what you want can't be done in a lambda, you'd be forced to accept different binding behaviour as well. Also, I'm not convinced that people think of defs and lambdas as differing like that. I suspect that they would have the same expectation if a def were written inside the loop instead of a lambda. -- Greg
Terry Reedy wrote:
What if I *want* a closure to be over variables, as normal, instead of values.
You can always assign the loop value to another variable inside the loop. Or use some other approach, such as a while loop. -- Greg
On Friday 03 October 2008, Greg Ewing wrote:
For what percentage of your for loops will this matter? For me, 0, though I did have one IIRC before I rewrote it. I imagine most people are no different. My point is that it's a lot of added complexity, and possible bugs, gotchas, and performance penalties for an extremely basic part of the language. If we need to have another keyword anyway, why do we need to deal with all that just to save a line in a handful of loops?
Ah, my apologies; I seem to have totally spaced reading the relevant part of the GP.
+1 from me too. Neil Toronto wrote:
Spanking good point. To hack this "properly" all cell variables closed over within the loop would have to go into the per-iteration scope.
Agreed. And to preserve current semantics these values would need to be copied to the new scope of every next iteration (if it's closed over).
It seems an odd sort of scope that lets rebindings inside it fall through outwards.
True, but a lot of Python programs would depend on this - even new ones because of social inertia.
It'll unfortunately have to wait til python 4000 :). I like finally fixing this problem, which I've also run into. But I don't like the idea of introducing a new keyword to create new scopes. I think all variable that are assigned to in a loop body and closed over should be put into a cell as Greg proposes, not just the index variable. (In both for and while loops.) At the end of each loop iteration all such variables would need to be copied to the 'next' scope, which could be the parent scope or the next iteration. I'm trying really hard to think about cases that would break if this new behaviour was introduced, but I can't think about anything. The only thing that would 'break' is if you would want the standard example of lst = [] for i in range(10): lst.append(lambda: i) for f in lst: print f() to actually print 9 times 10. But if you want your bunch of lambda functions that you create in the loop body to all refer to the last value of i, why on earth would you even attempt to create a whole bunch of lambdas in this way?? (except to show that for loops are broken) Dillon Collins wrote:
Are you suggesting doing something with the scope of all block constructs? If the variables of the block are copied into their outer scope at the end, there really is allmost no difference between, say, an if block with it's own scope and the if blocks we have now. The only time it matters is if the block is executed multiple times and if a variable is closed over in it. So that's only in loops that create functions/lambdas in their bodies. If you are suggesting to only introduce a new 'scope' keyword or something like that and leave loops alone, I'd say I would prefer to fix the loops without introducing new grammar. Greg wrote:
The (IMO) natural approach that many functional-minded languages take is to have each block-like structure create a new scope. I think your proposal with cells for loop variables would fix it for loops, if it is applied to all variables closed over in a loop. But in other block structures that aren't executed repeatedly, this problem doesn't come up. Are there any other problem cases that aren't taken care of by this proposal or the 'nonlocal' keyword? Andrew Clover wrote:
This does point at an implementation gotcha. The global i and the i in the 'current' loop scope have to be kept in sync. But actually having two variables and keeping them synchronized so they appear to be one is near impossible due to pythons dynamic nature and multithreading. So this would require the variable i to be both a global variable and a cell variable. The module namespace dict would need to point to the cell containing i. This would require the python interpreter to check on every global lookup if it is looking at the value itself or at a cell containing the value. Perhaps this could be done with an extra tag bit on the value pointers in the module namespace dictionaries. So for some annotated code:
Whether a global variable contains a value or a cell is something the interpreter needs to check at runtime, but I think it would not have to be very expensive.
2008/10/6 Jan Kanis <jan.kanis@phil.uu.nl>:
How do you want this to behave? lst = [] a = [0] for i in range(10): a[0] = i lst.append(lambda: a[0]) for f in lst: print(f()) How about this? for a[0] in range(10): lst.append(lambda: a[0]) for f in lst: print(f()) ATM, I think this proposal will only make things more complicated from every point of view. -- Arnaud
On 06/10/2008, Arnaud Delobelle <arnodel@googlemail.com> wrote:
In my previous post I argued not to distinguish between the loop index variable and other 'loop-scope' variables, so both pieces of code should show the same behaviour. In what that behaviour is, there are two possibilities: 1) Treat arbitrary location specifications like a[0] the same as normal local/global variables. That would imply that both examples print 0, 1, ... 9. It would also imply that lists and other arbitrary python data structures need to be able to hold cells (over which the python interpreter transparently indirects, so they would not be directly visible to python code). So at the end of the second loop, the python memory layout looks like this: +---+ a ---> |[0]| -----------\ +---+ | lst | | | v | +---+ v |[9]| ---> lambda ---> cell ---> 9 |[ ]| |[8]| ---> lambda ---> cell ---> 8 |[ ]| |[7]| ---> lambda ---> cell ---> 7 |[ ]| ... (Note in case ascii art breaks: a[0] is pointing to the cell that holds the 9, the same one lst[9]'s lambda is pointing at.) But I think this would be a better solution: 2) Treat location specifiers other than local and global variables (variables you can write down without using dots or square brackets) the same as they are treated today. In that case, both loops would print ten times 9. I would want to argue this is the better approach, because when you write down a[0] you are specifically thinking of a and a[0] in terms of objects, while when you use a local variable you just need some place to store a temporary result, and not be bothered with it any more than absolutely necessary. However, I don't think anyone in their right mind would write loops using a[0] as the index variable. Have you ever had the need to do this?
ATM, I think this proposal will only make things more complicated from every point of view.
Partly true, more (variants of) proposals makes decisions harder. But you should read my proposal as merely extending Gregs original proposal of using cells in loop indexes to all variables that are used in loops, obviating the need for a separate scope/let construct. So I think it has a place in this discussion. Jan
On Wed, Oct 8, 2008 at 6:08 AM, Jan Kanis <jan.kanis@phil.uu.nl> wrote:
I don't think this is better. Not that I'm proposing we add macros to the language but if we did then macro find(a, x, i): for i in range(len(x)): if x[i]: return lambda: x[i] would operate differently for find(a, x, i) and find(a, x, i[0]). I think that's a bad idea. --- Bruce
On 08/10/2008, Bruce Leban <bruce@leapyear.org> wrote:
No, it wouldn't. The loop stops iterating as soon as one lambda is created, so the whole situation which started this thread, with multiple lambdas being created in a loop and each next iteration 'overwriting' the variable the previous lambdas refer to, does not occur. Your example would behave the same under current semantics, under my proposal, and under my proposal with complex variables also being 'celled' (the variant I don't prefer). Also, as you say, Python doesn't actually have macros and isn't going to get them any time soon. Saying that my proposal doesn't interact nicely with feature X which python doesn't have and is not going to get in the forseeable future is not really a convincing argument. You are more likely to convince me if you can show an actual example in which a[0] or an other non-simple variable is used as index variable. Like I said before, I don't think such a situation occurs, and if it does you'd want the behaviour I defended previously. Jan
On Fri, Oct 3, 2008 at 3:37 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
However, it's not lambda that's broken, it's the for loop.
I disagree. If you propose to change the for-loop to create new cells, you would also need to introduce new syntax for introducing new cells in other contexts. While it is common (especially when demonstrating the problem) to use a for loop variable in the lambda, the same problem exists when the variable referenced is constructed via other means. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Like this? >>> i = 0 >>> f = lambda: i >>> i = 1 >>> f() 1 Whether the for loop is broken really depends on what's meant by "broken". Some would say the very idea of cells is broken because *values* ought to be closed over. That's bunk, of course. ;) But I think most people are comfortable with the above example. If you unroll the loop, current behavior makes perfect sense: >>> f = [] >>> for i in [0, 1]: f.append(lambda: i) ... >>> [g() for g in f] [1, 1] >>> f = [] >>> i = 0 >>> f.append(lambda: i) >>> i = 1 >>> f.append(lambda: i) >>> [g() for g in f] [1, 1] But the deal with loops is that we don't usually unroll them in our heads when we reason about their behavior. We generally consider each iteration in isolation, or as an abstract single iteration. (After all, it only appears once in the program text.) If iterations need to depend on each other, it's almost always done via something that already exists outside the loop. The semantics that best matches that kind of reasoning is fully scoped loops. Python 4000 for scoped blocks? Neil
On Fri, Oct 3, 2008 at 3:51 PM, Neil Toronto <ntoronto@cs.byu.edu> wrote:
No, I was thinking of examples like this:
This leads me to reject claims that "the for-loop is broken" and in particular clamoring for fixing the for-loop without allowing us to fix this example.
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
I don't think the for loop is broken. If you want scopes other than global and function, then you should add that explicitly, maybe something like: a = [] for i in range(10): local k: k = i**2 a.append(lambda: k) # k is not accessible here a[3]() => 9 which is roughly equivalent to: a = [] for i in range(10): def local_k(): k = i**2 a.append(lambda: k) local_k() --- Bruce
Guido van Rossum wrote:
Yeah, I never said the for-loop was the *only* thing that's broken. :-) Perhaps "broken" is too strong a word. What I really mean is that it's designed in a way that interacts badly with nested functions. More generally, Python's inability to distinguish clearly between creating new bindings and changing existing bindings interacts badly with nested functions. I agree that the wider problem needs to be addressed somehow, and perhaps that should be the starting point. Depending on the solution adopted, we can then look at whether a change to the for-loop is still needed. -- Greg
On Fri, Oct 3, 2008 at 10:56 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
Since this idea didn't get much steam, a more modest proposal would be to relax the restriction on cells: allow the creation of new cells and the rebinding of func_closure in pure Python. Then one could explicitly create a new scope without any other change in the language through a 'localize' decorator that would create a new cell for every free variable (i.e. global or value from an enclosing scope) of the function: lst = [] for i in range(10): @localize def f(): print i lst.append(f) lst.append(localize(lambda: i**2)) I'd love to be proven wrong but I don't think localize() can be implemented in current Python. George
I think you probably can in CPython, but that would involve bytecode introspection and using ctypes.pythonapi.PyCell_New, and it would be terribly inefficient. I wrote a similar decorator that takes a function and bind some of its variables to some values. e.g @bind(x=40, y=2) def foo(): return x+y
foo() 42
It's useless of course. -- Arnaud
On 13 Oct 2008, at 15:22, George Sakkis wrote:
When I was saying it was useless, I was talking about my bind decorator of course! It's useless because the above can be written def foo(x=40, y=2): return x+y It's inefficient because it works by deconstructing and reconstructing the function bytecode. If I have the time I will post an implementation of your localize decorator in CPython later (I think it would be easy, if one ignores nonlocal variables in nested functions). -- Arnaud
On Mon, Oct 13, 2008 at 2:05 PM, Arnaud Delobelle <arnodel@googlemail.com>wrote:
But the whole point of this thread is that this is an abuse of default arguments since it changes f's signature; f(5,6), f(1), f(y=10) should all raise TypeError. It's inefficient because it works by deconstructing and reconstructing the
function bytecode.
But this happens only once at decoration time, not every time f is called, right ? George
On Monday 13 October 2008, Arnaud Delobelle wrote:
It's inefficient because it works by deconstructing and reconstructing the function bytecode.
That's not necessary. Just make a new closure for it. Here's some code (I was bored/curious). The function reclose_kwds is provided for fun. Enjoy: def cell(v): """Create a cell containing the arg via a dummy function""" def noop(): return v return noop.func_closure[0] def reclose(func, closure): """copy func, but use the given closure""" return function(func.func_code, func.func_globals, func.func_name, func.func_defaults, closure) def reclose_kwds(func, **kwds): """update func's closure using the names/values given as keywords""" cinfo = zip(func.func_code.co_freevars, func.func_closure) closure = tuple(cell(kwds[v]) if v in kwds else c for v,c in cinfo) return reclose(func, closure) def close(*names): """lock the given (non-global) variable names to their current values for function""" def _close(func): cinfo = zip(func.func_code.co_freevars, func.func_closure) closure = tuple(cell(c.cell_contents) if v in names else c for v,c in cinfo) return reclose(func, closure) return _close def close_all(func): """lock all non-global variables in function to their current values""" closure = tuple(cell(c.cell_contents) for c in func.func_closure) return reclose(func, closure) def g(): j=1 def f(): ret = [] for i in range(3): #q=lambda x:x*i*j def q(x): return x*i*j ret.append(q) return ret return f() def g2(): j=1 def f(): ret = [] for i in range(3): #q=close('i')(lambda x:x*i*j) @close('i') def q(x): return x*i*j ret.append(q) return ret return f() q1, q2, q3 = g() p1, p2, p3 = g2() print q1, q1(2) print q2, q2(2) print q3, q3(2) print print p1, p1(2) print p2, p2(2) print p3, p3(2) print
On Tuesday 14 October 2008, Arnaud Delobelle wrote:
Nope. However, I expect that globals would be even easier. If you want to freeze all the variables, you can just replace func_globals with func_globals.copy(). Otherwise, you can replace it with some proxy object that would read from your dict and fall back to the real globals if necessary. (Probably subclass dict with the __missing__ method.) Or at least I think that would work. I don't have time to try it now...
2008/10/14 Dillon Collins <dillonco@comcast.net>:
I guess you're right. My version was just an adaptation of the 'bind' decorator that I mentioned above, where only *some* non local variables were 'frozen', so I had to change the bytecode for that. I tried just to adapt it but it was the wrong approach! Anyway here is another version, without using bytecode introspection or ctypes: def new_closure(vals): args = ','.join('x%i' % i for i in range(len(vals))) f = eval("lambda %s:lambda:(%s)" % (args, args)) return f(*vals).func_closure def localize(f): f_globals = dict((n, f.func_globals[n]) for n in f.func_code.co_names) f_closure = ( f.func_closure and new_closure([c.cell_contents for c in f.func_closure]) ) return type(f)(f.func_code, f_globals, f.func_name, f.func_defaults, f_closure) -- Arnaud
On 13 Oct 2008, at 01:24, George Sakkis wrote:
Here is a (very) quick and dirty implementation in CPython (requires ctypes). I'm sure it breaks in all sorts of ways but I don't have more time to test it :) Tests follow the implementation. ------------------------- localize.py --------------------- from itertools import * import sys import ctypes from array import array from opcode import opmap, HAVE_ARGUMENT new_cell = ctypes.pythonapi.PyCell_New new_cell.restype = ctypes.py_object new_cell.argtypes = [ctypes.py_object] from types import CodeType, FunctionType LOAD_GLOBAL = opmap['LOAD_GLOBAL'] LOAD_DEREF = opmap['LOAD_DEREF'] LOAD_FAST = opmap['LOAD_FAST'] STORE_GLOBAL = opmap['STORE_GLOBAL'] STORE_DEREF = opmap['STORE_DEREF'] STORE_FAST = opmap['STORE_FAST'] code_args = ( 'argcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names', 'varnames', 'filename', 'name', 'firstlineno', 'lnotab', 'freevars', 'cellvars' ) def copy_code(code_obj, **kwargs): "Make a copy of a code object, maybe changing some attributes" for arg in code_args: if not kwargs.has_key(arg): kwargs[arg] = getattr(code_obj, 'co_%s' % arg) return CodeType(*map(kwargs.__getitem__, code_args)) def code_walker(code): l = len(code) code = array('B', code) i = 0 while i < l: op = code[i] if op >= HAVE_ARGUMENT: yield op, code[i+1] + (code[i+2] << 8) i += 3 else: yield op, None i += 1 class CodeMaker(object): def __init__(self): self.code = array('B') def append(self, opcode, arg=None): app = self.code.append app(opcode) if arg is not None: app(arg & 0xFF) app(arg >> 8) def getcode(self): return self.code.tostring() def localize(f): if not isinstance(f, FunctionType): return f nonlocal_vars = [] new_cells = [] frame = sys._getframe(1) values = dict(frame.f_globals) values.update(frame.f_locals) co = f.func_code deref = co.co_cellvars + co.co_freevars names = co.co_names varnames = co.co_varnames offset = len(deref) varindex = {} new_code = CodeMaker() # Disable CO_NOFREE in the code object's flags flags = co.co_flags & (0xFFFF - 0x40) # Count the number of arguments of f, including *args & **kwargs argcount = co.co_argcount if flags & 0x04: argcount += 1 if flags & 0x08: argcount += 1 # Change the code object so that the non local variables are # bound to new cells which are initialised to the current value # of the variable with that name in the surrounding frame. for opcode, arg in code_walker(co.co_code): vname = None if opcode in (LOAD_GLOBAL, STORE_GLOBAL): vname = names[arg] elif opcode in (LOAD_DEREF, STORE_DEREF): vname = deref[arg] else: new_code.append(opcode, arg) continue try: vi = varindex[vname] except KeyError: nonlocal_vars.append(vname) new_cells.append(new_cell(values[vname])) vi = varindex[vname] = offset offset += 1 if opcode in (LOAD_GLOBAL, LOAD_DEREF): new_code.append(LOAD_DEREF, vi) else: new_code.append(STORE_DEREF, vi) co = copy_code(co, code=new_code.getcode(), freevars=co.co_freevars + tuple(nonlocal_vars), flags=flags) return FunctionType(co, f.func_globals, f.func_name, f.func_defaults, (f.func_closure or ()) + tuple(new_cells)) ------------------------ /localize.py --------------------- Some examples:
-- Arnaud
Greg Ewing wrote:
There's been another discussion on c.l.py about the problem of
The behavior is a fact. Calling it a 'problem' is an opinion, one that I disagree with. So to me, your 'solution' is a solution to a non-problem.
If one understands that 'lambda: i' is essentially 'def f(): return i' and that code bodies are only executed when called, the behavior is obvious.
The usual response is to do
lst.append(lambda i=i: i)
Here are 5 more alternatives that have the same effect: (The 3rd is new since my c.l.p response) lst = [] for i in range(10): lst.append(eval("lambda: %d" %i)) lst = [] def f(i): return lambda: i for i in range(10): lst.append(f(i)) lst = [] def f(i): lst.append(lambda:i) for i in range(10): f(i) def populate(n): n -= 1 if n >= 0: return populate(n)+[lambda:n] else: return [] lst = populate(10) def populate(i,n,lst): if i < n: return populate(i+1,n,lst+[lambda:i]) else: return lst lst = populate(0,10,[])
but this is not a very satisfying solution.
To you.
For one thing, it's still abusing default arguments,
Use is a fact, abuse is an opinion.
something that lexical scoping was supposed to have removed the need for,
Lexical scoping allows reading of variables that vary (get rebound). In 2.6/3.0, one can also write this from within the closure. This addition was anticipated from the beginning; it just took a while for Guido to decide on the syntax from among the 10-20 proposals. Modifying func.__defaults__ is much more awkward (I somehow thought it to be read-only, but in 3.0 it is not.)
So use another method. Are not 5 others enough?
A respondant on c.l.p pointed out that Python works the same as C and Common Lisp. There are quite a few differences between Scheme/Haskell and Python. I believe neither is as widely known and used as Python.
As I understand this, you are proposing that for i in it: body be rewritten as def _(i): body for i in it: _(i) which is my third alternative above and only takes about 15 additional keystrokes, and only those are needed by an anti-default purist. Someone who wants this semantic should write it explicitly. I believe this sort of automagic would make Python even harder to learn and understand. One should be able to learn and use loops and simple functions before learning about nested functions and closures. If the loop is inside a function, as is typical for real code, and the loop body rebinds names outside the loop, then automagic addition of nonlocal declarations would be needed.
My rewrite above does not require this. [snip]
The benefit would be that almost all code involving loops and nested functions would behave intuitively,
To you, perhaps, but not to all.
Python would free itself from any remaining perception of having broken scope rules,
That is a very idiosyncratic perception.
By ruining the language? Just to save a few keystrokes? No thanks. -1000 (Overblown rhetoric meets overblown rhetoric ;-) Terry Jan Reedy
On Friday 03 October 2008, Terry Reedy wrote:
Or better yet: for i in range(10): lst.append((lambda i: lambda:i)(i)) But I'm probably not helping ;). I'd have to say, though, that if this was a problem, it'd have to be with lambda. Most people don't expect control blocks to have their own context, they only expect functions to have them, and then a 'global' one. Nested functions are awkward because they have their own context but can fall back to the parent if need be and people don't really see the sort of local-global aspect of closures. Also, how awful would the 'nonlocal' boilerplate be: count = 0 for i in lst: nonlocal count if i is not None: count += i And, unless I'm mistaken, this would make for loops incompatible with comprehensions:
That's not good.
Dillon Collins wrote:
And, unless I'm mistaken, this would make for loops incompatible with comprehensions:
No, the same thing would be done in list comprehensions as well (except for preserving the final value, since LC variables no longer leak). -- Greg
Terry Reedy wrote:
It's not just about lack of understanding -- even when you fully understand what's going on, you have to do something about it, and the available solutions are, to me, less than satisfying. Yes, that's an opinion. Most things in programming language design are. I'm discussing this to find whether anyone shares my opinion.
Here are 5 more alternatives that have the same effect:
All of which are even worse, to my eyes.
For one thing, it's still abusing default arguments,
Use is a fact, abuse is an opinion.
The reason I call it "abuse" is that the intended use of default argument values is as just that, a default to use if no other value is passed in. In this case there's no intention of passing a value in, so you're using the feature for something other than its intended purpose. What's more, if a value does happen to get passed in, it *breaks* what you're trying to do. So it only works in favourable circumstances, and isn't a general solution.
as if the function needs to take a variable number of arguments.
Usually it doesn't, but it could, if the API you're passing the function to requires it to.
A respondant on c.l.p pointed out that Python works the same as C and Common Lisp.
Yes, but... in Lisp or its derivatives, you *don't* normally write the equivalent of a for-loop by rebinding an existing control variable. You use a mapping function of some sort, in which the whole loop body is a lambda, and therefore receives a new binding for each loop value. This is the natural way to code in such languages. Most of the time you create new bindings rather than change existing ones, and this interacts well with nested functions. Python's assignment rules and lack of variable declarations, on the other hand, interact rather badly with nested functions. The most natural way of writing code often ends up rebinding where a new binding would be more appropriate. I'm suggesting a change to the for-loop because it's a place where, if it matters at all, a new binding is almost certainly what you want. To address the rest of the cases, there would be a 'let' statement or some such to introduce new bindings.
Only conceptually. It would be unacceptably inefficient to actually implement it that way in current CPython. This translation isn't quite equivalent, because if the body assigns to i, in your version the change won't be seen from outside the loop during that iteration. In my version, it will.
Since loops without any nested functions would be completely unaffected, either conceptually or implementation-wise, I don't see how this would interfere with learning about loops before closures. -- Greg
Greg Ewing wrote:
If you mean worse than the simple default arg hack, I could be persuaded. But the default arg issue is a red herring. There is no need for what you call abuse, as I amply demonstrated, and there is no way for you to stop others from (mis)using them without removing them. If you mean worse than turning for-loops into an esoteric CS monster (my view), we disagree. My deeper objection is this. Your intended-to-be-motivating example, similar to what others have occasionally posted, is a toy snippet that illustrates some points of Python behavior, but which I see no use for in real application code. Given def f(i): return i; your lst[i]() is equivalent to f(i). So just write and use the function. OK, to be persnickety, we need more code, of about the same length as needed to generate lst: def f(i): if not isinstance(i,int): raise TypeError("requires int i") if not -10 <=i <10: raise ValueError("requires -10 <= i < 10") return i So, as near as I can see, your list of identical functions with variant closure cells simulates type and range checking. What is the point of that? Perhaps you have an undisclosed real use case with much more complicated closures. It would still be true that the array index could instead to fed to the function as an arg. If there were really a reason not to do that, the 15 keystroke overhead would be relatively much smaller for more complicated (and realistic) closures. Terry Jan Reedy
Terry Reedy wrote:
My example wasn't intended to prove the existence of the problem, only refer to an already-acknowledged one. Its existence is attested by the fact that people regularly get tripped up by it. Here's a more realistic example: menu_items = [ ("New Game", 'new'), ("Resume", 'resume'), ("Quit", 'quit') ] buttons = [] for title, action in menu_items: buttons.append(Button(title, lambda: getattr(game, action)())) which gives you three buttons that all execute the 'quit' action. -- Greg Given def f(i): return i; your lst[i]() is
On 4 Oct 2008, at 05:12, Greg Ewing wrote:
Isn't this better as: buttons.append(Button(title, getattr(game, action))) Unless you want late binding of 'game', but that would be confusing.
which gives you three buttons that all execute the 'quit' action.
-- Arnaud
Arnaud Delobelle wrote:
Well, you might, for example if you implement restoring a saved game by unpickling a Game object and assigning it to game. There's always some way to rearrange it so that it works, but the point is that it's easy to write things like this that don't work, unless you really keep your wits about you. -- Greg
Greg Ewing wrote:
To me, this example and comment proves my point. If you want 'action' interpolated immediately, to not actually be a variable of each function, while you want 'game' left to be a true free variable, then overtly say so specifically in one way or another without magic. In my opinion, the 'evil default arg hack' does this nicely (though not completely), especially if a new name is used for the lambda local. lambda a=action: getattr(game,a) This is only 4 extra keystrokes. If the function is left anonymous and only called by clicking a button, there is no danger of a 'user' accidentally calling the function with an extra arg that overrides the default. This fact to me eliminates the main objection to the usage. If one still does not like that for whatever reason, complete value interpolation is nicely done by eval("lambda: getattr(game,%s)" % action) That is 13 extra spaces, including 2 spaces for easier reading.
There's always some way to rearrange it so that it works,
Also my point.
It is also easy to write things that don't work, unless you really keep your wits about you, with lists and other mutables, and with floats ;-). Terry Jan Reedy
participants (14)
-
Aahz -
Andrew Clover -
Arnaud Delobelle -
Bruce Leban -
Dillon Collins -
George Sakkis -
Greg Ewing -
Guido van Rossum -
Jan Kanis -
Leif Walsh -
Neil Toronto -
Oleg Broytmann -
Ron Adam -
Terry Reedy