In the PEP 572 threads there's some grumbling about class scopes. Here's a random brainstorm. How about we change the scoping rules so that only functions defined with 'def' (or inside one of those) cannot see the class scope, and comprehensions and lambdas treat the class scope as an outer scope so they can close over class variables? Not sure what should happen to nested classes, they should probably be lumped in with the 'def' scopes. -- --Guido van Rossum (python.org/~guido)
On Sun, Mar 25, 2018 at 5:51 PM Guido van Rossum <guido@python.org> wrote:
I would expect classes to behave like comprehensions - i.e. code that is not delayed will see the class' scope as an enclosing scope (since it is not namespaced yet so there's no other way to refer to it) and delayed code (e.g. lambdas or defs) will not. Elazar
On 26 March 2018 at 01:50, Guido van Rossum <guido@python.org> wrote:
I think it's mainly in comprehensions that folks may find the current behaviour surprising, as that's the case where we define a function and immediately call it, so it mostly looks like regular inline code execution. Once explicitly delayed invocation is involved (lambdas, def statements), folks seem to be more comfortable with the idea "Oh, of course that's going to behave like a method at class scope". Generator expressions also mostly get a pass, since the *iteration* is delayed, even though the generator-iterator itself is created immediately. One possibility that has occurred to me (but I've never investigated to check the technical feasibility) is to see whether we might be able to define a notion of "implied arguments" for the implicit comprehension and generator expression function definitions. The gist of that idea would be: - when walking the symbol table for a comprehension or genexp, keep track of every variable name mentioned in that subtree which is not resolved in that subtree (which we already have the machinery to do, as it's needed for compiling functions in general) - treat all those names as *implied arguments* to the implicit function, and call it with the right references in the right positions While we'd likely still want to keep evaluation of the outermost iterator outside the nested scope for the sake of better genexp tracebacks, that would still be enough to enable cases like: class C: y = 1 values = [x+y for x in range(10)] The trick would be to make it so that that comprehension gets expanded as: def _listcomp(_outermost_iter, y): result = [] for x in _outermost_iter: result.append(x+y) return result _listcomp_result = _listcomp(range(10), y) One of the more attractive aspects of this possibility (assuming it can be made to work) is that it isn't even a special case in the runtime name lookup rules - it's just changing the definition of the implicit functions that we create for comprehensions and generator expressions to make sure that they can refer to *any* name that resolves in the namespace where they're called (even invisible-to-the-compiler names like those injected via __prepare__ methods, or assignments via locals() at class scope). Cheers, Nick. P.S. I suspect we'd also find that this served as a performance optimisation, since it wouldn't only prebind class locals, it would prebind references to builtins and module globals as well. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Mar 26, 2018 at 6:33 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Honestly that sounds way too complex. In my ideal world, only functions defined with 'def' are methods whose __get__ auto-binds the first argument (unless @classmethod or @staticmethod is present), and do not get to "see" the class scope. This is how you define methods after all. Lambdas used at class scope do not get this benefit, and they see the class scope as just another closure; ditto for comprehensions and genexprs. I don't think that the "is the call delayed" idea is the proper way to distinguish the two cases here. IMO the key concept is "is it used to define a method". I betcha if you see a lambda here it's because there's some calculation going on whose result will be stored as a class variable. -- --Guido van Rossum (python.org/~guido)
On 27 March 2018 at 01:11, Guido van Rossum <guido@python.org> wrote:
Aye, I don't disagree with that. The implicit functions used in the comprehension & generator expression cases are just potentially simpler to handle, as we don't care about their API signatures, which means we can freely pollute their APIs with eager name bindings if we choose to do so. Lambda expressions are different, since their public API matters, so we can't mess with it freely to pass in extra name references. What we potentially *could* do though is allow implicit additions to their __closure__ attributes in a way that isn't permitted for regular function definitions, such that in the following code: class C: x = 1 f = staticmethod(lambda: print(x)) we would do the following things differently for lambdas (vs def functions): 1. We'd pass "x" down as a candidate for nonlocal name resolution while compiling the lambda 2. If "x" did get referenced from a lambda, we'd start allowing non-optimised code objects to have a list of cell vars (the way functions already do), and define a new "STORE_CLASSDEREF" opcode (as the counterpart to LOAD_CLASSDEREF) What STORE_CLASSDEREF would need to do is write updates both to the current locals namespace (so the expectations of a class body are met) *and* to the nominated cell object (so any references from lambda expressions are updated). The biggest downside I'd see to this is that for an incredibly long time, we've pushed the view that "lambda expressions are just regular functions that don't know their own name". Making it so that lambdas can close over class attributes breaks that equivalence, and if we were to use the cell based approach I suggest above, the seams would be visible in the case where a lambda expression references an attribute that gets rebound after class creation: >>> C.f() 1 >>> C.x = 2 >>> C.f() # With a cell based approach, this wouldn't change 1 All of those potential complexities are specific to lambda functions though - since the implicit functions created for lambda expressions and generator expressions are called and then immediately thrown away, they don't need a persistent cell reference to the closed over variable name. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 27 March 2018 at 10:56, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes - but that would be the intention of the code beign written as in your example - class C: x = 1 f = staticmethod(lambda: print(x)) While, the classic behavior can be attained by doing: class C: x = 1 f = classmethod(lambda cls: print(cls.x)) And the behavior in both cases if one of no-surprises for me. For coders who don't have the mechanism of class creation in their mind, that could come as a surprise, but it is better than being inconsistent. TL;DR: +1 for your approach - I am just saying your perceived drawback is not one IMHO.
On 27 March 2018 at 15:32, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
I wouldn't describe myself as "having the mechanism of class creation in my mind", but I'm not 100% sure that's a necessary criterion here. In an ideal world, Python's semantics is supposed to be intuitive, which means that understanding subtle details shouldn't be necessary to anticipate the behaviour of certain constructs. Looking at the following definitions: class C: x = 1 y1 = x y2 = (lambda: x)() def meth1(self): return self.x meth2 = lambda self: self.x @staticmethod def sm1(): return x sm2 = staticmethod(lambda: x) @classmethod def cm1(cls): return cls.x cm2 = classmethod(lambda cls: cls.x) I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect sm2 to fail for the same reason - namely that sm1 has no access to the class or an instance of it, so it should not be able to reference x. And so we should get NameError. These to me don't imply any sort of "understanding of how classes are created" - they simply need an understanding of how methods (normal, static and class) get access to the class/instance. They also require that you *don't* expect a class statement to create a scope that can be closed over. I didn't really think about that before I started analysing this code, but once I did, I realised that I've never expected that. So my reaction to a bare nonlocal variable reference in a method (whether defined in a def statement or as a lambda) would be "wait, what would that mean? I guess it's the global". I wouldn't even be looking at the x defined in the class at that point. The definition of y2 follows that rule as well - even though the fact that it's not defining a method, but it's "just" a bare lambda, slightly muddies the water. The odd one out is y1. I actually can't quite explain the logic that allows y1 to refer to the value of x, even though it's the most "natural" case. As I said, I don't think of a class as defining a new scope, rather I think of it as bundling together a set of statements that will be run to populate the class (maybe that's "having the mechanism of class creation in my mind"?). So I guess y1 = x is just normal statement sequencing. So I guess I'm saying that I don't really see a problem with the current behaviour here. Classes *don't* create a scope. So you don't close over class variables. For further emphasis of this point of view: @staticmethod def sm1(): nonlocal x return x gives nonlocal x ^ SyntaxError: no binding for nonlocal 'x' found which again I can understand as "there's no outer scope containing x". Having said all this, I guess you could say that I'm too close to the current behaviour to see its deficiencies. Maybe that's true. But I don't actually *use* any of this on a regular basis. I worked out all of the above based on intuition from how I understood Python's class model, plus a few small experiments that mostly just confirmed what I expected. If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means complicating the behaviour to the point where my mental model needs to involve injecting arguments into nested scopes, I'm a strong -1.If there's a need to "fix" comprehensions, then I'd much rather that we do so in a way that doesn't change the current behaviour or execution model. Specifically, I'd actually much *rather* that these 3 cases continue giving NameError. The comprehension cases: y3 = [x+i for i in (0,1,2)] #1 y4 = [1+i for i in (x, x+1)] #2 (#1 fails with NameError, and #2 works) strike me as odd corner cases that are a bit odd, but are not worth disrupting the normal cases for. And they should be viewed as oddities in how comprehensions look names up, and not as peculiarities of class scope. Sorry - I intended that to be an "it's all pretty simple and obvious" comment, but it turned into a rather long post. But I do think the basic idea remains simple. Paul [1] The other cases don't need to refer to a bare x, they already have perfectly good ways of accessing the class variable x.
Well, there is an idiom to "keep everything as is", and work around the current limitations: class A: def b(): x = 1 d = [i + x for i in range(2)] return locals() locals().update(b()) del b Maybe if we could find a syntactic sugar for this idiom (with an abuse of the `with` keyword, for example), people could be happy, with class bodies working by default as they are now, and enabling the reuse of defined members by using the special syntax - class A: with class: x = 1 d = [i + x for in range(2)] On 27 March 2018 at 12:27, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 March 2018 at 16:51, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
I'd actually like to see some real world use cases to get a feel for whether this is even worth worrying about. (I'm not saying it isn't, just that it's hard to get a feel for the importance based on artificial examples). BTW, for an alternative workaround that avoids nested scopes altogether:
No, I'm not suggesting that's clearer - but it does (at least in my mind) make it more obvious that the problem is with comprehensions, not with class scopes. Paul
On Tue, Mar 27, 2018 at 9:52 AM, Paul Moore <p.f.moore@gmail.com> wrote:
The only reason I brought it up was because I ran into this about three weeks ago porting some old Python 2 to 3, where there was a class level comprehension that referenced a class variable on the lhs. I simply removed it by enumerating the cases by hand, no big deal, but it did take me a while to figure out why that no longer worked. My specific case looked approximately like this: class Plugin: plugin_dir = 'somepath' plugin_names = [os.join(plugin_dir, name) for name in ('list', 'of', 'names')]
On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope issues for comprehensions and genexprs. There may still be breakage, when the code defines a global x that is overridden by a class-level x, and a class-level comprehension references x assuming it to be the global. So we need to tread carefully even here -- but this case is weird already: x = 42 class C: x = [1, 2, 3] z = [x+y for y in x] # [43, 44, 45] -- --Guido van Rossum (python.org/~guido)
On Tue, Mar 27, 2018 at 11:43 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Assuming you're concerned about leaking names out of the comprehension into the class scope, that shouldn't be a problem. (The solution actually involves leaking names *into* the comprehension scope, but I'm not sure that should be called "leaking". :-) I do notice that we probably can't easily solve this using the existing closure machinery ("cells") because if we were to have cells in the class dict, it would confuse everything else that looks in the class namespace. -- --Guido van Rossum (python.org/~guido)
On 27 March 2018 at 19:43, Ethan Furman <ethan@stoneleaf.us> wrote:
To me, that would be the ideal. I assume there are significant technical challenges, though, as otherwise I'd have thought that would have been the approach taken when Python 3 fixed the name leaking issue from Python 2. Paul
On 28 March 2018 at 04:47, Paul Moore <p.f.moore@gmail.com> wrote:
It isn't avoiding the names leaking that's particularly challenging (there are several viable ways to do that), it's making sure that inner scopes can still see them. For example: lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)] for lazy, eager in lazy_and_eager: print(lazy(), eager()) # prints: 2 0 2 1 2 2 While it's *technically* feasible to hide "i" from the outer scope without hiding it from inner scopes with a variable renaming based approach, you end up having to design and document a separate lexical scoping mechanism that's distinct from the one that functions already use. I really didn't want to do that, so I proposed just using the existing scoping mechanism instead, and while that has definitely had its quirks, I'm still happy enough with it a decade later to call it a successful approach. Cheers, Nick. P.S. This is also a contributing factor to one of the aspects of the sublocals proposal: disallowing closing over them means that a renaming based approach *can* be used to keep them separate from regular local variable names. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 March 2018 at 16:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm really not sure what point you're trying to make here - are you saying that this is good or bad? Correct or incorrect? I don't really have any intuition about what's going on here, so I'd just have to work it out in terms of the defined scoping rules. And I'd then tell whoever wrote it to rewrite it more clearly ;-) Maybe a real-life case where this was important would clarify what counts as intuitive here - but as it stands,I don't really care. I don't even care that much about compatibility. Unless someone were to come along and demonstrate a serious breakage in their code, I think it's perfectly OK to change the behaviour in this situation, if that's what you're suggesting (with suitable deprecation, of course).
I do think the current implementation is a pretty good compromise. I'd be reasonably OK with not changing anything in this area. But this discussion was prompted by some of the debates around statement local variables, so "not changing anything" includes, in my mind, "not trying to make statement local variables work" as they interact badly with the current scoping behaviour in this area. Paul
On 30 March 2018 at 02:53, Paul Moore <p.f.moore@gmail.com> wrote:
In this context, it's neither good nor bad, it just is :)
And I'd then tell whoever wrote it to rewrite it more clearly ;-)
Aye, we get to do that in a code review, but the compiler doesn't - it has to figure out how to make it do something at least reasonably defensible.
Maybe a real-life case where this was important would clarify what counts as intuitive here - but as it stands,I don't really care.
The fact that deep nesting of lexical scopes within an expression is almost always going to be an unreadable mess in practice is one of the reasons I don't think there's a strong case for either backwards compatibility breaks *or* significant increases in the overall semantic complexity. Any sensible coding style is already going to say "Don't do that, it's too hard to read", so we're mainly caring about it at all based on our own senses of engineering aesthetics and a general dislike of implementation details leaking through as user visible semantic differences.
It's specifically lexical closures that lead to things getting weird, and that's a key rationale for PEP 572's current recommendation that statement locals be completely invisible to nested scopes. Trying to have a subscope that isn't visible to the rest of the function it's in, while still being visible to nested scopes defined within that subscope, seems to lead to sufficient conflicts in reference lifecycle and visibility that just using a real nested scope instead ends up being preferable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 30 March 2018 at 06:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
OK, cool. That I can completely agree with :-)
OK. I can accept your point that lexical closure weirdness is the reason for PEP 572 preferring to make statement locals ignore nested scopes. I'm not sure I fully agree, but I can accept it. My point is that having a construct that introduces new names but makes them invisible to nested scopes is *also* weird enough to be a problem. As a result, I think that PEP 572 faces a problem in that it's hit a problem for which it doesn't have a good solution. Whether that problem is bad enough to kill the PEP is a matter of judgement. In my view: * The benefits of the PEP are marginal * We haven't yet established a suitably readable syntax (IMO, := is probably the best suggestion so far, but it still has its problems) * I remain unconvinced that the PEP improves readability of comprehensions * Plus the above scoping issue So for me, the balance is in favour of keeping the status quo. If the scoping issue could be resolved, the remaining issues are much more obviously matters of taste and my view on the PEP would move to "not worth the effort" rather than against. Paul PS I should also add that the "just make them assignments" resolution to PEP 572's scope issues is the only suggestion so far that *doesn't* have scoping weirdnesses of its own (because it uses existing scoping mechanisms). The problem with that one is that it's prone to leaking in ways that I think people are uncomfortable with ("I think" because I don't have time right now to review the examples given to see for myself how compelling they are).
On 27 March 2018 at 19:43, Ethan Furman <ethan@stoneleaf.us> wrote:
If you mean this [i for i in range(5)] i # NameError then yes, this is possible. Serhiy outlined the implementation few moths ago. The rough idea is to use automatic re-naming. The only problem with this is that if someone will step into debugger one will see a name like <comp>.0.i instead of i. But this can be solved in the debuggers. -- Ivan
On Tue, Mar 27, 2018 at 11:51 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Oh, sorry, I misread what you were talking about. You're proposing going back to the Python 2 shared namespace. I'm not at all excited about that, and I'm not convinced that adjusting debuggers to hide the name mangling is effective -- it does nothing about other forms of introspection. Also depending on how PEP 572 falls there may be assignments in there. Plus there may be object lifetime consequences. -- --Guido van Rossum (python.org/~guido)
Wow! I had to try it myself! If I had came across something like the following in a code review : x = [1, 2] class C: x = [3, 4, 5] z = [x for _ in x] I would have expected C.z to equal either `[[1, 2], [1, 2]]` or `[[3, 4, 5], [3, 4, 5], [3, 4, 5]]`, but surely not `[[1, 2], [1, 2], [1, 2]]`! Is that intentional, or the result of other way-more-logical decisions? - Brice
On 28 March 2018 at 03:19, Guido van Rossum <guido@python.org> wrote:
Yeah, the "immediately call the function and throw it away" aspect makes them much easier to deal with, and they're also the case that currently feels weird to me (since the nested function is *supposed* to be a hidden implementation detail, but it's existence currently leaks through here). I don't have that niggle with lambdas, since they're visibly defining a new nested scope, so having them behave like method definitions doesn't feel strange.
Hmm, potentially more concerning might be cases where methods are currently ignored, but in a future Python start shadowing builtins for class level comprehensions and genexps. We're getting to the level of "rare case" (methods shadowing builtins) intersecting with "rare case" (using a comprehension or genexp at class scope) there though, so it should be feasible to handle through a regular deprecation cycle. So I'd guess this would need a future import ("from __future__ import implicit_scopes", perhaps? I'm struggling to come up with a good name, since it's essentially "implicit comprehension and generator expression nested scopes, version two"), and an associated deprecation warning for free variable references that would start resolving differently. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nevertheless, treating lambdas differently from "def functions" in this context alone can be complicated. Even if not the most intuitive thing on writing code, everywhere else lambda s can't even be distinguished from "regular functions" once created. And I have created a lot of "one line methods" as lambdas over the years. (that is a lot of code that would be broke with no reason). Even calling "type" with a couple "lambda" expressionsin the namespace parameter is something very useful, and might be crippled if lambda's can't be methods at all (although that is a different context, I don't think things should be changed at this point) But there are two different things about lambdas in class bodies: - enabling a lambda to "see" the outer class scope: that is ok, as it would just make things simpler, and won't break existing code, as this ability did not exist before. - preventing a lambda to become a method though its `__get__`: this breakes stuff. As it is not incompatible with a lambda that can "see" the class scope, I see no reason for doing that (code might not work if a lambda with cell classes pointing to the class body, but my feeling is that it would just work correctly if such cells are permitted to start with) js -><-
On Sun, Mar 25, 2018 at 5:51 PM Guido van Rossum <guido@python.org> wrote:
I would expect classes to behave like comprehensions - i.e. code that is not delayed will see the class' scope as an enclosing scope (since it is not namespaced yet so there's no other way to refer to it) and delayed code (e.g. lambdas or defs) will not. Elazar
On 26 March 2018 at 01:50, Guido van Rossum <guido@python.org> wrote:
I think it's mainly in comprehensions that folks may find the current behaviour surprising, as that's the case where we define a function and immediately call it, so it mostly looks like regular inline code execution. Once explicitly delayed invocation is involved (lambdas, def statements), folks seem to be more comfortable with the idea "Oh, of course that's going to behave like a method at class scope". Generator expressions also mostly get a pass, since the *iteration* is delayed, even though the generator-iterator itself is created immediately. One possibility that has occurred to me (but I've never investigated to check the technical feasibility) is to see whether we might be able to define a notion of "implied arguments" for the implicit comprehension and generator expression function definitions. The gist of that idea would be: - when walking the symbol table for a comprehension or genexp, keep track of every variable name mentioned in that subtree which is not resolved in that subtree (which we already have the machinery to do, as it's needed for compiling functions in general) - treat all those names as *implied arguments* to the implicit function, and call it with the right references in the right positions While we'd likely still want to keep evaluation of the outermost iterator outside the nested scope for the sake of better genexp tracebacks, that would still be enough to enable cases like: class C: y = 1 values = [x+y for x in range(10)] The trick would be to make it so that that comprehension gets expanded as: def _listcomp(_outermost_iter, y): result = [] for x in _outermost_iter: result.append(x+y) return result _listcomp_result = _listcomp(range(10), y) One of the more attractive aspects of this possibility (assuming it can be made to work) is that it isn't even a special case in the runtime name lookup rules - it's just changing the definition of the implicit functions that we create for comprehensions and generator expressions to make sure that they can refer to *any* name that resolves in the namespace where they're called (even invisible-to-the-compiler names like those injected via __prepare__ methods, or assignments via locals() at class scope). Cheers, Nick. P.S. I suspect we'd also find that this served as a performance optimisation, since it wouldn't only prebind class locals, it would prebind references to builtins and module globals as well. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Mar 26, 2018 at 6:33 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Honestly that sounds way too complex. In my ideal world, only functions defined with 'def' are methods whose __get__ auto-binds the first argument (unless @classmethod or @staticmethod is present), and do not get to "see" the class scope. This is how you define methods after all. Lambdas used at class scope do not get this benefit, and they see the class scope as just another closure; ditto for comprehensions and genexprs. I don't think that the "is the call delayed" idea is the proper way to distinguish the two cases here. IMO the key concept is "is it used to define a method". I betcha if you see a lambda here it's because there's some calculation going on whose result will be stored as a class variable. -- --Guido van Rossum (python.org/~guido)
On 27 March 2018 at 01:11, Guido van Rossum <guido@python.org> wrote:
Aye, I don't disagree with that. The implicit functions used in the comprehension & generator expression cases are just potentially simpler to handle, as we don't care about their API signatures, which means we can freely pollute their APIs with eager name bindings if we choose to do so. Lambda expressions are different, since their public API matters, so we can't mess with it freely to pass in extra name references. What we potentially *could* do though is allow implicit additions to their __closure__ attributes in a way that isn't permitted for regular function definitions, such that in the following code: class C: x = 1 f = staticmethod(lambda: print(x)) we would do the following things differently for lambdas (vs def functions): 1. We'd pass "x" down as a candidate for nonlocal name resolution while compiling the lambda 2. If "x" did get referenced from a lambda, we'd start allowing non-optimised code objects to have a list of cell vars (the way functions already do), and define a new "STORE_CLASSDEREF" opcode (as the counterpart to LOAD_CLASSDEREF) What STORE_CLASSDEREF would need to do is write updates both to the current locals namespace (so the expectations of a class body are met) *and* to the nominated cell object (so any references from lambda expressions are updated). The biggest downside I'd see to this is that for an incredibly long time, we've pushed the view that "lambda expressions are just regular functions that don't know their own name". Making it so that lambdas can close over class attributes breaks that equivalence, and if we were to use the cell based approach I suggest above, the seams would be visible in the case where a lambda expression references an attribute that gets rebound after class creation: >>> C.f() 1 >>> C.x = 2 >>> C.f() # With a cell based approach, this wouldn't change 1 All of those potential complexities are specific to lambda functions though - since the implicit functions created for lambda expressions and generator expressions are called and then immediately thrown away, they don't need a persistent cell reference to the closed over variable name. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 27 March 2018 at 10:56, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes - but that would be the intention of the code beign written as in your example - class C: x = 1 f = staticmethod(lambda: print(x)) While, the classic behavior can be attained by doing: class C: x = 1 f = classmethod(lambda cls: print(cls.x)) And the behavior in both cases if one of no-surprises for me. For coders who don't have the mechanism of class creation in their mind, that could come as a surprise, but it is better than being inconsistent. TL;DR: +1 for your approach - I am just saying your perceived drawback is not one IMHO.
On 27 March 2018 at 15:32, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
I wouldn't describe myself as "having the mechanism of class creation in my mind", but I'm not 100% sure that's a necessary criterion here. In an ideal world, Python's semantics is supposed to be intuitive, which means that understanding subtle details shouldn't be necessary to anticipate the behaviour of certain constructs. Looking at the following definitions: class C: x = 1 y1 = x y2 = (lambda: x)() def meth1(self): return self.x meth2 = lambda self: self.x @staticmethod def sm1(): return x sm2 = staticmethod(lambda: x) @classmethod def cm1(cls): return cls.x cm2 = classmethod(lambda cls: cls.x) I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect sm2 to fail for the same reason - namely that sm1 has no access to the class or an instance of it, so it should not be able to reference x. And so we should get NameError. These to me don't imply any sort of "understanding of how classes are created" - they simply need an understanding of how methods (normal, static and class) get access to the class/instance. They also require that you *don't* expect a class statement to create a scope that can be closed over. I didn't really think about that before I started analysing this code, but once I did, I realised that I've never expected that. So my reaction to a bare nonlocal variable reference in a method (whether defined in a def statement or as a lambda) would be "wait, what would that mean? I guess it's the global". I wouldn't even be looking at the x defined in the class at that point. The definition of y2 follows that rule as well - even though the fact that it's not defining a method, but it's "just" a bare lambda, slightly muddies the water. The odd one out is y1. I actually can't quite explain the logic that allows y1 to refer to the value of x, even though it's the most "natural" case. As I said, I don't think of a class as defining a new scope, rather I think of it as bundling together a set of statements that will be run to populate the class (maybe that's "having the mechanism of class creation in my mind"?). So I guess y1 = x is just normal statement sequencing. So I guess I'm saying that I don't really see a problem with the current behaviour here. Classes *don't* create a scope. So you don't close over class variables. For further emphasis of this point of view: @staticmethod def sm1(): nonlocal x return x gives nonlocal x ^ SyntaxError: no binding for nonlocal 'x' found which again I can understand as "there's no outer scope containing x". Having said all this, I guess you could say that I'm too close to the current behaviour to see its deficiencies. Maybe that's true. But I don't actually *use* any of this on a regular basis. I worked out all of the above based on intuition from how I understood Python's class model, plus a few small experiments that mostly just confirmed what I expected. If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means complicating the behaviour to the point where my mental model needs to involve injecting arguments into nested scopes, I'm a strong -1.If there's a need to "fix" comprehensions, then I'd much rather that we do so in a way that doesn't change the current behaviour or execution model. Specifically, I'd actually much *rather* that these 3 cases continue giving NameError. The comprehension cases: y3 = [x+i for i in (0,1,2)] #1 y4 = [1+i for i in (x, x+1)] #2 (#1 fails with NameError, and #2 works) strike me as odd corner cases that are a bit odd, but are not worth disrupting the normal cases for. And they should be viewed as oddities in how comprehensions look names up, and not as peculiarities of class scope. Sorry - I intended that to be an "it's all pretty simple and obvious" comment, but it turned into a rather long post. But I do think the basic idea remains simple. Paul [1] The other cases don't need to refer to a bare x, they already have perfectly good ways of accessing the class variable x.
Well, there is an idiom to "keep everything as is", and work around the current limitations: class A: def b(): x = 1 d = [i + x for i in range(2)] return locals() locals().update(b()) del b Maybe if we could find a syntactic sugar for this idiom (with an abuse of the `with` keyword, for example), people could be happy, with class bodies working by default as they are now, and enabling the reuse of defined members by using the special syntax - class A: with class: x = 1 d = [i + x for in range(2)] On 27 March 2018 at 12:27, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 March 2018 at 16:51, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
I'd actually like to see some real world use cases to get a feel for whether this is even worth worrying about. (I'm not saying it isn't, just that it's hard to get a feel for the importance based on artificial examples). BTW, for an alternative workaround that avoids nested scopes altogether:
No, I'm not suggesting that's clearer - but it does (at least in my mind) make it more obvious that the problem is with comprehensions, not with class scopes. Paul
On Tue, Mar 27, 2018 at 9:52 AM, Paul Moore <p.f.moore@gmail.com> wrote:
The only reason I brought it up was because I ran into this about three weeks ago porting some old Python 2 to 3, where there was a class level comprehension that referenced a class variable on the lhs. I simply removed it by enumerating the cases by hand, no big deal, but it did take me a while to figure out why that no longer worked. My specific case looked approximately like this: class Plugin: plugin_dir = 'somepath' plugin_names = [os.join(plugin_dir, name) for name in ('list', 'of', 'names')]
On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope issues for comprehensions and genexprs. There may still be breakage, when the code defines a global x that is overridden by a class-level x, and a class-level comprehension references x assuming it to be the global. So we need to tread carefully even here -- but this case is weird already: x = 42 class C: x = [1, 2, 3] z = [x+y for y in x] # [43, 44, 45] -- --Guido van Rossum (python.org/~guido)
On Tue, Mar 27, 2018 at 11:43 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Assuming you're concerned about leaking names out of the comprehension into the class scope, that shouldn't be a problem. (The solution actually involves leaking names *into* the comprehension scope, but I'm not sure that should be called "leaking". :-) I do notice that we probably can't easily solve this using the existing closure machinery ("cells") because if we were to have cells in the class dict, it would confuse everything else that looks in the class namespace. -- --Guido van Rossum (python.org/~guido)
On 27 March 2018 at 19:43, Ethan Furman <ethan@stoneleaf.us> wrote:
To me, that would be the ideal. I assume there are significant technical challenges, though, as otherwise I'd have thought that would have been the approach taken when Python 3 fixed the name leaking issue from Python 2. Paul
On 28 March 2018 at 04:47, Paul Moore <p.f.moore@gmail.com> wrote:
It isn't avoiding the names leaking that's particularly challenging (there are several viable ways to do that), it's making sure that inner scopes can still see them. For example: lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)] for lazy, eager in lazy_and_eager: print(lazy(), eager()) # prints: 2 0 2 1 2 2 While it's *technically* feasible to hide "i" from the outer scope without hiding it from inner scopes with a variable renaming based approach, you end up having to design and document a separate lexical scoping mechanism that's distinct from the one that functions already use. I really didn't want to do that, so I proposed just using the existing scoping mechanism instead, and while that has definitely had its quirks, I'm still happy enough with it a decade later to call it a successful approach. Cheers, Nick. P.S. This is also a contributing factor to one of the aspects of the sublocals proposal: disallowing closing over them means that a renaming based approach *can* be used to keep them separate from regular local variable names. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 March 2018 at 16:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm really not sure what point you're trying to make here - are you saying that this is good or bad? Correct or incorrect? I don't really have any intuition about what's going on here, so I'd just have to work it out in terms of the defined scoping rules. And I'd then tell whoever wrote it to rewrite it more clearly ;-) Maybe a real-life case where this was important would clarify what counts as intuitive here - but as it stands,I don't really care. I don't even care that much about compatibility. Unless someone were to come along and demonstrate a serious breakage in their code, I think it's perfectly OK to change the behaviour in this situation, if that's what you're suggesting (with suitable deprecation, of course).
I do think the current implementation is a pretty good compromise. I'd be reasonably OK with not changing anything in this area. But this discussion was prompted by some of the debates around statement local variables, so "not changing anything" includes, in my mind, "not trying to make statement local variables work" as they interact badly with the current scoping behaviour in this area. Paul
On 30 March 2018 at 02:53, Paul Moore <p.f.moore@gmail.com> wrote:
In this context, it's neither good nor bad, it just is :)
And I'd then tell whoever wrote it to rewrite it more clearly ;-)
Aye, we get to do that in a code review, but the compiler doesn't - it has to figure out how to make it do something at least reasonably defensible.
Maybe a real-life case where this was important would clarify what counts as intuitive here - but as it stands,I don't really care.
The fact that deep nesting of lexical scopes within an expression is almost always going to be an unreadable mess in practice is one of the reasons I don't think there's a strong case for either backwards compatibility breaks *or* significant increases in the overall semantic complexity. Any sensible coding style is already going to say "Don't do that, it's too hard to read", so we're mainly caring about it at all based on our own senses of engineering aesthetics and a general dislike of implementation details leaking through as user visible semantic differences.
It's specifically lexical closures that lead to things getting weird, and that's a key rationale for PEP 572's current recommendation that statement locals be completely invisible to nested scopes. Trying to have a subscope that isn't visible to the rest of the function it's in, while still being visible to nested scopes defined within that subscope, seems to lead to sufficient conflicts in reference lifecycle and visibility that just using a real nested scope instead ends up being preferable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 30 March 2018 at 06:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
OK, cool. That I can completely agree with :-)
OK. I can accept your point that lexical closure weirdness is the reason for PEP 572 preferring to make statement locals ignore nested scopes. I'm not sure I fully agree, but I can accept it. My point is that having a construct that introduces new names but makes them invisible to nested scopes is *also* weird enough to be a problem. As a result, I think that PEP 572 faces a problem in that it's hit a problem for which it doesn't have a good solution. Whether that problem is bad enough to kill the PEP is a matter of judgement. In my view: * The benefits of the PEP are marginal * We haven't yet established a suitably readable syntax (IMO, := is probably the best suggestion so far, but it still has its problems) * I remain unconvinced that the PEP improves readability of comprehensions * Plus the above scoping issue So for me, the balance is in favour of keeping the status quo. If the scoping issue could be resolved, the remaining issues are much more obviously matters of taste and my view on the PEP would move to "not worth the effort" rather than against. Paul PS I should also add that the "just make them assignments" resolution to PEP 572's scope issues is the only suggestion so far that *doesn't* have scoping weirdnesses of its own (because it uses existing scoping mechanisms). The problem with that one is that it's prone to leaking in ways that I think people are uncomfortable with ("I think" because I don't have time right now to review the examples given to see for myself how compelling they are).
On 27 March 2018 at 19:43, Ethan Furman <ethan@stoneleaf.us> wrote:
If you mean this [i for i in range(5)] i # NameError then yes, this is possible. Serhiy outlined the implementation few moths ago. The rough idea is to use automatic re-naming. The only problem with this is that if someone will step into debugger one will see a name like <comp>.0.i instead of i. But this can be solved in the debuggers. -- Ivan
On Tue, Mar 27, 2018 at 11:51 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Oh, sorry, I misread what you were talking about. You're proposing going back to the Python 2 shared namespace. I'm not at all excited about that, and I'm not convinced that adjusting debuggers to hide the name mangling is effective -- it does nothing about other forms of introspection. Also depending on how PEP 572 falls there may be assignments in there. Plus there may be object lifetime consequences. -- --Guido van Rossum (python.org/~guido)
Wow! I had to try it myself! If I had came across something like the following in a code review : x = [1, 2] class C: x = [3, 4, 5] z = [x for _ in x] I would have expected C.z to equal either `[[1, 2], [1, 2]]` or `[[3, 4, 5], [3, 4, 5], [3, 4, 5]]`, but surely not `[[1, 2], [1, 2], [1, 2]]`! Is that intentional, or the result of other way-more-logical decisions? - Brice
On 28 March 2018 at 03:19, Guido van Rossum <guido@python.org> wrote:
Yeah, the "immediately call the function and throw it away" aspect makes them much easier to deal with, and they're also the case that currently feels weird to me (since the nested function is *supposed* to be a hidden implementation detail, but it's existence currently leaks through here). I don't have that niggle with lambdas, since they're visibly defining a new nested scope, so having them behave like method definitions doesn't feel strange.
Hmm, potentially more concerning might be cases where methods are currently ignored, but in a future Python start shadowing builtins for class level comprehensions and genexps. We're getting to the level of "rare case" (methods shadowing builtins) intersecting with "rare case" (using a comprehension or genexp at class scope) there though, so it should be feasible to handle through a regular deprecation cycle. So I'd guess this would need a future import ("from __future__ import implicit_scopes", perhaps? I'm struggling to come up with a good name, since it's essentially "implicit comprehension and generator expression nested scopes, version two"), and an associated deprecation warning for free variable references that would start resolving differently. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nevertheless, treating lambdas differently from "def functions" in this context alone can be complicated. Even if not the most intuitive thing on writing code, everywhere else lambda s can't even be distinguished from "regular functions" once created. And I have created a lot of "one line methods" as lambdas over the years. (that is a lot of code that would be broke with no reason). Even calling "type" with a couple "lambda" expressionsin the namespace parameter is something very useful, and might be crippled if lambda's can't be methods at all (although that is a different context, I don't think things should be changed at this point) But there are two different things about lambdas in class bodies: - enabling a lambda to "see" the outer class scope: that is ok, as it would just make things simpler, and won't break existing code, as this ability did not exist before. - preventing a lambda to become a method though its `__get__`: this breakes stuff. As it is not incompatible with a lambda that can "see" the class scope, I see no reason for doing that (code might not work if a lambda with cell classes pointing to the class body, but my feeling is that it would just work correctly if such cells are permitted to start with) js -><-
participants (10)
-
Brice Parent -
Elazar -
Eric Fahlgren -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
Ivan Levkivskyi -
Joao S. O. Bueno -
Nick Coghlan -
Paul Moore