Unbound locals in class scopes
Hello, There appeared a question in the discussion on http://bugs.python.org/issue24129 about documenting the behavior that unbound local variables in a class definition do not follow the normal rules. Guido said 13 years ago that this behavior should not be changed: https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior.
On 06/20/2015 02:51 AM, Ivan Levkivskyi wrote:
Hello,
There appeared a question in the discussion on http://bugs.python.org/issue24129 about documenting the behavior that unbound local variables in a class definition do not follow the normal rules.
Guido said 13 years ago that this behavior should not be changed: https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior.
Guido's comment still stands as far as references inside methods work in regards to the class body. (they must use a self name to access the class name space.) But the execution of the class body does use lexical scope, otherwise it would print xtop instead of xlocal here. x = "xtop" y = "ytop" def func(): x = "xlocal" y = "ylocal" class C: print(x) print(y) y = 1 func() prints xlocal ytop Maybe a better way to put this is, should the above be the same as this?
x = "xtop" y = "ytop" def func(): ... x = "xlocal" ... y = "ylocal" ... def _C(): ... print(x) ... print(y) ... y = 1 ... return locals() ... C = type("C", (), _C()) ... func() xlocal Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 9, in func File "<stdin>", line 6, in _C UnboundLocalError: local variable 'y' referenced before assignment
I think yes, but I'm not sure how it may be different in other ways. Cheers, Ron
On 06/20/2015 12:12 PM, Ron Adam wrote:
On 06/20/2015 02:51 AM, Ivan Levkivskyi wrote:
Guido said 13 years ago that this behavior should not be changed: https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior.
Guido's comment still stands as far as references inside methods work in regards to the class body. (they must use a self name to access the class name space.) But the execution of the class body does use lexical scope, otherwise it would print xtop instead of xlocal here.
Minor corrections: Methods can access but not write to the class scope without using self. So that is also equivalent to the function version using type(). The methods capture the closure they were defined in, which is interesting. And the self name refers to the object's names space not the class name space.
On 20 June 2015 at 18:39, Ron Adam
On 06/20/2015 12:12 PM, Ron Adam wrote:
On 06/20/2015 02:51 AM, Ivan Levkivskyi wrote:
Guido said 13 years ago that this behavior should not be changed:
https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior.
Guido's comment still stands as far as references inside methods work in
regards to the class body. (they must use a self name to access the class name space.) But the execution of the class body does use lexical scope, otherwise it would print xtop instead of xlocal here.
Minor corrections:
Methods can access but not write to the class scope without using self. So that is also equivalent to the function version using type(). The methods capture the closure they were defined in, which is interesting.
And the self name refers to the object's names space not the class name space.
It is still not clear whether Guido's comment still stands for not raising an UnboundLocalError in class definitions but using globals instead. This is the only questionable point for me currently.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
On Sun, Jun 21, 2015 at 6:22 PM, Ivan Levkivskyi
It is still not clear whether Guido's comment still stands for not raising an UnboundLocalError in class definitions but using globals instead.
Can you phrase this in the form of an example, showing what it currently does and what you think it should do, instead? -- --Guido van Rossum (python.org/~guido)
On 21 June 2015 at 22:05, Guido van Rossum
On Sun, Jun 21, 2015 at 6:22 PM, Ivan Levkivskyi
wrote:
It is still not clear whether Guido's comment still stands for not
raising an UnboundLocalError in class definitions but using globals instead.
Can you phrase this in the form of an example, showing what it currently does and what you think it should do, instead?
Here is an example: x = "xtop" y = "ytop" def func(): x = "xlocal" y = "ylocal" class C: print(x) print(y) y = 1 func() prints xlocal ytop Intuitively, one might think that it should raise UnboundLocalError or print ylocal instead of ytop. This question was discussed 13 years ago and then you said that this lookup in globals is an intentional behavior. This behavior is not documented, and I started an issue on bug tracker about documenting it. Then, Eric proposed to ask you again, whether this is still an intentional behavior.
On 22 June 2015 at 08:46, Ivan Levkivskyi
On 21 June 2015 at 22:05, Guido van Rossum
wrote: On Sun, Jun 21, 2015 at 6:22 PM, Ivan Levkivskyi
wrote: It is still not clear whether Guido's comment still stands for not raising an UnboundLocalError in class definitions but using globals instead.
Can you phrase this in the form of an example, showing what it currently does and what you think it should do, instead?
Here is an example:
x = "xtop" y = "ytop" def func(): x = "xlocal" y = "ylocal" class C: print(x) print(y) y = 1 func()
prints
xlocal ytop
Intuitively, one might think that it should raise UnboundLocalError or print ylocal instead of ytop. This question was discussed 13 years ago and then you said that this lookup in globals is an intentional behavior.
This behavior is not documented, and I started an issue on bug tracker about documenting it. Then, Eric proposed to ask you again, whether this is still an intentional behavior.
Diving into some bytecode details: We added LOAD_CLASSDEREF (https://docs.python.org/3/library/dis.html#opcode-LOAD_CLASSDEREF) a while back to account for the fact that __prepare__ may inject locals into a class body that the compiler doesn't know about. Unlike LOAD_DEREF, LOAD_CLASSDEREF checks the locals before it checks the closure cell. However, neither LOAD_CLASSDEREF *nor* LOAD_DEREF is used for names that are *assigned* in the class body - those get flagged as local variables, so we end up emitting LOAD_NAME for them, and hence ignore any nonlocal variables with that name. If a module global or builtin exists, we'll find that, otherwise we'll throw NameError. With nested_scopes having been the default since 2.2, and accounting for the fact that folks *do* write code like "name = name" at class scope and expect it to "do the right thing", it seems reasonable to me to just make this work properly, rather than having to explain why it doesn't work as one might expected based on the behaviour of module level class definitions and other references to nonlocal variables from a class body. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jun 22, 2015 at 3:03 AM, Nick Coghlan
On 22 June 2015 at 08:46, Ivan Levkivskyi
wrote: On 21 June 2015 at 22:05, Guido van Rossum
wrote: On Sun, Jun 21, 2015 at 6:22 PM, Ivan Levkivskyi
wrote: It is still not clear whether Guido's comment still stands for not raising an UnboundLocalError in class definitions but using globals
Can you phrase this in the form of an example, showing what it currently does and what you think it should do, instead?
Here is an example:
x = "xtop" y = "ytop" def func(): x = "xlocal" y = "ylocal" class C: print(x) print(y) y = 1 func()
prints
xlocal ytop
Intuitively, one might think that it should raise UnboundLocalError or
instead. print
ylocal instead of ytop. This question was discussed 13 years ago and then you said that this lookup in globals is an intentional behavior.
This behavior is not documented, and I started an issue on bug tracker about documenting it. Then, Eric proposed to ask you again, whether this is still an intentional behavior.
Diving into some bytecode details:
In particular, the bytecode for C is:
dis.dis(func.__code__.co_consts[3]) 6 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 6 LOAD_CONST 0 ('func.<locals>.C') 9 STORE_NAME 2 (__qualname__)
7 12 LOAD_NAME 3 (print) 15 LOAD_CLASSDEREF 0 (x) 18 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 21 POP_TOP 8 22 LOAD_NAME 3 (print) 25 LOAD_NAME 4 (y) 28 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 31 POP_TOP 9 32 LOAD_CONST 1 (1) 35 STORE_NAME 4 (y) 38 LOAD_CONST 2 (None) 41 RETURN_VALUE
We added LOAD_CLASSDEREF (https://docs.python.org/3/library/dis.html#opcode-LOAD_CLASSDEREF) a while back to account for the fact that __prepare__ may inject locals into a class body that the compiler doesn't know about. Unlike LOAD_DEREF, LOAD_CLASSDEREF checks the locals before it checks the closure cell.
However, neither LOAD_CLASSDEREF *nor* LOAD_DEREF is used for names that are *assigned* in the class body - those get flagged as local variables, so we end up emitting LOAD_NAME for them, and hence ignore any nonlocal variables with that name. If a module global or builtin exists, we'll find that, otherwise we'll throw NameError.
With nested_scopes having been the default since 2.2, and accounting for the fact that folks *do* write code like "name = name" at class scope and expect it to "do the right thing", it seems reasonable to me to just make this work properly, rather than having to explain why it doesn't work as one might expected based on the behaviour of module level class definitions and other references to nonlocal variables from a class body.
But what *is* the correct behavior? I suspect people's intuitions differ. If you think of this as similar to function scopes you're likely to be wrong. Also, since classes inside functions are commonly used in unit tests (at least mine :-), I worry that *any* changes in this behavior might break working code (no matter how much that "working" could be considered an accident, it's still going to be a bear to debug if this happens to you). -- --Guido van Rossum (python.org/~guido)
On 22 June 2015 at 17:13, Guido van Rossum
But what *is* the correct behavior? I suspect people's intuitions differ. If you think of this as similar to function scopes you're likely to be wrong.
For me, there's a correct answer for *new* users based on the combination of: 1. For module level class definitions, if you look up a name before binding it locally, it's looked up the same way it would be if you never bound it locally at all 2. For function level class definitions, If the name isn't bound locally in the class body, nonlocals are visible as if the class body was running in the scope of the function defining the class If folks never experienced Python as it existed prior to 2.2, the current behaviour isn't likely to be at all intuitive to them (I barely make it into that category myself - prior to adopting 2.2.2 for DSP testing in 2002 or so, my sole experience with Python 1.5.2 was a single university level networking class*)
Also, since classes inside functions are commonly used in unit tests (at least mine :-), I worry that *any* changes in this behavior might break working code (no matter how much that "working" could be considered an accident, it's still going to be a bear to debug if this happens to you).
Agreed - I think any change here would need to go through a deprecation period where we warned about cases where LOAD_NAME would be changed to LOAD_CLASSDEREF and there was a nonlocal defined with that name. Since it's trivial easy to workaround by renaming a local variable to use a name that differs from the nested class attribute, I'd also be entirely happy with leaving the current behaviour as an obscure quirk - I'm only +0 on changing it, and also +0 on declaring it "not worth the hassle of changing". Cheers, Nick. P.S. *Two fun facts about that class (this was in the late 90's): a) the lecturer *required* us to use Python as he assumed there'd be very few people that already knew it, so we'd be starting off on a more level playing field; and b) I asked if I could use Java instead, since I already knew that at the time (he said no, and Python really *was* very easy to pick up for the assignments we needed to do. It made it up again an easy choice several years later) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 23 June 2015 at 05:33, Nick Coghlan
Also, since classes inside functions are commonly used in unit tests (at least mine :-), I worry that *any* changes in this behavior might break working code (no matter how much that "working" could be considered an accident, it's still going to be a bear to debug if this happens to you).
Agreed - I think any change here would need to go through a deprecation period where we warned about cases where LOAD_NAME would be changed to LOAD_CLASSDEREF and there was a nonlocal defined with that name.
Since it's trivial easy to workaround by renaming a local variable to use a name that differs from the nested class attribute, I'd also be entirely happy with leaving the current behaviour as an obscure quirk - I'm only +0 on changing it, and also +0 on declaring it "not worth the hassle of changing".
This is also my attitude on this issue. Looks like everyone agrees that this is a strange behavior, but not strange enough to be changed.
participants (4)
-
Guido van Rossum
-
Ivan Levkivskyi
-
Nick Coghlan
-
Ron Adam