Tightening up the specification for locals()
An exchange in one of the enum threads prompted me to write down something I've occasionally thought about regarding locals(): it is currently severely underspecified, and I'd like to make the current CPython behaviour part of the language/library specification. (We recently found a bug in the interaction between the __prepare__ method and lexical closures that was indirectly related to this underspecification) Specifically, rather than the current vague "post-modification of locals may not work", I would like to explicitly document the expected behaviour at module, class and function scope (as well as clearly documenting the connection between modules, classes and the single- and dual-namespace variants of exec() and eval()): * at module scope, as well as when using exec() or eval() with a single namespace, locals() must return the same thing as globals(), which must be the actual execution namespace. Subsequent execution may change the contents of the returned mapping, and changes to the returned mapping must change the execution environment. * at class scope, as well as when using exec() or eval() with separate global and local namespaces, locals() must return the specified local namespace (which may be supplied by the metaclass __prepare__ method in the case of classes). Subsequent execution may change the contents of the returned mapping, and changes to the returned mapping must change the execution environment. For classes, this mapping will not be used as the actual class namespace underlying the defined class (the class creation process will copy the contents to a fresh dictionary that is only accessible by going through the class machinery). * at function scope, locals() must return a *snapshot* of the current locals and free variables. Subsequent execution must not change the contents of the returned mapping and changes to the returned mapping must not change the execution environment. Rather than adding this low level detail to the library reference docs, I would suggest adding it to the data model section of the language reference, with a link to the appropriate section from the docs for the locals() builtin. The warning in the locals() docs would be softened to indicate that modifications won't work at function scope, but are supported at module and class scope. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/05/13 11:29, Nick Coghlan wrote:
Fixing the underspecification is good. Enshrining a limitation as the one correct way, not so good.
If we were designing the language from scratch, with no concern for optimizing function execution, would we want this as a language feature? I don't believe that there is anyone who would say: "I really want locals() to behave differently inside functions from how it behaves inside classes and the global scope, as a feature in and of itself." Obviously CPython introduces that limitation for good reason, and I don't wish to suggest that this is the wrong thing to do, but it is a trade-off, and some implementations may wish to make other trade-offs, or even find a way to avoid it altogether. E.g. IronPython and Jython both allow this:
And why not? In and of itself, writing to locals() inside a function is no worse a thing to do than writing to locals() inside a class or global scope. It's not something actively harmful that must be prohibited, so why prohibit it? I think that conforming Python implementations should be allowed a choice between two fully-specified behaviours, the choice between them being a "quality of implementation" issue: - locals() may return a read-only or frozen mapping containing a snapshot of the current locals and free variable, in which case subsequent execution must not change the contents of the returned mapping, and changing the returned mapping is not possible; - locals() may return an ordinary dict, in which case it must be the actual execution namespace, or a proxy to it. Subsequent execution will change the contents of the returned mapping, and changes to the mapping must change the execution environment. Code can determine at runtime which capability is provided by inspecting the type of the returned mapping: if isinstance(locals(), dict) then you have support for modifying the executable environment, if not, you don't. Obviously if you wish to write platform-agnostic code, you have to target the least behaviour, which would be read-only locals. But there's lots of code that runs only under Jython or IronPython, and if somebody really needs to write to locals(), they can target an implementation that provides that feature. -- Steven
On Fri, 03 May 2013 12:43:41 +1000 Steven D'Aprano <steve@pearwood.info> wrote:
I have to say, I agree with Steven here. Mutating locals() is currently an implementation detail, and it should IMHO stay that way. Only reading a non-mutated locals() should be well-defined. Regards Antoine.
On Sun, May 12, 2013 at 10:01 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
At global and class scope (and, equivalently, in exec), I strongly disagree. There, locals() is (or should be) well defined, either as identical to globals(), as the value returned from __prepare__() (and will be passed to the metaclass as the namespace). The exec case corresponds to those two instances, depending on whether the single namespace or dual namespace version is performed. What Steven was objecting to was my suggestion that CPython's current behaviour where mutating locals() may not change the local namespace be elevated to an actual requirement where mutating locals *must not* change the local namespace. He felt that was overspecifying a CPython-specific limitation, and I think he's right - at function scope, the best we can say is that modifying the result of locals() may or may not make those changes visible to other code in that function (or closures that reference the local variables in that function). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, May 12, 2013 at 11:28 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Right, the main reason for the proposal is to lock down "locals() is globals()" for module namespaces and "locals() is the namespace that was returned from __prepare__ and will be passed to the metaclass constructor" for class bodies. The change to exec merely follows because the single argument form corresponds to module execution and the two argument form to class body execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, May 12, 2013 at 2:01 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Like it or not, people rely on this behavior. I don't think CPython (or PyPy) can actually afford to change it. If so, documenting it sounds like a better idea than leaving it undocumented only known to the "inner shrine" Cheers, fijal
On 5/2/2013 9:29 PM, Nick Coghlan wrote:
Except that, apparently, subsequent execution *does* change the returned mapping when tracing in on. Some of the loose specification is intentional. http://bugs.python.org/issue7083 locals() behaviour differs when tracing is in effect -- Terry Jan Reedy
On 03/05/13 11:29, Nick Coghlan wrote:
Fixing the underspecification is good. Enshrining a limitation as the one correct way, not so good.
If we were designing the language from scratch, with no concern for optimizing function execution, would we want this as a language feature? I don't believe that there is anyone who would say: "I really want locals() to behave differently inside functions from how it behaves inside classes and the global scope, as a feature in and of itself." Obviously CPython introduces that limitation for good reason, and I don't wish to suggest that this is the wrong thing to do, but it is a trade-off, and some implementations may wish to make other trade-offs, or even find a way to avoid it altogether. E.g. IronPython and Jython both allow this:
And why not? In and of itself, writing to locals() inside a function is no worse a thing to do than writing to locals() inside a class or global scope. It's not something actively harmful that must be prohibited, so why prohibit it? I think that conforming Python implementations should be allowed a choice between two fully-specified behaviours, the choice between them being a "quality of implementation" issue: - locals() may return a read-only or frozen mapping containing a snapshot of the current locals and free variable, in which case subsequent execution must not change the contents of the returned mapping, and changing the returned mapping is not possible; - locals() may return an ordinary dict, in which case it must be the actual execution namespace, or a proxy to it. Subsequent execution will change the contents of the returned mapping, and changes to the mapping must change the execution environment. Code can determine at runtime which capability is provided by inspecting the type of the returned mapping: if isinstance(locals(), dict) then you have support for modifying the executable environment, if not, you don't. Obviously if you wish to write platform-agnostic code, you have to target the least behaviour, which would be read-only locals. But there's lots of code that runs only under Jython or IronPython, and if somebody really needs to write to locals(), they can target an implementation that provides that feature. -- Steven
On Fri, 03 May 2013 12:43:41 +1000 Steven D'Aprano <steve@pearwood.info> wrote:
I have to say, I agree with Steven here. Mutating locals() is currently an implementation detail, and it should IMHO stay that way. Only reading a non-mutated locals() should be well-defined. Regards Antoine.
On Sun, May 12, 2013 at 10:01 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
At global and class scope (and, equivalently, in exec), I strongly disagree. There, locals() is (or should be) well defined, either as identical to globals(), as the value returned from __prepare__() (and will be passed to the metaclass as the namespace). The exec case corresponds to those two instances, depending on whether the single namespace or dual namespace version is performed. What Steven was objecting to was my suggestion that CPython's current behaviour where mutating locals() may not change the local namespace be elevated to an actual requirement where mutating locals *must not* change the local namespace. He felt that was overspecifying a CPython-specific limitation, and I think he's right - at function scope, the best we can say is that modifying the result of locals() may or may not make those changes visible to other code in that function (or closures that reference the local variables in that function). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, May 12, 2013 at 11:28 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Right, the main reason for the proposal is to lock down "locals() is globals()" for module namespaces and "locals() is the namespace that was returned from __prepare__ and will be passed to the metaclass constructor" for class bodies. The change to exec merely follows because the single argument form corresponds to module execution and the two argument form to class body execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, May 12, 2013 at 2:01 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Like it or not, people rely on this behavior. I don't think CPython (or PyPy) can actually afford to change it. If so, documenting it sounds like a better idea than leaving it undocumented only known to the "inner shrine" Cheers, fijal
On 5/2/2013 9:29 PM, Nick Coghlan wrote:
Except that, apparently, subsequent execution *does* change the returned mapping when tracing in on. Some of the loose specification is intentional. http://bugs.python.org/issue7083 locals() behaviour differs when tracing is in effect -- Terry Jan Reedy
participants (7)
-
Antoine Pitrou -
Benjamin Peterson -
Fábio Santos -
Maciej Fijalkowski -
Nick Coghlan -
Steven D'Aprano -
Terry Jan Reedy