[Python-Dev] [PEP 558] thinking through locals() semantics
Nathaniel Smith
njs at pobox.com
Mon May 27 11:15:01 EDT 2019
First, I want to say: I'm very happy with PEP 558's changes to
f_locals. It solves the weird threading bugs, and exposes the
fundamental operations you need for debugging in a simple and clean
way, while leaving a lot of implementation flexibility for future
Python VMs. It's a huge improvement over what we had before.
I'm not as sure about the locals() parts of the proposal. It might be
fine, but there are some complex trade-offs here that I'm still trying
to wrap my head around. The rest of this document is me thinking out
loud to try to clarify these issues.
##### What are we trying to solve?
There are two major questions, which are somewhat distinct:
- What should the behavior of locals() be in CPython?
- How much of that should be part of the language definition, vs
CPython implementation details?
The status quo is that for locals() inside function scope, the
behavior is quite complex and subtle, and it's entirely implementation
defined. In the current PEP draft, there are some small changes to the
semantics, and also it promotes them becoming part of the official
language semantics.
I think the first question, about semantics, is the more important
one. If we're promoting them to the language definition, the main
effect is just to make it more important we get the semantics right.
##### What are the PEP's proposed semantics for locals()?
They're kinda subtle. [Nick: please double-check this section, both
for errors and because I think it includes some edge cases that the
PEP currently doesn't mention.]
For module/class scopes, locals() has always returned a mapping object
which acts as a "source of truth" for the actual local environment –
mutating the environment directly changes the mapping object, and
vice-versa. That's not going to change.
In function scopes, things are more complicated. The *local
environment* is conceptually well-defined, and includes:
- local variables (current source of truth: "fast locals" array)
- closed-over variables (current source of truth: cell objects)
- any arbitrary key/values written to frame.f_locals that don't
correspond to local or closed-over variables, e.g. you can do
frame.f_locals[object()] = 10, and then later read it out again.
However, the mapping returned by locals() does not directly reflect
this local environment. Instead, each function frame has a dict
associated with it. locals() returns this dict. The dict always holds
any non-local/non-closed-over variables, and also, in certain
circumstances, we write a snapshot of local and closed-over variables
back into the dict.
Specifically, we write back:
- Whenever locals() is called
- Whenever exec() or eval() is called without passing an explicit
locals argument
- After every trace/profile event, if a Python-level tracing/profiling
function is registered.
(Note: in CPython, the use of Python-level tracing/profiling functions
is extremely rare. It's more common in alternative implementations
like PyPy. For example, the coverage package uses a C-level tracing
function on CPython, which does not trigger locals updates, but on
PyPy it uses a Python-level tracing function, which does trigger
updates.)
In addition, the PEP doesn't say, but I think that any writes to
f_locals immediately update both the environment and the locals dict.
These semantics have some surprising consequences. Most obviously, in
function scope (unlike other scopes), mutating locals() does not
affect the actual local environment:
def f():
a = 1
locals()["a"] = 2
assert a == 1
The writeback rules can also produce surprising results:
def f():
loc1 = locals()
# Since it's a snapshot created at the time of the call
# to locals(), it doesn't contain 'loc1':
assert "loc1" not in loc1
loc2 = locals()
# Now loc1 has changed:
assert "loc1" in loc1
However, the results here are totally different if a Python-level
tracing/profiling function is installed – in particular, the first
assertion fails.
The interaction between f_locals and and locals() is also subtle:
def f():
a = 1
loc = locals()
assert "loc" not in loc
# Regular variable updates don't affect 'loc'
a = 2
assert loc["a"] == 1
# But debugging updates do:
sys._getframe().f_locals["a"] = 3
assert a == 3
assert loc["a"] == 3
# But it's not a full writeback
assert "loc" not in loc
# Mutating 'loc' doesn't affect f_locals:
loc["a"] = 1
assert sys._getframe().f_locals["a"] == 1
# Except when it does:
loc["b"] = 3
assert sys._getframe().f_locals["b"] == 3
Again, the results here are totally different if a Python-level
tracing/profiling function is installed.
And you can also hit these subtleties via 'exec' and 'eval':
def f():
a = 1
loc = locals()
assert "loc" not in loc
# exec() triggers writeback, and then mutates the locals dict
exec("a = 2; b = 3")
# So now the current environment has been reflected into 'loc'
assert "loc" in loc
# Also loc["a"] has been changed to reflect the exec'ed assignments
assert loc["a"] == 2
# But if we look at the actual environment, directly or via
# f_locals, we can see that 'a' has not changed:
assert a == 1
assert sys._getframe().f_locals["a"] == 1
# loc["b"] changed as well:
assert loc["b"] == 3
# And this *does* show up in f_locals:
assert sys._getframe().f_locals["b"] == 3
Of course, many of these edge cases are pretty obscure, so it's not
clear how much they matter. But I think we can at least agree that
this isn't the one obvious way to do it :-).
##### What's the landscape of possible semantics?
I did some brainstorming, and came up with 4 sets of semantics that
seem plausible enough to at least consider:
- [PEP]: the semantics in the current PEP draft.
- [PEP-minus-tracing]: same as [PEP], except dropping the writeback on
Python-level trace/profile events.
- [snapshot]: in function scope, each call to locals() returns a new,
*static* snapshot of the local environment, removing all this
writeback stuff. Something like:
def locals():
frame = get_caller_frame()
if is_function_scope(frame):
# make a point-in-time copy of the "live" proxy object
return dict(frame.f_locals)
else:
# in module/class scope, return the actual local environment
return frame.f_locals
- [proxy]: Simply return the .f_locals object, so in all contexts
locals() returns a live mutable view of the actual environment:
def locals():
return get_caller_frame().f_locals
##### How to evaluate our options?
I can think of a lot of criteria that all-else-being-equal we would
like Python to meet. (Of course, in practice they conflict.)
Consistency across APIs: it's surprising if locals() and
frame.f_locals do different things. This argues for [proxy].
Consistency across contexts: it's surprising if locals() has acts
differently in module/class scope versus function scope. This argues
for [proxy].
Consistent behavior when the environment shifts, or small maintenance
changes are made: it's nice if code that works today keeps working
tomorrow. On this criterion, I think [snapshot] > [proxy] >
[PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because
a very rare environmental change that almost no-one tests and mostly
only happens when debugging (i.e., enabling tracing) causes a radical
change in semantics.
Simplicity of explaining to users: all else being equal, it's nice if
our docs are short and clear and the language fits in your head. On
this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] >
[PEP]. As evidence that the current behavior is confusing, see:
- Ned gets confused and writes a long blog post after he figures it
out: https://nedbatchelder.com/blog/201211/tricky_locals.html
- A linter that warns against mutating locals():
https://lgtm.com/rules/10030096/
Simplicity of implementation: "If the implementation is easy to
explain, it may be a good idea." Since we need the proxy code anyway
to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one
'if' statement) > ([PEP] = [PEP-minus-tracing]).
Impact on other interpreter implementations: all else being equal,
we'd like to give new interpreters maximal freedom to do clever
things. (And local variables are a place where language VMs often
expend a lot of cleverness.) [proxy] and [snapshot] are both easily
implemented in terms of f_locals, so they basically don't constrain
alternative implementations at all. I'm not as sure about [PEP] and
[PEP-minus-tracing]. I originally thought they must be horrible. On
further thought, I'm not convinced they're *that* bad, since the need
to support people doing silly stuff like frame.f_locals[object()] = 10
means that implementations will already need to sometimes attach
something like a dict object to their function frames. But perhaps
alternative implementations would like to disallow this, or are OK
with making it really slow but care about locals() performance more.
Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] =
[PEP-minus-tracing]), but I'm not sure whether the '>' is large or
small.
Backwards compatibility: help(locals) says:
NOTE: Whether or not updates to this dictionary will affect
name lookups in the local scope and vice-versa is
*implementation dependent* and not covered by any backwards
compatibility guarantees.
So that claims that there are ~no backwards compatibility issues here.
I'm going to ignore that; no matter what the docs say, we still don't
want to break everyone's code. And unfortunately, I can't think of any
realistic way to do a gradual transition, with like deprecation
warnings and all that (can anyone else?), so whatever we do will be a
flag-day change.
Of our four options, [PEP] is intuitively the closest to what CPython
has traditionally done. But what exactly breaks under the different
approaches? I'll split this off into its own section.
##### Backwards compatibility
I'll split this into three parts: code that treats locals() as
read-only, exec()/eval(), and code that mutates locals().
I believe (but haven't checked) that the majority of uses of locals()
are in simple cases like:
def f():
....
print("{a} {b}".format(**locals()))
Luckily, this code remains totally fine under all four of our candidates.
exec() and eval() are an interesting case. In Python 2, exec'ing some
assignments actually *did* mutate the local environment, e.g. you
could do:
# Python 2
def f():
exec "a = 1"
assert a == 1
In Python 3, this was changed, so now exec() inside a function cannot
mutate the enclosing scope. We got some bug reports about this change,
and there are a number of questions on stackoverflow about it, e.g.:
- https://bugs.python.org/issue4831
- https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-variable-in-a-exec-function
- https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-variable-python
In all released versions of Python, eval() was syntactically unable to
rebind variables, so eval()'s interaction with the local environment
was undefined. However, in 3.8, eval() *will* be able to rebind
variables using the ':=' operator, so this interaction will become
user-visible. Presumably we'll want eval() to match exec().
OK, with that background out of the way, let's look at our candidates.
If we adopt [proxy], then that will mean exec()-inside-functions will
go back to the Python 2 behavior, where executing assignments in the
enclosing scope actually changes the enclosing scope. This will likely
break some code out there that's relying on the Python 3 behavior,
though I don't know how common that is. (I'm guessing not too common?
Using the same variable inside and outside an 'exec' and trusting that
they *won't* be the same seems like an unusual thing to do. But I
don't know.)
With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With
[snapshot], there's technically a small difference: if you call
locals() and then exec(), the exec() no longer triggers an implicit
writeback to the dict that locals() returned. I think we can ignore
this, and say that for all three of these, exec() is unlikely to
produce backwards compatibility issues.
OK, finally, let's talk about code that calls locals() and then
mutates the return value. The main difference between our candidates
is how they handle mutation, so this seems like the most important
case to focus on.
Conveniently, Mark Shannon and friends have statically analyzed a
large corpus of Python code and made a list of cases where people do
this: https://lgtm.com/rules/10030096/alerts/
Thanks! I haven't gone through the whole list, but I read through the
first few in the hopes of getting a better sense of what kind of code
does this in the real world and how it would be impacted by our
different options.
https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d00c7d6ace85341d9/files/xarray/plot/plot.py?sort=name&dir=ASC&mode=heatmap#L701
Current: raises if a Python-level trace/profile function is set
[PEP]: raises if a Python-level trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: always raises
Comment: uses locals() to capture a bunch of passed in kwargs so it
can pass them as **kwargs to another function, and treats them like a
snapshot. The authors were clearly aware of the dangers, because this
pattern appears multiple times in this file, and all the other places
make an explicit copy of locals() before using it, but this place
apparently got missed. Fix is trivial: just do that here too.
https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api.mustache
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: ok
Comment: this is inside a tool used to generate Python wrappers for
REST APIs. The vast majority of entries in the lgtm database are from
code generated by this tool. This was tricky to analyze, because it's
complex templated code, and it does mutate locals(). But I'm pretty
confident that the mutations end up just... not mattering, and all
possible generated code works under all of our candidates. Which is
lucky, because if we did have to fix this it wouldn't be trivial:
fixing up the code generator itself wouldn't be too hard, but it'll
take years for everyone to regenerate their old wrappers.
https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e3f20fd297d8ba2006/files/salt/utils/thin.py?sort=name&dir=ASC&mode=heatmap#L258
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: raises
[proxy]: ok
Comment: the use of locals() here is totally superfluous – it
repeatedly reads and writes to locals()[mod], and in all cases this
could be replaced by a simple variable, or any other dict. And it's
careful not to assign to any name that matches an actual local
variable, so it works fine with [proxy] too. But it does assume that
multiple calls to locals() return the same dict, so [snapshot] breaks
it. In this case the fix is trivial: just use a variable.
https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040a4d35f2cf2b3506e6e3d7cdf77db/files/pymatgen/ext/jhu.py?sort=name&dir=ASC&mode=heatmap#L52
Current: buggy if a trace/profile function is set
[PEP]: buggy if a trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: raises
Comment: Another example of collecting kwargs into a dict. Actually
this code is always buggy, because they seem to think that
dict.pop("a", "b") removes both "a" and "b" from the dict... but it
would raise an exception on [proxy], because they use one of those
local variables after popping it from locals(). Fix is trivial: take
an explicit snapshot of locals() before modifying it.
##### Conclusion so far
[PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or
better on all criteria, and actually *more* compatible with all the
legacy code I looked at, even though it's technically less consistent
with what CPython used to do. Unless someone points out some new
argument, I think we can reject the writeback-when-tracing part of the
PEP draft.
Choosing between the remaining three is more of a judgement call.
I'm leaning towards saying that on net, [snapshot] beats
[PEP-minus-tracing]: it's dramatically simpler, and the backwards
incompatibilities that we've found so far seem pretty minor, on par
with what we do in every point release. (In fact, in 3/4 of the cases
I looked at, [snapshot] is actually what users seemed to trying to use
in the first place.)
For [proxy] versus [snapshot], a lot depends on what we think of
changing the semantics of exec(). [proxy] is definitely more
consistent and elegant, and if we could go back in time I think it's
what we'd have done from the start. Its compatibility is maybe a bit
worse than [snapshot] on non-exec() cases, but this seems pretty minor
overall (it often doesn't matter, and if it does just write
dict(locals()) instead of locals(), like you would in non-function
scope). But the change in exec() semantics is an actual language
change, even though it may not affect much real code, so that's what
stands out for me.
I'd very much like to hear about any considerations I've missed, and
any opinions on the "judgement call" part.
--
Nathaniel J. Smith -- https://vorpus.org
More information about the Python-Dev
mailing list