[Python-Dev] [PEP 558] thinking through locals() semantics

[Steven D'Aprano]

On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote:
[...]
> I'm not as sure about the locals() parts of the proposal. It might be
> fine, but there are some complex trade-offs here that I'm still trying
> to wrap my head around. The rest of this document is me thinking out
> loud to try to clarify these issues.

Wow. Thanks for the detail on this, I think the PEP should link to this 
thread, you've done some great work here.


[...]
> In function scopes, things are more complicated. The *local
> environment* is conceptually well-defined, and includes:
> - local variables (current source of truth: "fast locals" array)
> - closed-over variables (current source of truth: cell objects)

I don't think closed-over variables are *local* variables. They're 
"nonlocal", and you need a special keyword to write to them.


> - any arbitrary key/values written to frame.f_locals that don't
> correspond to local or closed-over variables, e.g. you can do
> frame.f_locals[object()] = 10, and then later read it out again.

Today I learned something new.


> However, the mapping returned by locals() does not directly reflect
> this local environment. Instead, each function frame has a dict
> associated with it. locals() returns this dict. The dict always holds
> any non-local/non-closed-over variables, and also, in certain
> circumstances, we write a snapshot of local and closed-over variables
> back into the dict.

I'm going to try to make a case for your "snapshot" scenario.

The locals dict inside a function is rather weird:

- unlike in the global or class scope, writing to the dict does not 
  update the variables

- writing to it is discouraged, but its not a read-only proxy

- it seems to be a snapshot of the state of variables when you 
  called locals():

      # inside a function
      x = 1
      d = locals()
      x = 2
      assert d['x'] == 1

- but it's not a proper, static, snapshot, because sometimes it
  will mutate without you touching it.

That last point is Action At A Distance, and while it is explicable 
("there's only one locals dict, and calling locals() updates it") its 
also rather unintuitive and surprising and violates the Principle Of 
Least Surprise.

[Usual disclaimers about "surprising to whom?" applies.]

Unless I missed something, it doesn't seem that any of the code you 
(Nathan) analysed is making use of this AAAD behaviour, at least not 
deliberately. At least one of the examples took steps to avoid it by 
making an explicit copy after calling locals(), but missed one leaving 
that function possibly buggy.

Given how weirdly the locals dict behaves, and how tricky it is to 
explain all the corner cases, I'm going to +1 your "snapshot" idea: 

- we keep the current behaviour for locals() in the global and class 
  scopes;

- we keep the PEP's behaviour for writebacks when locals() or exec()
  (and eval with walrus operator) are called, for the frame dict;

- but we change locals() to return a copy of that dict, rather than
  the dict itself.

(I think I've got the details right... please correct me if I've 
misunderstood anything.)

Being a backwards-incompatible change, that means that folks who were 
relying on that automagical refresh of the snapshot will need to change 
their code to explicitly refresh:

    # update in place:
    d.update(locals())

    # or get a new snapshot
    d = locals()


Or they explicitly grab a reference to the frame dict instead of 
calling locals(). Either way is likely to be less surprising than the 
status quo and less likely to lead to accidental, unexpected updates of 
the local dictionary without your knowledge.


[...]
> Of course, many of these edge cases are pretty obscure, so it's not
> clear how much they matter. But I think we can at least agree that
> this isn't the one obvious way to do it :-).

Indeed. I thought I was doing well to know that writing to locals() 
inside a function didn't necessarily update the variable, but I had no 
idea of the levels of complexity actually involved!


> I can think of a lot of criteria that all-else-being-equal we would
> like Python to meet. (Of course, in practice they conflict.)
> 
> Consistency across APIs: it's surprising if locals() and
> frame.f_locals do different things. This argues for [proxy].

I don't think it is that surprising, since frame.f_locals is kinda 
obscure (the average Python coder wouldn't know a frame if one fell 
on them) and locals() has been documented as weird for decades.

In any case, at least "its a copy of ..." is simple and understandable.


> Consistency across contexts: it's surprising if locals() has acts
> differently in module/class scope versus function scope. This argues
> for [proxy].

True as far as it goes, but its also true that for the longest time, in 
most implementations, locals() has acted differently. So no change there.

On the other hand, locals() currently returns a dict everywhere. It 
might be surprising for it to start returning a proxy object inside 
functions instead of a dict.


-- 
Steven