On Mon, May 27, 2019 at 9:16 AM Guido van Rossum wrote:

I re-ran your examples and found that some of them fail.

On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith wrote:

[...]

...

The interaction between f_locals and and locals() is also subtle:

def f(): a = 1 loc = locals() assert "loc" not in loc # Regular variable updates don't affect 'loc' a = 2 assert loc["a"] == 1 # But debugging updates do: sys._getframe().f_locals["a"] = 3 assert a == 3

That assert fails; `a` is still 2 here for me.

I think you're running on current Python, and I'm talking about the semantics in the current PEP 558 draft, which redefines f_locals so that the assert passes. Nick has a branch here if you want to try it: https://github.com/python/cpython/pull/3640 (Though I admit I was lazy, and haven't tried running my examples at all -- they're just based on the text.)

...

assert loc["a"] == 3 # But it's not a full writeback assert "loc" not in loc # Mutating 'loc' doesn't affect f_locals: loc["a"] = 1 assert sys._getframe().f_locals["a"] == 1 # Except when it does: loc["b"] = 3 assert sys._getframe().f_locals["b"] == 3

All of this can be explained by realizing `loc is sys._getframe().f_locals`. IOW locals() always returns the dict in f_locals.

That's not true in the PEP version of things. locals() and frame.f_locals become radically different. locals() is still a dict stored in the frame object, but f_locals is a magic proxy object that reads/writes to the fast locals array directly.

...

Again, the results here are totally different if a Python-level tracing/profiling function is installed.

And you can also hit these subtleties via 'exec' and 'eval':

def f(): a = 1 loc = locals() assert "loc" not in loc # exec() triggers writeback, and then mutates the locals dict exec("a = 2; b = 3") # So now the current environment has been reflected into 'loc' assert "loc" in loc # Also loc["a"] has been changed to reflect the exec'ed assignments assert loc["a"] == 2 # But if we look at the actual environment, directly or via # f_locals, we can see that 'a' has not changed: assert a == 1 assert sys._getframe().f_locals["a"] == 1 # loc["b"] changed as well: assert loc["b"] == 3 # And this *does* show up in f_locals: assert sys._getframe().f_locals["b"] == 3

This works indeed. My understanding is that the bytecode interpreter, when accessing the value of a local variable, ignores f_locals and always uses the "fast" array. But exec() and eval() don't use fast locals, their code is always compiled as if it appears in a module-level scope.

While the interpreter is running and no debugger is active, in a function scope f_locals is not used at all, the interpreter only interacts with the fast array and the cells. It is initialized by the first locals() call for a function scope, and locals() copies the fast array and the cells into it. Subsequent calls in the same function scope keep the same value for f_locals and re-copy fast and cells into it. This also clears out deleted local variables and emptied cells, but leaves "strange" keys (like "b" in the examples) unchanged.

The truly weird case happen when Python-level tracers are present, then the contents of f_locals is written back to the fast array and cells at certain points. This is intended for use by pdb (am I the only user of pdb left in the world?), so one can step through a function and mutate local variables. I find this essential in some cases.

Right, the original goal for the PEP was to remove the "truly weird case" but keep pdb working

...

Of course, many of these edge cases are pretty obscure, so it's not clear how much they matter. But I think we can at least agree that this isn't the one obvious way to do it :-).

##### What's the landscape of possible semantics?

I did some brainstorming, and came up with 4 sets of semantics that seem plausible enough to at least consider:

- [PEP]: the semantics in the current PEP draft.

To be absolutely clear this copies the fast array and cells to f_locals when locals() is called, but never copies back, except when Python-level tracing/profiling is on.

In the PEP draft, it never copies back at all, under any circumstance.

...

- [PEP-minus-tracing]: same as [PEP], except dropping the writeback on Python-level trace/profile events.

But this still copies the fast array and cells to f_locals when a Python trace function is called, right? It just doesn't write back.

No, when I say "writeback" in this email I always mean PyFrame_FastToLocals. The PEP removes PyFrame_LocalsToFast entirely.

...

- [snapshot]: in function scope, each call to locals() returns a new, *static* snapshot of the local environment, removing all this writeback stuff. Something like:

def locals(): frame = get_caller_frame() if is_function_scope(frame): # make a point-in-time copy of the "live" proxy object return dict(frame.f_locals) else: # in module/class scope, return the actual local environment return frame.f_locals

This is the most extreme variant, and in this case there is no point in having f_locals at all for a function scope (since nothing uses it). I'm not 100% sure that you understand this.

Yes, this does suggest an optimization: you should be able to skip allocating a dict for every frame in most cases. I'm not sure how much of a difference it makes. In principle we could implement that optimization right now by delaying the dict allocation until the first time f_locals or locals() is used, but we currently don't bother. And even if we adopt this, we'll still need to keep a slot in the frame struct to allocate the dict if we need to, because people can still be obnoxious and do frame.f_locals["unique never before seen name"] = blah and expect to be able to read it back later, which means we need somewhere to store that. (In fact Trio does do this right now, as part of its control-C handling stuff, because there's literally no other place where you can store information that a signal handler can see when it's walking the stack.) We could deprecate writing new names to f_locals like this, but that's a longer-term thing.

...

- [proxy]: Simply return the .f_locals object, so in all contexts locals() returns a live mutable view of the actual environment:

def locals(): return get_caller_frame().f_locals

So this is PEP without any writeback. But there is still copying from the fast array and cells to f_locals. Does that only happen when locals() is called? Or also when a Python-level trace/profiling function is called? My problem with all variants except what's in the PEP is that it would leave pdb *no* way (short of calling into the C API using ctypes) of writing back local variables.

No, this option is called [proxy] because this is the version where locals() and f_locals *both* give you magic proxy objects where __getitem__ and __setitem__ access the fast locals array directly, as compared to the PEP where only f_locals gives you that magic object. -n -- Nathaniel J. Smith -- https://vorpus.org

Note that the weird, Action At A Distance behavior is also visible for locals() called at module scope (since there, locals() is globals(), which returns the actual dict that's the module's __dict__, i.e. the Source Of Truth. So I think it's unavoidable in general, and we would do wise not to try and "fix" it just for function locals. (And I certainly don't want to mess with globals().) This is another case for the proposed [proxy] semantics, assuming we can get over our worry about backwards incompatibility. On Mon, May 27, 2019 at 7:31 PM Steven D'Aprano wrote:

On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote: [...]

...

I'm not as sure about the locals() parts of the proposal. It might be fine, but there are some complex trade-offs here that I'm still trying to wrap my head around. The rest of this document is me thinking out loud to try to clarify these issues.

Wow. Thanks for the detail on this, I think the PEP should link to this thread, you've done some great work here.

[...]

...

In function scopes, things are more complicated. The *local environment* is conceptually well-defined, and includes: - local variables (current source of truth: "fast locals" array) - closed-over variables (current source of truth: cell objects)

I don't think closed-over variables are *local* variables. They're "nonlocal", and you need a special keyword to write to them.

...

- any arbitrary key/values written to frame.f_locals that don't correspond to local or closed-over variables, e.g. you can do frame.f_locals[object()] = 10, and then later read it out again.

Today I learned something new.

...

However, the mapping returned by locals() does not directly reflect this local environment. Instead, each function frame has a dict associated with it. locals() returns this dict. The dict always holds any non-local/non-closed-over variables, and also, in certain circumstances, we write a snapshot of local and closed-over variables back into the dict.

I'm going to try to make a case for your "snapshot" scenario.

The locals dict inside a function is rather weird:

- unlike in the global or class scope, writing to the dict does not update the variables

- writing to it is discouraged, but its not a read-only proxy

- it seems to be a snapshot of the state of variables when you called locals():

# inside a function x = 1 d = locals() x = 2 assert d['x'] == 1

- but it's not a proper, static, snapshot, because sometimes it will mutate without you touching it.

That last point is Action At A Distance, and while it is explicable ("there's only one locals dict, and calling locals() updates it") its also rather unintuitive and surprising and violates the Principle Of Least Surprise.

[Usual disclaimers about "surprising to whom?" applies.]

Unless I missed something, it doesn't seem that any of the code you (Nathan) analysed is making use of this AAAD behaviour, at least not deliberately. At least one of the examples took steps to avoid it by making an explicit copy after calling locals(), but missed one leaving that function possibly buggy.

Given how weirdly the locals dict behaves, and how tricky it is to explain all the corner cases, I'm going to +1 your "snapshot" idea:

- we keep the current behaviour for locals() in the global and class scopes;

- we keep the PEP's behaviour for writebacks when locals() or exec() (and eval with walrus operator) are called, for the frame dict;

- but we change locals() to return a copy of that dict, rather than the dict itself.

(I think I've got the details right... please correct me if I've misunderstood anything.)

Being a backwards-incompatible change, that means that folks who were relying on that automagical refresh of the snapshot will need to change their code to explicitly refresh:

# update in place: d.update(locals())

# or get a new snapshot d = locals()

Or they explicitly grab a reference to the frame dict instead of calling locals(). Either way is likely to be less surprising than the status quo and less likely to lead to accidental, unexpected updates of the local dictionary without your knowledge.

[...]

...

Of course, many of these edge cases are pretty obscure, so it's not clear how much they matter. But I think we can at least agree that this isn't the one obvious way to do it :-).

Indeed. I thought I was doing well to know that writing to locals() inside a function didn't necessarily update the variable, but I had no idea of the levels of complexity actually involved!

...

I can think of a lot of criteria that all-else-being-equal we would like Python to meet. (Of course, in practice they conflict.)

Consistency across APIs: it's surprising if locals() and frame.f_locals do different things. This argues for [proxy].

I don't think it is that surprising, since frame.f_locals is kinda obscure (the average Python coder wouldn't know a frame if one fell on them) and locals() has been documented as weird for decades.

In any case, at least "its a copy of ..." is simple and understandable.

...

Consistency across contexts: it's surprising if locals() has acts differently in module/class scope versus function scope. This argues for [proxy].

True as far as it goes, but its also true that for the longest time, in most implementations, locals() has acted differently. So no change there.

On the other hand, locals() currently returns a dict everywhere. It might be surprising for it to start returning a proxy object inside functions instead of a dict.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Tue, 28 May 2019 at 06:00, Nathaniel Smith wrote:

- There's the "justified" action-at-a-distance that currently happens at module scope, where locals().__setitem__ affects variable lookup, and variable mutation affects locals().__getitem__. This can produce surprising results if you pass locals() into something that's expecting a regular dict, but it's also arguably the point of an environment introspection API, and like you say, it's unavoidable and expected at module scope and when using globals().

If I understand you, this "justified" action at a distance is exactly similar to how os.environ works, making it very reasonable that locals and globals work the same (indeed, explaining that locals *doesn't* do this at the moment is why (IMO) people consider locals() as "a bit weird").

- And then there's the "spooky" action-at-a-distance that currently happens at function scope, where calling locals() has the side-effect of mutating the return value from previous calls to locals(), and the objects returned from locals may or may not spontaneously mutate themselves depending on whether some other code registered a trace function. This is traditional, but extremely surprising if you aren't deeply familiar with internals of CPython's implementation.

Yep, this is the one that most non-experts commenting on this thread (including me) seem to have been surprised by.

Of the four designs:

[PEP] and [PEP-minus-tracing] both have "spooky" action-at-a-distance (worse in [PEP]), but they don't have "justified" action-at-a-distance.

[proxy] adds "justified" action-at-a-distance, and removes "spooky" action at a distance.

[snapshot] gets rid of both kinds of action-at-a-distance (at least in function scope).

This summary leaves me very strongly feeling that I prefer [proxy] first, then [snapshot], with the two [PEP] variants a distant third and fourth. Of course, all of this is only if we have decided to formalise the semantics and change CPython to conform. I've never personally been affected by any of the edge cases with locals(), so on a purely personal basis, I'm equally happy with "do nothing" :-) Paul

(I'll likely write a more detailed reply once I'm back on an actual computer, but wanted to send an initial response while folks in the US are still awake, as the detailed reply may not be needed) Thanks for this write-up Nathaniel - I think you've done a good job of capturing the available design alternatives. The one implicit function locals() update case that you missed is in the accessor for "frame.f_locals", so I think dropping CPython's implicit update for all Python trace functions would be a clear win (I actually almost changed the PEP to do that once I realized it was already pretty redundant given the frame accessor behaviour). I'm OK with either [PEP-minus-tracing] or [snapshot], with a slight preference for [snapshot] (since it's easier to explain). The only design option I wouldn't be OK with is [proxy], as I think that poses a significant potential backwards compatibility problem, and I trust Armin Rigo's perspective that it would be hellish for JIT-compiled Python implementations to handle without taking the same kind of performance hit they do when they need to emulate the frame API. By contrast, true snapshot semantics will hopefully make life *easier* for JIT compilers, and folks that actually want the update() behaviour can either rely on frame.f_locals, or do an explicit update. This would also create a possible opportunity to simplify the fast locals proxy semantics: if it doesn't need to emulate the current behaviour of allowing arbitrary keys to be added and preserved between calls to locals(), then it could dispense with its internal dict cache entirely, and instead reject any operations that try to add new keys that aren't defined on the underlying code object (removing keys and then adding them back would still be permitted, and handled like a code level del statement). Cheers, Nick.

On Wed., 29 May 2019, 2:29 pm Guido van Rossum, wrote:

So why is it “hellish” for JITs if locals() returns a proxy, while frame.f_locals being a proxy is okay?

As I understand it, they already drop out of compiled mode if they detect that the code is tinkering with frame objects. Having a single locals() call de-optimize an entire function would be far from ideal. Cheers, Nick. )

Indeed. On Tue, May 28, 2019 at 11:07 PM Greg Ewing wrote:

Nick Coghlan wrote:

...

Having a single locals() call de-optimize an entire function would be far from ideal.

I don't see what would be so bad about that. The vast majority of functions have no need for locals().

-- Greg _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido (mobile)

Nick Coghlan wrote:

If there was a compelling use case for letting "a = 1; exec(src); print(a)" print something other than "1" at function scope, then I'd be more amenable to the idea of the associated compatibility break and potential performance regression in other implementations.

However, there isn't any such use case - if there were, we wouldn't have deliberately changed the semantics from the old Python 2 ones to the current Python 3 ones in PEP 3100 [1].

I get the impression that was done because everyone involved thought it wasn't worth the ugliness of maintaining all the fast/locals swapping stuff, not because of any principle that the current behaviour is right or better in any way. Given a locals proxy object, it would be much easier to support the old behaviour (which seems obvious and correct to me) without eval or exec having to be anything special. -- Greg

Nick Coghlan wrote:

So for me, getting rid of write backs via exec and "import *" was a matter of "Yay, we finally closed those unfortunate loopholes" rather than being any kind of regrettable necessity.

If that were the reasoning, the principled thing to do would be to raise an exception if an eval or exec tries to write to a local, rather than mostly ignore it. In any case, I don't really agree with that philosophy. Python is at its essence a dynamic language. Things like JIT and static type analysis are only possible to the extent that you refrain from using some of its dynamic features. Removing features entirely just because they *can* interfere with these things goes against the spirit of the language, IMO. -- Greg

Hi, On Wed, 29 May 2019 at 08:07, Greg Ewing wrote:

Nick Coghlan wrote:

...

Having a single locals() call de-optimize an entire function would be far from ideal.

I don't see what would be so bad about that. The vast majority of functions have no need for locals().

You have the occasional big function that benefits a lot from being JIT-compiled but which contains ``.format(**locals())``. That occurs in practice, and that's why PyPy is happy that there is a difference between ``locals()`` and ``sys._getframe().f_locals``. PyPy could be made to support the full mutable view, but that's extra work that isn't done so far and is a bit unlikely to occur at this point. It also raises the significantly the efforts for other JIT implementations of Python if they have to support a full-featured ``locals()``; supporting ``_getframe().f_locals`` is to some extent optional, but supporting ``locals()`` is not. A bientôt, Armin.

On Sun, Jun 02, 2019 at 11:52:02PM +1200, Greg Ewing wrote:

Armin Rigo wrote:

...

You have the occasional big function that benefits a lot from being JIT-compiled but which contains ``.format(**locals())``.

There should be a lot less need for that now that we have f-strings.

I think you're forgetting that a lot of code (especially libraries) either have to support older versions of Python, and so cannot use f-strings at all, or was written using **locals before f-strings came along, and hasn't been touched since. Another case where f-strings don't help is when the template is dynamically generated. It may be that there will be less new code written using **locals() but I don't think that the **locals() trick will disappear any time before Python 5000. -- Steven

On Wed, May 29, 2019, at 01:25, Nick Coghlan wrote:

Having a single locals() call de-optimize an entire function would be far from ideal.

What if there were a way to explicitly de-optimize a function, rather than guessing the user's intent based on looking for locals and exec calls (both of which are builtins which could be shadowed or assigned to other variables)? Also, regardless of anything else, maybe in an optimized function locals should return a read-only mapping?

On Tue, May 28, 2019 at 6:48 PM Greg Ewing wrote:

Nathaniel Smith wrote:

...

- [proxy]: Simply return the .f_locals object, so in all contexts locals() returns a live mutable view of the actual environment:

def locals(): return get_caller_frame().f_locals

Not sure I quite follow this -- as far as I can see, f_locals currently has the same snapshot behaviour as locals().

I'm assuming you mean to change things so that locals() returns a mutable view tracking the environment in both directions. That sounds like a much better idea all round to me. No weird shared-snapshot behaviour, and no need for anything to behave differently when tracing.

Yeah, I made the classic mistake and forgot that my audience isn't as immersed in this as I am :-). Throughout the email I'm assuming we're going to adopt PEP 558's proposal about replacing f_locals with a new kind of mutable view object, and then given that, asking what we should do about locals(). -n -- Nathaniel J. Smith -- https://vorpus.org