On Thu, 26 Aug 2021 at 18:29, Nick Coghlan <ncoghlan@gmail.com> wrote:
[Guido wrote]
> > PS. The mapping from varname to position should be on the code object, not on the frame. This is how Mark does it (though his implementation would need to be extended to take cells into account).
>
> It's taking cells into account that forces the lookup mapping to be on
> the frame: different executions of the same code object may reference
> different cell objects.
Technically, if you design the fast refs mapping so that even cell
references have to do an indirection through the fast locals array,
then you can put the name-to-fast-locals-offset mapping on the code
object. Mine doesn't do that though, it references the cells directly
instead.
I don't think it makes much difference in practice though, and I don't
like the idea of storing a lazily initialised cache on nominally
immutable code objects.
Only the part visible to the user is immutable. The specializing interpreter stores all sorts of internal mutable data on there -- and at least since 3.9 we've had the inline cache stored there. So as long as this mapping is invisible to the user, please let's put it on the code object -- we have a lot more frame objects than code objects, so saving a pointer on the frame object and adding it to the code object is advantageous. The cost of the extra indirection is irrelevant, this is always going to be a slow interface meant for occasional use in a debugger.
OK, that makes sense - I'll add a todo note to the implementation PR.
For the main topic of PEP 667 (reducing the cache consistency question to purely a matter of PyEval_GetLocals() compatibility), I think I can see a way to make the extra code complexity of the 5 new custom accessory types (iterator, reversed iterator, keys set, values multi set, items set) worthwhile to my mind: write the latter 3 in terms of the first two and the generic mapping API, and expose them for use in other custom mapping implementations (either directly in the types module, or as optional C accelerators for the collections module). (I wouldn't make exposing them part of the PEP, I'd just aim to write them so only the forward and reverse iterators were specific to proxy objects)
With that done, popitem() is trivial to rewrite to depend on the iterator code instead of the cache.
That would leave len() and value comparison as the only proxy operations that don't adhere to the expected algorithmic complexity of mapping objects, and writing up the comparison with PEP 667 finally convinced me that those are quirks that API users could manage more easily than 558's current lazy caching semantics.
I'd still keep the full value cache under the hood, though, as I don't see enough benefit in getting rid of it when the cost is an unnecessary API compatibility break.
However, the PEP 667 write up and your own points *have* persuaded me that the extra C code needed to offer assuredly consistent views through all the proxy mapping methods is worthwhile.
That will then leave the proposed C APIs as the only differences between the PEPs - the Python level behaviour will be aligned with Mark's proposal.
Cheers,
Nick.
P.S. I don't want to rely on Python code anywhere in the fast locals proxy implementation, as that could cause weird interactions if a trace hook is enabled during the initial start up of the interpreter and tries to trace the proxy implementation code.