[Python-ideas] Storing a reference to the function object (if any) inside frame objects

[Nathaniel Smith]

Hi all,

Currently, given a frame object (e.g. from sys._getframe or
inspect.getouterframes), there's no way to get back to the function
object that created it. This creates an obstacle for various sorts of
introspection. In particular, in the unusual but real situation where
you need to "mark" a function in a way that can be detected via stack
introspection, then the natural way to do that – attaching an
attribute to the function object – doesn't work. Instead, you have to
mangle the frame's locals.

Example: In pytest, if you want to mark a function as uninteresting
and not worth showing in tracebacks (a common situation for assertion
helpers), then you can't do this with a decorator like
@pytest.tracebackhide; instead, you have to set a __magic__ local
variable: https://docs.pytest.org/en/latest/example/simple.html#writing-well-integrated-assertion-helpers

Example: In trio, I want to mark certain Python functions as
"protected" from receiving KeyboardInterrupt exceptions – kind of like
how CPython doesn't allow KeyboardInterrupt in internal functions [1].
If it were possible for my signal handler to get at function objects
when it walks the stack, then the necessary decorator would be
something like:

def keyboard_interrupt_protected(fn):
    fn._trio_keyboard_interrupt_protected = True
    return fn

Instead right now it's this mess:
https://github.com/python-trio/trio/blob/64119b12309ffeaf3a35622ef08d3b03e438006e/trio/_core/_ki.py#L108-L150

And worse, the current mess has a race condition, because a
KeyboardInterrupt could arrive while we're in the middle of executing
the code to enable the KeyboardInterrupt protection. OTOH if this
information was carried on the function object, the protection would
be be automatically set up when the frame was created.

Another benefit is that it would make it easy to include
__qualname__'s in exception tracebacks, which is often nice. For
example, here's a current traceback:

File "/home/njs/trio/trio/_sync.py", line 374, in acquire_nowait
  return self._lock.acquire_nowait()
File "/home/njs/trio/trio/_sync.py", line 277, in acquire_nowait
  raise WouldBlock

And here's what it could look like if the traceback machinery could
access frame.f_func.__qualname__:

File "/home/njs/trio/trio/_sync.py", line 374, in Condition.acquire_nowait
  return self._lock.acquire_nowait()
File "/home/njs/trio/trio/_sync.py", line 277, in Lock.acquire_nowait
  raise WouldBlock

Here's some semi-terrifying code attempting to work around this issue:
https://stackoverflow.com/questions/14817788/python-traceback-with-module-names

To be fair, this last issue could also be solved by adding a
f_qualname field to frame objects, like generators and coroutines
already have. But another way to think about it is that if we had the
whole function object then generators and coroutines wouldn't have to
track the qualname separately anymore :-).

Q: But what about frames that don't have any associated function
object, like those created by exec()?
A: Then frame.f_func would be NULL/None, no biggie.

Q: Won't this increase memory usage?
A: In principle this could increase memory usage by keeping function
objects alive longer than otherwise, but I think the effect should be
pretty minimal. It doesn't introduce any reference loops, and the vast
majority of function objects outlive their frames in any case.
Specifically I think the only (?) case where it would make a
difference is if you create a new function and then call it
immediately without storing it in a temporary, like: (lambda: 1)().

So basically: it's a bit obscure, but I think would be a nice little
addition and I don't see any major downsides. Anything I'm missing?

-n

[1] There's a lot more detail on this here:
https://vorpus.org/blog/control-c-handling-in-python-and-trio/
but it's probably unnecessary for the current discussion.

-- 
Nathaniel J. Smith -- https://vorpus.org