Storing a reference to the function object (if any) inside frame objects

Hi all, Currently, given a frame object (e.g. from sys._getframe or inspect.getouterframes), there's no way to get back to the function object that created it. This creates an obstacle for various sorts of introspection. In particular, in the unusual but real situation where you need to "mark" a function in a way that can be detected via stack introspection, then the natural way to do that – attaching an attribute to the function object – doesn't work. Instead, you have to mangle the frame's locals. Example: In pytest, if you want to mark a function as uninteresting and not worth showing in tracebacks (a common situation for assertion helpers), then you can't do this with a decorator like @pytest.tracebackhide; instead, you have to set a __magic__ local variable: https://docs.pytest.org/en/latest/example/simple.html#writing-well-integrate... Example: In trio, I want to mark certain Python functions as "protected" from receiving KeyboardInterrupt exceptions – kind of like how CPython doesn't allow KeyboardInterrupt in internal functions [1]. If it were possible for my signal handler to get at function objects when it walks the stack, then the necessary decorator would be something like: def keyboard_interrupt_protected(fn): fn._trio_keyboard_interrupt_protected = True return fn Instead right now it's this mess: https://github.com/python-trio/trio/blob/64119b12309ffeaf3a35622ef08d3b03e43... And worse, the current mess has a race condition, because a KeyboardInterrupt could arrive while we're in the middle of executing the code to enable the KeyboardInterrupt protection. OTOH if this information was carried on the function object, the protection would be be automatically set up when the frame was created. Another benefit is that it would make it easy to include __qualname__'s in exception tracebacks, which is often nice. For example, here's a current traceback: File "/home/njs/trio/trio/_sync.py", line 374, in acquire_nowait return self._lock.acquire_nowait() File "/home/njs/trio/trio/_sync.py", line 277, in acquire_nowait raise WouldBlock And here's what it could look like if the traceback machinery could access frame.f_func.__qualname__: File "/home/njs/trio/trio/_sync.py", line 374, in Condition.acquire_nowait return self._lock.acquire_nowait() File "/home/njs/trio/trio/_sync.py", line 277, in Lock.acquire_nowait raise WouldBlock Here's some semi-terrifying code attempting to work around this issue: https://stackoverflow.com/questions/14817788/python-traceback-with-module-na... To be fair, this last issue could also be solved by adding a f_qualname field to frame objects, like generators and coroutines already have. But another way to think about it is that if we had the whole function object then generators and coroutines wouldn't have to track the qualname separately anymore :-). Q: But what about frames that don't have any associated function object, like those created by exec()? A: Then frame.f_func would be NULL/None, no biggie. Q: Won't this increase memory usage? A: In principle this could increase memory usage by keeping function objects alive longer than otherwise, but I think the effect should be pretty minimal. It doesn't introduce any reference loops, and the vast majority of function objects outlive their frames in any case. Specifically I think the only (?) case where it would make a difference is if you create a new function and then call it immediately without storing it in a temporary, like: (lambda: 1)(). So basically: it's a bit obscure, but I think would be a nice little addition and I don't see any major downsides. Anything I'm missing? -n [1] There's a lot more detail on this here: https://vorpus.org/blog/control-c-handling-in-python-and-trio/ but it's probably unnecessary for the current discussion. -- Nathaniel J. Smith -- https://vorpus.org

On 4 May 2017 at 18:59, Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
Currently, given a frame object (e.g. from sys._getframe or inspect.getouterframes), there's no way to get back to the function object that created it. This creates an obstacle for various sorts of introspection. In particular, in the unusual but real situation where you need to "mark" a function in a way that can be detected via stack introspection, then the natural way to do that – attaching an attribute to the function object – doesn't work. Instead, you have to mangle the frame's locals.
Eric Snow put together a mostly working patch for this quite some time ago: http://bugs.python.org/issue12857 It needs some naming adjustments to better account for generators and coroutines, and rebasing against a recent version of the tree, but I don't see any major barriers to getting the change made other than someone actually working through all the nitty gritty details of finalising the patch. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (2)
-
Nathaniel Smith
-
Nick Coghlan