Standard way to get caller name and easy call stack access
Seems like the only way to get caller name in Python is to use inspect module [1] It seems that this way is not recommended (I wonder why?) and there is no other. The stackoverflow question about how to get callers method namd [1] got 3k views over 1 year, so the recipe seems to be useful. I've developed a similar snippet [2] that also extracts a module and class name in addition to method. To me this snippet is very handy to insert in print statements to see who uses a certain function at run-time. The output from this function is: spyderlib.widgets.externalshell.introspection.NotificationThread.run Which no function of `inspect` and `traceback` modules can provide (correct me if I wrong, but I am pretty confident). The stuff they propose will look like: File "/mount/sdb5/projects/spyderlib/spyderlib/widgets/externalshell/introspection.py", line 183, in run write_packet(self.notify_socket, output) As a 'software engineer' I am more interested to get logical structure of call stack rather than physical location of code text pieces (which is important too, but secondary). Neither `traceback` nor `inspect` seems to be designed with this use case, and if you'll look closely - the traceback above misses class info completely. To get the name of the class from this traceback you need to know its location withing the lines of the file. So, I'd propose to include caller_name() function somewhere and name it somehow. But there is more to that. Maybe too late for Python 3, but still an idea - make call stack a normal Python feature (optional for stackless, but still a feature) and describe a standard for call stack names, to provide everybody a simple and safe way to see what's going on with execution stack (and an easy way to get caller names). Making it a feature means that there will be almost no overhead when looking up the name (id) of the caller callers. This will also be handy for logging. I wish I could shorten this proposal. The picture in my head is something like: __main__ spyderlib.widgets.externalshell.introspection.NotificationThread.run spyderlib.utils.bsdsocket.write_packet Every indented string is an unique id of function or other closure that makes up Python code. Would it rock? This will probably require describing all possible ways to run Python code to see where a call to caller_name() can appear to cover all possible situations. Is there already a list of these ways? 1. http://stackoverflow.com/questions/2654113/python-how-to-get-the-callers-met... 2. https://gist.github.com/2151727 -- anatoly t.
anatoly techtonik schrieb am Do, 22. Mär 2012, um 00:03:47 +0300:
Using introspection in production code is not recommended in general, with some exceptions, since the main purpose of introspection is debugging. *If* you want to do this at all, the recommended way is to use the inspect module. (And there is another way -- at least in CPython you can use 'sys._getframe().f_back.f_code.co_name'.)
The missing link here seems to be that code objects don't allow access to PEP 3155 qualified names -- if they did, this would be rather easy to do. How about adding a `co_qualname` attribute to code objects? It could point to the same string the function object points to, so the overhead would be minimal.
So, I'd propose to include caller_name() function somewhere and name it somehow.
I don't think this would be necessary if the qualified name could be accessed from the frame object in some way.
But there is more to that. Maybe too late for Python 3, but still an idea - make call stack a normal Python feature
Not sure what you are talking about here. Cheers, Sven
On Thu, Mar 22, 2012 at 3:17 AM, Sven Marnach <sven@marnach.net> wrote:
If sys._getframe() is not available in Jython, IronPython and PyPy then I won't use that for debug mode output in libraries. The mechanism with `inspect` is complicated, incomplete - seems like it was not designed with object oriented concepts in mind. Look at the comments in the gist I posted before - after several iterations I still don't know how to distinguish between function name and static method call. I suspect there is no such info at all, because the API level of `inspect` objects is too low level. Other proof for that is that gist is impossible to understand without thoroughly reading the documentation. The inspect module was added around the time new style object appeared, and debugging was concentrated around the byte code compilation. Nobody thought about big applications where all these frames, line numbers, stack space and indexes of last attempted instruction in bytecode don't make any sense. Writing helpers for debugging call stacks could be easy.
Thanks for PEP 3155. I read it, but completely forgot the word "qualified". Despite the discussion inside that "full name" is a bad choice - this is the first thing that comes to mind at the prompt of search engine. Adding co_qualname is an easy low-level solution, but looking back I don't want anybody to went through the path of getting through all the gory details about python frames, filtering all the confusing and irrelevant pieces just to get to call stack. So, the argument - that this part of Python should be simplified, because beautiful is better than ugly. More arguments at the end of the letter. There is also technical argument against using existing technique of frame access, which is explained in a big note here http://docs.python.org/library/inspect.html#the-interpreter-stack
For example you need to explain a Python beginner what a Stackless is. If you have an abstraction, you just do:
Not an abstraction - people can actually play with it, and then you say - "This is how CPython works. In Stackless there is no call stack - there is a queue": __main__ module.something -------------------------------- ^^^ we were here module2.we_are_here module.something __main__ So, the ability to operate the "call stack" concept is that I call the language feature. If people can see it, touch it and play with it - you can expect to see a lot of cool things appear about it. I could talk about perspectives of exploratory/incremental programming, and run-time rollbacks, but I better stop right here. -- anatoly t.
Anatoly, Trying to find out your calling function is hard because well-designed decoupled code with properly separated concerns *should never care who is calling it*, so improving the APIs for doing so is an *extremely low priority task*. There are some truly exceptional cases where it matters. For those, the onus is on the developers *implementing those exceptional cases* to figure out how to do this in a platform independent way (which may involve conditional code that does different things depending on the platform). These legitimate use cases are simply too rare for it to be worthwhile for the various implementations to agree on a standard way to expose the functionality and promise never to change it. If all you are trying to do is print some debugging info and you can use Python 3.2, then use the "stack_info=True" argument to the logging APIs. That's what it's for. If you can't use Python 3.2, then look at the way logging does it in 3.2 and see if you can adapt that for your own needs. There's no compelling reason that writing helpers for debugging call stacks needs to be easy. It should definitely be *possible*, but 99.99% of all Python programmers are just going to use the native debugging facilities of the interpreter, or use those of an IDE that someone else wrote. Once again, you're painting with a broad sweeping brush "ah, it's horrible, it's too hard to do anything worthwhile" (despite the obvious contradiction that many projects already implement such things quite effectively) instead of suggesting *small*, *incremental* changes that might actually improve the status quo. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan schrieb am Do, 22. Mär 2012, um 18:01:31 +1000:
To be fair, Anatoly *did* suggest an incremental change -- adding the function caller_name(). And I think this raises a good point: It is outright impossible now to access the qualified name of a calling function. Adding an 'co_qualname' attribute to code objects would facilitate this. Another point Anatoly raised is that the mere function names mentioned in tracebacks often are less useful than fully qualified names. So how about including the qualified names in the traceback? Cheers, Sven
On Thu, Mar 22, 2012 at 11:01 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
You're saying that somebody specifically designed this part to be hard, because those who program properly should never request the name of calling function. Is that what you're wanted to say? Correct me, but I think you were trying to say that nobody is interested in improving the way to get caller function, because people should not write the code that requests it. There are two counter arguments: 1. De facto existence of the code in standard library that is used a lot and that makes Python programs slow - logging 2. Practicality that beats purity. Debugging and introspection capabilities of the language are as important for development and maintenance as writing well-designed decoupled code with properly separated concerns (which mostly exists in ideal world without time, readability and performance constraints). For people who study Python applications programming that goes beyond function reference most of the time is spent in reverse coding - debugging and troubleshooting, and accessing a call stack without unnecessary implications makes a big difference in understanding how application (and Python) works.
Is there an objective statistics how much a typical application accesses call stack information directly or indirectly? I have a Python checkout with its test suite - is there a way to patch test suite to count call stack accesses? As I am not a C expert - I can't immediately see any obvious problems with caching call stack information with qualified names on any platform. Straightforward idea is to annotate bytecode in a separate memory page during compilation.
I could respond with "In the face of ambiguity, refuse the temptation to guess.", but you're right - my original intention was to get the name of the caller for writing debug info into a file (without logging module though as seemed to be a reason for deadlock I was tying to debug). But when I saw how hard is it to understand Python call stack from the code, how complicated is to get to it (and how much time I spent to get what I need), and it the end information about Python caller is _still incomplete_ - I thought about other implications that this have on Python development as a language in a broader sense. There is a clojure-py project, Lisp written in Python, RPython and many other things, but understanding of code differences of these languages is obscured by the fact that the basic level in Python (on top of which all these things are written) is too low level. Python became a platform, and it would be nice when a platform is transparent and offers convenient debug capabilities (I tend to do Python development on Fedora, because it provides nice Python debugging commands in gdb).
Debugging facilities of the interpreter.. Huh. inspect.stack()[2][0].f_locals['self'].__class__.__name__ - is that the thing you keep in your head when breaking into Python console to get a name of a class for a calling method? And what should be invoked if a caller is not a method? I find it convenient to quickly insert print statements (or probes) at the points of interest to explore run-time behavior at these points instead of inserting awkward logging imports and formatting calls all over the place. I also don't know which IDE can better help with debugging multi-threaded application in Python with Qt bindings. So I just want to have native debugging capabilities of the interpreter to be convenient and actually useful.
Well, I proposed to have a caller_name() method. If it is not too universal, then a sys.call_stack() could be a better try: sys.call_stack() - Get current call stack. Return the list with a list of qualified names starting with the oldest. Rationale: make logging calls and module based filters zero cost. -- anatoly t.
On Tue, Mar 27, 2012 at 9:02 PM, anatoly techtonik <techtonik@gmail.com> wrote:
No, I'm saying the existence of frame objects and a call stack in the first place is inherently an implementation detail and hence it is impossible to provide full access to it in an implementation independent fashion. We don't want to make it part of the language spec, because we want to give implementors freedom to break those APIs and still claim compliance with the Python language spec (many implementors try to provide those APIs *anyway*, but they're well within their rights not to do so).
No, I am merely saying that such code is necessarily implementation dependent. As far as I am aware, the reason printing tracebacks (either directly or via logging) slows PyPy down is *because it has to make sure the stack exists to be printed*, thus forcing PyPy to turn off a bunch of optimisations. Similarly, IronPython by default doesn't create a full frame stack - you have to specify additional options to request creation of either full or partial frames during execution in order to use modules that rely on those features.
The call stack is necessarily tightly coupled to the implementation, and not all implementations will be easily able to introspect their calling environment (since the ability to do so efficiently depends on the underlying runtime). Guido made a deliberate choice a long time ago to exclude that feature from the language requirements by including the leading underscore in sys._getframe(). That and the similar underscore on sys._current_frames() are not an accident - they're there to give implementation authors additional freedom in how they choose to implement function calls (and that includes the authors of future versions of CPython).
You are thinking far too much about a single implementation here. The problem is not implementing call stack access for CPython (or any implementation that can provide the sys._getframe() API without much additional effort). The problem is placing that constraint on all current and future implementations of Python that claim conformance with the language spec.
No, I expect people to use pdb or just read the tracebacks printed when an exception is thrown. Runtime introspection of local variables without the aid of a debugger or other tool is a seriously advanced programming technique (and will *always* be highly implementation dependent). The following two options can be very useful for exploring problematic parts of a program: import pdb; pdb.set_trace() # Break into pdb in a running program import pbd; pdb.pm() # Start pdb after an exception was thrown (very useful in conjunction with the use of "python -i" to drop into the interactive prompt instead of terminating)
As others pointed out, I did indeed miss the concrete suggestions you made. The idea of making qualname an attribute of the code object rather than just the function certainly has merit, although it would be quite an extensive patch to achieve it (as it would involve changes to several parts of the code generation and execution chain, from the AST evaluation step through to the main eval loop and the function and class object constructors). Now, while 3.3 is still in alpha, is definitely the right time for such a patch to be put forward (it would be awkward, although possibly still feasible, to make such a change in a backwards compatible way for 3.4 if the current implementation is released as is for 3.3). There are other potentially beneficial use cases for the new qualname attribute as well (e.g. the suggestion of using it in tracebacks instead of __name__ is a good idea) As far as the call_stack() or caller_name() idea goes, it would definitely be significantly less restrictive than requiring that implementations expose the full frame machinery that CPython uses. However, the other implementations (especially PyPy) would need to be consulted regarding the possible performance implications. Use cases would also need to be investigated to ensure that just the qualified name is sufficient. Tracebacks and warnings, for example, require at least the filename and line number, and other use cases may require the module name. Any such discussion really needs to be driven by the implications for Python runtimes that don't natively use CPython-style execution frames, to ensure providing a rarely used introspection API doesn't end up slowing down *all* applications on those platforms. That particular performance concern doesn't affect CPython, since the de facto API for stack introspection is the one that CPython uses internally anyway and exposing it as is to users (albeit with the "here be dragons" implementation dependence warning) is a relatively cheap exercise. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan schrieb am Tue, 27. Mar 2012, um 23:00:49 +1000:
I think that including the qualified name in tracebacks would indeed be an worthwhile improvement, and it would be nice if this could be included in 3.3. It would put __qualname__ to an excellent use. There are some open questions to adding a 'co_qualname' attribute. 1. What should be done for code objects that don't correspond to something having a __qualname__, like list comprehensions? There are two options: Using "function_name.<locals>.<listcomp>", similar to lambda functions, or simply using 'None' or the same string as 'co_name' to avoid the overhead of computing a qualified name for every code object. 2. What about module names and PEP 395 qualified module names? One option would of course be to add both of them to the code object as well. Steven D'Aprano suggested giving full access to the function objects instead. The function object cannot be referenced in the code object, though, because this would create reference cycles in CPython. It *can* be referenced in the frame object (and this way, the change would only affect implementations having stack frames in the first place). This would only partially solve the use case of including qualified names in the traceback, since it only covers functions, not modules and classes. (For classes, we can't do something like this anyway, since the class object does not yet exist while the class body code executes.) What would have to be done to push this proposal? Cheers, Sven
On Tue, Mar 27, 2012 at 12:32 PM, Sven Marnach <sven@marnach.net> wrote:
Not if you use weakrefs. And frankly, even without weakrefs, so what? Functions are typically immortal; waiting until gc to collect the oddballs isn't such a terrible cost.
If the name needs to be available, that affects all implementations, perhaps even to the point of requiring them to create pseudo-frames, or causing them to add the equivalent of an always-active trace function. So yes, there is cost.
Both the name and the namespace do, these may serve as a proxies for most uses. Or repurpose the super() magic. -jJ
Jim Jewett schrieb am Tue, 27. Mar 2012, um 16:06:51 -0400:
I didn't carefully think about this one. It's impossible to reference the function object in the code object for a different reason: there can be many function objects corresponding to the same code object. So the discussion whether the reference cycle is tolerable is moot -- it's impossible anyway. If we want access to the function objects from the stack trace, the only way to go is to add a reference to frame objects. Cheers, Sven
On Tue, Mar 27, 2012 at 10:32 AM, Sven Marnach <sven@marnach.net> wrote:
There is already an outstanding issue on co_qualname [1].
There are some open questions to adding a 'co_qualname' attribute.
The same questions stand for adding a co_func (for the function whose definition resulted in the code object) [2]. I'd rather have a co_func than a co_qualname. Then you can do code.co_func.__qualname__.
I'm all for introspection, but I think it needs to be exposed carefully. Otherwise those APIs become attractive nuisances. An implicit self-referencing name (like __function__) in a function's locals is one such example where it would be nice, but would likely get abused (a concern that Guido had)[3]. -eric [1] http://bugs.python.org/issue13672 [2] discussion starting here: http://mail.python.org/pipermail/python-ideas/2011-August/011053.html [3] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html
Eric Snow schrieb am Tue, 27. Mar 2012, um 19:24:37 -0600:
How is this supposed to work? The code object is created at compilation time, while the function object is only created when the function definition is executed. For functions nested inside another function, a new function object is created everytime the outer function runs, reusing the same code object that was created at compilation time. Cheers, Sven
anatoly techtonik wrote: [...]
I don't have an opinion on whether this is a good idea or a bad idea, but if people are going to make radical changes with respect to inspecting the call stack, I think it would be much better to get a list of the actual calling functions, not their names. Reasons: * If you have the function, you can do anything with it, including getting the name, the docstring, any annotations it may have, etc., but if all you have is the name, it isn't so easy, and may not actually be possible to unambiguously find the function object given only it's name. * Not all functions have meaningful names (think lambdas); or their name may be ambiguous (think functions produced by factory functions). -- Steven
On 3/27/12 2:33 PM, Steven D'Aprano wrote:
One problem with this is that some of the intermediate frames (at the very least, the root frame) won't be functions but rather executed code in the form of module-level code or exec'ed strings. Furthermore, in CPython at least, it is difficult to go from the code object (which is what you have available) to its owning function object (if there is one). You can get the (well, a) name from the code object, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Tue, Mar 27, 2012 at 9:14 AM, Robert Kern <robert.kern@gmail.com> wrote:
Here's a solution for the "this frame is the result of calling which function?" question: http://bugs.python.org/issue12857 I think it strikes the right balance between providing something new/useful and not making it too accessible. -eric
anatoly techtonik wrote:
sys._getframe() is available in PyPy and Jython, I'm not sure if IronPython supports it, I think it does. A simpler and more concrete suggestion might be to simply rename sys._getframe() as sys.getframe(), since all the main implementations support it anyway. Cheers, Mark.
On 22 March 2012 09:24, Mark Shannon <mark@hotpy.org> wrote:
But at least now those of us using it know they're treading on thin ice. E.g. for some reason sys._current_frames() is not available as widely as sys._getframe(). Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Thu, Mar 22, 2012 at 09:24:40AM +0000, Mark Shannon wrote:
Apparently sys._getframe is only partially supported in PyPy. It works but probably shouldn't be encouraged because it has a huge performance hit: "sys._getframe(), sys.exc_info() work, but they give performance penalty that can be huge, by disabling the JIT. Use only for specialized use cases (like a debugger) where performance does not matter. "One unobvious usecase where this is used is the logging module. Don't use logging module if you want things to be fast." https://bitbucket.org/pypy/pypy/wiki/JitFriendliness -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Mar 22, 2012, at 11:37 AM, Andrew McNabb wrote:
"One unobvious usecase where this is used is the logging module. Don't use logging module if you want things to be fast."
Similarly, the i18n use case that motivated sys._getframe() in the first place shouldn't be considered performance critical. -Barry
On Mar 22, 2012, at 12:17 AM, Sven Marnach wrote:
(And there is another way -- at least in CPython you can use 'sys._getframe().f_back.f_code.co_name'.)
Note that back in the day, the motivating factor for sys._getframe() was an observation I made about internationalizing Python programs. This boils down to a DRY argument. For example, you could do this: def here_kitty(person, pet): print(_('$person has a $pet').safe_substitute(person=person, pet=pet)) So much error-prone DRY. With sys._getframe(), a package like flufl.i18n can allow you to write this like so: def here_kitty(person, pet): print(_('$person has a $pet')) because _() can walk up to the caller's frame and pull out the required variables. Now imagine writing hundreds of translatable strings in your application, which would you prefer? It would be impossible to port flufl.i18n to implementations that don't provide an equivalent of sys._getframe(), so it would be nice if there were perhaps a standard way of spelling this. OTOH, it was recognized at the time that this was pretty specialized, and thus it was hidden in an underscore function. Cheers, -Barry
anatoly techtonik schrieb am Do, 22. Mär 2012, um 00:03:47 +0300:
Using introspection in production code is not recommended in general, with some exceptions, since the main purpose of introspection is debugging. *If* you want to do this at all, the recommended way is to use the inspect module. (And there is another way -- at least in CPython you can use 'sys._getframe().f_back.f_code.co_name'.)
The missing link here seems to be that code objects don't allow access to PEP 3155 qualified names -- if they did, this would be rather easy to do. How about adding a `co_qualname` attribute to code objects? It could point to the same string the function object points to, so the overhead would be minimal.
So, I'd propose to include caller_name() function somewhere and name it somehow.
I don't think this would be necessary if the qualified name could be accessed from the frame object in some way.
But there is more to that. Maybe too late for Python 3, but still an idea - make call stack a normal Python feature
Not sure what you are talking about here. Cheers, Sven
On Thu, Mar 22, 2012 at 3:17 AM, Sven Marnach <sven@marnach.net> wrote:
If sys._getframe() is not available in Jython, IronPython and PyPy then I won't use that for debug mode output in libraries. The mechanism with `inspect` is complicated, incomplete - seems like it was not designed with object oriented concepts in mind. Look at the comments in the gist I posted before - after several iterations I still don't know how to distinguish between function name and static method call. I suspect there is no such info at all, because the API level of `inspect` objects is too low level. Other proof for that is that gist is impossible to understand without thoroughly reading the documentation. The inspect module was added around the time new style object appeared, and debugging was concentrated around the byte code compilation. Nobody thought about big applications where all these frames, line numbers, stack space and indexes of last attempted instruction in bytecode don't make any sense. Writing helpers for debugging call stacks could be easy.
Thanks for PEP 3155. I read it, but completely forgot the word "qualified". Despite the discussion inside that "full name" is a bad choice - this is the first thing that comes to mind at the prompt of search engine. Adding co_qualname is an easy low-level solution, but looking back I don't want anybody to went through the path of getting through all the gory details about python frames, filtering all the confusing and irrelevant pieces just to get to call stack. So, the argument - that this part of Python should be simplified, because beautiful is better than ugly. More arguments at the end of the letter. There is also technical argument against using existing technique of frame access, which is explained in a big note here http://docs.python.org/library/inspect.html#the-interpreter-stack
For example you need to explain a Python beginner what a Stackless is. If you have an abstraction, you just do:
Not an abstraction - people can actually play with it, and then you say - "This is how CPython works. In Stackless there is no call stack - there is a queue": __main__ module.something -------------------------------- ^^^ we were here module2.we_are_here module.something __main__ So, the ability to operate the "call stack" concept is that I call the language feature. If people can see it, touch it and play with it - you can expect to see a lot of cool things appear about it. I could talk about perspectives of exploratory/incremental programming, and run-time rollbacks, but I better stop right here. -- anatoly t.
Anatoly, Trying to find out your calling function is hard because well-designed decoupled code with properly separated concerns *should never care who is calling it*, so improving the APIs for doing so is an *extremely low priority task*. There are some truly exceptional cases where it matters. For those, the onus is on the developers *implementing those exceptional cases* to figure out how to do this in a platform independent way (which may involve conditional code that does different things depending on the platform). These legitimate use cases are simply too rare for it to be worthwhile for the various implementations to agree on a standard way to expose the functionality and promise never to change it. If all you are trying to do is print some debugging info and you can use Python 3.2, then use the "stack_info=True" argument to the logging APIs. That's what it's for. If you can't use Python 3.2, then look at the way logging does it in 3.2 and see if you can adapt that for your own needs. There's no compelling reason that writing helpers for debugging call stacks needs to be easy. It should definitely be *possible*, but 99.99% of all Python programmers are just going to use the native debugging facilities of the interpreter, or use those of an IDE that someone else wrote. Once again, you're painting with a broad sweeping brush "ah, it's horrible, it's too hard to do anything worthwhile" (despite the obvious contradiction that many projects already implement such things quite effectively) instead of suggesting *small*, *incremental* changes that might actually improve the status quo. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan schrieb am Do, 22. Mär 2012, um 18:01:31 +1000:
To be fair, Anatoly *did* suggest an incremental change -- adding the function caller_name(). And I think this raises a good point: It is outright impossible now to access the qualified name of a calling function. Adding an 'co_qualname' attribute to code objects would facilitate this. Another point Anatoly raised is that the mere function names mentioned in tracebacks often are less useful than fully qualified names. So how about including the qualified names in the traceback? Cheers, Sven
On Thu, Mar 22, 2012 at 11:01 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
You're saying that somebody specifically designed this part to be hard, because those who program properly should never request the name of calling function. Is that what you're wanted to say? Correct me, but I think you were trying to say that nobody is interested in improving the way to get caller function, because people should not write the code that requests it. There are two counter arguments: 1. De facto existence of the code in standard library that is used a lot and that makes Python programs slow - logging 2. Practicality that beats purity. Debugging and introspection capabilities of the language are as important for development and maintenance as writing well-designed decoupled code with properly separated concerns (which mostly exists in ideal world without time, readability and performance constraints). For people who study Python applications programming that goes beyond function reference most of the time is spent in reverse coding - debugging and troubleshooting, and accessing a call stack without unnecessary implications makes a big difference in understanding how application (and Python) works.
Is there an objective statistics how much a typical application accesses call stack information directly or indirectly? I have a Python checkout with its test suite - is there a way to patch test suite to count call stack accesses? As I am not a C expert - I can't immediately see any obvious problems with caching call stack information with qualified names on any platform. Straightforward idea is to annotate bytecode in a separate memory page during compilation.
I could respond with "In the face of ambiguity, refuse the temptation to guess.", but you're right - my original intention was to get the name of the caller for writing debug info into a file (without logging module though as seemed to be a reason for deadlock I was tying to debug). But when I saw how hard is it to understand Python call stack from the code, how complicated is to get to it (and how much time I spent to get what I need), and it the end information about Python caller is _still incomplete_ - I thought about other implications that this have on Python development as a language in a broader sense. There is a clojure-py project, Lisp written in Python, RPython and many other things, but understanding of code differences of these languages is obscured by the fact that the basic level in Python (on top of which all these things are written) is too low level. Python became a platform, and it would be nice when a platform is transparent and offers convenient debug capabilities (I tend to do Python development on Fedora, because it provides nice Python debugging commands in gdb).
Debugging facilities of the interpreter.. Huh. inspect.stack()[2][0].f_locals['self'].__class__.__name__ - is that the thing you keep in your head when breaking into Python console to get a name of a class for a calling method? And what should be invoked if a caller is not a method? I find it convenient to quickly insert print statements (or probes) at the points of interest to explore run-time behavior at these points instead of inserting awkward logging imports and formatting calls all over the place. I also don't know which IDE can better help with debugging multi-threaded application in Python with Qt bindings. So I just want to have native debugging capabilities of the interpreter to be convenient and actually useful.
Well, I proposed to have a caller_name() method. If it is not too universal, then a sys.call_stack() could be a better try: sys.call_stack() - Get current call stack. Return the list with a list of qualified names starting with the oldest. Rationale: make logging calls and module based filters zero cost. -- anatoly t.
On Tue, Mar 27, 2012 at 9:02 PM, anatoly techtonik <techtonik@gmail.com> wrote:
No, I'm saying the existence of frame objects and a call stack in the first place is inherently an implementation detail and hence it is impossible to provide full access to it in an implementation independent fashion. We don't want to make it part of the language spec, because we want to give implementors freedom to break those APIs and still claim compliance with the Python language spec (many implementors try to provide those APIs *anyway*, but they're well within their rights not to do so).
No, I am merely saying that such code is necessarily implementation dependent. As far as I am aware, the reason printing tracebacks (either directly or via logging) slows PyPy down is *because it has to make sure the stack exists to be printed*, thus forcing PyPy to turn off a bunch of optimisations. Similarly, IronPython by default doesn't create a full frame stack - you have to specify additional options to request creation of either full or partial frames during execution in order to use modules that rely on those features.
The call stack is necessarily tightly coupled to the implementation, and not all implementations will be easily able to introspect their calling environment (since the ability to do so efficiently depends on the underlying runtime). Guido made a deliberate choice a long time ago to exclude that feature from the language requirements by including the leading underscore in sys._getframe(). That and the similar underscore on sys._current_frames() are not an accident - they're there to give implementation authors additional freedom in how they choose to implement function calls (and that includes the authors of future versions of CPython).
You are thinking far too much about a single implementation here. The problem is not implementing call stack access for CPython (or any implementation that can provide the sys._getframe() API without much additional effort). The problem is placing that constraint on all current and future implementations of Python that claim conformance with the language spec.
No, I expect people to use pdb or just read the tracebacks printed when an exception is thrown. Runtime introspection of local variables without the aid of a debugger or other tool is a seriously advanced programming technique (and will *always* be highly implementation dependent). The following two options can be very useful for exploring problematic parts of a program: import pdb; pdb.set_trace() # Break into pdb in a running program import pbd; pdb.pm() # Start pdb after an exception was thrown (very useful in conjunction with the use of "python -i" to drop into the interactive prompt instead of terminating)
As others pointed out, I did indeed miss the concrete suggestions you made. The idea of making qualname an attribute of the code object rather than just the function certainly has merit, although it would be quite an extensive patch to achieve it (as it would involve changes to several parts of the code generation and execution chain, from the AST evaluation step through to the main eval loop and the function and class object constructors). Now, while 3.3 is still in alpha, is definitely the right time for such a patch to be put forward (it would be awkward, although possibly still feasible, to make such a change in a backwards compatible way for 3.4 if the current implementation is released as is for 3.3). There are other potentially beneficial use cases for the new qualname attribute as well (e.g. the suggestion of using it in tracebacks instead of __name__ is a good idea) As far as the call_stack() or caller_name() idea goes, it would definitely be significantly less restrictive than requiring that implementations expose the full frame machinery that CPython uses. However, the other implementations (especially PyPy) would need to be consulted regarding the possible performance implications. Use cases would also need to be investigated to ensure that just the qualified name is sufficient. Tracebacks and warnings, for example, require at least the filename and line number, and other use cases may require the module name. Any such discussion really needs to be driven by the implications for Python runtimes that don't natively use CPython-style execution frames, to ensure providing a rarely used introspection API doesn't end up slowing down *all* applications on those platforms. That particular performance concern doesn't affect CPython, since the de facto API for stack introspection is the one that CPython uses internally anyway and exposing it as is to users (albeit with the "here be dragons" implementation dependence warning) is a relatively cheap exercise. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan schrieb am Tue, 27. Mar 2012, um 23:00:49 +1000:
I think that including the qualified name in tracebacks would indeed be an worthwhile improvement, and it would be nice if this could be included in 3.3. It would put __qualname__ to an excellent use. There are some open questions to adding a 'co_qualname' attribute. 1. What should be done for code objects that don't correspond to something having a __qualname__, like list comprehensions? There are two options: Using "function_name.<locals>.<listcomp>", similar to lambda functions, or simply using 'None' or the same string as 'co_name' to avoid the overhead of computing a qualified name for every code object. 2. What about module names and PEP 395 qualified module names? One option would of course be to add both of them to the code object as well. Steven D'Aprano suggested giving full access to the function objects instead. The function object cannot be referenced in the code object, though, because this would create reference cycles in CPython. It *can* be referenced in the frame object (and this way, the change would only affect implementations having stack frames in the first place). This would only partially solve the use case of including qualified names in the traceback, since it only covers functions, not modules and classes. (For classes, we can't do something like this anyway, since the class object does not yet exist while the class body code executes.) What would have to be done to push this proposal? Cheers, Sven
On Tue, Mar 27, 2012 at 12:32 PM, Sven Marnach <sven@marnach.net> wrote:
Not if you use weakrefs. And frankly, even without weakrefs, so what? Functions are typically immortal; waiting until gc to collect the oddballs isn't such a terrible cost.
If the name needs to be available, that affects all implementations, perhaps even to the point of requiring them to create pseudo-frames, or causing them to add the equivalent of an always-active trace function. So yes, there is cost.
Both the name and the namespace do, these may serve as a proxies for most uses. Or repurpose the super() magic. -jJ
Jim Jewett schrieb am Tue, 27. Mar 2012, um 16:06:51 -0400:
I didn't carefully think about this one. It's impossible to reference the function object in the code object for a different reason: there can be many function objects corresponding to the same code object. So the discussion whether the reference cycle is tolerable is moot -- it's impossible anyway. If we want access to the function objects from the stack trace, the only way to go is to add a reference to frame objects. Cheers, Sven
On Tue, Mar 27, 2012 at 10:32 AM, Sven Marnach <sven@marnach.net> wrote:
There is already an outstanding issue on co_qualname [1].
There are some open questions to adding a 'co_qualname' attribute.
The same questions stand for adding a co_func (for the function whose definition resulted in the code object) [2]. I'd rather have a co_func than a co_qualname. Then you can do code.co_func.__qualname__.
I'm all for introspection, but I think it needs to be exposed carefully. Otherwise those APIs become attractive nuisances. An implicit self-referencing name (like __function__) in a function's locals is one such example where it would be nice, but would likely get abused (a concern that Guido had)[3]. -eric [1] http://bugs.python.org/issue13672 [2] discussion starting here: http://mail.python.org/pipermail/python-ideas/2011-August/011053.html [3] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html
Eric Snow schrieb am Tue, 27. Mar 2012, um 19:24:37 -0600:
How is this supposed to work? The code object is created at compilation time, while the function object is only created when the function definition is executed. For functions nested inside another function, a new function object is created everytime the outer function runs, reusing the same code object that was created at compilation time. Cheers, Sven
anatoly techtonik wrote: [...]
I don't have an opinion on whether this is a good idea or a bad idea, but if people are going to make radical changes with respect to inspecting the call stack, I think it would be much better to get a list of the actual calling functions, not their names. Reasons: * If you have the function, you can do anything with it, including getting the name, the docstring, any annotations it may have, etc., but if all you have is the name, it isn't so easy, and may not actually be possible to unambiguously find the function object given only it's name. * Not all functions have meaningful names (think lambdas); or their name may be ambiguous (think functions produced by factory functions). -- Steven
On 3/27/12 2:33 PM, Steven D'Aprano wrote:
One problem with this is that some of the intermediate frames (at the very least, the root frame) won't be functions but rather executed code in the form of module-level code or exec'ed strings. Furthermore, in CPython at least, it is difficult to go from the code object (which is what you have available) to its owning function object (if there is one). You can get the (well, a) name from the code object, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Tue, Mar 27, 2012 at 9:14 AM, Robert Kern <robert.kern@gmail.com> wrote:
Here's a solution for the "this frame is the result of calling which function?" question: http://bugs.python.org/issue12857 I think it strikes the right balance between providing something new/useful and not making it too accessible. -eric
anatoly techtonik wrote:
sys._getframe() is available in PyPy and Jython, I'm not sure if IronPython supports it, I think it does. A simpler and more concrete suggestion might be to simply rename sys._getframe() as sys.getframe(), since all the main implementations support it anyway. Cheers, Mark.
On 22 March 2012 09:24, Mark Shannon <mark@hotpy.org> wrote:
But at least now those of us using it know they're treading on thin ice. E.g. for some reason sys._current_frames() is not available as widely as sys._getframe(). Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Thu, Mar 22, 2012 at 09:24:40AM +0000, Mark Shannon wrote:
Apparently sys._getframe is only partially supported in PyPy. It works but probably shouldn't be encouraged because it has a huge performance hit: "sys._getframe(), sys.exc_info() work, but they give performance penalty that can be huge, by disabling the JIT. Use only for specialized use cases (like a debugger) where performance does not matter. "One unobvious usecase where this is used is the logging module. Don't use logging module if you want things to be fast." https://bitbucket.org/pypy/pypy/wiki/JitFriendliness -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Mar 22, 2012, at 11:37 AM, Andrew McNabb wrote:
"One unobvious usecase where this is used is the logging module. Don't use logging module if you want things to be fast."
Similarly, the i18n use case that motivated sys._getframe() in the first place shouldn't be considered performance critical. -Barry
On Mar 22, 2012, at 12:17 AM, Sven Marnach wrote:
(And there is another way -- at least in CPython you can use 'sys._getframe().f_back.f_code.co_name'.)
Note that back in the day, the motivating factor for sys._getframe() was an observation I made about internationalizing Python programs. This boils down to a DRY argument. For example, you could do this: def here_kitty(person, pet): print(_('$person has a $pet').safe_substitute(person=person, pet=pet)) So much error-prone DRY. With sys._getframe(), a package like flufl.i18n can allow you to write this like so: def here_kitty(person, pet): print(_('$person has a $pet')) because _() can walk up to the caller's frame and pull out the required variables. Now imagine writing hundreds of translatable strings in your application, which would you prefer? It would be impossible to port flufl.i18n to implementations that don't provide an equivalent of sys._getframe(), so it would be nice if there were perhaps a standard way of spelling this. OTOH, it was recognized at the time that this was pretty specialized, and thus it was hidden in an underscore function. Cheers, -Barry
participants (11)
-
anatoly techtonik -
Andrew McNabb -
Barry Warsaw -
Eric Snow -
Floris Bruynooghe -
Jim Jewett -
Mark Shannon -
Nick Coghlan -
Robert Kern -
Steven D'Aprano -
Sven Marnach