Compressing excepthook output
I suggest adding an excepthook that prints out a compressed version of the stack trace. The new excepthook should be the default at least for interactive mode. The use case is this: you are using an interactive interpreter, or perhaps in eclipse's PyDev, experimenting with some code. The code happen to has an infinite recursion - maybe an erroneous boundary condition, maybe the recursion itself was an accident. You didn't catch the RuntimeError so you get a print of the traceback. This is by default a 2000 lines of highly repetitive call chain. Most likely, a single cycle repeating some 300 times. The main problem is that in most environments, by default, you have only a limited amount of lines kept in the window. So you can't just scroll up and see what was the error in the first place - where is the entry point into the cycle. You have to reproduce it, and catch RuntimeError. You can't just use printing for debugging, either, because you won't see them. And even if you can see it, you had lost much of your "history" for nothing. I have tried to implement an alternative for sys.excepthook (see below), which compresses the last simple cycle in the call graph. Turns out it's not trivial, since the traceback object is not well documented (and maybe it shouldn't be, as it is an implementation detail) so it's non trivial (if at all possible) to change the trace list in an existing traceback. I don't think it is reasonable to just send anyone interested in such a feature to implement it themselves - especially given that newcomers ate its main target - and even if we do, there is no simple way to make it a default. Such a compression will not always help, since the call graph may be arbitrarily complex, so there has to be some threshold below which there won't be any compression. this threshold should be chosen after considering the number of lines accessible by default in common environments (Linux/Windows terminals, eclipse's console, etc.). Needless to say, the output should be correct in all cases. I am not sure that my example implementation is. Another suggestion, related but distinct: `class RecursionLimitError(RuntimeError)` should be raised instead of a plain RuntimeError. One should be able to except this specific case, and "Exception messages are not part of the Python API". --- Example for the desired result (non interactive): Traceback (most recent call last): File "/workspace/compress.py", line 48, in <module> bar() File "/workspace/compress.py", line 46, in bar p() File "/workspace/compress.py", line 43, in p def p(): p0() File "/workspace/compress.py", line 41, in p0 def p0(): p2() File "/workspace/compress.py", line 39, in p2 def p2(): p() RuntimeError: maximum recursion depth exceeded 332.67 occurrences of cycle of size 3 detected Code: import traceback import sys def print_exception(name, value, count, size, newtrace): # this is ugly and fragile sys.stderr.write('Traceback (most recent call last):\n') sys.stderr.writelines(traceback.format_list(newtrace)) sys.stderr.write('{}: {}\n'.format(name ,value)) sys.stderr.write('{} occurrences of cycle of size {} detected\n'.format(count, size)) def analyze_cycles(tb): calls = set() size = 0 for i, call in enumerate(reversed(tb)): if size == 0: calls.add(call) if call == tb[-1]: size = i elif call not in calls: length = i break return size, length def cycle_detect_excepthook(exctype, value, trace): if exctype is RuntimeError: tb = traceback.extract_tb(trace) # Feels like a hack here if len(tb) >= sys.getrecursionlimit()-1: size, length = analyze_cycles(tb) count = round(length/size, 2) if count >= 2: print_exception(exctype.__name__, value, count, size, tb[:-length+size]) return sys.__excepthook__(exctype, value, tb) sys.excepthook = cycle_detect_excepthook if __name__ == '__main__': def p2(): p() def p0(): p2() def p(): p() def bar(): p() bar() ### Elazar
Better display of recursion errors sounds reasonable to me, as does giving them a dedicated subclass. Step one would be coming up with test cases and a solid implementation and display format for cyclic call detection. Once that is available, then using it in the default excepthook for the CPython REPL is a separate question. Cheers, Nick.
On Mon, Sep 16, 2013 at 01:39:46AM +0200, אלעזר wrote:
I suggest adding an excepthook that prints out a compressed version of the stack trace. The new excepthook should be the default at least for interactive mode. [...] I have tried to implement an alternative for sys.excepthook (see below), which compresses the last simple cycle in the call graph. Turns out it's not trivial, since the traceback object is not well documented (and maybe it shouldn't be, as it is an implementation detail) so it's non trivial (if at all possible) to change the trace list in an existing traceback. I don't think it is reasonable to just send anyone interested in such a feature to implement it themselves - especially given that newcomers ate its main target - and even if we do, there is no simple way to make it a default.
I like where this is going. Tracebacks for recursive function calls are extremely noisy, with the extra lines rarely giving any useful information. Have a look at the cgitb module in the standard library. I think you should start off by cleaning up your traceback handler to be less "ugly and fragile" (your words), if possible, and then consider publishing it on ActiveState's website as a Python recipe. That would be the first step in gathering user feedback and experience in the real world, and if it turns out to be useful in practice, at a later date we can look at adding it to the standard library. http://code.activestate.com/recipes/ -- Steven
On 16 September 2013 10:07, Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Sep 16, 2013 at 01:39:46AM +0200, אלעזר wrote:
I suggest adding an excepthook that prints out a compressed version of the stack trace. The new excepthook should be the default at least for interactive mode. [...] I have tried to implement an alternative for sys.excepthook (see below), which compresses the last simple cycle in the call graph. Turns out it's not trivial, since the traceback object is not well documented (and maybe it shouldn't be, as it is an implementation detail) so it's non trivial (if at all possible) to change the trace list in an existing traceback. I don't think it is reasonable to just send anyone interested in such a feature to implement it themselves - especially given that newcomers ate its main target - and even if we do, there is no simple way to make it a default.
I like where this is going. Tracebacks for recursive function calls are extremely noisy, with the extra lines rarely giving any useful information.
Have a look at the cgitb module in the standard library.
I think you should start off by cleaning up your traceback handler to be less "ugly and fragile" (your words), if possible, and then consider publishing it on ActiveState's website as a Python recipe. That would be the first step in gathering user feedback and experience in the real world, and if it turns out to be useful in practice, at a later date we can look at adding it to the standard library.
Another couple of potentially useful pointers: - the traceback.py source is a good place to get more details on how traceback objects work (http://hg.python.org/cpython/file/default/Lib/traceback.py) - you may want to try out the updated traceback extraction API proposed in http://bugs.python.org/issue17911 and see if that cleans up your code. If it helps, that would be good validation of the proposed new API, if it doesn't, it may provide hints for further improvement. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Nick Coghlan
-
Steven D'Aprano
-
אלעזר