Shrink recursion error tracebacks (was: Have REPL print less by default)
This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception. Of course, the same is true for actual programs, where tracebacks drown off everything else useful. For all we know, a typo caused a NameError in one of the functions which somehow triggered it to call itself over again until Python decided it was enough, but you can't know that because your entire scrollback is full of File "<stdin>", line 1, in func And, for programs that are user-facing, I take all tracebacks and redirect them to a file that the user can submit me so I can debug the issue, and having tens of hundreds of the same line hurts readability. The issue that the output from running a certain script in the REPL and as a script would differ has been raised, and I am of the opinion that in no way, shape or form should the two have different behaviours (the exception being sys.ps1, sys.ps2 and builtins._ in interactive mode, which are 100% fine). I'm suggesting a change to how Python handles specifically recursion limits, to cut after a sane number of times (someone suggested to shrink the output after 3 times), and simply stating how many more identical messages there are. I would also like to extend this to any arbitrary loop of callers (for example foo -> bar -> baz -> foo and so on would be counted as one "call" for the purposes of this proposal). Under this proposal, something similar to this would happen: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in func File "<stdin>", line 1, in func File "<stdin>", line 1, in func [Previous 1 message(s) repeated 996 more times] RecursionError: maximum recursion depth exceeded With multiple chained calls (I don't know how hard it would be to implement this, probably not trivial): Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz [Previous 3 message(s) repeated 330 more times] RecursionError: maximum recursion depth exceeded We can probably afford to cut out immediately as we notice the recursion, but even just 3 times with multiple chained calls isn't going to be anywhere near as long as it currently is. Thoughts? Suggestions? Something I forgot? -Emanuel
On 04/19/2016 09:07 PM, Émanuel Barry wrote:
Under this proposal, something similar to this would happen:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in func File "<stdin>", line 1, in func File "<stdin>", line 1, in func [Previous 1 message(s) repeated 996 more times] RecursionError: maximum recursion depth exceeded
With multiple chained calls (I don't know how hard it would be to implement this, probably not trivial):
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz [Previous 3 message(s) repeated 330 more times] RecursionError: maximum recursion depth exceeded
Thoughts? Suggestions? Something I forgot?
I don't think you'll find anyone opposed -- someone just needs to do the work, and volunteer hours are scarce. :( -- ~Ethan~
On 4/20/2016 12:07 AM, Émanuel Barry wrote:
This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception.
For those who don't know, the Windows console uses a circular buffer of lines. The default size was once 300, I believe -- too small to contain a run of the test suite. (I seems to be less on Win 10). The max with win 10 appears to be 999 (x 4? unclear). -- Terry Jan Reedy
On Wed, Apr 20, 2016 at 2:19 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 4/20/2016 12:07 AM, Émanuel Barry wrote:
This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception.
For those who don't know, the Windows console uses a circular buffer of lines. The default size was once 300, I believe -- too small to contain a run of the test suite. (I seems to be less on Win 10). The max with win 10 appears to be 999 (x 4? unclear).
I'm guessing you pulled the 999x4 from the Options tab of the properties dialog, but that option is actually for command history (i.e. the ability to use the up arrow to recall previous commands). That has a limit of 999 commands in the buffer, with a limit of 999 (not 4, which is the default) total buffers in memory across all open CMD.EXE instances. (Not sure what happens when you open more instances than that limit is set to.) The setting for scrollback is found under the Layout tab, and is disguised as the Height option in the Screen Buffer Size section. That has a limit of 9,999 lines, and I do believe it defaulted to 300 (I changed that setting so long ago that I no longer remember for sure what the stock default was).
Perhaps to simplify the implementation, one could just look for repetitions/cycles that include the LAST trace line. Rob Cliffe On 20/04/2016 05:07, Émanuel Barry wrote:
This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception.
Of course, the same is true for actual programs, where tracebacks drown off everything else useful. For all we know, a typo caused a NameError in one of the functions which somehow triggered it to call itself over again until Python decided it was enough, but you can't know that because your entire scrollback is full of
File "<stdin>", line 1, in func
And, for programs that are user-facing, I take all tracebacks and redirect them to a file that the user can submit me so I can debug the issue, and having tens of hundreds of the same line hurts readability.
The issue that the output from running a certain script in the REPL and as a script would differ has been raised, and I am of the opinion that in no way, shape or form should the two have different behaviours (the exception being sys.ps1, sys.ps2 and builtins._ in interactive mode, which are 100% fine). I'm suggesting a change to how Python handles specifically recursion limits, to cut after a sane number of times (someone suggested to shrink the output after 3 times), and simply stating how many more identical messages there are. I would also like to extend this to any arbitrary loop of callers (for example foo -> bar -> baz -> foo and so on would be counted as one "call" for the purposes of this proposal).
Under this proposal, something similar to this would happen:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in func File "<stdin>", line 1, in func File "<stdin>", line 1, in func [Previous 1 message(s) repeated 996 more times] RecursionError: maximum recursion depth exceeded
With multiple chained calls (I don't know how hard it would be to implement this, probably not trivial):
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz [Previous 3 message(s) repeated 330 more times] RecursionError: maximum recursion depth exceeded
We can probably afford to cut out immediately as we notice the recursion, but even just 3 times with multiple chained calls isn't going to be anywhere near as long as it currently is.
Thoughts? Suggestions? Something I forgot?
-Emanuel _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Yes of course. On 20/04/2016 11:41, Koos Zevenhoven wrote:
On Wed, Apr 20, 2016 at 12:41 PM, Rob Cliffe <rob.cliffe@btinternet.com> wrote:
Perhaps to simplify the implementation, one could just look for repetitions/cycles that include the LAST trace line. Rob Cliffe
Maybe you mean the last line before the exception message?
-Koos
On Wed, Apr 20, 2016 at 12:07:11AM -0400, Émanuel Barry wrote:
This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception.
Even if you have an effectively unlimited scrollback, getting thousands of identical lines is a PITA and rather distracting and annoying.
I'm suggesting a change to how Python handles specifically recursion limits,
We should not limit this to specifically *recursion* limits, despite the error message printed out, it is not just recursive calls that count towards the recursion limit. It is any function call. In practice, though, it is difficult to run into this limit with regular non-recursive calls, but not impossible.
to cut after a sane number of times (someone suggested to shrink the output after 3 times), and simply stating how many more identical messages there are.
That would be me :-)
I would also like to extend this to any arbitrary loop of callers (for example foo -> bar -> baz -> foo and so on would be counted as one "call" for the purposes of this proposal).
Seems reasonable.
We can probably afford to cut out immediately as we notice the recursion,
No you can't. A bunch of recursive calls may be followed by non- recursive calls, or a different set of recursive calls. It might not even be a RuntimeError at the end. For example, put these four functions in a file, "fact.py": def fact(n): if n < 1: return 1 return n*fact(n-1) def rec(n): if n < 20: return spam(n) return n + rec(n-1) def spam(n): return eggs(2*n) def eggs(arg): return fact(arg//2) Now I run this: import fact sys.setrecursionlimit(35) fact.rec(30) and I get this shorter traceback: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/steve/python/fact.py", line 9, in rec return n + rec(n-1) [...repeat previous line 10 times...] File "/home/steve/python/fact.py", line 8, in rec return spam(n) File "/home/steve/python/fact.py", line 12, in spam return eggs(2*n) File "/home/steve/python/fact.py", line 15, in eggs return fact(arg//2) File "/home/steve/python/fact.py", line 3, in fact return n*fact(n-1) [...repeat previous line 18 times...] File "/home/steve/python/fact.py", line 2, in fact if n < 1: return 1 RuntimeError: maximum recursion depth exceeded in comparison The point being, you cannot just stop processing traces when you find one repeated. -- Steve
From Steven D'Aprano Sent: Wednesday, April 20, 2016 10:46 PM
We should not limit this to specifically *recursion* limits, despite the error message printed out, it is not just recursive calls that count towards the recursion limit. It is any function call.
In practice, though, it is difficult to run into this limit with regular non-recursive calls, but not impossible.
See my second comment
We can probably afford to cut out immediately as we notice the recursion,
No you can't. A bunch of recursive calls may be followed by non- recursive calls, or a different set of recursive calls. It might not even be a RuntimeError at the end.
See my first comment *wink wink* My proposal is so that we strip out repeated lines to avoid unnecessary noise, *not* to strip out everything thereafter! I agree that my wording was misleading, but I do want to keep all relevant information - only to strip out consecutive and identical lines. If there's a 20-call recursion in the middle of the stack for some odd reason (call itself with n+1 until n == 20 and then do something else?), some of it would be stripped out but the rest would remain. The core of my proposal is to enhance ability to read and debug large tracebacks, which most of the times will be because of recursive calls. We're shrinking tracebacks for recursive calls, and if that happens to shrink other large tracebacks as well, that's a useful side effect and should be considered, but that's not the core of my idea. If a one-size-fits-all solution works here, I'll go for that and avoid dumping yet another bucket of special cases (which aren't special enough to break the rules) into the code. There are already enough.
The point being, you cannot just stop processing traces when you find one repeated.
Yep, don't want that at all indeed. Thanks for your input Steven :) -Emanuel
On Thu, Apr 21, 2016 at 2:55 PM, Émanuel Barry <vgr255@live.ca> wrote:
The core of my proposal is to enhance ability to read and debug large tracebacks, which most of the times will be because of recursive calls. We're shrinking tracebacks for recursive calls, and if that happens to shrink other large tracebacks as well, that's a useful side effect and should be considered, but that's not the core of my idea.
No problem with that. In my experience, most RecursionErrors come from *accidental* recursion, which is straight-forwardly infinite and usually involves a single function. Consider this idiom:
class id(int): ... _orig_id = id ... def __new__(cls, obj): ... return super().__new__(cls, cls._orig_id(obj)) ... def __repr__(self): ... return hex(self) ... obj = object() obj <object object at 0x7ffb12290110> id(obj) 0x7ffb12290110
If I muck something up and accidentally call id() inside the definition of __new__ (instead of cls._orig_id), it'll end up infinitely recursing. A traceback shortener that recognizes only the very simplest forms of repetition would work fine for this case. It doesn't need a huge amount of intelligence - the traceback would have a bunch of exactly identical lines, and it _would_ end with one of the identical ones. ChrisA
No problem with that. In my experience, most RecursionErrors come from *accidental* recursion, which is straight-forwardly infinite and usually involves a single function. It can come very easily with __getattr__, if __getattr__ itself uses an undefined attribute. This recursion is usually not what the user wanted! :)
On 04/21/2016 12:35 PM, Joseph Martinot-Lagarde wrote:
No problem with that. In my experience, most RecursionErrors come from *accidental* recursion, which is straight-forwardly infinite and usually involves a single function.
It can come very easily with __getattr__, if __getattr__ itself uses an undefined attribute. This recursion is usually not what the user wanted! :)
True, but I'm pretty sure that falls in to the *accidental recursion* category. ;) -- ~Ethan~
True, but I'm pretty sure that falls in to the *accidental recursion* category. ;) Of course, I just wanted to present a less complicated example (and more common) than the id example from Chris Angelico.
Anyway shortening the traceback is useful in all cases, the recursion being intentionnal or not.
I implemented a small patch for this, over at http://bugs.python.org/issue26823 Comments and feedback are very much welcome :) -Emanuel
participants (9)
-
Chris Angelico -
Ethan Furman -
Jonathan Goble -
Joseph Martinot-Lagarde -
Koos Zevenhoven -
Rob Cliffe -
Steven D'Aprano -
Terry Reedy -
Émanuel Barry