Two different (probably radical) ideas, with the same justification: reduce often-useless output in the REPL, which can flood out terminal history and overwhelm the user. 1. Limit the output per entered command: If you type into the REPL (AKA interactive shell), list(range(n)) and you forgot that you set n to 10**10, the interpreter should not print more than a page of output. Instead, it will print a few lines ("... and approximately X more lines"), and tell you how to print more. (E.g. "Call '_more()' for more. Call '_full()' for full output.") Alternatively, have "less"-like behavior. 2. Only print a few parts of the stack trace. In particular, for a recursive or mutually recursive function, if the error was due to maximum recursion (is this reasonably distinguishable? the error is `RuntimeError('maximum recursion depth exceeded')`), try to print each function on the stack once each. Again, there should be a message telling you how to get the full stacktrace printed. EXACTLY how, preferably in a way that is easy to type, so that a typo won't cause the trace to be lost. It should not use `sys.something()`, because the user's first few encounters with this message will result in, "NameError: name 'sys' is not defined". A few possible rules to reduce stacktrace size: - Always show the last (top) frame(?). - Hide any other stdlib funcs directly below (above) it. - If a function appears more than once in a row, show it once, with the note, "(and X recursive calls)". - If functions otherwise appears more than once (usually by mutual recursion?), and there is a run of them, list them as, "(Mutual recursion: 'x' (5 times), 'y' (144 times), 'z' (13 times).)". These two behaviors and their corresponding functions could go into a special module which is (by default) loaded by the interactive shell. The behaviors can be turned off with some command-line verbosity flag, or tuned with command-line parameters (e.g. how many lines/pages to print). Justification: - Excessive output floods the terminal window. Some terminals have a limit on output history (Windows *defaults* to 300 lines), or don't let you scroll up at all (at least, my noob self couldn't figure it out when I did get flooded). - Students learning Python, and also everyone else using Python, don't usually need 99% of a 10000-line output or stack trace. - People who want the full output are probably advanced users with, like, high-limit or unlimited window size, and advanced users are more likely to look for a verbosity flag, or use a third-party REPL. Default should be newbie friendly, because advanced users can work around it. Thoughts? Even if the specific proposals are unworkable, is limiting REPL output (by default) a reasonable goal?
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
For that specific example, when I run it, the output is quite short: $ python3 >>> n = 10**10 >>> list(range(n)) Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError I take the point though: some objects have a very long ‘repr’ output.
Perhaps you want a different “print the interactive-REPL-safe text representation of this object” function, and to have the interactive REPL use that new function to represent objects. -- \ “I am too firm in my consciousness of the marvelous to be ever | `\ fascinated by the mere supernatural …” —Joseph Conrad, _The | _o__) Shadow-Line_ | Ben Finney
On Apr 17, 2016 9:43 PM, "Ben Finney" <ben+python@benfinney.id.au> wrote:
You mean that the REPL should call `repr_limited(...)` instead of `repr(...)`, and not that class makers should implement `__repr_limited__`, right? I think one could make a `pprint` class for that. Thing is, I would also want the following to have limited output. >>> def dumb(n): ... for i in range(n): ... print(i) ... >>> dumb(10**10) That could print the first 50 or so lines, then the REPL buffers the rest and prints out the message about suppressed output. On the other hand, for a long-running computation with logging statements, I don't wanna start it, come back, and lose all that would have been printed, just because I forgot that the REPL does that now. Storing it all could be a use of memory that the computation might not be able to afford. Possible solution: 1. After output starts getting buffered, print out warnings every X seconds about Y lines being buffered. 2. Detect screen limit, and only store that many lines back. You don't lose anything that wasn't already going to be lost. (Optional, but possibly useful: Also warn about Z lines being lost due to screen limit.) It was going to be in the terminal's memory, anyway, so just store it (as efficiently as possible) in Python's memory. I think this only uses up to twice as much total system memory (actual screen mem + dump). Since you won't use the output as Python at least until the user comes back, it could be lazy about converting to `str`, and maybe even compress the buffer on the fly (if queue compression is cheaper than printing). It gets trickier when you want user input during the eval part of the REPL. 3. Allow the user to interrupt the eval to print the last page of output, or dump the entire current output, and then Python will continue the eval. (Include in the warning, "Press Ctrl+?? to show <thing>.") 4. If the eval is waiting for user input, show the last page. (I don't know how the user could ask for more pages without sending something that the eval is allowed to interrupt as input. I don't understand terminal keyboard control that much.)
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
The former, yes: changing the behaviour of the interactive session, without the need for the programmer to change their code. To be clear, I'm asking whether that would meet your requirements :-)
I think one could make a `pprint` class for that.
Surem, that would be a good place for it. I think that's much more feasible than changing the behaviour of ‘repr’ for this purpose.
Again, it's only the interactive session which has the behaviour you want changed. That code run from a non-interactive session simply won't output anything if there's no error, so I think ‘repr’ is not a problem by default. It's only because the interactive session has the *additional* behaviour of emitting the representation of the object resulting from evaluation, that it has the behaviour you describe. So this is increasing the likelihood that the best way to address your request is not “change default ‘repr’ behaviour”, but “change the representation function the interactive session uses for its special object-representation output”. -- \ “The conflict between humanists and religionists has always | `\ been one between the torch of enlightenment and the chains of | _o__) enslavement.” —Wole Soyinka, 2014-08-10 | Ben Finney
On Apr 17, 2016 11:24 PM, "Ben Finney" <ben+python@benfinney.id.au> wrote:
Surem, that would be a good place for it. I think that's much more feasible than changing the behaviour of ‘repr’ for this purpose.
Huh? I never meant for any change to happen to `repr`. The desired behavior is, "Set default: Each command entered into the REPL should have limited output (without loss of capability for a newbie)." The proposal is to have the REPL determine when it's going to output too much. My example, as originally written, `print`d the huge object. I realized that this was too specific, and I didn't *really* care about `print` as such, so I changed it to (what I thought was) a more general example. I should've just had multiple examples.
On 18 April 2016 at 15:32, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
A few tips for folks that want to play with this: - setting sys.displayhook controls how evaluated objects are displayed in the default REPL - setting sys.excepthook does the same for exceptions - the PYTHONSTARTUP env var runs the given file when the default REPL starts So folks are already free to change their REPL to work however they want it to: set sys.displayhook and sys.excepthook from a PYTHONSTARTUP file Changing the *default* REPL behaviour is a very different question, and forks off in a couple of different directions: - improved defaults for teaching novices? Perhaps the default REPL isn't the best environment for that - easier debugging at the REPL? Perhaps pprint should gain an "install_displayhook()" option that overwrites sys.displayhook and optionally allows enabling of an output pager Since the more the REPL does, the more opportunities there are for it to break when debugging, having the output hooks be as simple as possible is quite desirable. However, making it easier to request something more sophisticated via the pprint module seems like a plausible approach to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Apr 18, 2016 3:25 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 18 April 2016 at 15:32, Franklin? Lee <leewangzhong+python@gmail.com>
wrote: limited output (without loss of capability for a newbie)." The proposal is to have the REPL determine when it's going to output too much.
A few tips for folks that want to play with this:
- setting sys.displayhook controls how evaluated objects are displayed in
the default REPL
- improved defaults for teaching novices? Perhaps the default REPL isn't
I don't want arguments like, "This can already be done, for yourself, if you really need it." I use IPython's shell, with maximum output height, and it was years ago that I used a terminal which I couldn't scroll. I want arguments like, "This will break my workflow." the best environment for that Why not? I imagine that self-taught novices will use the default REPL more than the advanced users.
On 04/18/2016 08:41 AM, Franklin? Lee wrote:
On Apr 18, 2016 3:25 AM, "Nick Coghlan" wrote:
How about: the default REPL is a basic tool, and the limitations of basic tools are what drive folks to seek out advanced tools. ? Or Marc-Andre's:
Or even Nick's (that you snipped):
IMO those are all good reasons to leave the basic REPL alone. -- ~Ethan~
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
That's your prerogative. This forum is for discussing whether ideas are right for changing Python, though. Arguments such as “This can already be done, for yourself, if you need it” are salient and sufficient, and not to be dismissed.
I want arguments like, "This will break my workflow."
You're free to solicit those arguments. You'll need to ackonwledge, though, that they are in addition to the quite sufficient argument of “already possible for people to get this in Python as is, if they want it”. -- \ “I think Western civilization is more enlightened precisely | `\ because we have learned how to ignore our religious leaders.” | _o__) —Bill Maher, 2003 | Ben Finney
On Mon, Apr 18, 2016 at 11:51 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
The REPL is a basic tool for basic users, which is why it should "Do the right thing" for people who wouldn't know better. I'm asking whether this is the "right thing" for those basic users: Advanced users are the ones who can use more than basic info.
I think it's a high priority not to lose information that was hidden. In my proposal, I said that there should be a way to access it, right after you cause it, which is very different from needing to enable a variable. If not for (memory) performance anxiety, I would've proposed that the info would be stored indefinitely, or that you would always be able to access the last three (or something) outputs/stacktraces, because I was concerned about typos causing exceptions that would then push the interesting stacktrace out. (Hey, that's another possible proposal: maybe syntax/lookup errors on the "R" part of "REPL" shouldn't count toward `sys.last_traceback`. Has anyone here ever needed to debug or reprint syntax errors coming from compiling the command they just entered?)
I read this as, "It'll be more complicated, so it MIGHT be buggy." This seems solvable via design, and doesn't speak to whether it's good, in principle, to limit output per input. If you still think it's a good reason, then I probably didn't understand it correctly. On Mon, Apr 18, 2016 at 2:39 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
When they are presented as a solution to "my" problem, they are, to be short, irrelevant. They try to address the speaker's need for the feature on their own machine, when I am asking for opinions on both the usefulness and harmfulness of such a feature, and the principle behind the feature. They are distracting: people are talking more about whether I can do it "myself" than how it's bad to have this, or how it wouldn't help who I want to help. Even if they weren't trying to solve "my" problem, I had non-advanced users in mind, and these solutions tend to be about what advanced users can do. I felt the need to point out what I'm looking for in the arguments, because people are telling me how *I* can have this feature by, for example, writing a display hook, or plugging into pprint.
On 19 April 2016 at 05:52, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
Basic users should probably be using a tool like IDLE, which has a bit more support for beginners than the raw REPL. I view the REPL as more important for intermediate or advanced users who want to quickly test out an idea (at least, that's *my* usage of the REPL). As I disagree with your statement "the REPL is a basic tool for basic users" I don't find your conclusions compelling, sorry. Paul
On Tue, Apr 19, 2016 at 11:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I was once a basic user, but I still have no idea what "IDLE" is. Does it come with python? I have tried $ idle $ python -m idle $ python -m IDLE $ python --idle To be honest, I do remember seeing a shortcut to IDLE in one of my Windows python installations, and I've seen it come up in discussions. However, it does not seem to me that IDLE is something that beginners would know to turn to. I do use IPython. IPython is nice---too bad it starts up slowly and is not recommended by default. -- Koos
On Tue, Apr 19, 2016 at 7:35 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
The first one will often work, but it depends on exactly how your Python installation has been set up. (Also, not all Linux distros come with Idle by default; you may have to install a python-idle package.) The most reliable way to invoke Idle is: $ python3 -m idlelib.idle (or python2 if you want 2.7). It's more verbose than would be ideal, and maybe it'd help to add one of your other attempts as an alias, but normally Idle will be made available in your GUI, which is where a lot of people will look for it. ChrisA
On 19 April 2016 at 10:35, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Windows, it's set up as a shortcut in the start menu alongside Python, with a tooltip "Launches IDLE, the interactive environment for Python 3.5". And Python scripts have an "Edit with IDLE" option on the right click menu. So it's where most Windows users would naturally look when searching for "the Python GUI". Discoverability of IDLE on other platforms isn't something I know much about, sorry. (I should also point out that I'm a relentless command line user on Windows, so my comments above about Windows are based on my experience watching what my "normal Windows user" colleagues do, not on my personal habits...) Paul
On 4/19/2016 5:35 AM, Koos Zevenhoven wrote:
I was once a basic user, but I still have no idea what "IDLE" is. Does it come with python?
Yes, unless explicitly omitted either in packaging or installation. (Some redistributors might put it with a separate tkinter/tix/idle/turtle/turtledemo package. The windows installer has a box that can be unchecked.)
I have tried
$ idle
This and idle3 works on some but not all systems.
$ python -m idle
python -m idlelib (or idlelib.idle -- required in 2.x)
$ python -m IDLE $ python --idle
To be honest, I do remember seeing a shortcut to IDLE in one of my Windows python installations,
Right. Once run, the IDLE icon can be pinned to the taskbar.
Yet many do, perhaps because instructors and books suggest it and tell how to start it up. -- Terry Jan Reedy
On Apr 19, 2016 4:09 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 19 April 2016 at 05:52, Franklin? Lee <leewangzhong+python@gmail.com>
wrote:
You say "should"? Do you mean that it is likely, or do you mean that it is what would happen in an ideal world? My college had CS students SSH into the department's Linux server to compile and run their code, and many teachers don't believe that students should start with fancy IDE featues like, er, syntax highlighting. You (and most regulars on this list) can adjust your shell to the way you like it, or use a more sophisticated shell, like IPython or bpython. On the other hand, changing shells and adding display hooks to site.py is not an option for those who don't know it's an option.
But that doesn't answer my question: would the proposed change hurt your workflow?
On 19 April 2016 at 10:36, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
I mean that that is what I hear people on lists like this saying as "a reasonable beginner environment". It means that the people I've introduced to Python (on Windows) have tended to end up using IDLE (either from the start menu or via "edit with IDLE" from the right click menu on a script). My experience (in business, environments) is that people expect an IDE when introduced to a new programming language, and IDLE, like it or not, is what is available out of the box with Python.
As a "scripting expert" and consultant, I typically get asked to knock up scripts on a variety of environments. I do not normally have what I'd describe as "my shell", just a series of basic "out of the box" prompts people expect me to work at. So no, the luxury of configuring the default experience is *not* something I typically have.
Yes. If I get a stack trace, I want it all. And if I print something out, I want to see it by default. The REPL for me is an investigative environment for seeing exactly what's going on. (Also, having the REPL behave differently depending on what version of Python I have would be a problem - backward compatibility applies here as much as anywhere else). Paul
On Tue, Apr 19, 2016 at 5:36 AM Franklin? Lee <leewangzhong+python@gmail.com> wrote:
That's probably because your professors thought you were more advanced than other new Pythonistas, because you were CS students. If I were in their shoes, I might chose a different approach depending on the level of the course.
But that doesn't answer my question: would the proposed change hurt your workflow?
It might. Would it affect doctests? Would it make indirect infinite recursion more difficult to trace? Would it make me remember yet another command line option or REPL option to turn on complete reprs? Would it force me to explain yet another config setting to new programmers? I think a beginner understands when they've printed something too big. I see this happen frequently. They laugh, shake their heads, and retype whatever they need to. If they're using IDLE, they say, "OMG I crashed it!" then they close the window or restart IDLE. I'd say it's more a problem in IDLE than in the default REPL.
On 4/19/2016 10:03 AM, Michael Selik wrote:
Recursion limit tracebacks with 2000 short lines (under 80 chars) are not a problem for tk's Text widget. Long lines, from printing something line 'a'*10000 or [1]*5000 make the widget sluggish. Too many or too long may require a restart.
I'd say it's more a problem in IDLE than in the default REPL.
Its a problem with using an underlying text widget optimized for managing 'sensible' length '\n' delimited lines. -- Terry Jan Reedy
On Apr 19, 2016 9:12 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
Implicit side effects from hidden code break that mental equivalence -
it's why effective uses of metaclasses, monkeypatching, and other techniques for deliberately introducing changed implicit behaviour often also involve introducing some kind of more local signal to help convey what is going on (such as a naming convention, or ensuring the altered behaviour is used consistently across the entire project). But a local signal _is_ part of my original proposal: """Instead, it will print a few lines ("... and approximately X more lines"), and tell you how to print more. (E.g. "Call '_more()' for more. Call '_full()' for full output.")""" """Again, there should be a message telling you how to get the full stacktrace printed. EXACTLY how, preferably in a way that is easy to type, so that a typo won't cause the trace to be lost. It should not use `sys.something()`, because the user's first few encounters with this message will result in, "NameError: name 'sys' is not defined".""" This also satisfies Terry Reedy's "reversibility" condition. On Apr 19, 2016 10:03 AM, "Michael Selik" <mike@selik.org> wrote:
On Tue, Apr 19, 2016 at 5:36 AM Franklin? Lee <
leewangzhong+python@gmail.com> wrote: than other new Pythonistas, because you were CS students. If I were in their shoes, I might chose a different approach depending on the level of the course. I meant to have two separate clauses: - In my college, there was a server for compiling and running code. It allowed people to do so without installing a compiler (e.g. on a general-purpose school computer). It also gave a standard compiler to test against. I did not learn Python in school. - Many teachers, all over the world, do not believe in IDE features for beginners. (Therefore, there will be many *students* which don't learn about IDLE.) You can want them to stop thinking that way, but the students will still be out there.
Let's be clear about what I am proposing. I prioritize not losing info and making it obvious how to get at that info. My suggestion was to show less output *and* have a message at the end saying how to get the output. It obviously should not affect doctests, since the output there is not for humans. An implementation that affects doctests should be considered buggy. Perhaps have an option to allow it, but affecting doctests by default would be an actual backward incompatibility, since it changes how existing code *runs* (i.e. not just output, but different logical paths during runtime). My idea for tracebacks of mutually recursive calls: See which functions are on the stack more than once, and if they keep appearing, pack them up. The algorithm should not pack up calls which happen only once within an apparent cycle, and it should be clear which order the calls come in (outside of folded mutually-recursive calls, of course). Example: File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutually recursive calls hidden: f (300), g (360)] File "<stdin>", line 1, in h File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutual-recursive calls hidden: f (103), g (200)] RuntimeError: maximum recursion depth exceeded [963 calls hidden. Call _full_ex() to print full trace.] Or maybe the second f and g are also folded into the second hidden bit. And maybe it checks line numbers when deciding whether to print explicitly (but not when folding). Would that output make indirect infinite recursion more difficult for you to debug?
I am not proposing this as a better error message, but because a flooded terminal is loss of information. It is also good to have significant info close together, to reduce effort (and thus mental cache) between processing of related ideas. Have you never needed to see the output of previous lines? Re-entering the earlier line might not work if the state of the program has changed. This happens at the beginner level.
On Wed, Apr 20, 2016, at 14:25, Franklin? Lee wrote:
You know what would make complicated infinite recursion easier to debug? The arguments. Is there a reliable way to determine what in f_locals correspond to arguments? My toy example below only works for named positional arguments. def magic(frame): code = frame.f_code fname = code.co_name argcount = code.co_argcount args = code.co_varnames[:argcount] values = tuple(frame.f_locals[a] for a in args) result = '%s%r' % (fname, values) if len(result) > 64: return fname + '(...)' return result Maybe even tokenize the source line the error occurred on and print any other locals whose name matches any token on the line. I guess I'll leave it to the bikeshed design committee and say: WIBNI tracebacks printed relevant frame variables FSDO relevant?
On Wed, Apr 20, 2016 at 03:01:48PM -0400, Random832 wrote:
You know what would make complicated infinite recursion easier to debug? The arguments.
Check out the cgitb module, which installs an except hook which (among many other things) prints the arguments to the functions. Run this sample code: import cgitb cgitb.enable(format='text') def spam(arg): if arg == 0: raise ValueError('+++ out of cheese error, redo from start +++') return spam(arg - 1) def eggs(x): return spam(x) + 1 def cheese(a, b, c): return a or b or eggs(c) cheese(0, None, 1) and you will see output something like the following. (For brevity I have compressed some of the output.) ValueError Python 3.3.0rc3: /usr/local/bin/python3.3 Thu Apr 21 10:58:22 2016 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /home/steve/python/<stdin> in <module>() /home/steve/python/<stdin> in cheese(a=0, b=None, c=2) /home/steve/python/<stdin> in eggs(x=2) /home/steve/python/<stdin> in spam(arg=2) /home/steve/python/<stdin> in spam(arg=1) /home/steve/python/<stdin> in spam(arg=0) ValueError: +++ out of cheese error, redo from start +++ __cause__ = None __class__ = <class 'ValueError'> [...] __traceback__ = <traceback object> args = ('+++ out of cheese error, redo from start +++',) with_traceback = <built-in method with_traceback of ValueError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in cheese File "<stdin>", line 2, in eggs File "<stdin>", line 4, in spam File "<stdin>", line 4, in spam File "<stdin>", line 3, in spam ValueError: +++ out of cheese error, redo from start +++ -- Steve
On 19 April 2016 at 14:52, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
No, that's not what it means. It relates to one of the ways experienced developers are able to debug code: by running it in their heads, and comparing the result their brain computed with what actually happened. When there's a discrepancy, either their expectation is wrong or the code is wrong, and they need to investigate further to figure out which it is. When folks say "Python fits my brain" this is often what they're talking about. Implicit side effects from hidden code break that mental equivalence - it's why effective uses of metaclasses, monkeypatching, and other techniques for deliberately introducing changed implicit behaviour often also involve introducing some kind of more local signal to help convey what is going on (such as a naming convention, or ensuring the altered behaviour is used consistently across the entire project). The default REPL behaviour is appropriate for this "somewhat experienced Pythonista tinkering with code to see how it behaves" use case - keeping the results very close to what they would be if you typed the same line of code into a text file and ran it that way. It's not necessarily the best way to *learn* those equivalences, but that's also not what it's designed for. IPython's REPL is tailored for a different audience - their primary audience is research scientists, and they want to be able to better eyeball calculation results, rather than lower level Python instance representations. As a result, it's much cleverer than the default REPL, but it's also aiming to tap into people's intuitions about the shape of their data and the expected outcomes of the operations they're performing on it, rather than their ability to mentally run Python code specifically. A REPL designed specifically for folks learning Python, like the one in the Mu editor, or the direction IDLE seems to be going, would likely be better off choosing different default settings for sys displayhook and sys.excepthook, but those changes would be best selected based on direct observations of classrooms and workshops, and noting where folks get confused or intimidated by the default settings. For environments other than IDLE, they can also be iterated on at a much higher rate than we make CPython releases. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Apr 19, 2016 at 11:12:16PM +1000, Nick Coghlan wrote:
I mostly agree with what you say, but I would like to see one change to the default sys.excepthook: large numbers of *identical* traceback lines (as you often get with recursion errors) should be collapsed. For example: py> sys.setrecursionlimit(20) py> fact(30) # obvious recursive factorial Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 2, in fact RuntimeError: maximum recursion depth exceeded in comparison Try as I might, I just don't see the value of manually counting all those 'File "<stdin>", line 3, in fact' lines to find out where the recursive call failed :-) I think that it would be better if identical lines were collapsed, something like this: import sys import traceback from itertools import groupby TEMPLATE = " [...repeat previous line %d times...]\n" def collapse(seq, minimum, template=TEMPLATE): for key, group in groupby(seq): group = list(group) if len(group) < minimum: for item in group: yield item else: yield key yield template % (len(group)-1) def shortertb(*args): lines = traceback.format_exception(*args) sys.stderr.write(''.join(collapse(lines, 3))) sys.excepthook = shortertb which then gives tracebacks like this: py> sys.setrecursionlimit(200) py> a = fact(10000) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in fact [...repeat previous line 197 times...] File "<stdin>", line 2, in fact RuntimeError: maximum recursion depth exceeded in comparison -- Steve
On 4/19/2016 12:18 PM, Steven D'Aprano wrote:
Tracebacks produce *pairs* of lines: the location and the line itself. Replacing pairs with a count of repetitions would not lose information, and would make the info more visible. I would require at least, say, 3 repetitions before collapsing.
-- Terry Jan Reedy
On 4/20/2016 2:15 PM, Random832 wrote:
Aha. In IDLE's Shell, which generally simulates the interactive interpreter quite well, pairs are printed because interactive user input is exec'ed in a process running Python in normal batch mode. I expect any GUI shell to act similarly. I consider interactive interpreter omission of the line at fault to be a design buglet. For a single line of input, it is not much of a problem to look back up. But if one pastes a 20-line statement, finding line 13, for instance, is not so quick. And on Windows, a 1000 line traceback may erase the input so there is nothing to look at. -- Terry Jan Reedy
On Tue, Apr 19, 2016 at 06:28:40PM -0400, Terry Reedy wrote:
Only if the source code is available. The source isn't available for functions defined in the REPL, for C code, or for Python functions read from a .pyc file where the .py file is not available. The code I gave before works with pairs of location + source, without any change. If I move the definition of fact into a file, so that the source lines are included, we get: py> sys.setrecursionlimit(10) py> fact(30) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/steve/python/fact.py", line 3, in fact return n*fact(n-1) [...repeat previous line 7 times...] File "/home/steve/python/fact.py", line 2, in fact if n < 1: return 1 RuntimeError: maximum recursion depth exceeded in comparison I'd want to adjust the wording. Perhaps "previous entry" rather than previous line?
That's what my example does: it only collapses the line/(pair of lines) if there are at least three identical repetitions. -- Steve
On Wed, 20 Apr 2016 02:18:58 +1000, Steven D'Aprano wrote:
On Tue, Apr 19, 2016 at 11:12:16PM +1000, Nick Coghlan wrote:
That's a pretty powerful argument: running something in the REPL should give the same results as running it from a command line. When things fail differently in different environments, the environments themselves become suspects.
[...]
Yes, I see the smiley, but I would add specifically that the number of calls in the stack trace from a recursive function is rarely the important part. When I write something recursive, and get *that* stack trace, I don't have to scroll anywhere to look at anything to know that I blew the termination condition(s). (I'm agreeing with you: there is no value of counting the number of calls in the stack trace.) If I suddenly got tiny stack traces, I'd spend *more* time realizing what went wrong, until I retrained myself.
On 4/19/2016 9:12 AM, Nick Coghlan wrote:
The default REPL behavior should not throw away output. In the Windows console, what is not displayed cannot be retrieved. Reversible output editing is possible and appropriate in a GUI that can keep output separate from what is displayed.
There is a external Squeezer extension to IDLE that more or less does what this thread proposes and is also reversible. Deleted blocks are replaced by something that can be clicked on to expand the text. There is a proposal to incorporate Squeezer into IDLE. I have not reviewed the proposal yet because it would not solve any of my problems. I am more interested in way to put long output from help() into a separate text window instead of the shell.
-- Terry Jan Reedy
On Sun, 17 Apr 2016 21:04:11 -0400, Franklin? Lee wrote:
The existing pretty print module already does some of what you want. Instead of creating a new module, maybe extending that one would have greater benefits to more users.
Judging what others might find more friendly, or guessing what actions they are more likely to take, can be dangerous.
Thoughts? Even if the specific proposals are unworkable, is limiting REPL output (by default) a reasonable goal?
IMO, the default REPL should be as simple as possible, and respond as directly as possible to what I ask it to do. It should also be [highly] configurable for when you want more complexity, like summarizing stack traces and paginating output.
On Mon, Apr 18, 2016 at 2:33 AM, Dan Sommers <dan@tombstonezero.net> wrote:
There's also IPython's repr system as prior art: http://ipython.readthedocs.org/en/stable/api/generated/IPython.lib.pretty.ht... My impression (as I guess one of the few people who have tried to systematically add _repr_pretty_ callbacks to their libraries) is that it's only about 30% baked and has a number of flaws and limitations, but it's still a significant step beyond __repr__ or the stdlib pprint. (Some issues I've run into: there's something wonky in the line breaking that often produces misaligned lines; integration with __repr__ is awkward if you don't want to reimplement everything twice [1]; there's no built-in control for compressing output down to ellipses; handling the most common case of reprs that look like Foo(bar, baz, kwarg=1) is way more awkward than it needs to be [2]; and IIRC there were some awkward limitations in the formatting tools provided though I don't remember the details now. But despite all this it does make it easy to write composable reprs, so e.g. here's a syntax tree: In [1]: import patsy In [2]: patsy.parse_formula.parse_formula("y ~ 1 + (a * b)") Out[2]: ParseNode('~', Token('~', <Origin y ->~<- 1 + (a * b) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->y<- ~ 1 + (a * b) (0-1)>, extra='y'), []), ParseNode('+', Token('+', <Origin y ~ 1 ->+<- (a * b) (6-7)>), [ParseNode('ONE', Token('ONE', <Origin y ~ ->1<- + (a * b) (4-5)>, extra='1'), []), ParseNode('*', Token('*', <Origin y ~ 1 + (a ->*<- b) (11-12)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin y ~ 1 + (->a<- * b) (9-10)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin y ~ 1 + (a * ->b<-) (13-14)>, extra='b'), [])])])]) -n [1] https://github.com/pydata/patsy/blob/23ab37519276188c4ce09db03a1c40c5b9938bf... [2] https://github.com/pydata/patsy/blob/23ab37519276188c4ce09db03a1c40c5b9938bf... -- Nathaniel J. Smith -- https://vorpus.org
On 18.04.2016 03:04, Franklin? Lee wrote:
You should be able to write your own sys.displayhook to accomplish this.
Same here with sys.excepthook. FWIW: I see your point in certain situations, but don't think the defaults should be such that you have to enable some variable to see everything. This would make debugging harder than necessary, since often enough (following Murphy's law) the most interesting information would be hidden in some ellipsis. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 18 2016)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Franklin? Lee <leewangzhong+python@...> writes:
I don't know about the other ideas, but simplifying the output of recursion errors would be very useful. 990 repeats of the exact same lines add no value at all, the user being an expert or a beginner. Something like this is very clear in my mind: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/user/script.py", line 10, in f f() File "/home/user/script.py", line 10, in f f() ... File "/home/user/script.py", line 10, in f (994 repeats) f() ... File "/home/user/script.py", line 10, in f f() File "/home/user/script.py", line 10, in f f() RecursionError: maximum recursion depth exceeded
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
For that specific example, when I run it, the output is quite short: $ python3 >>> n = 10**10 >>> list(range(n)) Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError I take the point though: some objects have a very long ‘repr’ output.
Perhaps you want a different “print the interactive-REPL-safe text representation of this object” function, and to have the interactive REPL use that new function to represent objects. -- \ “I am too firm in my consciousness of the marvelous to be ever | `\ fascinated by the mere supernatural …” —Joseph Conrad, _The | _o__) Shadow-Line_ | Ben Finney
On Apr 17, 2016 9:43 PM, "Ben Finney" <ben+python@benfinney.id.au> wrote:
You mean that the REPL should call `repr_limited(...)` instead of `repr(...)`, and not that class makers should implement `__repr_limited__`, right? I think one could make a `pprint` class for that. Thing is, I would also want the following to have limited output. >>> def dumb(n): ... for i in range(n): ... print(i) ... >>> dumb(10**10) That could print the first 50 or so lines, then the REPL buffers the rest and prints out the message about suppressed output. On the other hand, for a long-running computation with logging statements, I don't wanna start it, come back, and lose all that would have been printed, just because I forgot that the REPL does that now. Storing it all could be a use of memory that the computation might not be able to afford. Possible solution: 1. After output starts getting buffered, print out warnings every X seconds about Y lines being buffered. 2. Detect screen limit, and only store that many lines back. You don't lose anything that wasn't already going to be lost. (Optional, but possibly useful: Also warn about Z lines being lost due to screen limit.) It was going to be in the terminal's memory, anyway, so just store it (as efficiently as possible) in Python's memory. I think this only uses up to twice as much total system memory (actual screen mem + dump). Since you won't use the output as Python at least until the user comes back, it could be lazy about converting to `str`, and maybe even compress the buffer on the fly (if queue compression is cheaper than printing). It gets trickier when you want user input during the eval part of the REPL. 3. Allow the user to interrupt the eval to print the last page of output, or dump the entire current output, and then Python will continue the eval. (Include in the warning, "Press Ctrl+?? to show <thing>.") 4. If the eval is waiting for user input, show the last page. (I don't know how the user could ask for more pages without sending something that the eval is allowed to interrupt as input. I don't understand terminal keyboard control that much.)
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
The former, yes: changing the behaviour of the interactive session, without the need for the programmer to change their code. To be clear, I'm asking whether that would meet your requirements :-)
I think one could make a `pprint` class for that.
Surem, that would be a good place for it. I think that's much more feasible than changing the behaviour of ‘repr’ for this purpose.
Again, it's only the interactive session which has the behaviour you want changed. That code run from a non-interactive session simply won't output anything if there's no error, so I think ‘repr’ is not a problem by default. It's only because the interactive session has the *additional* behaviour of emitting the representation of the object resulting from evaluation, that it has the behaviour you describe. So this is increasing the likelihood that the best way to address your request is not “change default ‘repr’ behaviour”, but “change the representation function the interactive session uses for its special object-representation output”. -- \ “The conflict between humanists and religionists has always | `\ been one between the torch of enlightenment and the chains of | _o__) enslavement.” —Wole Soyinka, 2014-08-10 | Ben Finney
On Apr 17, 2016 11:24 PM, "Ben Finney" <ben+python@benfinney.id.au> wrote:
Surem, that would be a good place for it. I think that's much more feasible than changing the behaviour of ‘repr’ for this purpose.
Huh? I never meant for any change to happen to `repr`. The desired behavior is, "Set default: Each command entered into the REPL should have limited output (without loss of capability for a newbie)." The proposal is to have the REPL determine when it's going to output too much. My example, as originally written, `print`d the huge object. I realized that this was too specific, and I didn't *really* care about `print` as such, so I changed it to (what I thought was) a more general example. I should've just had multiple examples.
On 18 April 2016 at 15:32, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
A few tips for folks that want to play with this: - setting sys.displayhook controls how evaluated objects are displayed in the default REPL - setting sys.excepthook does the same for exceptions - the PYTHONSTARTUP env var runs the given file when the default REPL starts So folks are already free to change their REPL to work however they want it to: set sys.displayhook and sys.excepthook from a PYTHONSTARTUP file Changing the *default* REPL behaviour is a very different question, and forks off in a couple of different directions: - improved defaults for teaching novices? Perhaps the default REPL isn't the best environment for that - easier debugging at the REPL? Perhaps pprint should gain an "install_displayhook()" option that overwrites sys.displayhook and optionally allows enabling of an output pager Since the more the REPL does, the more opportunities there are for it to break when debugging, having the output hooks be as simple as possible is quite desirable. However, making it easier to request something more sophisticated via the pprint module seems like a plausible approach to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Apr 18, 2016 3:25 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 18 April 2016 at 15:32, Franklin? Lee <leewangzhong+python@gmail.com>
wrote: limited output (without loss of capability for a newbie)." The proposal is to have the REPL determine when it's going to output too much.
A few tips for folks that want to play with this:
- setting sys.displayhook controls how evaluated objects are displayed in
the default REPL
- improved defaults for teaching novices? Perhaps the default REPL isn't
I don't want arguments like, "This can already be done, for yourself, if you really need it." I use IPython's shell, with maximum output height, and it was years ago that I used a terminal which I couldn't scroll. I want arguments like, "This will break my workflow." the best environment for that Why not? I imagine that self-taught novices will use the default REPL more than the advanced users.
On 04/18/2016 08:41 AM, Franklin? Lee wrote:
On Apr 18, 2016 3:25 AM, "Nick Coghlan" wrote:
How about: the default REPL is a basic tool, and the limitations of basic tools are what drive folks to seek out advanced tools. ? Or Marc-Andre's:
Or even Nick's (that you snipped):
IMO those are all good reasons to leave the basic REPL alone. -- ~Ethan~
"Franklin? Lee" <leewangzhong+python@gmail.com> writes:
That's your prerogative. This forum is for discussing whether ideas are right for changing Python, though. Arguments such as “This can already be done, for yourself, if you need it” are salient and sufficient, and not to be dismissed.
I want arguments like, "This will break my workflow."
You're free to solicit those arguments. You'll need to ackonwledge, though, that they are in addition to the quite sufficient argument of “already possible for people to get this in Python as is, if they want it”. -- \ “I think Western civilization is more enlightened precisely | `\ because we have learned how to ignore our religious leaders.” | _o__) —Bill Maher, 2003 | Ben Finney
On Mon, Apr 18, 2016 at 11:51 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
The REPL is a basic tool for basic users, which is why it should "Do the right thing" for people who wouldn't know better. I'm asking whether this is the "right thing" for those basic users: Advanced users are the ones who can use more than basic info.
I think it's a high priority not to lose information that was hidden. In my proposal, I said that there should be a way to access it, right after you cause it, which is very different from needing to enable a variable. If not for (memory) performance anxiety, I would've proposed that the info would be stored indefinitely, or that you would always be able to access the last three (or something) outputs/stacktraces, because I was concerned about typos causing exceptions that would then push the interesting stacktrace out. (Hey, that's another possible proposal: maybe syntax/lookup errors on the "R" part of "REPL" shouldn't count toward `sys.last_traceback`. Has anyone here ever needed to debug or reprint syntax errors coming from compiling the command they just entered?)
I read this as, "It'll be more complicated, so it MIGHT be buggy." This seems solvable via design, and doesn't speak to whether it's good, in principle, to limit output per input. If you still think it's a good reason, then I probably didn't understand it correctly. On Mon, Apr 18, 2016 at 2:39 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
When they are presented as a solution to "my" problem, they are, to be short, irrelevant. They try to address the speaker's need for the feature on their own machine, when I am asking for opinions on both the usefulness and harmfulness of such a feature, and the principle behind the feature. They are distracting: people are talking more about whether I can do it "myself" than how it's bad to have this, or how it wouldn't help who I want to help. Even if they weren't trying to solve "my" problem, I had non-advanced users in mind, and these solutions tend to be about what advanced users can do. I felt the need to point out what I'm looking for in the arguments, because people are telling me how *I* can have this feature by, for example, writing a display hook, or plugging into pprint.
On 19 April 2016 at 05:52, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
Basic users should probably be using a tool like IDLE, which has a bit more support for beginners than the raw REPL. I view the REPL as more important for intermediate or advanced users who want to quickly test out an idea (at least, that's *my* usage of the REPL). As I disagree with your statement "the REPL is a basic tool for basic users" I don't find your conclusions compelling, sorry. Paul
On Tue, Apr 19, 2016 at 11:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I was once a basic user, but I still have no idea what "IDLE" is. Does it come with python? I have tried $ idle $ python -m idle $ python -m IDLE $ python --idle To be honest, I do remember seeing a shortcut to IDLE in one of my Windows python installations, and I've seen it come up in discussions. However, it does not seem to me that IDLE is something that beginners would know to turn to. I do use IPython. IPython is nice---too bad it starts up slowly and is not recommended by default. -- Koos
On Tue, Apr 19, 2016 at 7:35 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
The first one will often work, but it depends on exactly how your Python installation has been set up. (Also, not all Linux distros come with Idle by default; you may have to install a python-idle package.) The most reliable way to invoke Idle is: $ python3 -m idlelib.idle (or python2 if you want 2.7). It's more verbose than would be ideal, and maybe it'd help to add one of your other attempts as an alias, but normally Idle will be made available in your GUI, which is where a lot of people will look for it. ChrisA
On 19 April 2016 at 10:35, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Windows, it's set up as a shortcut in the start menu alongside Python, with a tooltip "Launches IDLE, the interactive environment for Python 3.5". And Python scripts have an "Edit with IDLE" option on the right click menu. So it's where most Windows users would naturally look when searching for "the Python GUI". Discoverability of IDLE on other platforms isn't something I know much about, sorry. (I should also point out that I'm a relentless command line user on Windows, so my comments above about Windows are based on my experience watching what my "normal Windows user" colleagues do, not on my personal habits...) Paul
On 4/19/2016 5:35 AM, Koos Zevenhoven wrote:
I was once a basic user, but I still have no idea what "IDLE" is. Does it come with python?
Yes, unless explicitly omitted either in packaging or installation. (Some redistributors might put it with a separate tkinter/tix/idle/turtle/turtledemo package. The windows installer has a box that can be unchecked.)
I have tried
$ idle
This and idle3 works on some but not all systems.
$ python -m idle
python -m idlelib (or idlelib.idle -- required in 2.x)
$ python -m IDLE $ python --idle
To be honest, I do remember seeing a shortcut to IDLE in one of my Windows python installations,
Right. Once run, the IDLE icon can be pinned to the taskbar.
Yet many do, perhaps because instructors and books suggest it and tell how to start it up. -- Terry Jan Reedy
On Apr 19, 2016 4:09 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 19 April 2016 at 05:52, Franklin? Lee <leewangzhong+python@gmail.com>
wrote:
You say "should"? Do you mean that it is likely, or do you mean that it is what would happen in an ideal world? My college had CS students SSH into the department's Linux server to compile and run their code, and many teachers don't believe that students should start with fancy IDE featues like, er, syntax highlighting. You (and most regulars on this list) can adjust your shell to the way you like it, or use a more sophisticated shell, like IPython or bpython. On the other hand, changing shells and adding display hooks to site.py is not an option for those who don't know it's an option.
But that doesn't answer my question: would the proposed change hurt your workflow?
On 19 April 2016 at 10:36, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
I mean that that is what I hear people on lists like this saying as "a reasonable beginner environment". It means that the people I've introduced to Python (on Windows) have tended to end up using IDLE (either from the start menu or via "edit with IDLE" from the right click menu on a script). My experience (in business, environments) is that people expect an IDE when introduced to a new programming language, and IDLE, like it or not, is what is available out of the box with Python.
As a "scripting expert" and consultant, I typically get asked to knock up scripts on a variety of environments. I do not normally have what I'd describe as "my shell", just a series of basic "out of the box" prompts people expect me to work at. So no, the luxury of configuring the default experience is *not* something I typically have.
Yes. If I get a stack trace, I want it all. And if I print something out, I want to see it by default. The REPL for me is an investigative environment for seeing exactly what's going on. (Also, having the REPL behave differently depending on what version of Python I have would be a problem - backward compatibility applies here as much as anywhere else). Paul
On Tue, Apr 19, 2016 at 5:36 AM Franklin? Lee <leewangzhong+python@gmail.com> wrote:
That's probably because your professors thought you were more advanced than other new Pythonistas, because you were CS students. If I were in their shoes, I might chose a different approach depending on the level of the course.
But that doesn't answer my question: would the proposed change hurt your workflow?
It might. Would it affect doctests? Would it make indirect infinite recursion more difficult to trace? Would it make me remember yet another command line option or REPL option to turn on complete reprs? Would it force me to explain yet another config setting to new programmers? I think a beginner understands when they've printed something too big. I see this happen frequently. They laugh, shake their heads, and retype whatever they need to. If they're using IDLE, they say, "OMG I crashed it!" then they close the window or restart IDLE. I'd say it's more a problem in IDLE than in the default REPL.
On 4/19/2016 10:03 AM, Michael Selik wrote:
Recursion limit tracebacks with 2000 short lines (under 80 chars) are not a problem for tk's Text widget. Long lines, from printing something line 'a'*10000 or [1]*5000 make the widget sluggish. Too many or too long may require a restart.
I'd say it's more a problem in IDLE than in the default REPL.
Its a problem with using an underlying text widget optimized for managing 'sensible' length '\n' delimited lines. -- Terry Jan Reedy
On Apr 19, 2016 9:12 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
Implicit side effects from hidden code break that mental equivalence -
it's why effective uses of metaclasses, monkeypatching, and other techniques for deliberately introducing changed implicit behaviour often also involve introducing some kind of more local signal to help convey what is going on (such as a naming convention, or ensuring the altered behaviour is used consistently across the entire project). But a local signal _is_ part of my original proposal: """Instead, it will print a few lines ("... and approximately X more lines"), and tell you how to print more. (E.g. "Call '_more()' for more. Call '_full()' for full output.")""" """Again, there should be a message telling you how to get the full stacktrace printed. EXACTLY how, preferably in a way that is easy to type, so that a typo won't cause the trace to be lost. It should not use `sys.something()`, because the user's first few encounters with this message will result in, "NameError: name 'sys' is not defined".""" This also satisfies Terry Reedy's "reversibility" condition. On Apr 19, 2016 10:03 AM, "Michael Selik" <mike@selik.org> wrote:
On Tue, Apr 19, 2016 at 5:36 AM Franklin? Lee <
leewangzhong+python@gmail.com> wrote: than other new Pythonistas, because you were CS students. If I were in their shoes, I might chose a different approach depending on the level of the course. I meant to have two separate clauses: - In my college, there was a server for compiling and running code. It allowed people to do so without installing a compiler (e.g. on a general-purpose school computer). It also gave a standard compiler to test against. I did not learn Python in school. - Many teachers, all over the world, do not believe in IDE features for beginners. (Therefore, there will be many *students* which don't learn about IDLE.) You can want them to stop thinking that way, but the students will still be out there.
Let's be clear about what I am proposing. I prioritize not losing info and making it obvious how to get at that info. My suggestion was to show less output *and* have a message at the end saying how to get the output. It obviously should not affect doctests, since the output there is not for humans. An implementation that affects doctests should be considered buggy. Perhaps have an option to allow it, but affecting doctests by default would be an actual backward incompatibility, since it changes how existing code *runs* (i.e. not just output, but different logical paths during runtime). My idea for tracebacks of mutually recursive calls: See which functions are on the stack more than once, and if they keep appearing, pack them up. The algorithm should not pack up calls which happen only once within an apparent cycle, and it should be clear which order the calls come in (outside of folded mutually-recursive calls, of course). Example: File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutually recursive calls hidden: f (300), g (360)] File "<stdin>", line 1, in h File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutual-recursive calls hidden: f (103), g (200)] RuntimeError: maximum recursion depth exceeded [963 calls hidden. Call _full_ex() to print full trace.] Or maybe the second f and g are also folded into the second hidden bit. And maybe it checks line numbers when deciding whether to print explicitly (but not when folding). Would that output make indirect infinite recursion more difficult for you to debug?
I am not proposing this as a better error message, but because a flooded terminal is loss of information. It is also good to have significant info close together, to reduce effort (and thus mental cache) between processing of related ideas. Have you never needed to see the output of previous lines? Re-entering the earlier line might not work if the state of the program has changed. This happens at the beginner level.
On Wed, Apr 20, 2016, at 14:25, Franklin? Lee wrote:
You know what would make complicated infinite recursion easier to debug? The arguments. Is there a reliable way to determine what in f_locals correspond to arguments? My toy example below only works for named positional arguments. def magic(frame): code = frame.f_code fname = code.co_name argcount = code.co_argcount args = code.co_varnames[:argcount] values = tuple(frame.f_locals[a] for a in args) result = '%s%r' % (fname, values) if len(result) > 64: return fname + '(...)' return result Maybe even tokenize the source line the error occurred on and print any other locals whose name matches any token on the line. I guess I'll leave it to the bikeshed design committee and say: WIBNI tracebacks printed relevant frame variables FSDO relevant?
On Wed, Apr 20, 2016 at 03:01:48PM -0400, Random832 wrote:
You know what would make complicated infinite recursion easier to debug? The arguments.
Check out the cgitb module, which installs an except hook which (among many other things) prints the arguments to the functions. Run this sample code: import cgitb cgitb.enable(format='text') def spam(arg): if arg == 0: raise ValueError('+++ out of cheese error, redo from start +++') return spam(arg - 1) def eggs(x): return spam(x) + 1 def cheese(a, b, c): return a or b or eggs(c) cheese(0, None, 1) and you will see output something like the following. (For brevity I have compressed some of the output.) ValueError Python 3.3.0rc3: /usr/local/bin/python3.3 Thu Apr 21 10:58:22 2016 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /home/steve/python/<stdin> in <module>() /home/steve/python/<stdin> in cheese(a=0, b=None, c=2) /home/steve/python/<stdin> in eggs(x=2) /home/steve/python/<stdin> in spam(arg=2) /home/steve/python/<stdin> in spam(arg=1) /home/steve/python/<stdin> in spam(arg=0) ValueError: +++ out of cheese error, redo from start +++ __cause__ = None __class__ = <class 'ValueError'> [...] __traceback__ = <traceback object> args = ('+++ out of cheese error, redo from start +++',) with_traceback = <built-in method with_traceback of ValueError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in cheese File "<stdin>", line 2, in eggs File "<stdin>", line 4, in spam File "<stdin>", line 4, in spam File "<stdin>", line 3, in spam ValueError: +++ out of cheese error, redo from start +++ -- Steve
On 19 April 2016 at 14:52, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
No, that's not what it means. It relates to one of the ways experienced developers are able to debug code: by running it in their heads, and comparing the result their brain computed with what actually happened. When there's a discrepancy, either their expectation is wrong or the code is wrong, and they need to investigate further to figure out which it is. When folks say "Python fits my brain" this is often what they're talking about. Implicit side effects from hidden code break that mental equivalence - it's why effective uses of metaclasses, monkeypatching, and other techniques for deliberately introducing changed implicit behaviour often also involve introducing some kind of more local signal to help convey what is going on (such as a naming convention, or ensuring the altered behaviour is used consistently across the entire project). The default REPL behaviour is appropriate for this "somewhat experienced Pythonista tinkering with code to see how it behaves" use case - keeping the results very close to what they would be if you typed the same line of code into a text file and ran it that way. It's not necessarily the best way to *learn* those equivalences, but that's also not what it's designed for. IPython's REPL is tailored for a different audience - their primary audience is research scientists, and they want to be able to better eyeball calculation results, rather than lower level Python instance representations. As a result, it's much cleverer than the default REPL, but it's also aiming to tap into people's intuitions about the shape of their data and the expected outcomes of the operations they're performing on it, rather than their ability to mentally run Python code specifically. A REPL designed specifically for folks learning Python, like the one in the Mu editor, or the direction IDLE seems to be going, would likely be better off choosing different default settings for sys displayhook and sys.excepthook, but those changes would be best selected based on direct observations of classrooms and workshops, and noting where folks get confused or intimidated by the default settings. For environments other than IDLE, they can also be iterated on at a much higher rate than we make CPython releases. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Apr 19, 2016 at 11:12:16PM +1000, Nick Coghlan wrote:
I mostly agree with what you say, but I would like to see one change to the default sys.excepthook: large numbers of *identical* traceback lines (as you often get with recursion errors) should be collapsed. For example: py> sys.setrecursionlimit(20) py> fact(30) # obvious recursive factorial Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 3, in fact File "<stdin>", line 2, in fact RuntimeError: maximum recursion depth exceeded in comparison Try as I might, I just don't see the value of manually counting all those 'File "<stdin>", line 3, in fact' lines to find out where the recursive call failed :-) I think that it would be better if identical lines were collapsed, something like this: import sys import traceback from itertools import groupby TEMPLATE = " [...repeat previous line %d times...]\n" def collapse(seq, minimum, template=TEMPLATE): for key, group in groupby(seq): group = list(group) if len(group) < minimum: for item in group: yield item else: yield key yield template % (len(group)-1) def shortertb(*args): lines = traceback.format_exception(*args) sys.stderr.write(''.join(collapse(lines, 3))) sys.excepthook = shortertb which then gives tracebacks like this: py> sys.setrecursionlimit(200) py> a = fact(10000) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in fact [...repeat previous line 197 times...] File "<stdin>", line 2, in fact RuntimeError: maximum recursion depth exceeded in comparison -- Steve
On 4/19/2016 12:18 PM, Steven D'Aprano wrote:
Tracebacks produce *pairs* of lines: the location and the line itself. Replacing pairs with a count of repetitions would not lose information, and would make the info more visible. I would require at least, say, 3 repetitions before collapsing.
-- Terry Jan Reedy
On 4/20/2016 2:15 PM, Random832 wrote:
Aha. In IDLE's Shell, which generally simulates the interactive interpreter quite well, pairs are printed because interactive user input is exec'ed in a process running Python in normal batch mode. I expect any GUI shell to act similarly. I consider interactive interpreter omission of the line at fault to be a design buglet. For a single line of input, it is not much of a problem to look back up. But if one pastes a 20-line statement, finding line 13, for instance, is not so quick. And on Windows, a 1000 line traceback may erase the input so there is nothing to look at. -- Terry Jan Reedy
On Tue, Apr 19, 2016 at 06:28:40PM -0400, Terry Reedy wrote:
Only if the source code is available. The source isn't available for functions defined in the REPL, for C code, or for Python functions read from a .pyc file where the .py file is not available. The code I gave before works with pairs of location + source, without any change. If I move the definition of fact into a file, so that the source lines are included, we get: py> sys.setrecursionlimit(10) py> fact(30) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/steve/python/fact.py", line 3, in fact return n*fact(n-1) [...repeat previous line 7 times...] File "/home/steve/python/fact.py", line 2, in fact if n < 1: return 1 RuntimeError: maximum recursion depth exceeded in comparison I'd want to adjust the wording. Perhaps "previous entry" rather than previous line?
That's what my example does: it only collapses the line/(pair of lines) if there are at least three identical repetitions. -- Steve
On Wed, 20 Apr 2016 02:18:58 +1000, Steven D'Aprano wrote:
On Tue, Apr 19, 2016 at 11:12:16PM +1000, Nick Coghlan wrote:
That's a pretty powerful argument: running something in the REPL should give the same results as running it from a command line. When things fail differently in different environments, the environments themselves become suspects.
[...]
Yes, I see the smiley, but I would add specifically that the number of calls in the stack trace from a recursive function is rarely the important part. When I write something recursive, and get *that* stack trace, I don't have to scroll anywhere to look at anything to know that I blew the termination condition(s). (I'm agreeing with you: there is no value of counting the number of calls in the stack trace.) If I suddenly got tiny stack traces, I'd spend *more* time realizing what went wrong, until I retrained myself.
On 4/19/2016 9:12 AM, Nick Coghlan wrote:
The default REPL behavior should not throw away output. In the Windows console, what is not displayed cannot be retrieved. Reversible output editing is possible and appropriate in a GUI that can keep output separate from what is displayed.
There is a external Squeezer extension to IDLE that more or less does what this thread proposes and is also reversible. Deleted blocks are replaced by something that can be clicked on to expand the text. There is a proposal to incorporate Squeezer into IDLE. I have not reviewed the proposal yet because it would not solve any of my problems. I am more interested in way to put long output from help() into a separate text window instead of the shell.
-- Terry Jan Reedy
On Sun, 17 Apr 2016 21:04:11 -0400, Franklin? Lee wrote:
The existing pretty print module already does some of what you want. Instead of creating a new module, maybe extending that one would have greater benefits to more users.
Judging what others might find more friendly, or guessing what actions they are more likely to take, can be dangerous.
Thoughts? Even if the specific proposals are unworkable, is limiting REPL output (by default) a reasonable goal?
IMO, the default REPL should be as simple as possible, and respond as directly as possible to what I ask it to do. It should also be [highly] configurable for when you want more complexity, like summarizing stack traces and paginating output.
On Mon, Apr 18, 2016 at 2:33 AM, Dan Sommers <dan@tombstonezero.net> wrote:
There's also IPython's repr system as prior art: http://ipython.readthedocs.org/en/stable/api/generated/IPython.lib.pretty.ht... My impression (as I guess one of the few people who have tried to systematically add _repr_pretty_ callbacks to their libraries) is that it's only about 30% baked and has a number of flaws and limitations, but it's still a significant step beyond __repr__ or the stdlib pprint. (Some issues I've run into: there's something wonky in the line breaking that often produces misaligned lines; integration with __repr__ is awkward if you don't want to reimplement everything twice [1]; there's no built-in control for compressing output down to ellipses; handling the most common case of reprs that look like Foo(bar, baz, kwarg=1) is way more awkward than it needs to be [2]; and IIRC there were some awkward limitations in the formatting tools provided though I don't remember the details now. But despite all this it does make it easy to write composable reprs, so e.g. here's a syntax tree: In [1]: import patsy In [2]: patsy.parse_formula.parse_formula("y ~ 1 + (a * b)") Out[2]: ParseNode('~', Token('~', <Origin y ->~<- 1 + (a * b) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->y<- ~ 1 + (a * b) (0-1)>, extra='y'), []), ParseNode('+', Token('+', <Origin y ~ 1 ->+<- (a * b) (6-7)>), [ParseNode('ONE', Token('ONE', <Origin y ~ ->1<- + (a * b) (4-5)>, extra='1'), []), ParseNode('*', Token('*', <Origin y ~ 1 + (a ->*<- b) (11-12)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin y ~ 1 + (->a<- * b) (9-10)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin y ~ 1 + (a * ->b<-) (13-14)>, extra='b'), [])])])]) -n [1] https://github.com/pydata/patsy/blob/23ab37519276188c4ce09db03a1c40c5b9938bf... [2] https://github.com/pydata/patsy/blob/23ab37519276188c4ce09db03a1c40c5b9938bf... -- Nathaniel J. Smith -- https://vorpus.org
On 18.04.2016 03:04, Franklin? Lee wrote:
You should be able to write your own sys.displayhook to accomplish this.
Same here with sys.excepthook. FWIW: I see your point in certain situations, but don't think the defaults should be such that you have to enable some variable to see everything. This would make debugging harder than necessary, since often enough (following Murphy's law) the most interesting information would be hidden in some ellipsis. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 18 2016)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Franklin? Lee <leewangzhong+python@...> writes:
I don't know about the other ideas, but simplifying the output of recursion errors would be very useful. 990 repeats of the exact same lines add no value at all, the user being an expert or a beginner. Something like this is very clear in my mind: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/user/script.py", line 10, in f f() File "/home/user/script.py", line 10, in f f() ... File "/home/user/script.py", line 10, in f (994 repeats) f() ... File "/home/user/script.py", line 10, in f f() File "/home/user/script.py", line 10, in f f() RecursionError: maximum recursion depth exceeded
participants (16)
-
Ben Finney -
Chris Angelico -
Dan Sommers -
Ethan Furman -
Franklin? Lee -
Joseph Martinot-Lagarde -
Koos Zevenhoven -
M.-A. Lemburg -
Michael Selik -
Nathaniel Smith -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Random832 -
Steven D'Aprano -
Terry Reedy