On Tue, Feb 11, 2020 at 5:19 AM Andrew Barnert via Python-ideas wrote:

Meanwhile, for the implementation side, I don’t think there’s an issue with how traceback passes data around, but with what’s available in the first place. Look at what gets stored as line number information in a compiled code object. (You don’t want to read the whole details on how the lnotab is built or anything; just `dis.dis` some code.) Given the offset of an opcode, is there a correct algorithm to get the range of line numbers for the full expression that opcode is part of? If not, the only way traceback could do it would be re-parsing the body of the scope, which I don’t think you’d want to do. (At least not in general—in some applications it might be useful, and a PyPI library that hooked trackbacks to do that plus adding all kinds of useful heuristics might be nifty.)

Lemme put a slightly different spin on that. The traceback currently has the line number from the compiled code, and while it might be a bit of extra work to figure out exactly when to expand, this could be done entirely in userspace (eg as an excepthook), and could be an incredibly useful library. It wouldn't need any language changes, and could be published on PyPI as a straight-forward way to improve tracebacks in large applications. (I don't think this would be useful in the general case (there's a lot of times when I'd rather have a compact traceback than the expanded one), but you could easily activate this in any sort of application where it'd be helpful.) ChrisA

There are already several libraries which improve tracebacks in various ways, including the standard library module `cgitb`. These libraries often simply include multiple lines surrounding the primary one, which in most cases will give you the context you need and maybe some extra. I recently wrote a library [`stack_data`](https://github.com/alexmojaki/stack_data) which offers a generic way to get data to format tracebacks in your own way and thus support such libraries. I have pending PRs to integrate it into [IPython](https://github.com/ipython/ipython/pull/11886) and [stackprinter](https://github.com/cknd/stackprinter/pull/23). stack_data has a more strategic method of getting contextual lines. It parses the AST of a file and splits the lines up into *pieces*. Quoting my own docs:

A piece is a range of one or more lines in a file that should logically be grouped together. A piece contains either a single simple statement or a part of a compound statement (loops, if, try/except, etc) that doesn't contain any other statements. Most pieces are a single line, but a multi-line statement or if condition is a single piece.

So it's not quite what you want, since it will show a whole statement instead of just an expression, but it's pretty close and sometimes will even be preferable. If there's demand I could write a dependency-free version whose only job is get the lines of the main piece, and that could easily be merged into the traceback module. I haven't proposed it yet because I thought it would be better and easier to first prove the concept in libraries like IPython. Narrowing things down to the expression requires dark magic that analyses the bytecode such as [`executing`](https://github.com/alexmojaki/executing). This is actually fine, the hard work has been done already, but it will only work in some cases and I'm pretty sure it's never going to be in the standard library. FWIW I also think this would be a great improvement to tracebacks. Only seeing a single line is annoying.

if you expand _every_ step into multiple lines, that annoying 99-line deep traceback where you only care about lines 97-98 becomes an even more annoying 213-line deep trackback

Generally, no. 1. Most tracebacks are not that long to begin with in my experience. 2. I just did an analysis of the pieces in a bunch of real Python files. 86% of pieces are just one line. There are some spectacular outliers, so the maximum length of a piece needs to be capped. If we cap pieces at 6 lines, the mean number of lines is about 1.35. If we cap at 10, the mean is about 1.46. 3. Since half the lines in a traceback are filenames etc. instead of source lines, the length of a traceback would only increase by 23% (assuming that cap of 10). Also, scrolling vertically is easy, and filenames tend to have visual patterns that make it easy to find the part of a traceback you want.