Hi Mark, Thank you for the feedback. Let me address/elaborate some of the points that Pablo touched on.
PEP 657 proposes that locations for exceptions be treated as ranges, whereas tracing, profiling and debugging currently treat locations as points.
I don't think we're making strong claims that the full `(line, end_line, column, end_column)` should be the canonical representation for exception locations. The only concrete place we suggest their usage is in the printing of tracebacks. The information is not exposed in the exception or traceback objects intentionally as part of this. The place we make a reference to non-traceback tooling being able to use this information is coverage tools being able to perform expression-level granularity in coverage. As a quick example consider: x = True or f() might be marked as covered by a line coverage tool but this PEP potentially exposes extra information that might help show that the function call is not covered.
Consider this example: https://github.com/python/cpython/blob/main/Lib/test/test_compile.py#L861
And this example continues to work just as it does right now. There is no change to the tracing apis or the existing co_lines method. I think your concern is if tracing tools switched to using PEP 657 (which they are not obligated to), but even that case works out: co_lines() returns (6, 8, 4) for the CALL_METHOD. co_positions() from PEP 657 returns (4, 6, 5, 6) corresponding to (line, end_line, column, end_column) for the CALL_METHOD.
For example, how would one set a breakpoint on line 4 above?
Just as they would right now, they don't need to change how they set breakpoints.
PEP 657 claims it is fully backwards compatible, but it cannot be both backwards compatible and consistent.
I think there's a misunderstanding on what backwards compatibility means between us, can you concretely explain how this PEP or its implementation would break existing tools? I understand your concerns about having two potentially conflicting APIs for source locations but that is not a backwards compatibility problem.
1. Clarify, in detail, the impact on line-based tools like profilers, coverage.py and debuggers. This should include help on how to use the new APIs and where using the old APIs might result in behavioral changes.
As mentioned, we don't have an expectation for line-based tools to switch to the new API. Its primary consumer is meant to be the traceback mechanism. Usage of the old APIs will and must not lead to any behavioral changes.
2. Change the C API to a single function: int PyCode_Addr2Location(PyCodeObject *co, int addr, int *startline, int *startcolumn, int *endline, int *endcolumn)
Thank you, this is a great suggestion.
3. Drop the opt-out option. If the extra information is optional, then the compression scheme must allow for that; making the code more complex and potentially less efficient. Does opting out use the start of the range, or the old line, as the location?
In the future if a fantastic compression scheme is figured out that requires both the end lines and column information, I think it would be acceptable to make the opt-out only suppress the printing of the carets in the traceback while still maintaining the data in the code objects. This would still be backwards compatible.
4. Drop the limitation on column offsets. The data needs to be compressed anyway, so allowing arbitrary column offsets is effectively free.
Sure, I think there were plenty of good ideas thrown around compression and lazy-loading in the last PEP 657 thread so this is more of just a soft-limitation of the current implementation of the PEP. This limit can be increased in the future without any changes in the API or hurting backwards compatibility.
6. Store all location information in a single table (this applies more to the implementation than the PEP) Using four separate objects to hold the location info adds a lot of overhead for most functions.
I would just like to cap off and address this point together. The PEP is meant primarily to aid with debugging and improve tracebacks. The API is designed to be very limited and allow for a lot of room for expansion and optimization. It does not make strong prescriptive claims that tools must switch to the new richer information as your mail seems to suggest. The other smaller aspects like the internal storage formats, not storing data when opted out are concerns that can be addressed without making any breaking changes.