With CPython, tracebacks obtained from code written in C can be extremely clean compared with functionally equivalent code written in Python. Consider the following test file where I am using a local copy of Python's datetime.py module. ```py from local_datetime import date d = date(2021, 13, 1) ``` Executing this code results in the following, very "clean" traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) ValueError: month must be in 1..12 ``` A single line of code is shown, which is the one written by the end-user of this library; all code details from inside the library module are hidden. As it turns out, the datetime module imports (if it is available) the _datetime module written in C. If we comment out this import, so that the pure Python code is used instead, here is the new traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 847, in __new__ year, month, day = _check_date_fields(year, month, day) File "C:\Users\andre\local_datetime.py", line 422, in _check_date_fields raise ValueError('month must be in 1..12', month) ValueError: ('month must be in 1..12', 13) ``` In addition to the call by the end-user of that library, we see some inner code of that library that has no practical value to the end-user. We can suppress some lines of the traceback by replacing the line ```py year, month, day = _check_date_fields(year, month, day) ``` by ```py try: year, month, day = _check_date_fields(year, month, day) except Exception: _, _, tb = sys.exc_info() tb.tb_next = None raise ``` which simplifies the traceback somewhat: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 848, in __new__ year, month, day = _check_date_fields(year, month, day) ValueError: ('month must be in 1..12', 13) ``` but there is still one line of code from inside the library. === Idea: it would be nice if one could somehow define **callables** with a custom attribute that would indicate that tracebacks should not include frames "below" it. More explicitly, using the notation of the above example, imagine that we could write ```py date.__hide_tb__ = True ``` and that any call to `date()` that generate a traceback would not show frames below the calling frame, thus reproducing what we see when using the C code in this example. For library writers, this would be easily reverted when attempting to debug. For end users, this would result in much more readable tracebacks, unencumbered by extra lines of code that are outside of their control. André Roberge
pytest uses __tracebackhide__ https://doc.pytest.org/en/latest/example/simple.html#writing-well-integrated... Eg anyio sets __tracebackhide__ = __traceback_hide__ = True to remove internal frames from user Tracebacks On Sat, 29 May 2021, 19:21 André Roberge, <andre.roberge@gmail.com> wrote:
With CPython, tracebacks obtained from code written in C can be extremely clean compared with functionally equivalent code written in Python. Consider the following test file where I am using a local copy of Python's datetime.py module.
```py from local_datetime import date d = date(2021, 13, 1) ```
Executing this code results in the following, very "clean" traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) ValueError: month must be in 1..12 ```
A single line of code is shown, which is the one written by the end-user of this library; all code details from inside the library module are hidden.
As it turns out, the datetime module imports (if it is available) the _datetime module written in C. If we comment out this import, so that the pure Python code is used instead, here is the new traceback:
```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 847, in __new__ year, month, day = _check_date_fields(year, month, day) File "C:\Users\andre\local_datetime.py", line 422, in _check_date_fields raise ValueError('month must be in 1..12', month) ValueError: ('month must be in 1..12', 13) ```
In addition to the call by the end-user of that library, we see some inner code of that library that has no practical value to the end-user.
We can suppress some lines of the traceback by replacing the line
```py year, month, day = _check_date_fields(year, month, day) ``` by ```py try: year, month, day = _check_date_fields(year, month, day) except Exception: _, _, tb = sys.exc_info() tb.tb_next = None raise ```
which simplifies the traceback somewhat: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 848, in __new__ year, month, day = _check_date_fields(year, month, day) ValueError: ('month must be in 1..12', 13) ``` but there is still one line of code from inside the library. === Idea: it would be nice if one could somehow define **callables** with a custom attribute that would indicate that tracebacks should not include frames "below" it. More explicitly, using the notation of the above example, imagine that we could write
```py date.__hide_tb__ = True ``` and that any call to `date()` that generate a traceback would not show frames below the calling frame, thus reproducing what we see when using the C code in this example. For library writers, this would be easily reverted when attempting to debug. For end users, this would result in much more readable tracebacks, unencumbered by extra lines of code that are outside of their control.
André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WQN7QJ... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, May 29, 2021 at 3:25 PM Thomas Grainger <tagrain@gmail.com> wrote:
pytest uses __tracebackhide__
https://doc.pytest.org/en/latest/example/simple.html#writing-well-integrated...
Thanks for the reminder. Pytest takes care of traceback formatting for users. Individual projects can of course do that. But I want to do the reverse: I want to limit what users of my projects will see in a traceback, without having them specify something in their code. What I am interested in is having a project-agnostic standardised Python way to specify what to show to an end-user when an exception is raised.
Eg anyio sets __tracebackhide__ = __traceback_hide__ = True to remove internal frames from user Tracebacks
Do you have a specific link to this? I have looked in the documentation as well as in a few modules (_core/_exceptions.py in particular) without seeing any reference to this. I am not interested in custom formatting for testing, which is something comparatively easy to do.
https://github.com/agronholm/anyio/blob/9eb4671547b01f5e3ba0e0ca602b6aceec15... On Sat, 29 May 2021, 20:24 André Roberge, <andre.roberge@gmail.com> wrote:
On Sat, May 29, 2021 at 3:25 PM Thomas Grainger <tagrain@gmail.com> wrote:
pytest uses __tracebackhide__
https://doc.pytest.org/en/latest/example/simple.html#writing-well-integrated...
Thanks for the reminder. Pytest takes care of traceback formatting for users. Individual projects can of course do that. But I want to do the reverse: I want to limit what users of my projects will see in a traceback, without having them specify something in their code.
What I am interested in is having a project-agnostic standardised Python way to specify what to show to an end-user when an exception is raised.
Eg anyio sets __tracebackhide__ = __traceback_hide__ = True to remove internal frames from user Tracebacks
Do you have a specific link to this? I have looked in the documentation as well as in a few modules (_core/_exceptions.py in particular) without seeing any reference to this. I am not interested in custom formatting for testing, which is something comparatively easy to do.
You can control what the traceback of exceptions you are emitting: "raise e.with_traceback(None)" should clear everything before the current frame. Or you can get clever and construct a traceback with only the frames you want. On Saturday, May 29, 2021, 08:27:18 PM GMT+1, André Roberge <andre.roberge@gmail.com> wrote: What I am interested in is having a project-agnostic standardised Python way to specify what to show to an end-user when an exception is raised.
On Sat, May 29, 2021 at 4:54 PM Irit Katriel <iritkatriel@yahoo.com> wrote:
You can control what the traceback of exceptions you are emitting:
"raise e.with_traceback(None)" should clear everything before the current frame.
I'm sorry, but I still don't see how. This particular line would still show up. Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 852, in __new__ raise e.with_traceback(None) ValueError: ('month must be in 1..12', 13) Sorry if I am missing something obvious. In the example I gave, as a user calling date() [from a library], I would only want to see that line show up.
Or you can get clever and construct a traceback with only the frames you want.
On Saturday, May 29, 2021, 08:27:18 PM GMT+1, André Roberge < andre.roberge@gmail.com> wrote:
What I am interested in is having a project-agnostic standardised Python way to specify what to show to an end-user when an exception is raised.
Hello, On Sat, 29 May 2021 17:29:26 -0300 André Roberge <andre.roberge@gmail.com> wrote:
On Sat, May 29, 2021 at 4:54 PM Irit Katriel <iritkatriel@yahoo.com> wrote:
You can control what the traceback of exceptions you are emitting:
"raise e.with_traceback(None)" should clear everything before the current frame.
I'm sorry, but I still don't see how. This particular line would still show up.
Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 852, in __new__ raise e.with_traceback(None) ValueError: ('month must be in 1..12', 13)
Sorry if I am missing something obvious. In the example I gave, as a user calling date() [from a library], I would only want to see that line show up.
It would be *very* confusing to see something like:
File "some/where/deep.py", line 2123, in <module> foo = bar(x, *y) ValueError: ('month must be in 1..12', 13)
You can start your stopwatch and see how much time it will take you to figure out that "bar" is a variable referencing "date" and sequence "y" has "month" somewhere in it. That already happens in CPython in some places, and it's quite confusing when you hit it: https://mail.python.org/archives/list/python-ideas@python.org/thread/MINLM7B... On the topic of "functions implemented in C provide clean tracebacks", on a couple of occasions, I found them to be terse up to being obfuscated. I really consider one sweet day to be able to implement feature to bring them closer to Python level of user-friendliness, e.g.
Traceback (most recent call last): File "foo.py", line 2, in test builtin_func(a) File "~/projects/foo_module/src1.c", line 34, in builtin_func_impl ValueError("wrong phase of moon")
So it was possible to debug code across C extensions boundary more easily (instead of grepping loads of C code for confusing/generic error messages guessing which one was actually hit). Oh, and btw CPython importlib also implements some traceback obfuscation: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f797... Besides user satisfaction ("clean tracebacks"), I guess it contributes largely to the fact that only a few people know how Python import machinery works and are able to customize it. (At least, I found it pretty hard to work with - you literally need to go thru the entire importlib source code and remove those pesky _call_with_frames_removed() calls, and only then you start to get a picture what breaks where and how it all works together).
Or you can get clever and construct a traceback with only the frames you want.
On Saturday, May 29, 2021, 08:27:18 PM GMT+1, André Roberge < andre.roberge@gmail.com> wrote:
What I am interested in is having a project-agnostic standardised Python way to specify what to show to an end-user when an exception is raised.
-- Best regards, Paul mailto:pmiscml@gmail.com
See this issue: https://bugs.python.org/issue31299 On Saturday, May 29, 2021, 07:19:32 PM GMT+1, André Roberge <andre.roberge@gmail.com> wrote: With CPython, tracebacks obtained from code written in C can be extremely clean compared with functionally equivalent code written in Python. Consider the following test file where I am using a local copy of Python's datetime.py module. ```pyfrom local_datetime import date d = date(2021, 13, 1) ``` Executing this code results in the following, very "clean" traceback:```pytbTraceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) ValueError: month must be in 1..12 ``` A single line of code is shown, which is the one written by the end-user of this library; all code details from inside the library module are hidden. As it turns out, the datetime module imports (if it is available) the _datetime module written in C. If we comment out this import, so that the pure Python code is used instead,here is the new traceback: ```pytbTraceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 847, in __new__ year, month, day = _check_date_fields(year, month, day) File "C:\Users\andre\local_datetime.py", line 422, in _check_date_fields raise ValueError('month must be in 1..12', month) ValueError: ('month must be in 1..12', 13) ``` In addition to the call by the end-user of that library, we see some inner code of that library that has no practical value to the end-user. We can suppress some lines of the traceback by replacing the line ```pyyear, month, day = _check_date_fields(year, month, day) ```by```pytry: year, month, day = _check_date_fields(year, month, day) except Exception: _, _, tb = sys.exc_info() tb.tb_next = None raise ``` which simplifies the traceback somewhat:```pytbTraceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 848, in __new__ year, month, day = _check_date_fields(year, month, day) ValueError: ('month must be in 1..12', 13) ```but there is still one line of code from inside the library.===Idea: it would be nice if one could somehow define **callables** with a custom attribute thatwould indicate that tracebacks should not include frames "below" it.More explicitly, using the notation of the above example, imagine that we could write ```pydate.__hide_tb__ = True```and that any call to `date()` that generate a traceback would not show frames below thecalling frame, thus reproducing what we see when using the C code in this example.For library writers, this would be easily reverted when attempting to debug.For end users, this would result in much more readable tracebacks, unencumbered by extra lines of code that are outside of their control. André Roberge _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WQN7QJ... Code of Conduct: http://python.org/psf/codeofconduct/
See this issue: https://bugs.python.org/issue31299
This issue refers to https://bugs.python.org/issue16217 which talks about
On Sat, May 29, 2021 at 3:38 PM Irit Katriel <iritkatriel@yahoo.com> wrote: lines "above" the one of interest (compared with those that I describe as being "below"). It also talks about filtering some specific modules -- something I already do in one of my projects. Still, interesting to see; thanks for the link.
On Saturday, May 29, 2021, 07:19:32 PM GMT+1, André Roberge < andre.roberge@gmail.com> wrote:
With CPython, tracebacks obtained from code written in C can be extremely clean compared with functionally equivalent code written in Python. Consider the following test file where I am using a local copy of Python's datetime.py module.
```py from local_datetime import date d = date(2021, 13, 1) ```
Executing this code results in the following, very "clean" traceback: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) ValueError: month must be in 1..12 ```
A single line of code is shown, which is the one written by the end-user of this library; all code details from inside the library module are hidden.
As it turns out, the datetime module imports (if it is available) the _datetime module written in C. If we comment out this import, so that the pure Python code is used instead, here is the new traceback:
```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 847, in __new__ year, month, day = _check_date_fields(year, month, day) File "C:\Users\andre\local_datetime.py", line 422, in _check_date_fields raise ValueError('month must be in 1..12', month) ValueError: ('month must be in 1..12', 13) ```
In addition to the call by the end-user of that library, we see some inner code of that library that has no practical value to the end-user.
We can suppress some lines of the traceback by replacing the line
```py year, month, day = _check_date_fields(year, month, day) ``` by ```py try: year, month, day = _check_date_fields(year, month, day) except Exception: _, _, tb = sys.exc_info() tb.tb_next = None raise ```
which simplifies the traceback somewhat: ```pytb Traceback (most recent call last): File "test.py", line 2, in <module> d = date(2021, 13, 1) File "C:\Users\andre\local_datetime.py", line 848, in __new__ year, month, day = _check_date_fields(year, month, day) ValueError: ('month must be in 1..12', 13) ``` but there is still one line of code from inside the library. === Idea: it would be nice if one could somehow define **callables** with a custom attribute that would indicate that tracebacks should not include frames "below" it. More explicitly, using the notation of the above example, imagine that we could write
```py date.__hide_tb__ = True ``` and that any call to `date()` that generate a traceback would not show frames below the calling frame, thus reproducing what we see when using the C code in this example. For library writers, this would be easily reverted when attempting to debug. For end users, this would result in much more readable tracebacks, unencumbered by extra lines of code that are outside of their control.
André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WQN7QJ... Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
André Roberge -
Irit Katriel -
Paul Sokolovsky -
Thomas Grainger