Re: Inadequate error reporting during function call setup stage
On 21/02/2021 23:06, Terry Reedy wrote:
The way I understand it's not about print(), it's about disambiguating multiple functions with the same name. Example: PS > type .\ambiguous_names.py import random def do_stuff(): pass f = do_stuff def do_stuff(a, b): pass g = do_stuff random.choice([f, g])(42) PS > py .\ambiguous_names.py Traceback (most recent call last): File "...\ambiguous_names.py", line 13, in <module> random.choice([f, g])(42) TypeError: do_stuff() missing 1 required positional argument: 'b' The traceback gives no clue which of the two do_stuff() functions caused the error, you have to check both implementations. If that is a comman problem one might consider including module name and co_firstlineno in the message, or at least adding the relevant do_stuff() function to the exception's args.
Hello, On Mon, 22 Feb 2021 10:44:19 +0100 Peter Otten <__peter__@web.de> wrote:
Thanks, that's exactly what I meant, and a repro with random "roulette" is also what I had in mind, I just didn't get to it yet ;-).
As my original message argues, that's a workaround. Python tracebacks already have places where they show source file and line number - namely, the individual traceback entries. So, instead of cramming that info into the exception message, there should be additional last (latest in the order of execution) traceback entry, pointing to the exact function which had parameter mismatch. As I mentioned, I implemented that in my Python dialect, which happened to have exactly the same problem (code is not based on CPython). It looks like: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> File ".../xforms.py", line 25, in print TypeError: unexpected keyword argument 'noann' - that makes clear that it's "print" function of "xforms.py" module, line 25, which got an unexpected keyword argument. -- Best regards, Paul mailto:pmiscml@gmail.com
You are proposing to fake a stack frame that I have to know is not a stack frame but is in fact the location of the function in the exception? I'm -1 on that as its confusing. Having checked that its python code and not a C extension function you could use the info in fn.__code__ to get the filename and line of where the function is defined and put that info into the exception. Example of the info: | >>> os.path.join.__code__ <code object join at 0x7fb19aebc7c0, file "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/posixpath.py", line 71> I use repr(fn.__code__) a lot when debugging complex code. Barry
Hello, On Mon, 22 Feb 2021 19:47:04 +0000 Barry Scott <barry@barrys-emacs.org> wrote:
No, I'm proposing to stop faking lack of the last stack frame due to CPython's implementation details. See the original message for more info.
Could use crystal ball, even.
-- Best regards, Paul mailto:pmiscml@gmail.com
On 2/22/21 3:06 PM, Paul Sokolovsky wrote:
I'm trying to understand what last stack frame (or lack of a last stack frame, or the faking of a lack of a last stack frame) you are talking about. Your original message shows some tracebacks, but doesn't have the code that produced them. It's hard to understand what you are referring to. --Ned.
My take on this is that it's actually a valid suggestion, albeit wrapped in some unhelpful language. This is a more common example, in my opinion: === fileA.py === class MyClass: def do_thing(self, arg_a, arg_b): pass === fileB.py === ... class MySubClass(MyClass): pass === main.py === ... MySubclass().do_thing(1) If you run the above (with trivial imports filled in etc.) Then the traceback gives: Traceback (most recent call last): File "c.py", line 3, in <module> B().do_thing() TypeError: do_thing() takes exactly 3 arguments (1 given) So Python has identified that a function 'do_thing' is being called incorrectly, but /where/ is do_thing defined? This problem gets much harder if there are multiple definitions of 'do_thing' in the codebase, as alluded to in the original mail. Paul's suggestion is that python should add the source location of the function being called into the error message somewhere. The "it is/isn't a stack frame" issue is one possible solution, and comes down to a potential *semantic only* disagreement on how functions are called with python. If you take traditional runtime-less languages, then the responsibility for unpacking and handling arguments is considered part of the function being called. With C, for example, typically the compiler inserts a header (can't remember the better term for this right now) at the start of a function, which moves the function arguments around as needed, so any failure match the arguments originates from within the called function (this is even more obvious with VA_ARGS for example). Obviously, in languages with a runtime, such as python, the responsibility for matching arguments with function signatures can sit outside of the runtime function call context, and runtime-internal machinery information is not typically included in stack-traces. Therefore, it appears (although I don't think this was part of any specific plan?) that the error originates from the call site (where the function is being called from) because this is the last point that was executing any python code. So, the suggestion, as I understand it, is that a small conceptual shift is made in how we think about the mechanics of calling a function within python, such that the assignment and binding of function arguments are conceptually performed 'inside' the function being called (I assume you could equate this to the function 'def' line being executed). If we do this, then it becomes easy to consider the TypeError as originating from within the called function, and therefore python will print the exact location of the function that's being called *as well as* the location of the call site (in the frame above). Internally, the python runtime would have to go out of its way to enter this 'fake' frame before generating the exception as, in reality the frame isn't entered (as I understand it) until after the arguments have been assigned correctly to the signature. But this seems like it should be relatively simple and non-invasive to do. The above makes sense to me, apologies if it's not very understandable to others. Thanks Steve On Mon, Feb 22, 2021 at 10:39 PM Ned Batchelder <ned@nedbatchelder.com> wrote:
On Tue, Feb 23, 2021 at 10:10 AM Stestagg <stestagg@gmail.com> wrote:
So Python has identified that a function 'do_thing' is being called incorrectly, but /where/ is do_thing defined? This problem gets much harder if there are multiple definitions of 'do_thing' in the codebase, as alluded to in the original mail.
Paul's suggestion is that python should add the source location of the function being called into the error message somewhere.
Thank you for explaining. I'm glad I wasn't the only one confused by the original post :) I think this is a nice-to-have, rather than being a serious bug to be fixed. When you get an error about a function call, it's entirely possible that the target function is the problem, or that the wrong function is being referenced; but you can get similar problems with all kinds of mismatches (like "a, b, c = thing" and getting a ValueError - maybe you'd need to check the line where 'thing' originated). If this can be done easily, great, but otherwise it might be the domain of traceback enhancement tools rather than the core language. Does execution ever (in the normal case) actually hit the 'def' line? If so, I wouldn't be averse to having a traceback line mentioning it. But if (as I suspect) tracing successful execution wouldn't hit that line, then it doesn't seem right to feign execution of it. My understanding of the def statement is that it runs ONLY when the function is defined; when the function's called, you go straight into the body. ChrisA
Hello, On Tue, 23 Feb 2021 12:31:37 +1100 Chris Angelico <rosuav@gmail.com> wrote:
It's not a serious bug, it's "astonishing overlook". Astonishing, as it was there for so many years, unpatched. And of course, it's rare enough. But when it happens, you're in the maze. I spend much time communicating with Python critics (well, haters), and sometimes read quite funny criticisms like: error reporting (stacktraces) in Python is poor. I always considered that "random last resort nitpick", as error reporting in Python has always been much better than in other scripting languages (I guess by now everyone caught up). But now imagine error like that happens in some 3rd-party library. Which happened to do some first-class function/higher order programming. With functions from another 3rd-party library. Which released a new version with "slightly updated" API. All that happening in production. Someone who ever caught such (or similar) case would forever get imprint of "Python has poor error reporting". Because just imagine what you report to the maintainer of the first 3rd-party lib: "Your lib calls something wrong in file X line Y" - "Ok, let's look into that. What does it call?" - "Umm, I don't know, something called 'foo'" (or "<lambda>").
Good point. But right analogy here is: suppose error happens in "a, b, c = thing". But Python doesn't show that line in the backtrace, it stops 1 entry before it. Go figure now. Because that's the right analogy of what happens with a function - in the function "prolog", there's a series of assignments: "formal_paramX = actual_argX". Conceptually, those *are* in the function context (or you would not be able to reference function's formal params).
If this can be done easily, great, but otherwise it might be the domain of traceback enhancement tools rather than the core language.
For Pycopy it was easy. I will look into CPython one of these weeks.
For another analogy, you can look at how native functions are defined and where they check params, and from where exceptions originate there (by obvious reasons, Python backtraces don't show C source code function/linenos ;-)).
ChrisA
[] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Feb 23, 2021 at 5:33 PM Paul Sokolovsky <pmiscml@gmail.com> wrote:
But they aren't actual lines of code. Your analogy breaks down because you're trying to put a line into a backtrace for a mythical and conceptual action that happens as part of the function call. That's why I dispute that this is an "astonishing oversight". It would be nice to have the extra information, but given that Python and Python programmers have survived for thirty years without it, I don't think it's nearly as serious as you're implying.
Yes, or you could look at CPython byte code and how it does a bunch of stack operations, but some of them don't actually happen. Looking at the implementation, especially in a completely different language, doesn't really help here :) The way I see it, this is perfect as an "additional information" field. For instance, when I compile certain buggy C programs, gcc tells me things like this: demo.c: In function ‘main’: demo.c:4:9: warning: passing argument 1 of ‘printf’ makes pointer from integer without a cast [-Wint-conversion] printf(12345); ^~~~~ In file included from demo.c:1: /usr/include/stdio.h:332:43: note: expected ‘const char * restrict’ but argument is of type ‘int’ extern int printf (const char *__restrict __format, ...); ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ It starts by reporting the error, and then says "note: the thing you're calling came from over here". It's separate from the error itself, but provides extremely useful information. Python doesn't currently have a standardized way to report this, but IMO that would be a far better way than synthesizing a fake backtrace entry. Taking your original example, it might look something like this: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() got an unexpected keyword argument 'noann' Note: The function is defined here: File "whatever.py", line 123, in <module> def print(thingy, thongy, whop): ChrisA
Hello, On Tue, 23 Feb 2021 17:52:20 +1100 Chris Angelico <rosuav@gmail.com> wrote:
But that's the whole point - the externally visible behavior should correspond to the conceptual model, even if a particular implementation does something different underlyingly. So again, it's CPython's implementation detail that, for bytecode functions only, it implements: def foo(params): # foo takes care of its params ... foo(args) as: param_check_and_call(foo, __param_meta__[foo], *args) And due to this implementation detail raises exceptions from param_check_and_call(). []
The way I see it, this is perfect as an "additional information" field.
I started there, and proceeded to where no "additional information" hacks are needed, and everything fits well into the existing framework. Come along with me. [] -- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Tue, 23 Feb 2021 17:52:20 +1100 Chris Angelico <rosuav@gmail.com> wrote: []
Ah, missed to reply to that one. Can judge it by myself: that's definitely not the first time I see that issue, where there was "something wrong" when getting exceptions when dealing with first-class function values. But I didn't even have a good understanding of what's wrong, and cases were either simple and obvious, or I reduced to print debugging. But now I'm working on a project where I consider UX to be important. And I consider a good UX for Python project not when it catch-all's exceptions and obfuscates the problem with a generic error message, and likewise not when there're 5 screenfuls of chained exceptions. I consider it's when there's human-sized backtrace, clearly pointing at the problem. And for that case, I figured that the backtrace is senseless and doesn't give a user of my program enough information to understand the issue. So, I stopped with writing my program, sat and scratched my head to understand what's wrong, then sat again to try a few different solutions (the original post should mirror that), until finally came to one which makes sense to the end user, and doesn't require any crutches. I won't be surprised if that's how other Python programmers dealt with it over decades - some didn't even dig into "what's wrong", "something is wrong" was enough for them to do in-mind-debugging. The other gritted teeth and recursed to print debugging, again without thinking too much what's wrong on the meta-level. And the remaining? Well, they now curse Python for poor error reporting on reddits and hackernewses. You can also see that on this very thread - at first, the issue was at all unclear (even though I tried to write a detailed message, packing different sides into it). Now issue becomes clear, but (besides the usual "never heard" response), the thinking is along the lines of "additional info hacks", which was my first motion either. Let's see where that leads us. (And that's why I posted to the mailing list first - chances, a bugs.p.o report would be closed with "discuss first on the mailing list" suggestion). [] -- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Mon, 22 Feb 2021 17:32:23 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
Well, if you looked at that stack trace, you saw the code: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() got an unexpected keyword argument 'noann' So yes, the code is: --- first_class_function_value(func, **pass_params) --- That's the "calling code". And the rest of the code? But's that's the whole point, that the current CPython's error reporting doesn't tell me where it is! It just tells me the function name, and go make a full-text search thru the entire sys.path to find its location, especially if there're many functions of that name. -- Best regards, Paul mailto:pmiscml@gmail.com
On 2/23/21 12:56 AM, Paul Sokolovsky wrote:
Perhaps instead of fiddling with stack frames, we can solve this problem in a more straightforward way. The error message has a description of the function: "print()". What if we extend that description: "print() at path/to/myprog.py:173". Then the existing stack traces would have the information you want, without having to debate the meaning of stack frames, and where execution happens. --Ned.
Hello, On Tue, 23 Feb 2021 11:52:57 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
That's what I implemented initially, saw this: File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() at path/to/myprog.py:173 got an unexpected keyword argument 'noann' and wondered - why am I messing with the error *message*, if above it is a backtrace entry, which naturally shows the location information ("File "pseudoc_tool.py", line 91"). Then all things clicked into their places. In my implementation, I didn't need to "fiddle with stack frames", I just needed to pre-populate the exception object with a first (shown last) backtrace entry for the called function, and let the usual processing append to it.
Without getting to the root cause, there will be only workarounds of different level of ugliness. For example, any reasonable JIT would implement proper semantics - param checking *inside* functions (just like the C code does now).
--Ned.
-- Best regards, Paul mailto:pmiscml@gmail.com
On Wed, Feb 24, 2021 at 4:59 AM Paul Sokolovsky <pmiscml@gmail.com> wrote:
Why do you assume that this is "proper semantics"? I'm confused. What is it that makes function parameter matching inherently part of the body of the function? For one thing, that notion is semantically incompatible with multiple dispatch, where matching parameter numbers or types determines which function is even going to be called. ChrisA
Hello, On Wed, 24 Feb 2021 05:01:37 +1100 Chris Angelico <rosuav@gmail.com> wrote:
The same what causes C-level function implementations to do parameter checking themselves. There's simply nowhere else to put that checking, and of course nobody calls C functions via an intermediary - that's hilariously inefficient. All those points apply to JITted functions too (you JIT to make it fast, then putting a slow bottleneck inbetween, which will also confuse CPU branch predictors - doesn't make sense).
Python doesn't support multiple dispatch. Majority of languages don't support multiple dispatch. For a well known reason - its benefits vs its inefficiency ratio is very low (inefficiency is high). So, whichever runtime system implements multiple dispatch, will need to think about its issues, including aspects of error reporting, I don't see how that's relevant to Python discussion. (Multiple dispatch is also effectively a poorman's, implicit pattern matching which works only between code blocks factored out into functions.) [] -- Best regards, Paul mailto:pmiscml@gmail.com
On 2/23/21 12:55 PM, Paul Sokolovsky wrote:
I don't think you are helping your cause by talking down to people. "any reasonable JIT" and "proper semantics" are judgemental without adding anything to the discussion. You wanted to know what function caused the problem. I proposed a simple solution that gave you precisely the information you wanted. But you've called it an ugly workaround for some reason. --Ned.
Hello, On Tue, 23 Feb 2021 13:09:39 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
Well, keeping repeating "we can put location info into the error message", disregarding an argument that *backtrace entries* are for location info - that should be classified as "judgemental" too then?
Because I saw them both (by implementing them both), and even in the order you suggest, "location info in the message" first, felt not satisfied, spent more time to implement the alternative, and yes, that looks both more naturally and more clearly. So, please don't be judgemental to my suggestion, but consider them both, not just first which came to your mind (which is also exactly the one which came to my mind first, peace).
--Ned.
-- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Mon, 22 Feb 2021 10:44:19 +0100 Peter Otten <__peter__@web.de> wrote:
Thanks, that's exactly what I meant, and a repro with random "roulette" is also what I had in mind, I just didn't get to it yet ;-).
As my original message argues, that's a workaround. Python tracebacks already have places where they show source file and line number - namely, the individual traceback entries. So, instead of cramming that info into the exception message, there should be additional last (latest in the order of execution) traceback entry, pointing to the exact function which had parameter mismatch. As I mentioned, I implemented that in my Python dialect, which happened to have exactly the same problem (code is not based on CPython). It looks like: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> File ".../xforms.py", line 25, in print TypeError: unexpected keyword argument 'noann' - that makes clear that it's "print" function of "xforms.py" module, line 25, which got an unexpected keyword argument. -- Best regards, Paul mailto:pmiscml@gmail.com
You are proposing to fake a stack frame that I have to know is not a stack frame but is in fact the location of the function in the exception? I'm -1 on that as its confusing. Having checked that its python code and not a C extension function you could use the info in fn.__code__ to get the filename and line of where the function is defined and put that info into the exception. Example of the info: | >>> os.path.join.__code__ <code object join at 0x7fb19aebc7c0, file "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/posixpath.py", line 71> I use repr(fn.__code__) a lot when debugging complex code. Barry
Hello, On Mon, 22 Feb 2021 19:47:04 +0000 Barry Scott <barry@barrys-emacs.org> wrote:
No, I'm proposing to stop faking lack of the last stack frame due to CPython's implementation details. See the original message for more info.
Could use crystal ball, even.
-- Best regards, Paul mailto:pmiscml@gmail.com
On 2/22/21 3:06 PM, Paul Sokolovsky wrote:
I'm trying to understand what last stack frame (or lack of a last stack frame, or the faking of a lack of a last stack frame) you are talking about. Your original message shows some tracebacks, but doesn't have the code that produced them. It's hard to understand what you are referring to. --Ned.
My take on this is that it's actually a valid suggestion, albeit wrapped in some unhelpful language. This is a more common example, in my opinion: === fileA.py === class MyClass: def do_thing(self, arg_a, arg_b): pass === fileB.py === ... class MySubClass(MyClass): pass === main.py === ... MySubclass().do_thing(1) If you run the above (with trivial imports filled in etc.) Then the traceback gives: Traceback (most recent call last): File "c.py", line 3, in <module> B().do_thing() TypeError: do_thing() takes exactly 3 arguments (1 given) So Python has identified that a function 'do_thing' is being called incorrectly, but /where/ is do_thing defined? This problem gets much harder if there are multiple definitions of 'do_thing' in the codebase, as alluded to in the original mail. Paul's suggestion is that python should add the source location of the function being called into the error message somewhere. The "it is/isn't a stack frame" issue is one possible solution, and comes down to a potential *semantic only* disagreement on how functions are called with python. If you take traditional runtime-less languages, then the responsibility for unpacking and handling arguments is considered part of the function being called. With C, for example, typically the compiler inserts a header (can't remember the better term for this right now) at the start of a function, which moves the function arguments around as needed, so any failure match the arguments originates from within the called function (this is even more obvious with VA_ARGS for example). Obviously, in languages with a runtime, such as python, the responsibility for matching arguments with function signatures can sit outside of the runtime function call context, and runtime-internal machinery information is not typically included in stack-traces. Therefore, it appears (although I don't think this was part of any specific plan?) that the error originates from the call site (where the function is being called from) because this is the last point that was executing any python code. So, the suggestion, as I understand it, is that a small conceptual shift is made in how we think about the mechanics of calling a function within python, such that the assignment and binding of function arguments are conceptually performed 'inside' the function being called (I assume you could equate this to the function 'def' line being executed). If we do this, then it becomes easy to consider the TypeError as originating from within the called function, and therefore python will print the exact location of the function that's being called *as well as* the location of the call site (in the frame above). Internally, the python runtime would have to go out of its way to enter this 'fake' frame before generating the exception as, in reality the frame isn't entered (as I understand it) until after the arguments have been assigned correctly to the signature. But this seems like it should be relatively simple and non-invasive to do. The above makes sense to me, apologies if it's not very understandable to others. Thanks Steve On Mon, Feb 22, 2021 at 10:39 PM Ned Batchelder <ned@nedbatchelder.com> wrote:
On Tue, Feb 23, 2021 at 10:10 AM Stestagg <stestagg@gmail.com> wrote:
So Python has identified that a function 'do_thing' is being called incorrectly, but /where/ is do_thing defined? This problem gets much harder if there are multiple definitions of 'do_thing' in the codebase, as alluded to in the original mail.
Paul's suggestion is that python should add the source location of the function being called into the error message somewhere.
Thank you for explaining. I'm glad I wasn't the only one confused by the original post :) I think this is a nice-to-have, rather than being a serious bug to be fixed. When you get an error about a function call, it's entirely possible that the target function is the problem, or that the wrong function is being referenced; but you can get similar problems with all kinds of mismatches (like "a, b, c = thing" and getting a ValueError - maybe you'd need to check the line where 'thing' originated). If this can be done easily, great, but otherwise it might be the domain of traceback enhancement tools rather than the core language. Does execution ever (in the normal case) actually hit the 'def' line? If so, I wouldn't be averse to having a traceback line mentioning it. But if (as I suspect) tracing successful execution wouldn't hit that line, then it doesn't seem right to feign execution of it. My understanding of the def statement is that it runs ONLY when the function is defined; when the function's called, you go straight into the body. ChrisA
Hello, On Tue, 23 Feb 2021 12:31:37 +1100 Chris Angelico <rosuav@gmail.com> wrote:
It's not a serious bug, it's "astonishing overlook". Astonishing, as it was there for so many years, unpatched. And of course, it's rare enough. But when it happens, you're in the maze. I spend much time communicating with Python critics (well, haters), and sometimes read quite funny criticisms like: error reporting (stacktraces) in Python is poor. I always considered that "random last resort nitpick", as error reporting in Python has always been much better than in other scripting languages (I guess by now everyone caught up). But now imagine error like that happens in some 3rd-party library. Which happened to do some first-class function/higher order programming. With functions from another 3rd-party library. Which released a new version with "slightly updated" API. All that happening in production. Someone who ever caught such (or similar) case would forever get imprint of "Python has poor error reporting". Because just imagine what you report to the maintainer of the first 3rd-party lib: "Your lib calls something wrong in file X line Y" - "Ok, let's look into that. What does it call?" - "Umm, I don't know, something called 'foo'" (or "<lambda>").
Good point. But right analogy here is: suppose error happens in "a, b, c = thing". But Python doesn't show that line in the backtrace, it stops 1 entry before it. Go figure now. Because that's the right analogy of what happens with a function - in the function "prolog", there's a series of assignments: "formal_paramX = actual_argX". Conceptually, those *are* in the function context (or you would not be able to reference function's formal params).
If this can be done easily, great, but otherwise it might be the domain of traceback enhancement tools rather than the core language.
For Pycopy it was easy. I will look into CPython one of these weeks.
For another analogy, you can look at how native functions are defined and where they check params, and from where exceptions originate there (by obvious reasons, Python backtraces don't show C source code function/linenos ;-)).
ChrisA
[] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Feb 23, 2021 at 5:33 PM Paul Sokolovsky <pmiscml@gmail.com> wrote:
But they aren't actual lines of code. Your analogy breaks down because you're trying to put a line into a backtrace for a mythical and conceptual action that happens as part of the function call. That's why I dispute that this is an "astonishing oversight". It would be nice to have the extra information, but given that Python and Python programmers have survived for thirty years without it, I don't think it's nearly as serious as you're implying.
Yes, or you could look at CPython byte code and how it does a bunch of stack operations, but some of them don't actually happen. Looking at the implementation, especially in a completely different language, doesn't really help here :) The way I see it, this is perfect as an "additional information" field. For instance, when I compile certain buggy C programs, gcc tells me things like this: demo.c: In function ‘main’: demo.c:4:9: warning: passing argument 1 of ‘printf’ makes pointer from integer without a cast [-Wint-conversion] printf(12345); ^~~~~ In file included from demo.c:1: /usr/include/stdio.h:332:43: note: expected ‘const char * restrict’ but argument is of type ‘int’ extern int printf (const char *__restrict __format, ...); ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ It starts by reporting the error, and then says "note: the thing you're calling came from over here". It's separate from the error itself, but provides extremely useful information. Python doesn't currently have a standardized way to report this, but IMO that would be a far better way than synthesizing a fake backtrace entry. Taking your original example, it might look something like this: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() got an unexpected keyword argument 'noann' Note: The function is defined here: File "whatever.py", line 123, in <module> def print(thingy, thongy, whop): ChrisA
Hello, On Tue, 23 Feb 2021 17:52:20 +1100 Chris Angelico <rosuav@gmail.com> wrote:
But that's the whole point - the externally visible behavior should correspond to the conceptual model, even if a particular implementation does something different underlyingly. So again, it's CPython's implementation detail that, for bytecode functions only, it implements: def foo(params): # foo takes care of its params ... foo(args) as: param_check_and_call(foo, __param_meta__[foo], *args) And due to this implementation detail raises exceptions from param_check_and_call(). []
The way I see it, this is perfect as an "additional information" field.
I started there, and proceeded to where no "additional information" hacks are needed, and everything fits well into the existing framework. Come along with me. [] -- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Tue, 23 Feb 2021 17:52:20 +1100 Chris Angelico <rosuav@gmail.com> wrote: []
Ah, missed to reply to that one. Can judge it by myself: that's definitely not the first time I see that issue, where there was "something wrong" when getting exceptions when dealing with first-class function values. But I didn't even have a good understanding of what's wrong, and cases were either simple and obvious, or I reduced to print debugging. But now I'm working on a project where I consider UX to be important. And I consider a good UX for Python project not when it catch-all's exceptions and obfuscates the problem with a generic error message, and likewise not when there're 5 screenfuls of chained exceptions. I consider it's when there's human-sized backtrace, clearly pointing at the problem. And for that case, I figured that the backtrace is senseless and doesn't give a user of my program enough information to understand the issue. So, I stopped with writing my program, sat and scratched my head to understand what's wrong, then sat again to try a few different solutions (the original post should mirror that), until finally came to one which makes sense to the end user, and doesn't require any crutches. I won't be surprised if that's how other Python programmers dealt with it over decades - some didn't even dig into "what's wrong", "something is wrong" was enough for them to do in-mind-debugging. The other gritted teeth and recursed to print debugging, again without thinking too much what's wrong on the meta-level. And the remaining? Well, they now curse Python for poor error reporting on reddits and hackernewses. You can also see that on this very thread - at first, the issue was at all unclear (even though I tried to write a detailed message, packing different sides into it). Now issue becomes clear, but (besides the usual "never heard" response), the thinking is along the lines of "additional info hacks", which was my first motion either. Let's see where that leads us. (And that's why I posted to the mailing list first - chances, a bugs.p.o report would be closed with "discuss first on the mailing list" suggestion). [] -- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Mon, 22 Feb 2021 17:32:23 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
Well, if you looked at that stack trace, you saw the code: Traceback (most recent call last): File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() got an unexpected keyword argument 'noann' So yes, the code is: --- first_class_function_value(func, **pass_params) --- That's the "calling code". And the rest of the code? But's that's the whole point, that the current CPython's error reporting doesn't tell me where it is! It just tells me the function name, and go make a full-text search thru the entire sys.path to find its location, especially if there're many functions of that name. -- Best regards, Paul mailto:pmiscml@gmail.com
On 2/23/21 12:56 AM, Paul Sokolovsky wrote:
Perhaps instead of fiddling with stack frames, we can solve this problem in a more straightforward way. The error message has a description of the function: "print()". What if we extend that description: "print() at path/to/myprog.py:173". Then the existing stack traces would have the information you want, without having to debate the meaning of stack frames, and where execution happens. --Ned.
Hello, On Tue, 23 Feb 2021 11:52:57 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
That's what I implemented initially, saw this: File "pseudoc_tool.py", line 91, in <module> first_class_function_value(func, **pass_params) TypeError: print() at path/to/myprog.py:173 got an unexpected keyword argument 'noann' and wondered - why am I messing with the error *message*, if above it is a backtrace entry, which naturally shows the location information ("File "pseudoc_tool.py", line 91"). Then all things clicked into their places. In my implementation, I didn't need to "fiddle with stack frames", I just needed to pre-populate the exception object with a first (shown last) backtrace entry for the called function, and let the usual processing append to it.
Without getting to the root cause, there will be only workarounds of different level of ugliness. For example, any reasonable JIT would implement proper semantics - param checking *inside* functions (just like the C code does now).
--Ned.
-- Best regards, Paul mailto:pmiscml@gmail.com
On Wed, Feb 24, 2021 at 4:59 AM Paul Sokolovsky <pmiscml@gmail.com> wrote:
Why do you assume that this is "proper semantics"? I'm confused. What is it that makes function parameter matching inherently part of the body of the function? For one thing, that notion is semantically incompatible with multiple dispatch, where matching parameter numbers or types determines which function is even going to be called. ChrisA
Hello, On Wed, 24 Feb 2021 05:01:37 +1100 Chris Angelico <rosuav@gmail.com> wrote:
The same what causes C-level function implementations to do parameter checking themselves. There's simply nowhere else to put that checking, and of course nobody calls C functions via an intermediary - that's hilariously inefficient. All those points apply to JITted functions too (you JIT to make it fast, then putting a slow bottleneck inbetween, which will also confuse CPU branch predictors - doesn't make sense).
Python doesn't support multiple dispatch. Majority of languages don't support multiple dispatch. For a well known reason - its benefits vs its inefficiency ratio is very low (inefficiency is high). So, whichever runtime system implements multiple dispatch, will need to think about its issues, including aspects of error reporting, I don't see how that's relevant to Python discussion. (Multiple dispatch is also effectively a poorman's, implicit pattern matching which works only between code blocks factored out into functions.) [] -- Best regards, Paul mailto:pmiscml@gmail.com
On 2/23/21 12:55 PM, Paul Sokolovsky wrote:
I don't think you are helping your cause by talking down to people. "any reasonable JIT" and "proper semantics" are judgemental without adding anything to the discussion. You wanted to know what function caused the problem. I proposed a simple solution that gave you precisely the information you wanted. But you've called it an ugly workaround for some reason. --Ned.
Hello, On Tue, 23 Feb 2021 13:09:39 -0500 Ned Batchelder <ned@nedbatchelder.com> wrote:
Well, keeping repeating "we can put location info into the error message", disregarding an argument that *backtrace entries* are for location info - that should be classified as "judgemental" too then?
Because I saw them both (by implementing them both), and even in the order you suggest, "location info in the message" first, felt not satisfied, spent more time to implement the alternative, and yes, that looks both more naturally and more clearly. So, please don't be judgemental to my suggestion, but consider them both, not just first which came to your mind (which is also exactly the one which came to my mind first, peace).
--Ned.
-- Best regards, Paul mailto:pmiscml@gmail.com
participants (6)
-
Barry Scott -
Chris Angelico -
Ned Batchelder -
Paul Sokolovsky -
Peter Otten -
Stestagg