Add more SyntaxWarnings?
https://bugs.python.org/issue15248 is about situations like the following:
[(1,2,3) (4,5,6)] Traceback (most recent call last): File "<pyshell#4>", line 2, in <module> (4,5,6)] TypeError: 'tuple' object is not callable
The original poster requested that the error message be augmented with something like "(missing preceding comma?)" Ezio Melotti suggested a FAQ entry like https://docs.python.org/3/faq/programming.html#why-am-i-getting-an-unboundlo... (I think such entries below in a separate doc and will try to post on python-ideas when I have a prototype.) Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a proof-of-concept diff that handled the above and many similar cases. The diff is based on the idea that while we can only positively identify 'callables' at runtime, we *can* negatively identify many non-callables when compiling. Ditto for subscriptables and indexables. Serhiy concluded with "This patch was inspired by usability improvements in GCC 8. https://developers.redhat.com/blog/2018/03/15/gcc-8-usability-improvements/ I haven't created a pull request because I have doubts about that this should be in the compiler rather of a third-party linter. But if several other core developers will support this idea I'll continue working in this direction." I was impressed with how clear and readable the patch is and consider it a plausible enhancement. I would like other core developers to comment. -- Terry Jan Reedy
On 2019-01-24, Terry Reedy wrote:
Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a proof-of-concept diff that handled the above and many similar cases.
I believe that in general we should give better errors or warnings if we can do it without huge difficulty. Serhiy's patch is quite simple. The same check *could* be done by a linting tool. Putting it in CPython will make it more widely available. These checks could be helpful to beginners who probably won't have linting tools setup. I think we should not make it an error, otherwise we have changed Python "the language". We don't want to force other Python implementations to do the same check. It might be hard for them to implement. So, SyntaxWarning seems like a reasonable compromise. Regards, Neil
In this case I agree that a SyntaxWarning is a good idea, per Serhiy's patch. I would be even more conservative, and only warn if the first object is a tuple -- the case of the missing comma in the original example appears likely enough, but I don't expect people to write e.g. `[[1, 2], (3, 4)]` very often, so leaving the comma out there would be very unlikely. Regarding the issue of when it's appropriate to issue a SyntaxWarning vs. when to leave it up to linters, again I would recommend great caution and only warn about code that is *definitely* going to fail when executed. (Or at least is *definitely* not going to please the programmer -- one of the first cases where we added a SyntaxWarning was actually `assert(condition, message)`, which "fails" by never failing. :-) But this is one of those cases. On Thu, Jan 24, 2019 at 2:10 PM Neil Schemenauer <nas-python@arctrix.com> wrote:
On 2019-01-24, Terry Reedy wrote:
Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a proof-of-concept diff that handled the above and many similar cases.
I believe that in general we should give better errors or warnings if we can do it without huge difficulty. Serhiy's patch is quite simple. The same check *could* be done by a linting tool. Putting it in CPython will make it more widely available. These checks could be helpful to beginners who probably won't have linting tools setup.
I think we should not make it an error, otherwise we have changed Python "the language". We don't want to force other Python implementations to do the same check. It might be hard for them to implement. So, SyntaxWarning seems like a reasonable compromise.
Regards,
Neil _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Thu, Jan 24, 2019 at 04:01:26PM -0600, Neil Schemenauer wrote:
On 2019-01-24, Terry Reedy wrote:
Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a proof-of-concept diff that handled the above and many similar cases.
I believe that in general we should give better errors or warnings if we can do it without huge difficulty. Serhiy's patch is quite simple. The same check *could* be done by a linting tool. Putting it in CPython will make it more widely available. These checks could be helpful to beginners who probably won't have linting tools setup.
+1 to what Neil says here.
I think we should not make it an error, otherwise we have changed Python "the language".
We're allowed to, we're the ones who say what the language is :-) The remainder of my comments are more speculative.
We don't want to force other Python implementations to do the same check. It might be hard for them to implement. So, SyntaxWarning seems like a reasonable compromise.
We could say that implementations are allowed to raise errors at compile time instead of run time, but aren't required to. Then it becomes a matter of "quality of implementation". For literal ints, strings, None, etc we can tell at compile time that an error will occur. All of these cannot fail to raise (short of changing the interpreter, in which case you're not using Python anymore): 1 + "1" # raises TypeError None[x] # TypeError 1.234(x) # TypeError "spam".idnex("a") # AttributeError In these specific cases, there is nothing wrong with the *syntax*, but a compiler should be permitted to immediately raise the same exception that would otherwise occur at run time. This is a form of peephole optimization, I guess. If people truly wanted to delay the exception, they could either turn off the peephole optimizer, or work around it: a = 1; a + "1" Thoughts? -- Steve
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve@pearwood.info> wrote:
We could say that implementations are allowed to raise errors at compile time instead of run time, but aren't required to. Then it becomes a matter of "quality of implementation".
For literal ints, strings, None, etc we can tell at compile time that an error will occur. All of these cannot fail to raise (short of changing the interpreter, in which case you're not using Python anymore):
1 + "1" # raises TypeError None[x] # TypeError 1.234(x) # TypeError "spam".idnex("a") # AttributeError
In these specific cases, there is nothing wrong with the *syntax*, but a compiler should be permitted to immediately raise the same exception that would otherwise occur at run time. This is a form of peephole optimization, I guess.
+1. If it's something that the peephole optimizer is already allowed to change (eg "1"+"1" is constant-folded) and there is absolutely no way that it can ever be changed at run time, then raising at compile time can't hurt [1]. It'd be as implementation-dependent and version-dependent as the peephole optimizer itself. Does there need to be a new subclass of SyntaxError for "Technically Valid But Not Meaningful" problems? Is there value in distinguishing "InevitableTypeError" from "InevitableAttributeError"? ChrisA [1] Yes, I know about XKCD 1172, but if someone's saying "if shouldnt_happen: None[None]" then that's their problem.
On 1/24/2019 5:52 PM, Chris Angelico wrote:
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve@pearwood.info> wrote:
We could say that implementations are allowed to raise errors at compile time instead of run time, but aren't required to. Then it becomes a matter of "quality of implementation".
For literal ints, strings, None, etc we can tell at compile time that an error will occur. All of these cannot fail to raise (short of changing the interpreter, in which case you're not using Python anymore):
1 + "1" # raises TypeError None[x] # TypeError 1.234(x) # TypeError "spam".idnex("a") # AttributeError
In these specific cases, there is nothing wrong with the *syntax*, but a compiler should be permitted to immediately raise the same exception that would otherwise occur at run time. This is a form of peephole optimization, I guess.
+1. If it's something that the peephole optimizer is already allowed to change (eg "1"+"1" is constant-folded) and there is absolutely no way that it can ever be changed at run time, then raising at compile time can't hurt [1]. It'd be as implementation-dependent and version-dependent as the peephole optimizer itself.
It would be a change if the code is never called. I'm not sure we care about code that's never called, but it is a change. Eric
On 1/24/2019 6:16 PM, Eric V. Smith wrote:
On 1/24/2019 5:52 PM, Chris Angelico wrote:
+1. If it's something that the peephole optimizer is already allowed to change (eg "1"+"1" is constant-folded) and there is absolutely no way that it can ever be changed at run time, then raising at compile time can't hurt [1]. It'd be as implementation-dependent and version-dependent as the peephole optimizer itself.
It would be a change if the code is never called. I'm not sure we care about code that's never called, but it is a change.
Which Chris already pointed out, in some text that I inadvertently deleted. I didn't catch his meaning, which is my fault. Eric
On Fri, 25 Jan 2019 at 10:15, Eric V. Smith <eric@trueblade.com> wrote:
It would be a change if the code is never called. I'm not sure we care about code that's never called, but it is a change.
The biggest problem with converting runtime errors to compile time errors is that it means affected dead code goes from being a readability & maintainability problem to "Python X.Y breaks my project". SyntaxWarning splits the difference nicely, since the compiler complains about the same dead code that would confuse a human reader, but also ignores it and moves on. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Jan 24, 2019 at 2:55 PM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve@pearwood.info> wrote:
We could say that implementations are allowed to raise errors at compile time instead of run time, but aren't required to. Then it becomes a matter of "quality of implementation".
For literal ints, strings, None, etc we can tell at compile time that an error will occur. All of these cannot fail to raise (short of changing the interpreter, in which case you're not using Python anymore):
1 + "1" # raises TypeError None[x] # TypeError 1.234(x) # TypeError "spam".idnex("a") # AttributeError
In these specific cases, there is nothing wrong with the *syntax*, but a compiler should be permitted to immediately raise the same exception that would otherwise occur at run time. This is a form of peephole optimization, I guess.
+1. If it's something that the peephole optimizer is already allowed to change (eg "1"+"1" is constant-folded) and there is absolutely no way that it can ever be changed at run time, then raising at compile time can't hurt [1]. It'd be as implementation-dependent and version-dependent as the peephole optimizer itself.
I'm -1 on all of these cases. There's nothing mysterious about e.g. `TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated tuples. (Surely people get errors about int+str all the time, and they've never complained -- unlike the tuple tuple case.)
Does there need to be a new subclass of SyntaxError for "Technically Valid But Not Meaningful" problems? Is there value in distinguishing "InevitableTypeError" from "InevitableAttributeError"?
I don't think there's a *general* problem to be solved here.
ChrisA
[1] Yes, I know about XKCD 1172, but if someone's saying "if shouldnt_happen: None[None]" then that's their problem.
-- --Guido van Rossum (python.org/~guido)
. There's nothing mysterious about e.g. `TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated tuples. (Surely people get errors about int+str all the time, and they've never complained -- unlike the tuple tuple case.)
Well, yes, that particular example is pretty clear. But as a rule, there are a LOT of errors that can be pretty mysterious to newbies. I would love to see Python become generally more informative with errors. In this case, you’d probably get a similar error, but it’s still nice to get it sooner, and if the hooks are in place, We could have others that are really helpful. -CHB Is there value in distinguishing
"InevitableTypeError" from "InevitableAttributeError"?
I don’t think so — what we need are helpful error messages. If it will be raised at compile time, then it won’t generally be catchable in s try-except— so the actual exception type isn’t very important. -CHB
On Thu, Jan 24, 2019 at 3:45 PM Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
. There's nothing mysterious about e.g. `TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated tuples. (Surely people get errors about int+str all the time, and they've never complained -- unlike the tuple tuple case.)
Well, yes, that particular example is pretty clear. But as a rule, there are a LOT of errors that can be pretty mysterious to newbies.
I would love to see Python become generally more informative with errors.
In this case, you’d probably get a similar error, but it’s still nice to get it sooner, and if the hooks are in place, We could have others that are really helpful.
I doubt people are writing `42 + "abc"` by accident. They'll write `x + y` and by accident the types won't match. So better error messages at runtime would help. But I doubt we'll see much mileage out of the syntax checks. And making things work different based on whether it's a literal or a variable isn't very Pythonic. -- --Guido van Rossum (python.org/~guido)
Chris Barker - NOAA Federal via Python-Dev writes:
Well, yes, that particular example is pretty clear. But as a rule, there are a LOT of errors that can be pretty mysterious to newbies.
Isn't that the very definition of "newbie"? That's half a joke, but I really don't think that programmers new to Python should be the standard. The problematic cases are those where even a relatively experienced Python programmer needs to be told why an error is raised, because it's too hard to figure out from background knowledge of the language, you need to know about implementation internals.
I would love to see Python become generally more informative with errors.
I would love to see all automated systems become more informative with errors! One thing I like about Python the development community is that people are doing something about it. Slowly but surely ....
I don’t think so — what we need are helpful error messages. If it will be raised at compile time, then it won’t generally be catchable in s try-except— so the actual exception type isn’t very important.
+1 In general depending much on exception type is a hit-or-miss affair.
But as a rule,
there are a LOT of errors that can be pretty mysterious to newbies.
Isn't that the very definition of "newbie"? That's half a joke, but I really don't think that programmers new to Python should be the standard.
Python is broadly advocated (and used) as a first language to learn. Because it is. So I think anything we can do to help newbies, *that doesn’t make the language less powerful, or even more annoying for experienced developers* is a good thing. That is: prioritizing newbie-friendliness is good. Prioritizing it over other important things is not. -CHB
The problematic cases are those where even a relatively experienced Python programmer needs to be told why an error is raised, because it's too hard to figure out from background knowledge of the language, you need to know about implementation internals.
I would love to see Python become generally more informative with errors.
I would love to see all automated systems become more informative with errors! One thing I like about Python the development community is that people are doing something about it. Slowly but surely ....
I don’t think so — what we need are helpful error messages. If it will be raised at compile time, then it won’t generally be catchable in s try-except— so the actual exception type isn’t very important.
+1
In general depending much on exception type is a hit-or-miss affair.
Chris Barker - NOAA Federal writes:
But as a rule, there are a LOT of errors that can be pretty mysterious to newbies.
Isn't that the very definition of "newbie"? That's half a joke, but I really don't think that programmers new to Python should be the standard.
Python is broadly advocated (and used) as a first language to learn. Because it is.
So I think anything we can do to help newbies, *that doesn’t make the language less powerful, or even more annoying for experienced developers* is a good thing.
That is: prioritizing newbie-friendliness is good. Prioritizing it over other important things is not.
I don't disagree. I disagree with the conclusion that it's worth the effort to try to improve all error messages that confuse new users, because new users (by definition) don't know enough to respond usefully in many cases. In those cases, they need to be told what's going on and why, where more experienced users can figure it out from their background knowledge of Python semantics. Embedding a "theory of operations" note in every error message would be possible, but I don't think it's a good idea -- it would certainly make the language more annoying for experienced developers. That's why I proposed the criterion
The problematic cases are those where even a relatively experienced Python programmer needs to be told why an error is raised, because it's too hard to figure out from background knowledge of the language, you need to know about implementation internals.
Maybe that's too strict. The pleasant hacks of giving ``quit`` and ``help`` "values" as variables are certainly useful. On the contrary, getting rid of the print statement was perhaps a better idea. :-) Steve
On Tue, 29 Jan 2019 at 14:47, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I don't disagree. I disagree with the conclusion that it's worth the effort to try to improve all error messages that confuse new users, because new users (by definition) don't know enough to respond usefully in many cases. In those cases, they need to be told what's going on and why, where more experienced users can figure it out from their background knowledge of Python semantics. Embedding a "theory of operations" note in every error message would be possible, but I don't think it's a good idea -- it would certainly make the language more annoying for experienced developers.
FWIW, we have pretty decent evidence that error messages don't have to provide a wonderful explanation on their own in order to be helpful: they just need to be distinctive enough that a web search will reliably get you to a page that gives you relevant information. Pre-seeded answers on Stack Overflow are excellent for handling the second half of that approach (see [1] for a specific example). Cheers, Nick. [1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-p... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2019-01-29 13:44, Nick Coghlan wrote:
On Tue, 29 Jan 2019 at 14:47, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I don't disagree. I disagree with the conclusion that it's worth the effort to try to improve all error messages that confuse new users, because new users (by definition) don't know enough to respond usefully in many cases. In those cases, they need to be told what's going on and why, where more experienced users can figure it out from their background knowledge of Python semantics. Embedding a "theory of operations" note in every error message would be possible, but I don't think it's a good idea -- it would certainly make the language more annoying for experienced developers.
FWIW, we have pretty decent evidence that error messages don't have to provide a wonderful explanation on their own in order to be helpful: they just need to be distinctive enough that a web search will reliably get you to a page that gives you relevant information.
Pre-seeded answers on Stack Overflow are excellent for handling the second half of that approach (see [1] for a specific example).
Cheers, Nick.
[1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-p...
I have a vague recollection that a certain computer system (Amiga?) had a 'why' command. If it reported an error, you could type "why" and it would give you more details. I suspect that all that was happening was that when the error occurred it would store the additional details somewhere that the 'why' command would simply retrieve.
Hello, On Tue, 29 Jan 2019 18:55:37 +0000 MRAB <python@mrabarnett.plus.com> wrote: []
https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-p... I have a vague recollection that a certain computer system (Amiga?) had a 'why' command. If it reported an error, you could type "why" and it would give you more details.
I suspect that all that was happening was that when the error occurred it would store the additional details somewhere that the 'why' command would simply retrieve.
Well, a typical Linux install includes "errno" util: $ errno 13 EACCES 13 Permission denied So, hail to error numbers. And indeed, even if some may think they're artifact of 1980'ies, they're alive, well, and useful: $ pylint --help-msg W1651 :deprecated-itertools-function (W1651): *Accessing a deprecated function on the itertools module* Used when accessing a function on itertools that has been removed in Python 3. This message belongs to the python3 checker. It can't be emitted when using Python >= 3.0. -- Best regards, Paul mailto:pmiscml@gmail.com
MRAB schrieb am 29.01.19 um 19:55:
On 2019-01-29 13:44, Nick Coghlan wrote:
FWIW, we have pretty decent evidence that error messages don't have to provide a wonderful explanation on their own in order to be helpful: they just need to be distinctive enough that a web search will reliably get you to a page that gives you relevant information.
Pre-seeded answers on Stack Overflow are excellent for handling the second half of that approach (see [1] for a specific example). [1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-p...
I have a vague recollection that a certain computer system (Amiga?) had a 'why' command. If it reported an error, you could type "why" and it would give you more details.
I suspect that all that was happening was that when the error occurred it would store the additional details somewhere that the 'why' command would simply retrieve.
So … are you suggesting to use the webbrowser module inside of the REPL to look up the exception message of the previously printed stack trace in stack overflow when a user types "why()"? I faintly recall someone implementing something in that direction. It's probably in some package on PyPI. Stefan
On 2019-01-30 18:12, Stefan Behnel wrote:
MRAB schrieb am 29.01.19 um 19:55:
On 2019-01-29 13:44, Nick Coghlan wrote:
FWIW, we have pretty decent evidence that error messages don't have to provide a wonderful explanation on their own in order to be helpful: they just need to be distinctive enough that a web search will reliably get you to a page that gives you relevant information.
Pre-seeded answers on Stack Overflow are excellent for handling the second half of that approach (see [1] for a specific example). [1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-p...
I have a vague recollection that a certain computer system (Amiga?) had a 'why' command. If it reported an error, you could type "why" and it would give you more details.
I suspect that all that was happening was that when the error occurred it would store the additional details somewhere that the 'why' command would simply retrieve.
So … are you suggesting to use the webbrowser module inside of the REPL to look up the exception message of the previously printed stack trace in stack overflow when a user types "why()"?
No, I was just suggesting it as a possible way of providing newbies with more information about an error without annoying more experienced users with excessively long messages every time.
I faintly recall someone implementing something in that direction. It's probably in some package on PyPI.
On Wed, Jan 30, 2019 at 07:12:21PM +0100, Stefan Behnel <stefan_ml@behnel.de> wrote:
I have a vague recollection that a certain computer system (Amiga?) had a 'why' command. If it reported an error, you could type "why" and it would give you more details.
I suspect that all that was happening was that when the error occurred it would store the additional details somewhere that the 'why' command would simply retrieve.
So ??? are you suggesting to use the webbrowser module inside of the REPL to look up the exception message of the previously printed stack trace in stack overflow when a user types "why()"?
I faintly recall someone implementing something in that direction. It's probably in some package on PyPI.
It doesn't have to be all web. Compare how ``help()`` works at the python REPL, ``pydoc name`` at the command line, and ``pydoc -p`` + a browser.
Stefan
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Thu, 31 Jan 2019 at 04:16, Stefan Behnel <stefan_ml@behnel.de> wrote:
I faintly recall someone implementing something in that direction. It's probably in some package on PyPI.
You may be thinking of Doug Hellman's https://pypi.org/project/whatthewhat/ Cheers, Nick. P.S. More horrifyingly (in an amusing way), there's also https://github.com/drathier/stack-overflow-import -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
At the cost of breaking threading a bit, I'm going to reply to a few different people in one post. On Fri, Jan 25, 2019 at 09:52:53AM +1100, Chris Angelico wrote:
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve@pearwood.info> wrote:
We could say that implementations are allowed to raise errors at compile time instead of run time, but aren't required to. Then it becomes a matter of "quality of implementation". [...]
Does there need to be a new subclass of SyntaxError for "Technically Valid But Not Meaningful" problems? Is there value in distinguishing "InevitableTypeError" from "InevitableAttributeError"?
If we did this (which is looking less and less likely...) I would argue for plain old TypeError and AttributeError. There should be no difference in the exception raised, only in the timing (compile time versus run time). On Thu, Jan 24, 2019 at 11:45:49PM +0000, Chris Barker wrote:
I would love to see Python become generally more informative with errors.
In this case, you’d probably get a similar error, but it’s still nice to get it sooner
That is precisely my thinking. On Thu, Jan 24, 2019 at 03:24:41PM -0800, Guido van Rossum wrote:
I'm -1 on all of these cases. There's nothing mysterious about e.g. `TypeError: unsupported operand type(s) for +: 'int' and 'str'`
Its not about the error being mysterious, its about raising the exception as early as possible. It shouldn't take 20 years to discover that this snippet buried deep in a large code base: if datetime.date.today().year == 2029: spam = 1 + '1' will fail :-) On Thu, Jan 24, 2019 at 03:58:50PM -0800, Guido van Rossum wrote:
But I doubt we'll see much mileage out of the syntax checks.
Frankly, I agree, so I'm not going to argue hard for this. It was more speculation than a solid proposal. A project manager I used to work with would have dismissed this as "turd polishing", but I prefer to think of it as "craftsmanship" -- a bit like a carpenter ensuring that his screws don't protrude above the surface of the wood. Sure, its not going to hold the wood together any better, but its just a nicer end-user experience :-) I expect that the cost of implementing these checks would probably exceed the benefit of detecting the errors early, but I could be wrong. Maybe it would be easy to implement. But if there is strong opposition to it, I'm not going to champion the idea any further. -- Steve
participants (14)
-
Chris Angelico -
Chris Barker - NOAA Federal -
Eric V. Smith -
Greg Ewing -
Guido van Rossum -
MRAB -
Neil Schemenauer -
Nick Coghlan -
Oleg Broytman -
Paul Sokolovsky -
Stefan Behnel -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy