The friendly-traceback project (in early alpha) is working on making traceback messages more helpful and translatable - https://aroberge.github.io/friendly-traceback-docs/docs/html/ On Tue, 14 Jan 2020 at 12:43, Guido van Rossum wrote:

On the subject of replacing the current parser, I am actively working on that. See GitHub.com/gvanrossum/pegen.

On Tue, Jan 14, 2020 at 10:32 Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:

...

On Jan 14, 2020, at 05:22, Σταύρος Ντέντος wrote:

...

Hello there,

If I have simply missed a double colon starting a for loop

File "./bbq.py", line 160 for config_file in config_files ^ SyntaxError: invalid syntax

the message is not as straightforward.

I think almost everyone would prefer it if the compiler could say “SyntaxError: missing colon at end of a compound statement header” or something more useful.

And that probably goes even more for this case:

spam = eggs(cheese, (foo, bar) cheese = spam*2

The problem is to come up with a rule that could be applied to detect these cases given the information the simple LR(1) parser has available at the time of failure. I suspect there’s no way to do that without radically changing the parser architecture, keeping track of a lot more state, or partially re-parsing things in the error handler. (If it were easy, Guido would have done it back in 1.x.)

But maybe there’s a way to heuristically detect that these problems are _likely_ causes of the error (without having to be as ridiculously complicated as what Clang does with C++ code)? If you could find a way to make the error say “SyntaxError: invalid syntax (possibly missing colon at end of compound statement header)” in most simple “forgot the colon” cases and very few other cases, without massively disrupting everything, I think people would be happy with that.

You might even be able to take advantage of re-parsing without having to solve all the problems that go with that. For example, technically, you can’t even access the last logical line to reparse; practically, you can get it in the same cases the traceback can print it, and those are probably the only cases you need to heuristically improve the error handling. You could even maybe do a quick & dirty proof of concept in Python in an import hook, if you don’t want to dive into the middle of the C compiler code.

As an alternative, there are lots of projects to use more powerful parser algorithms on Python. There’s not much call to replace CPython’s parser, because there aren’t any benefits to offset the costs. (At least assuming that the language is going to stay LR(1), to make it easy to parse in your head.) But if you could improve most of the most annoying error handling cases, that might be a different story. And these might also be easier to play with. (Some have pure Python implementations, and even the ones in C aren’t embedded in the middle of the compiler code.) IIRC, early Java did something clever with a GLR parser that has LR(1) performance on all valid code and strictly bounded complexity on error recovery (so it may get as bad as worst-case cubic, but cubic on N<=5 so who cares) so they could usually produce error messages as good as most C compilers without the horrible mess of parsing that most C compilers need.

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ILJNAN... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OX7USO... Code of Conduct: http://python.org/psf/codeofconduct/

-- Naomi Ceder @NaomiCeder • https://www.naomiceder.tech https://www.manning.com/books/the-quick-python-book-third-edition

For the sake of completeness: friendly-traceback is already handling this. @Naomi Thank you for bringing this up :-) https://github.com/aroberge/friendly-traceback/issues/42 -- Yours faithfully, Ntentos Stavros

On 14 Jan 2020, at 18:42, Guido van Rossum wrote:

On the subject of replacing the current parser, I am actively working on that. See GitHub.com/gvanrossum/pegen.

Will that allow me to write this? if a > 3 and x < 6: doit() Barry

On Tue, Jan 14, 2020 at 10:32 Andrew Barnert via Python-ideas mailto:python-ideas@python.org> wrote: On Jan 14, 2020, at 05:22, Σταύρος Ντέντος mailto:stdedos@gmail.com> wrote:

...

Hello there,

If I have simply missed a double colon starting a for loop

File "./bbq.py", line 160 for config_file in config_files ^ SyntaxError: invalid syntax

the message is not as straightforward.

I think almost everyone would prefer it if the compiler could say “SyntaxError: missing colon at end of a compound statement header” or something more useful.

And that probably goes even more for this case:

spam = eggs(cheese, (foo, bar) cheese = spam*2

The problem is to come up with a rule that could be applied to detect these cases given the information the simple LR(1) parser has available at the time of failure. I suspect there’s no way to do that without radically changing the parser architecture, keeping track of a lot more state, or partially re-parsing things in the error handler. (If it were easy, Guido would have done it back in 1.x.)

But maybe there’s a way to heuristically detect that these problems are _likely_ causes of the error (without having to be as ridiculously complicated as what Clang does with C++ code)? If you could find a way to make the error say “SyntaxError: invalid syntax (possibly missing colon at end of compound statement header)” in most simple “forgot the colon” cases and very few other cases, without massively disrupting everything, I think people would be happy with that.

You might even be able to take advantage of re-parsing without having to solve all the problems that go with that. For example, technically, you can’t even access the last logical line to reparse; practically, you can get it in the same cases the traceback can print it, and those are probably the only cases you need to heuristically improve the error handling. You could even maybe do a quick & dirty proof of concept in Python in an import hook, if you don’t want to dive into the middle of the C compiler code.

As an alternative, there are lots of projects to use more powerful parser algorithms on Python. There’s not much call to replace CPython’s parser, because there aren’t any benefits to offset the costs. (At least assuming that the language is going to stay LR(1), to make it easy to parse in your head.) But if you could improve most of the most annoying error handling cases, that might be a different story. And these might also be easier to play with. (Some have pure Python implementations, and even the ones in C aren’t embedded in the middle of the compiler code.) IIRC, early Java did something clever with a GLR parser that has LR(1) performance on all valid code and strictly bounded complexity on error recovery (so it may get as bad as worst-case cubic, but cubic on N<=5 so who cares) so they could usually produce error messages as good as most C compilers without the horrible mess of parsing that most C compilers need.

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org mailto:python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org mailto:python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ILJNAN... https://mail.python.org/archives/list/python-ideas@python.org/message/ILJNAN... Code of Conduct: http://python.org/psf/codeofconduct/ http://python.org/psf/codeofconduct/ -- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OX7USO... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jan 14, 2020, at 18:15, David Mertz wrote:

For what it's worth, after 20+ years of using Python, forgetting the colon for blocks remains the most common error I make by a fairly wide margin. Of course, once I see the error message—even being not all that descriptive of the real issue—I immediately know what to fix too.

What if the colon were made optional, with an eye to perhaps eventually no longer using it as the preferred style for new code? We had a post a while ago about the possibility of using the lack of a colon as an implicit line continuation (like with parentheses, e.g. "if a\nand b:", and this was (reasonably) rejected. But if a line beginning as a compound statement and ending without a colon is *never* going to have a valid meaning as something else... what's the point of the colon, otherwise? Seems like just grit on the screen.

What if the colon were made optional, with an eye to perhaps eventually no longer using it as the preferred style for new code?

But if a line beginning as a compound statement and ending without a

colon is *never* going to have a valid meaning as something else... what's the point of the colon, otherwise? Seems like just grit on the screen.

That's a very interesting idea, and could perhaps be a viable design decision for making a new language. But, I strongly suspect that the current usage of the colon is far too fundamentally nested in the foundations of Python, and is likely not going to change. Even if it was simply made optional, it would no doubt introduce quite a lot of confusion and add an extra cost to Python's learning curve, for both newer and older developers. Introductory Python guides all over the place would also have to updated, to mention that in version x, developers may see this code example with or without the colon. In order to justify this, it has to be worth the cost. You main argument for it seems to be that it's unnecessary, and ends up being an extra character without a real purpose. I don't find this to be worth the cost associated with making it optional. Also, that assumes that it *is* entirely without purpose. Personally, I find that it helps as a visual queue, to indicate that the next line is within the block of the statement that proceeds the colon. This isn't a huge deal in one line statements, but when it carries over to the next line it greatly helps to distinguish one level from the next. For example, PEP8 mentions the following code formats as "acceptable options" of handling multi-line statements: (1) # No extra indentation. if (this_is_one_thing and that_is_another_thing): do_something() (2) # Add a comment, which will provide some distinction in editors # supporting syntax highlighting. if (this_is_one_thing and that_is_another_thing): # Since both conditions are true, we can frobnicate. do_something() (3) # Add some extra indentation on the conditional continuation line. if (this_is_one_thing and that_is_another_thing): do_something() Now, let's take the colon away and see what it looks like: (1) # No extra indentation. if (this_is_one_thing and that_is_another_thing) do_something() (2) # Add a comment, which will provide some distinction in editors # supporting syntax highlighting. if (this_is_one_thing and that_is_another_thing) # Since both conditions are true, we can frobnicate. do_something() (3) # Add some extra indentation on the conditional continuation line. if (this_is_one_thing and that_is_another_thing) do_something() This might be a bit subjective, but to my eyes this makes the block that's within the statement rather difficult to distinguish; particularly for (1) and (2). The effect would only be amplified in more complex code. On Thu, Jan 16, 2020 at 1:15 PM Random832 wrote:

On Tue, Jan 14, 2020, at 18:15, David Mertz wrote:

...

For what it's worth, after 20+ years of using Python, forgetting the colon for blocks remains the most common error I make by a fairly wide margin. Of course, once I see the error message—even being not all that descriptive of the real issue—I immediately know what to fix too.

What if the colon were made optional, with an eye to perhaps eventually no longer using it as the preferred style for new code?

We had a post a while ago about the possibility of using the lack of a colon as an implicit line continuation (like with parentheses, e.g. "if a\nand b:", and this was (reasonably) rejected. But if a line beginning as a compound statement and ending without a colon is *never* going to have a valid meaning as something else... what's the point of the colon, otherwise? Seems like just grit on the screen. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/J47TEY... Code of Conduct: http://python.org/psf/codeofconduct/

On Sat, 18 Jan 2020 at 02:32, Josh Rosenberg wrote:

The colon remains syntactically necessary in some cases, particularly to disambiguate cases involving one-lining (no block involved). Stupid example: If the colon is optional, what does:

if d +d

mean? Is it a test of the value of d, followed by invoking the unary plus operator as a one-liner (that is, was it "if d: +d")? Or is it testing d + d, and opening a block on the next line ("if d + d:")?

Or, to turn this around, if a + b > 0 What is the correct indentation for the following line? if a + b > 0: should be indented if a: + b > 0 should *not* be indented. That sort of context-dependent interpretation is very hard to parse, both for the computer and for the user. Even if it's technically possible to parse for the compiler, I'd not want to have to worry when reading a line of code which is quoted out of context (in a traceback from a bug report, for example) that I needed to know what the surrounding lines were just to interpret the meaning. Yes, this would be a very rare thing to happen, but the lack of such weird edge cases is key to why Python feels natural and easy to read for me. Paul

note that there is a FAQ about this: https://docs.python.org/3/faq/design.html#why-are-colons-required-for-the-if... And if you google hard enough (my 30 seconds wasn't enough), I'm pretty sure you can find a description of actual usability tests (if informal) mack in the day that indicated that the colon was helpful for readability, if not for computer parsing. -CHB On Sun, Jan 19, 2020 at 7:49 AM Random832 wrote:

On Fri, Jan 17, 2020, at 21:32, Josh Rosenberg wrote:

...

The colon remains syntactically necessary in some cases, particularly to disambiguate cases involving one-lining (no block involved). Stupid example: If the colon is optional, what does:

I was only proposing making it optional in the multi-line case.

The following in the grammar:

[anywhere it appears]: ... ':' suite suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

would be replaced with

[whatever]: ... compound_body compound_body: ':' simple_stmt | [':'] NEWLINE INDENT stmt+ DEDENT

(I don't understand how TYPE_COMMENT interacts with NEWLINE/INDENT/DEDENT well enough to do the same for func_body_suite in terms of the actual grammar) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UGS2WP... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 2020-01-16 21:08, Rob Cliffe via Python-ideas wrote:

On 16/01/2020 18:14:18, Random832 wrote:

...

On Tue, Jan 14, 2020, at 18:15, David Mertz wrote:

...

For what it's worth, after 20+ years of using Python, forgetting the colon for blocks remains the most common error I make by a fairly wide margin. Of course, once I see the error message—even being not all that descriptive of the real issue—I immediately know what to fix too. I don't know about "most common" but I do it fairly frequently. What if the colon were made optional, with an eye to perhaps eventually no longer using it as the preferred style for new code?

We had a post a while ago about the possibility of using the lack of a colon as an implicit line continuation (like with parentheses, e.g. "if a\nand b:", and this was (reasonably) rejected. But if a line beginning as a compound statement and ending without a colon is *never* going to have a valid meaning as something else... what's the point of the colon, otherwise? Seems like just grit on the screen. +1

FWIW, I've hardly ever forgotten the colon, even when Python was new to me.