On 9/20/2021 11:48 AM, Eric V. Smith wrote:
When I initially wrote f-strings, it was an explicit design goal to be just like existing strings, but with a new prefix. That's why there are all of the machinations in the parser for scanning within f-strings: the parser had already done its duty, so there needed to be a separate stage to decode inside the f-strings. Since they look just like regular strings, most tools could add the lowest possible level of support just by adding 'f' to existing prefixes they support: 'r', 'b', 'u'.
Which is what I did with IDLE. Of course 'just add' was complicated by uppercase being allowed and 'f' being compatible with 'r' but not 'u' or 'b'.
I definitely share your concern about making f-strings more complicated to parse for tool vendors: basically all editors, alternative implementations, etc.: really anyone who parses python source code. But maybe we've already crossed this bridge with the PEG parser.
I think we are on the far side of the bridge with contextual keywords. I don't believe the new code for highlighting the new match statement is exactly correct. As I remember, properly classifying '_' in all the examples we created was too difficult, and maybe not possible.
Although I realize there's a difference between lexing and parsing. While the PEG parser just makes parsing more complicated, this change would make what was lexing into a more sophisticated parsing problem.
I have no love for the RE code. I would try ast.parse if I was not sure it would be too slow. I would be happy if a simplified and fast minimal lexer/parser were added for everyone to use. It would not have to make exactly the same distinctions that IDLE currently does. -- Terry Jan Reedy