To bring this back on track, I'll try and answer the questions from your original email.

I have started a project to move the parsing off-strings to the parser and the grammar. Appart

from some maintenance improvements (we can drop a considerable amount of hand-written code),

there are some interesting things we **could** (emphasis on could) get out of this and I wanted

to discuss what people think about them.

I think this is all awesome.

My position is that if we make zero syntactic changes to f-strings, and leave the functionality exactly as it is today, I think we should still move the logic into the parser and grammar, as you suggested. As you say, this would eliminate a lot of code, and in addition likely get us better error messages. As for the things we could possibly add:

* The parser will likely have "\n" characters and backslashes in f-strings expressions, which currently is impossible:

>>> f"blah blah {'\n'} blah"
  File "<stdin>", line 1
    f"blah blah {'\n'} blah"
                            ^
SyntaxError: f-string expression part cannot include a backslash

I think supporting backslashes in strings inside of f-string expression (the part inside {}) would be a big win, and should be the first thing we allow. I often have to do this:

nl = '\n'
x = f"blah {nl if condition else ' '}"

Being able to write this more naturally would be a big win.

I don't recall exactly why, but I disallowed backslashes inside expressions at the last minute before 3.6 was released. It might have been because I was interpreting them in a way that didn't make sense if a "real" parser were inspecting f-strings. The idea, even back then, was to re-allow them when/if we moved f-string parsing into the parser itself. I think it's time.

* The parser will allow nesting quote characters. This means that we **could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"

* The parser will naturally allow more control over error messages and AST positions.

* The **grammar** of the f-string will be fully specified without ambiguity. Currently, the "grammar" that we have in the docs

(https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals) is not really formal grammar because

not only is mixing lexing details with grammar details (the definition of "literal_char") but also is not compatible with the current python

lexing schema (for instance, it recognizes "{{" as its own token, which the language doesn't allow because something like "{{a:b}:c}"

is tokenized as "{", "{", "a" ... not as "{{", "a". Adding a formal grammar could help syntax highlighters, IDEs, parsers and other tools

to make sure they properly recognize everything that there is.

There may be some other advantages that we have not explored still.

The work is at a point where the main idea works (all the grammar is already there and working), but we need to make sure that all existing

errors and specifics are properly ported to the new code, which is a considerable amount of work still so I wanted to make sure we are on the

same page before we decide to invest more time on this (Batuhan is helping me with this and Lyssandros will likely join us). We are doing

this work in this branch: https://github.com/we-like-parsers/cpython/blob/fstring-grammar

Tell me what you think.

P.S. If you are interested to help with this project, please reach out to me. If we decide to go ahead we can use your help! :)

I'm interested in helping.

Thanks for your work on this.

Eric