Let’s make escaping in f-literals impossible
Hi, I originially posted this via google groups, which didn’t make it through to the list proper, sorry! Read it here please: https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s My arguments are basically: 1. f-literals are semantically not strings, but expressions. 2. Their escape sequences in the code parts are fundamentally both detrimental and superfluous (they’re only in for convenience, as confirmed by Guido in the quote below) 3. They’re detrimental because Syntax highlighters are (by design) unable to handle this part of Python 3.6a4’s grammar. This will cause code to be highlighted as parts of a string and therefore overlooked. i’m very sure this will cause bugs. 4. The fact that people see the embedded expressions as somehow “part of the string” is confusing. My poposal is to redo their grammar: They shouldn’t be parsed as strings and post-processed, but be their own thing. This also opens the door to potentially extend to with something like JavaScript’s tagged templates) Without the limitations of the string tokenization code/rules, only the string parts would have escape sequences, and the expression parts would be regular python code (“holes” in the literal). Below the mentioned quote and some replies to the original thread: Guido van Rossum <guido@python.org> schrieb am Mi., 17. Aug. 2016 um 20:11 Uhr:
The explanation is honestly that the current approach is the most straightforward for the implementation (it's pretty hard to intercept the string literal before escapes have been processed) and nobody cares enough about the edge cases to force the implementation to jump through more hoops.
I really don't think this discussion should be reopened. If you disagree, please start a new thread on python-ideas.
I really think it should. Please look at python code with f-literals. if they’re highlighted as strings throughout, you won’t be able to spot which parts are code. if they’re highlighted as code, the escaping rules guarantee that most highlighters can’t correctly highlight python anymore. i think that’s a big issue for readability. Brett Cannon <brett@python.org> schrieb am Mi., 17. Aug. 2016 um 20:28 Uhr:
They are still strings, there is just post-processing on the string itself to do the interpolation.
Sounds hacky to me. I’d rather see a proper parser for them, which of course would make my vision easy.
By doing it this way the implementation can use Python itself to do the tokenizing of the string, while if you do the string interpolation beforehand you would then need to do it entirely at the C level which is very messy and painful since you're explicitly avoiding Python's automatic handling of Unicode, etc.
of course we reuse the tokenization for the string parts. as said, you can view an f-literal as interleaved sequence of strings and expressions with an attached format specification. <f'> starts the f-literal, string contents follow. the only difference to other strings is <{> which starts expression tokenization. once the expression ends, an optional <formatspec> follows, then a <}> to switch back to string tokenization this repeats until (in string parsing mode) a <'> is encountered which ends the f-literal. You also make it harder to work with Unicode-based variable names (or at
least explain it). If you have Unicode in a variable name but you can't use \N{} in the string to help express it you then have to say "normal Unicode support in the string applies everywhere *but* in the string interpolation part".
i think you’re just proving my point that the way f-literals work now is confusing. the embedded expressions are just normal python. the embedded strings just normal strings. you can simply switch between both using <{> and <[format]}>. unicode in variable names works exactly the same as in all other python code because it is regular python code. Or another reason is you can explain f-strings as "basically
str.format_map(**locals(), **globals()), but without having to make the actual method call" (and worrying about clashing keys but I couldn't think of a way of using dict.update() in a single line). But with your desired change it kills this explanation by saying f-strings aren't like this but some magical string that does all of this stuff before normal string normalization occurs.
no, it’s simply the expression parts (that for normal formatting are inside of the braces of .format(...)) are *interleaved* in between string parts. they’re not part of the string. just regular plain python code. Cheers, and i really hope i’ve made a strong case, philipp
On Fri, Aug 19, 2016 at 1:05 AM, Philipp A. <flying-sheep@web.de> wrote:
the embedded expressions are just normal python. the embedded strings just normal strings. you can simply switch between both using <{> and <[format]}>.
The trouble with that way of thinking is that, to a human, the braces contain something. They don't "uncontain" it. Those braced expressions are still part of a string; they just have this bit of magic that gets them evaluated. Consider this:
"This is a number: {:0\u07c4}".format(13) 'This is a number: 0013'
Format codes are just text, so I should be able to use Unicode escapes. Okay. Now let's make that an F-string.
f"This is a number: {13:0\u07c4}" 'This is a number: 0013'
Format codes are still just text. So you'd have to say that the rules of text stop at an unbracketed colon, which is a pretty complicated rule to follow. The only difference between .format and f-strings is that the bit before the colon is the actual expression, rather than a placeholder that drags the value in from the format arguments. In human terms, that's not all that significant. IMO it doesn't matter that much either way - people will have to figure stuff out anyway. I like the idea that everything in the quotes is a string (and then parts of it get magically evaluated), but could live with there being some non-stringy parts in it. My suspicion is that what's easiest to code (ie easiest for the CPython parser) is also going to be easiest for all or most other tools (eg syntax highlighters). ChrisA
On Thu, Aug 18, 2016, at 12:17, Chris Angelico wrote:
The trouble with that way of thinking is that, to a human, the braces contain something. They don't "uncontain" it. Those braced expressions are still part of a string; they just have this bit of magic that gets them evaluated. Consider this:
There's a precedent. "$()" works this way in bash - call it a recursive parser context or whatever you like, but the point is that "$(command "argument with spaces")" works fine, and humans don't seem to have any trouble with it. Really it all comes down to what exactly the "bit of magic" is and how magical it is.
IMO it doesn't matter that much either way - people will have to figure stuff out anyway. I like the idea that everything in the quotes is a string (and then parts of it get magically evaluated), but could live with there being some non-stringy parts in it. My suspicion is that what's easiest to code (ie easiest for the CPython parser) is also going to be easiest for all or most other tools (eg syntax highlighters).
Except the parser has to actually parse string literals into what string they represent (so it can apply a further transformation to the result). Syntax highlighters generally don't.
On Thu, Aug 18, 2016 at 12:26:26PM -0400, Random832 wrote:
There's a precedent. "$()" works this way in bash - call it a recursive parser context or whatever you like, but the point is that "$(command "argument with spaces")" works fine, and humans don't seem to have any trouble with it.
This is the first time I've ever seen anyone claim that humans don't have any trouble with bash escaping and evaluation rules. -- Steve
Chris Angelico wrote:
f"This is a number: {13:0\u07c4}"
If I understand correctly, the proposal intends to make it easier for a syntax hightlighter to treat f"This is a number: {foo[42]:0\u07c4}" as f"This is a number: { foo[42] :0\u07c4}" --------------------- ------- ---------- highlight as string hightlight highlight as string as code I'm not sure an RE-based syntax hightlighter would have any easier a time with that, because for the second part it would need to recognise ':' as starting a string, but only if it followed some stuff that was preceded by the beginning of an f-string. I'm not very familiar with syntax higlighters, so I don't know if they're typically smart enought to cope with things like that. -- Greg
On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote:
Format codes are just text,
I really think that is wrong. They're more like executable code. https://www.python.org/dev/peps/pep-0498/#expression-evaluation "Just text" implies it is data: result = "function(arg)" like the string on the right hand side of the = is data. You wouldn't say that a function call was data (although it may *return* data): result = function(arg) or that it was "just text", and you shouldn't say the same about: result = f"{function(arg)}" either since they are functionally equivalent. Format codes are "just text" only in the sense that source code is "just text". Its technically correct and horribly misleading.
so I should be able to use Unicode escapes. Okay. Now let's make that an F-string.
f"This is a number: {13:0\u07c4}" 'This is a number: 0013'
If your aim is to write obfuscated code, then, yes, you should be able to write something like that. *wink* I seem to recall that Java allows string escapes in ordinary expressions, so that instead of writing: result = function(arg) you could write: result = \x66\x75\x6e\x63\x74\x69\x6f\x6e\x28\x61\x72\x67\x29 instead. We can't, and shouldn't, allow anything like this in Python code. Should we allow it inside f-strings? result = f"{\x66\x75\x6e\x63\x74\x69\x6f\x6e\x28\x61\x72\x67\x29}" -- Steve
On Fri, Aug 19, 2016 at 10:18 AM, Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote:
Format codes are just text,
I really think that is wrong. They're more like executable code.
https://www.python.org/dev/peps/pep-0498/#expression-evaluation
"Just text" implies it is data:
result = "function(arg)"
like the string on the right hand side of the = is data. You wouldn't say that a function call was data (although it may *return* data):
result = function(arg)
or that it was "just text", and you shouldn't say the same about:
result = f"{function(arg)}"
either since they are functionally equivalent. Format codes are "just text" only in the sense that source code is "just text". Its technically correct and horribly misleading.
By "format code", I'm talking about the bit after the colon, which isn't executable code, but is a directive that says how the result is to be formatted. These have existed since str.format() was introduced, and have always been text, not code. ChrisA
On 8/18/2016 8:18 PM, Steven D'Aprano wrote:
On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote:
Format codes are just text,
I really think that is wrong. They're more like executable code.
https://www.python.org/dev/peps/pep-0498/#expression-evaluation
I agree with you here. I just note that the strings passed to exec, eval, and compile are also executable code strings (and nothing but!). But I don't remember a suggestion that *they* should by colored as anything other than a string. However, this thread has suggested to me that perhaps there *should* be a way to syntax check such strings in the editor rather than waiting for the runtime call. -- Terry Jan Reedy
בתאריך יום ו׳, 19 באוג' 2016, 08:29, מאת Terry Reedy <tjreedy@udel.edu>:
On 8/18/2016 8:18 PM, Steven D'Aprano wrote:
On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote:
Format codes are just text,
I really think that is wrong. They're more like executable code.
https://www.python.org/dev/peps/pep-0498/#expression-evaluation
I agree with you here. I just note that the strings passed to exec, eval, and compile are also executable code strings (and nothing but!). But I don't remember a suggestion that *they* should by colored as anything other than a string.
But these are objects of type str, not string literals. If they were, I guess someone would have suggested such coloring. ~Elazar
On 8/19/2016 2:07 AM, אלעזר wrote:
בתאריך יום ו׳, 19 באוג' 2016, 08:29, מאת Terry Reedy <tjreedy@udel.edu <mailto:tjreedy@udel.edu>>:
On 8/18/2016 8:18 PM, Steven D'Aprano wrote: > On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote: > >> Format codes are just text, > > I really think that is wrong. They're more like executable code. > > https://www.python.org/dev/peps/pep-0498/#expression-evaluation
I agree with you here. I just note that the strings passed to exec, eval, and compile are also executable code strings (and nothing but!). But I don't remember a suggestion that *they* should by colored as anything other than a string.
But these are objects of type str, not string literals. If they were, I guess someone would have suggested such coloring.
I was referring to strings created by string literals in the code sitting in an editor. -- Terry Jan Reedy
I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals. I stepped out of the original discussion near the start as it looked like we were going to end up with interleaved strings and normal expressions, but if that's not the case then it is going to make it very difficult to provide a nice coding experience for them. On 18Aug2016 0805, Philipp A. wrote:
My poposal is to redo their grammar: They shouldn’t be parsed as strings and post-processed, but be their own thing. This also opens the door to potentially extend to with something like JavaScript’s tagged templates)
Without the limitations of the string tokenization code/rules, only the string parts would have escape sequences, and the expression parts would be regular python code (“holes” in the literal).
This is where I thought we'd end up - the '{' character (unless escaped by, e.g. \N, which addresses a concern below) would terminate the string literal and start an expression, which may be followed by a ':' and a format code literal. The '}' character would open the next string literal, and this continues until the closing quote.
They are still strings, there is just post-processing on the string itself to do the interpolation.
Sounds hacky to me. I’d rather see a proper parser for them
I believe the proper parser is already used, but the issue is that escapes have already been dealt with. Of course, it shouldn't be too difficult for the tokenizer to recognize {} quoted expressions within an f-literal and not modify escapes. There are multiple ways to handle this.
Or another reason is you can explain f-strings as "basically str.format_map(**locals(), **globals()), but without having to make the actual method call" (and worrying about clashing keys but I couldn't think of a way of using dict.update() in a single line). But with your desired change it kills this explanation by saying f-strings aren't like this but some magical string that does all of this stuff before normal string normalization occurs.
no, it’s simply the expression parts (that for normal formatting are inside of the braces of .format(...)) are *interleaved* in between string parts. they’re not part of the string. just regular plain python code.
Agreed. The .format_map() analogy breaks down very quickly when you consider f-literals like:
f'a { \'b\' }' 'a b'
If the contents of the braces were simply keys in the namespace then we wouldn't be able to put string literals in there. But because it is an arbitrary expression, if we want to put string literals in the f-literal (bearing in mind that we may be writing something more like f'{x.partition(\'-\')[0]}'), the escaping rules become very messy very quickly. I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes, and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying the entire expression in a different colour). So I too would like to see escapes made unnecessary within the expression part of a f-literal. Possibly if we put together a simple enough patch for the tokenizer it will be accepted? Cheers, Steve
On 18Aug2016 0950, Steve Dower wrote:
I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals.
I also really don't like the subject line. "Do not require string escapes within expressions in f-literals" more accurately represents the topic and the suggestion. "Let's make <anything> impossible" is just asking for a highly emotionally-charged discussion, which is best avoided in basically all circumstances, especially for less-frequent contributors to a community, and extra-especially when you haven't met most of the other contributors in person. Cheers, Steve
On 8/18/2016 12:50 PM, Steve Dower wrote:
I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals.
I consider these separate issues. IDLE currently provides filename completion support within strings while still highlighting the string one color. Even if it were enhanced to do name completion within an f-string, I am not sure I would want to put a mixture of colors within the string rather than leave if all one color.
I stepped out of the original discussion near the start as it looked like we were going to end up with interleaved strings and normal expressions, but if that's not the case then it is going to make it very difficult to provide a nice coding experience for them.
This is the crux of this thread. Is an f-string a single string that contains magically handled code, or interleaved strings using { and } as closing and opening quotes (which is backwards from their normal function of being opener and closer) and expressions? The latter view makes the grammar context sensitive, I believe, as } could only open a string if there is a previous f-tagged string an indefinite number of alternations back. It is not uncommon to write strings that consist completely of code. "for i in iterable: a.append(f(i))" to be written out or eval()ed or exec()ed. Does your environment have a mode to provide syntax highlighting and completion support for such things? What I think would be more useful would be the ability to syntax check such code strings while editing. A python-coded editor could just pass the extracted string to compile().
I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes,
I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation.
and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying the entire expression in a different colour).
Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.partition('if')[0]}', the 'if' is and will be erroneously highlighted as a keyword. I consider this breakage unacceptible. -- Terry Jan Reedy
On Thu, Aug 18, 2016, at 15:15, Terry Reedy wrote:
This is the crux of this thread. Is an f-string a single string that contains magically handled code, or interleaved strings using { and } as closing and opening quotes (which is backwards from their normal function of being opener and closer)
I'd rather conceptualize it as a sequence of two* kinds of thing: literal character sequences [as sequences of characters other than {] and expressions [started with {, and ended with a } that is not otherwise part of the expression] rather than treating { as a closing quote. In particular, treating } as an opening quote doesn't really work, since expressions can contain both strings (which may contain an unbalanced }) and dictionary/set literals (which contain balanced }'s which are not in quotes) - what ends the expression is a } at the top level. *or three, considering that escapes are used in the non-expression parts.
and expressions? The latter view makes the grammar context sensitive, I believe, as } could only open a string if there is a previous f-tagged string an indefinite number of alternations back.
} at the top level is otherwise a syntax error. I don't know enough about the theoretical constructs involved to know if this makes it formally 'context sensitive' or not - I don't know that it's any more context sensitive than ) being valid if there is a matching (. Honestly, I'd be more worried about : than }.
On 18Aug2016 1215, Terry Reedy wrote:
On 8/18/2016 12:50 PM, Steve Dower wrote:
I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes,
I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation.
That's a fair counter-example. Though f'{x.partition(\' + \')[0]}' still reads like string concatenation to me at first glance. YMMV.
and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying the entire expression in a different colour).
Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.partition('if')[0]}', the 'if' is and will be erroneously highlighted as a keyword. I consider this breakage unacceptible.
Won't it be broken anyway because of the new prefix? I'm sure there's a fairly straightforward way for a regex to say that a closing quote must not be preceded immediately by a backslash or by an open brace at all without a closing brace in between. Not having escapes within the expression makes it harder for everyone except the Python developer, in my opinion, and the rest of us ought to go out of our way for them. Cheers, Steve
On 8/18/2016 3:30 PM, Steve Dower wrote:
On 18Aug2016 1215, Terry Reedy wrote:
On 8/18/2016 12:50 PM, Steve Dower wrote:
I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes,
Why are you reusing the single quote', which needs the escaping that you don't like, instead of any of at least 6 alternatives that do not need any escaping? f'{x.partition("-")[0]}' f'{x.partition("""-""")[0]}' f"{x.partition('-')[0]}" f'''{x.partition('-')[0]}''' f"""{x.partition('-')[0]}""" f"""{x.partition('''-''')[0]}""" It seems to me that that this is at least somewhat a strawman issue. If you want to prohibit backslashed quote reuse in expressions, as in f'{x.partition(\'-\')[0]}', that is okay with me, as this is unnecessary* and arguably bad. The third alternative above is better. What breaks colorizers, and what I therefore object to, is the innovation of adding magical escaping of ' or " without \. Or add a new style rule to PEP 8. F-strings: avoid unnecessary escaping in the expression part of f-strings. Good: f"{x.partition('-')[0]}" Bad: f'{x.partition(\'-\')[0]}' Then PEP-8 checkers will flag such usage. *I am sure that there are possible complex expressions that would be prohibited by the rule that would be otherwise possible. But they should be extremely rare and possibly not the best solution anyway.
I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation.
That's a fair counter-example. Though f'{x.partition(\' + \')[0]}' still reads like string concatenation to me at first glance. YMMV.
When the outer and inner quotes are no longer the same, the effect is greatly diminished if not eliminated.
and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying the entire expression in a different colour).
Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.partition('if')[0]}', the 'if' is and will be erroneously highlighted as a keyword. I consider this breakage unacceptible.
Won't it be broken anyway because of the new prefix?
No. IDLE currently handles f-strings just fine other than not coloring the 'f'. This is a minor issue and easily fixed by adding '|f' and if allowed, '|F' at the end of the current stringprefix re.
I'm sure there's a fairly straightforward way for a regex to say that a closing quote must not be preceded immediately by a backslash or by an open brace at all without a closing brace in between.
I do not know that this is possible. Here is IDLE's current re for an unprefixed single quoted string. r"'[^'\\\n]*(\\.[^'\\\n]*)*'?" The close quote is optional because it must match a string that is in the process of being typed and is not yet closed. I consider providing a tested augmented re to be required for this proposal. Even then, making the parsing out of strings in Python code for colorizing version dependent is a problem in itself for colorizers not tied to a particular x.y version. Leaving prefixes aside, I can't remember string delimiter syntax changing since I learned it in 1.3.
Not having escapes within the expression makes it harder for everyone except the Python developer, in my opinion, and the rest of us ought to go out of our way for them.
I am not sure that this says what you mean. -- Terry Jan Reedy
On 19.08.16 08:07, Terry Reedy wrote:
On 8/18/2016 3:30 PM, Steve Dower wrote:
On 18Aug2016 1215, Terry Reedy wrote:
On 8/18/2016 12:50 PM, Steve Dower wrote:
I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes,
Why are you reusing the single quote', which needs the escaping that you don't like, instead of any of at least 6 alternatives that do not need any escaping?
f'{x.partition("-")[0]}' f'{x.partition("""-""")[0]}' f"{x.partition('-')[0]}" f'''{x.partition('-')[0]}''' f"""{x.partition('-')[0]}""" f"""{x.partition('''-''')[0]}"""
It seems to me that that this is at least somewhat a strawman issue.
If you want to prohibit backslashed quote reuse in expressions, as in f'{x.partition(\'-\')[0]}', that is okay with me, as this is unnecessary* and arguably bad. The third alternative above is better. What breaks colorizers, and what I therefore object to, is the innovation of adding magical escaping of ' or " without \.
Or add a new style rule to PEP 8.
+1. It is even possible to add a SyntaxWarning in future.
On 19 August 2016 at 06:07, Terry Reedy <tjreedy@udel.edu> wrote:
It seems to me that that this is at least somewhat a strawman issue.
If you want to prohibit backslashed quote reuse in expressions, as in f'{x.partition(\'-\')[0]}', that is okay with me, as this is unnecessary* and arguably bad. The third alternative above is better. What breaks colorizers, and what I therefore object to, is the innovation of adding magical escaping of ' or " without \.
Or add a new style rule to PEP 8.
F-strings: avoid unnecessary escaping in the expression part of f-strings. Good: f"{x.partition('-')[0]}" Bad: f'{x.partition(\'-\')[0]}'
Then PEP-8 checkers will flag such usage.
+1. While substantial IDEs like PyCharm or PTVS may use a full-scale parser to do syntax highlighting, I suspect that many tools just use relatively basic regex parsing (Vim certainly does). For those tools, unescaped nested quotes will likely be extremely difficult, if not impossible, to parse correctly. Whereas the current behaviour is "just" standard string highlighting. So if the Python parser were to change as proposed, I'd still argue strongly for a coding style that never uses any construct that would be interpreted differently from current behaviour (i.e., the changed behaviour should essentially be irrelevant). Given this, I thing the argument to change, whether it's theoretically an improvement or not, is irrelevant, and practicality says there's no point in bothering. (Python's parser is intentionally simple, to make it easy for humans and tools to parse Python code - I'm not sure the proposed change to syntax meets that guideline for simple syntax). Paul
On 8/18/2016 3:15 PM, Terry Reedy wrote:
On 8/18/2016 12:50 PM, Steve Dower wrote: I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation.
and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying the entire expression in a different colour).
Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.partition('if')[0]}', the 'if' is and will be erroneously highlighted as a keyword. I consider this breakage unacceptible.
Right. Because all strings (regardless of prefixes) are first parsed as strings, and then have their prefix "operator" applied, it's easy for a parser to ignore any sting prefix character. So something that parses or scans a Python file and currently understands u, b, and r to be string prefixes, just needs to add f to the prefixes it uses, and it can now at least understand f-strings (and fr-strings). It doesn't need to implement a full-blown expression parser just to find out where the end of a f-string is. Eric.
On 8/18/2016 8:27 PM, Eric V. Smith wrote:
On 8/18/2016 3:15 PM, Terry Reedy wrote:
Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.partition('if')[0]}', the 'if' is and will be erroneously highlighted as a keyword. I consider this breakage unacceptible.
Right. Because all strings (regardless of prefixes) are first parsed as strings, and then have their prefix "operator" applied, it's easy for a parser to ignore any sting prefix character.
So something that parses or scans a Python file and currently understands u, b, and r to be string prefixes, just needs to add f to the prefixes it uses, and it can now at least understand f-strings (and fr-strings). It doesn't need to implement a full-blown expression parser just to find out where the end of a f-string is.
Indeed, IDLE has one prefix re, which has changed occasionally and which I need to change for 3.6, and 4 res for the 4 unprefixed strings, which have been the same, AFAIK, for decades. It that prefixes all 4 string res with the prefix re and o or's the results together to get the 'string' re. -- Terry Jan Reedy
On 8/19/2016 1:16 AM, Terry Reedy wrote:
On 8/18/2016 8:27 PM, Eric V. Smith wrote:
So something that parses or scans a Python file and currently understands u, b, and r to be string prefixes, just needs to add f to the prefixes it uses, and it can now at least understand f-strings (and fr-strings). It doesn't need to implement a full-blown expression parser just to find out where the end of a f-string is.
Indeed, IDLE has one prefix re, which has changed occasionally and which I need to change for 3.6, and 4 res for the 4 unprefixed strings, which have been the same, AFAIK, for decades. It that prefixes all 4 string res with the prefix re and o or's the results together to get the 'string' re.
For something else that would become significantly more complicated to implement, you need look no further than the stdlib's own tokenizer module. So Python itself would require changes to parsers/lexers in Python/ast.c, IDLE, and Lib/tokenizer.py. In addition it would require adding tokens to Include/tokens.h and the generated Lib/token.py, and everyone using those files would need to adapt. Not that it's impossible, of course. But don't underestimate the amount of work this proposal would cause to the many places in and outside of Python that examine Python code. Eric.
On 19 August 2016 at 18:27, Eric V. Smith <eric@trueblade.com> wrote:
For something else that would become significantly more complicated to implement, you need look no further than the stdlib's own tokenizer module. So Python itself would require changes to parsers/lexers in Python/ast.c, IDLE, and Lib/tokenizer.py. In addition it would require adding tokens to Include/tokens.h and the generated Lib/token.py, and everyone using those files would need to adapt.
Not that it's impossible, of course. But don't underestimate the amount of work this proposal would cause to the many places in and outside of Python that examine Python code.
And if folks want to do something more clever than regex based single colour string highlighting, Python's own AST module is available to help them out: >>> tree = ast.parse("f'example{parsing:formatting}and trailer'") >>> ast.dump(tree) "Module(body=[Expr(value=JoinedStr(values=[Str(s='example'), FormattedValue(value=Name(id='parsing', ctx=Load()), conversion=-1, format_spec=Str(s='formatting')), Str(s='and trailer')]))])" Extracting the location of the field expression for syntax highlighting: >>> ast.dump(tree.body[0].value.values[1].value) "Name(id='parsing', ctx=Load())" (I haven't shown it in the example, but AST nodes have lineno and col_offset fields so you can relocate the original source code for processing) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Aug 18, 2016 at 08:27:50PM -0400, Eric V. Smith wrote:
Right. Because all strings (regardless of prefixes) are first parsed as strings, and then have their prefix "operator" applied, it's easy for a parser to ignore any sting prefix character.
Is that why raw strings can't end with a backspace? If so, that's the first time I've seen an explanation of that fact which makes sense! -- Steve
On Thu, 18 Aug 2016 at 08:32 Philipp A. <flying-sheep@web.de> wrote:
[SNIP] Brett Cannon <brett@python.org> schrieb am Mi., 17. Aug. 2016 um 20:28 Uhr:
They are still strings, there is just post-processing on the string itself to do the interpolation.
Sounds hacky to me. I’d rather see a proper parser for them, which of course would make my vision easy.
You say "hacky", I say "pragmatic". And Python's code base is actually rather top-notch and so it isn't bad code, but simply a design decision you are disagreeing with. Please remember that you're essentially asking people to spend their personal time to remove working code and re-implement something that you have not volunteered to actually code up yourself. Don't forget that none of us get paid to work on Python full-time; a lucky couple of us get to spend one day a week on Python and we all take time away from our family to work on things when we can. Insulting someone's hard work that they did for free to try and improve Python is not going to motivate people to want to help out with this idea. And considering Eric Smith who originally implemented all of this is possibly the person in the best position to implement your idea just had his work called "hacky" by you is not really a great motivator for him. IOW you really need to be mindful of the tone of your emails (as does anyone else who ever asks for something to change while not being willing to put in the time and effort to actually produce the code to facilitate the change). You have now had both Steve and me point out your tone and so you're quickly approaching a threshold where people will stop pointing this out and simply ignore your emails, so please be mindful of how you phrase things.
On 8/18/2016 11:05 AM, Philipp A. wrote:
Hi, I originially posted this via google groups, which didn’t make it through to the list proper, sorry! Read it here please: https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s
Hi, Philipp. I'm including your original proposal here, so that it's archived properly. Here's what you'd like to have work: f'foo{repr('bar\n') * 3}baz' == 'foo"bar\n""bar\n""bar\n"baz' We've had this discussion before, but I can't find it in the archives. It's possible it happened off-list. The problem I have with your proposal is that it greatly complicates the implementation and it makes it harder for humans to reason about the code (for the same reasons). This is not just an ease of implementation issue: it's a cognitive burden issue. As it currently stands, Python strings, of all 'flavors' (raw, unicode, binary, and f-), are parsed the same way: an optional prefix, one or three quote characters, the body of the string, then matching quote characters. This is sufficiently simple that it can be (and often is) implemented with regular expressions. You also need to support combinations of prefixes, like raw f-strings (fr or rf). With your proposal, it's much more difficult to find the end of an f-string. I do not think this is a reasonable requirement. For example, consider the following: f'a{func({'a{':1,'3}':[{}, ')', '{']})}b' A regex will not be able to deal with the matching braces needed to find the end of the expression. You need to keep track of the nesting level of parens, braces, brackets, and quotes (at least those, I might have left something off). The way this currently is written in Python 3.6a4: f"a{func({'a{':1,'3}':[{}, ')', '{']})}b" It's trivially easy to find the end of the string. It's easy for both humans and the parsers. Now admittedly in order to execute or syntax highlight the existing f-strings, you need to perform this parsing. But of the 3 parsers that ship with Python (ast.c, tokenize.py, IDLE), only ast.c needs to do that currently. I don't think tokenize.py ever will, and IDLE might (but could use the ast module). I think many parsers (e.g. python-mode.el, etc.) will just be able to simply consume the f-strings without looking inside them and move one, and we shouldn't unnecessarily complicate them.
My arguments are basically:
1. f-literals are semantically not strings, but expressions. 2. Their escape sequences in the code parts are fundamentally both detrimental and superfluous (they’re only in for convenience, as confirmed by Guido in the quote below)
I disagree that they're detrimental and superfluous. I'd say they're consistent with all other strings.
3. They’re detrimental because Syntax highlighters are (by design) unable to handle this part of Python 3.6a4’s grammar. This will cause code to be highlighted as parts of a string and therefore overlooked. i’m very sure this will cause bugs.
I disagree.
4. The fact that people see the embedded expressions as somehow “part of the string” is confusing.
My poposal is to redo their grammar: They shouldn’t be parsed as strings and post-processed, but be their own thing. This also opens the door to potentially extend to with something like JavaScript’s tagged templates)
Without the limitations of the string tokenization code/rules, only the string parts would have escape sequences, and the expression parts would be regular python code (“holes” in the literal).
Below the mentioned quote and some replies to the original thread:
Guido van Rossum <guido@python.org <mailto:guido@python.org>> schrieb am Mi., 17. Aug. 2016 um 20:11 Uhr:
The explanation is honestly that the current approach is the most straightforward for the implementation (it's pretty hard to intercept the string literal before escapes have been processed) and nobody cares enough about the edge cases to force the implementation to jump through more hoops.
I really don't think this discussion should be reopened. If you disagree, please start a new thread on python-ideas.
I really think it should. Please look at python code with f-literals. if they’re highlighted as strings throughout, you won’t be able to spot which parts are code. if they’re highlighted as code, the escaping rules guarantee that most highlighters can’t correctly highlight python anymore. i think that’s a big issue for readability.
Maybe I'm the odd man out, but I really don't care if my editor ever syntax highlights within f-strings. I don't plan on putting anything more complicated than variable names in my f-strings, and I think PEP 8 should recommend something similar.
Brett Cannon <brett@python.org <mailto:brett@python.org>> schrieb am Mi., 17. Aug. 2016 um 20:28 Uhr:
They are still strings, there is just post-processing on the string itself to do the interpolation.
Sounds hacky to me. I’d rather see a proper parser for them, which of course would make my vision easy.
You're saying if such a parser existed, it would be easy to use it to parse your version of f-strings? True enough! And as Nick points out, such a thing already exists in the ast module. But if your proposal were accepted, using such an approach would not be optional (only if you care about the inside of an f-string), it would be required to parse any python code even if you don't care about the contents of f-strings.
By doing it this way the implementation can use Python itself to do the tokenizing of the string, while if you do the string interpolation beforehand you would then need to do it entirely at the C level which is very messy and painful since you're explicitly avoiding Python's automatic handling of Unicode, etc.
of course we reuse the tokenization for the string parts. as said, you can view an f-literal as interleaved sequence of strings and expressions with an attached format specification.
<f'> starts the f-literal, string contents follow. the only difference to other strings is <{> which starts expression tokenization. once the expression ends, an optional <formatspec> follows, then a <}> to switch back to string tokenization this repeats until (in string parsing mode) a <'> is encountered which ends the f-literal.
You also make it harder to work with Unicode-based variable names (or at least explain it). If you have Unicode in a variable name but you can't use \N{} in the string to help express it you then have to say "normal Unicode support in the string applies everywhere *but* in the string interpolation part".
i think you’re just proving my point that the way f-literals work now is confusing.
the embedded expressions are just normal python. the embedded strings just normal strings. you can simply switch between both using <{> and <[format]}>.
unicode in variable names works exactly the same as in all other python code because it is regular python code.
Or another reason is you can explain f-strings as "basically str.format_map(**locals(), **globals()), but without having to make the actual method call" (and worrying about clashing keys but I couldn't think of a way of using dict.update() in a single line). But with your desired change it kills this explanation by saying f-strings aren't like this but some magical string that does all of this stuff before normal string normalization occurs.
no, it’s simply the expression parts (that for normal formatting are inside of the braces of .format(...)) are *interleaved* in between string parts. they’re not part of the string. just regular plain python code.
I think that's our disagreement. I do see them as part of the string.
Cheers, and i really hope i’ve made a strong case,
Thanks for your concern and your comments. But you've not swayed me.
philipp
Eric.
I don't think we should take action now. Would it make sense, as a precaution, to declare the PEP provisional for one release? Then we can develop a sense of whether the current approach causes real problems. We could also emit some kind of warning if the expression part contains an escaped quote, since that's where a potential change would cause breakage. (Or we could leave that to the linters.) -- --Guido van Rossum (python.org/~guido)
On 19Aug2016 1157, Guido van Rossum wrote:
I don't think we should take action now.
Would it make sense, as a precaution, to declare the PEP provisional for one release? Then we can develop a sense of whether the current approach causes real problems.
We could also emit some kind of warning if the expression part contains an escaped quote, since that's where a potential change would cause breakage. (Or we could leave that to the linters.)
After reading the responses, I like the idea of explicitly discouraging any sort of string escapes within the expression (whether quotes or special characters), but think it's best left for the linters and style guides. Basically, avoid writing a literal f'{expr}' where you'd need to modify expr at all to rewrite it as:
x = expr f'{x}'
We will almost certainly be looking to enable code completions and syntax highlighting in Visual Studio within expressions, and we can't easily process the string and then parse it for this purpose, but I think we'll be able to gracefully degrade in cases where escapes that are valid in strings are not valid in code. Cheers, Steve
On 8/19/2016 2:57 PM, Guido van Rossum wrote:
I don't think we should take action now.
Would it make sense, as a precaution, to declare the PEP provisional for one release? Then we can develop a sense of whether the current approach causes real problems.
We could also emit some kind of warning if the expression part contains an escaped quote, since that's where a potential change would cause breakage. (Or we could leave that to the linters.)
If anything, I'd make it an error to have any backslashes inside the brackets of an f-string for 3.6. We could always remove this restriction at a later date. I say this because as soon as f-strings make it into the wild, we're going to have a hard time breaking code in say 3.7 by saying "well, we told you that f-strings might change". Although frankly, other than be executive fiat (which I'm okay with), I don't see us ever resolving the issue if f-strings are strings first, or if the brackets put you into "non-string mode". There are good arguments on both sides. Moving to the implementation details, I'm not sure how easy it would be to even find backslashes, though. IIRC, backslashes are replaced early, before the f-string parser really gets to process the string. It might require a new implementation of the f-string parser independent of regular strings, which I likely would not have time for before beta 1. Although since this would be a reduction in functionality, maybe it doesn't have to get done by then. I also haven't thought of how this would affect raw f-strings. In any event, I'll take a look at adding this restriction, just to get an estimate of the magnitude of work involved. The easiest thing to do might be to disallow backslashes in any part of an f-string for 3.6, although that seems to be going too far. Eric.
On Aug 20, 2016 1:32 PM, "Eric V. Smith" <eric@trueblade.com> wrote:
On 8/19/2016 2:57 PM, Guido van Rossum wrote:
I don't think we should take action now.
Would it make sense, as a precaution, to declare the PEP provisional for one release? Then we can develop a sense of whether the current approach causes real problems.
We could also emit some kind of warning if the expression part contains an escaped quote, since that's where a potential change would cause breakage. (Or we could leave that to the linters.)
If anything, I'd make it an error to have any backslashes inside the
brackets of an f-string for 3.6. We could always remove this restriction at a later date.
I say this because as soon as f-strings make it into the wild, we're
going to have a hard time breaking code in say 3.7 by saying "well, we told you that f-strings might change".
Although frankly, other than be executive fiat (which I'm okay with), I
don't see us ever resolving the issue if f-strings are strings first, or if the brackets put you into "non-string mode". There are good arguments on both sides.
Moving to the implementation details, I'm not sure how easy it would be
to even find backslashes, though. IIRC, backslashes are replaced early, before the f-string parser really gets to process the string. It might require a new implementation of the f-string parser independent of regular strings, which I likely would not have time for before beta 1. Although since this would be a reduction in functionality, maybe it doesn't have to get done by then.
I also haven't thought of how this would affect raw f-strings.
In any event, I'll take a look at adding this restriction, just to get an
estimate of the magnitude of work involved. The easiest thing to do might be to disallow backslashes in any part of an f-string for 3.6, although that seems to be going too far.
Eric.
Speaking of which, how is this parsed? f"{'\n'}" If escape-handling is done first, the expression is a string literal holding an actual newline character (normally illegal), rather than an escape sequence which resolves to a newline character. If that one somehow works, how about this? f"{r'\n'}" I guess you'd have to write one of these: f"{'\\n'}" f"{'''\n''')" rf"{'\n'}"
On Sun, Aug 21, 2016 at 5:51 PM, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
Speaking of which, how is this parsed? f"{'\n'}" If escape-handling is done first, the expression is a string literal holding an actual newline character (normally illegal), rather than an escape sequence which resolves to a newline character.
It's illegal.
If that one somehow works, how about this? f"{r'\n'}"
Also illegal.
I guess you'd have to write one of these: f"{'\\n'}" f"{'''\n''')" rf"{'\n'}"
Modulo the typo in the second one, these all result in the same code:
dis.dis(lambda: f"{'\\n'}") 1 0 LOAD_CONST 1 ('\n') 2 FORMAT_VALUE 0 4 RETURN_VALUE f"{'\\n'}" '\n'
ChrisA
Sorry for replying to late, i had an email issue. First two important things: 1. mental model and intuition and 2. precendence. About how to think of them: I’m strongly of the opinion that the mental models of either an alternating sequence of strings and formatted expressions, or a string with “holes” for expressions, are better than “a string where parts are magically evaluated after it’s created”. That smells like “eval” instead of the usual separation of data and code. That’s the same reason I dislike calling them “f-strings”: they’re undoubtedly **expressions evaluating to strings**, not string literals. Precedence exists in ruby’s and CoffeeScript’s string interpolation, Bash’s $(), JavaScript’s template literals, and many more: https://en.wikipedia.org/wiki/String_interpolation !!! *All* of them that support arbitrary code (And that I ever encountered) work the way I propose for python to work @Brett Cannon: Pragmatism is only good as long as it only compromises on elegance, not usability. I think “partly evauable f-strings” are harder to use than “f-literals with string and expression parts” @Chris Angelo: All those things being illegal is surprising, which is another argument in favor of my proposal. @Guido van Rossum I’d rather not like them to be preliminarily in the language in this form, considering Python’s track record of not changing preliminary things anymore…but: @Eriv V. Smith: Great idea with banning all backslashes for now. This is so close to release, so we could ban escape sequences and use all of the existing code, then write a new RFC to make sure things are optimal (which in my eyes means the holes/alternating sequence model instead of the thing we have now) Thank you all for your contributions to the discussion and again sorry for messing up and only now posting this correctly. Best, Philipp Chris Angelico <rosuav@gmail.com> schrieb am So., 21. Aug. 2016 um 09:57 Uhr:
On Sun, Aug 21, 2016 at 5:51 PM, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
Speaking of which, how is this parsed? f"{'\n'}" If escape-handling is done first, the expression is a string literal holding an actual newline character (normally illegal), rather than an escape sequence which resolves to a newline character.
It's illegal.
If that one somehow works, how about this? f"{r'\n'}"
Also illegal.
I guess you'd have to write one of these: f"{'\\n'}" f"{'''\n''')" rf"{'\n'}"
Modulo the typo in the second one, these all result in the same code:
dis.dis(lambda: f"{'\\n'}") 1 0 LOAD_CONST 1 ('\n') 2 FORMAT_VALUE 0 4 RETURN_VALUE f"{'\\n'}" '\n'
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 21 August 2016 at 03:32, Eric V. Smith <eric@trueblade.com> wrote:
If anything, I'd make it an error to have any backslashes inside the brackets of an f-string for 3.6. We could always remove this restriction at a later date.
+1 for this if you can find a way to do it - it eliminates the problematic cases where the order of evaluation makes a difference, and ensures the parts within the braces can be reliably processed as normal Python code.
In any event, I'll take a look at adding this restriction, just to get an estimate of the magnitude of work involved. The easiest thing to do might be to disallow backslashes in any part of an f-string for 3.6, although that seems to be going too far.
Disallowing \t, \n, etc even in the plain text parts of the f-string would indeed be problematic. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8/23/2016 8:18 AM, Nick Coghlan wrote:
On 21 August 2016 at 03:32, Eric V. Smith <eric@trueblade.com> wrote:
If anything, I'd make it an error to have any backslashes inside the brackets of an f-string for 3.6. We could always remove this restriction at a later date.
+1 for this if you can find a way to do it - it eliminates the problematic cases where the order of evaluation makes a difference, and ensures the parts within the braces can be reliably processed as normal Python code.
I've been looking at this, and I agree it's the best thing to do, for now (and possibly forever). I'm just not convinced I can get it done before alpha 1. Assuming I can get the coding done, I think I should update PEP 498 to say there can be no backslashes inside the curly braces. That's my preferred outcome. If I can't get it done by alpha 1, then I think the options are: 1. Leave f-strings as they are now, and that's how they'll always be. 2. Leave f-strings as they are now, but mark them as provisional and warn people that the backslash restrictions will show up in an upcoming release. 3. Disallow any backslashes anywhere in f-strings for 3.6, and relax the restriction in 3.7 to make it only inside braces where the restriction is enforced. 4. Remove f-strings from 3.6, and add them in 3.7 with the "no backslash inside braces" restriction. I'm not wild about 2: people will ignore this and will write code that will break in 3.7. I'm also not wild about 3, since it's too restrictive. I'm open to suggestions. Eric.
Oops, I meant beta 1 where I said alpha 1. Eric. On 8/29/2016 5:12 PM, Eric V. Smith wrote:
On 8/23/2016 8:18 AM, Nick Coghlan wrote:
On 21 August 2016 at 03:32, Eric V. Smith <eric@trueblade.com> wrote:
If anything, I'd make it an error to have any backslashes inside the brackets of an f-string for 3.6. We could always remove this restriction at a later date.
+1 for this if you can find a way to do it - it eliminates the problematic cases where the order of evaluation makes a difference, and ensures the parts within the braces can be reliably processed as normal Python code.
I've been looking at this, and I agree it's the best thing to do, for now (and possibly forever).
I'm just not convinced I can get it done before alpha 1. Assuming I can get the coding done, I think I should update PEP 498 to say there can be no backslashes inside the curly braces. That's my preferred outcome.
If I can't get it done by alpha 1, then I think the options are: 1. Leave f-strings as they are now, and that's how they'll always be. 2. Leave f-strings as they are now, but mark them as provisional and warn people that the backslash restrictions will show up in an upcoming release. 3. Disallow any backslashes anywhere in f-strings for 3.6, and relax the restriction in 3.7 to make it only inside braces where the restriction is enforced. 4. Remove f-strings from 3.6, and add them in 3.7 with the "no backslash inside braces" restriction.
I'm not wild about 2: people will ignore this and will write code that will break in 3.7. I'm also not wild about 3, since it's too restrictive.
I'm open to suggestions.
Eric.
On 08/29/2016 02:16 PM, Eric V. Smith wrote:
I've been looking at this, and I agree it's the best thing to do, for now (and possibly forever).
I'm just not convinced I can get it done before alpha 1.
Isn't the f-string feature already in place? Update the PEP, then it's a bugfix. ;) -- ~Ethan~
On 8/29/2016 5:26 PM, Ethan Furman wrote:
On 08/29/2016 02:16 PM, Eric V. Smith wrote:
I've been looking at this, and I agree it's the best thing to do, for now (and possibly forever).
I'm just not convinced I can get it done before alpha 1.
Isn't the f-string feature already in place?
Yes. It's been in 3.6 for quite a while (maybe a year?).
Update the PEP, then it's a bugfix. ;)
Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way. Or, I could stop worrying and typing emails, and instead just get on with it! Eric.
On 29Aug2016 1433, Eric V. Smith wrote:
On 8/29/2016 5:26 PM, Ethan Furman wrote:
Update the PEP, then it's a bugfix. ;)
Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way.
Or, I could stop worrying and typing emails, and instead just get on with it!
I like this approach :) But I agree. Release Manager Ned has the final say, but I think this change can comfortably go in during the beta period. (I also disagree that it's a big change - nobody could agree on the 'obvious' behaviour of backslashes anyway, so chances are people would avoid them anyway, and there was strong consensus on advising people to avoid them.) Cheers, Steve
On 8/29/2016 5:40 PM, Steve Dower wrote:
On 29Aug2016 1433, Eric V. Smith wrote:
On 8/29/2016 5:26 PM, Ethan Furman wrote:
Update the PEP, then it's a bugfix. ;)
Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way.
Or, I could stop worrying and typing emails, and instead just get on with it!
I like this approach :)
But I agree. Release Manager Ned has the final say, but I think this change can comfortably go in during the beta period. (I also disagree that it's a big change - nobody could agree on the 'obvious' behaviour of backslashes anyway, so chances are people would avoid them anyway, and there was strong consensus on advising people to avoid them.)
By "big", I meant "a lot of C code changes". And you'd be surprised by the percentage of the tests that are devoted to backslashes inside braces! Eric.
Hi Eric, Very cool of you to get this going! I hope the outcome is to ban escapes within braced code parts of f-literals for 3.6 and add them “the right way” in 3.7: f'foo{ bar['\n'] }baz' It really is how things work in every single language that i ever encountered that has template literals / string interpolation / f-literals / whatchacallit, so in order to be logical and non-surprising, we should do it that way (eventually). Also the name “f-strings” is really misleading: They’re composite expressions that evaluate to strings. They can only be considered strings if you have no braced code parts in them. So I’m also still in favor of renaming them (e.g. to “f-literals”). Best, Philipp
On 08/30/2016 07:02 AM, Philipp A. wrote:
Very cool of you to get this going!
Thanks for raising the issue.
I hope the outcome is to ban escapes within braced code parts of f-literals for 3.6 and add them “the right way” in 3.7: f'foo{ bar['\n'] }baz'
There's debate on if that's the right way, and I personally think it's probably not. Personally, I'd be happy with the only change being to not allow backslashes inside braces. But that's not an argument that I'm willing to get into now, since I need to get this rolling for beta 1.
Also the name “f-strings” is really misleading: They’re composite expressions that evaluate to strings. They can only be considered strings if you have no braced code parts in them. So I’m also still in favor of renaming them (e.g. to “f-literals”).
I don't have much of an opinion here. I think there's not a lot of confusion to be had by calling them f-strings, but I think someone who works with teaching python might have a better handle on that. Eric.
On Tue, Aug 30, 2016 at 3:13 PM Eric V. Smith <eric@trueblade.com> wrote:
On 08/30/2016 07:02 AM, Philipp A. wrote: Also the name “f-strings” is really misleading: They’re composite expressions that evaluate to strings. They can only be considered strings if you have no braced code parts in them. So I’m also still in favor of renaming them (e.g. to “f-literals”). I don't have much of an opinion here. I think there's not a lot of confusion to be had by calling them f-strings, but I think someone who works with teaching python might have a better handle on that.
The problem is that "literal" is a technical term in the domain of compilers, and not a well known or self-explanatory term. Especially for beginners. I'd suggest something like "Format expression". ~Elazar
Eric V. Smith <eric@trueblade.com> schrieb am Di., 30. Aug. 2016 um 14:13 Uhr:
There's debate on if that's the right way, and I personally think it's probably not. Personally, I'd be happy with the only change being to not allow backslashes inside braces. But that's not an argument that I'm willing to get into now, since I need to get this rolling for beta 1.
And exactly that’s why i’m very happy with the way things are going: Banning escapes is the right short term solution, and possibly the best compromise if we can’t find consensus on how escapes should behave inside of those braces. אלעזר <elazarg@gmail.com> schrieb am Di., 30. Aug. 2016 um 14:37 Uhr:
The problem is that "literal" is a technical term in the domain of compilers, and not a well known or self-explanatory term. Especially for beginners. I'd suggest something like "Format expression".
That sounds fine as well! My issue is just that it’s as much of a string as a call of a (string returning) function/method or an expression concatenating strings: ''.join(things) # would you call this a string? '{!r}'.format(x) # or this? it’s basically the same as this “f-string”: f'{x!r}' 'start' + 'end' # or this? it’s a concatenation of two strings, just like f'start{ "end" }' Best, Philipp
On Tue, Aug 30, 2016 at 10:56 PM, Philipp A. <flying-sheep@web.de> wrote:
My issue is just that it’s as much of a string as a call of a (string returning) function/method or an expression concatenating strings:
''.join(things) # would you call this a string? '{!r}'.format(x) # or this? it’s basically the same as this “f-string”: f'{x!r}' 'start' + 'end' # or this? it’s a concatenation of two strings, just like f'start{ "end" }'
Yes, an f-string is really a form of expression, not a literal. But prior art has generally been to have similar constructs referred to as "interpolated strings" or similar terms: https://en.wikipedia.org/wiki/String_interpolation Plenty of languages have some such feature, and it's usually considered a type of string. Notice the close parallels between actual string literals used as format strings ("I have %d apples" % apples) and f-strings (f"I have {apples} apples"), and how this same parallel can be seen in many other languages. Yes, it may be a join expression to the compiler, but it's a string to the programmer. ChrisA
Sorry, but I'm afraid you are projecting your thinking onto others. The syntactical constructs are called “string interpolations”, not “interpolated strings”. I.e. they're interpolations (a certain type of action) on strings. Strings are the objects, not the subjects. Strings are data, we have code/expressions that look like strings with holes, but in reality, only the parts outside of the braces are strings. I hope I explained my semantics here adequately. Even if they're internally post-processed strings in the CPython code: that's an implementation detail, not a description of the way they work for Python users. Best, Philipp Chris Angelico <rosuav@gmail.com> schrieb am Di., 30. Aug. 2016, 15:43:
My issue is just that it’s as much of a string as a call of a (string returning) function/method or an expression concatenating strings:
''.join(things) # would you call this a string? '{!r}'.format(x) # or this? it’s basically the same as this “f-string”: f'{x!r}' 'start' + 'end' # or this? it’s a concatenation of two strings, just
On Tue, Aug 30, 2016 at 10:56 PM, Philipp A. <flying-sheep@web.de> wrote: like
f'start{ "end" }'
Yes, an f-string is really a form of expression, not a literal. But prior art has generally been to have similar constructs referred to as "interpolated strings" or similar terms:
https://en.wikipedia.org/wiki/String_interpolation
Plenty of languages have some such feature, and it's usually considered a type of string. Notice the close parallels between actual string literals used as format strings ("I have %d apples" % apples) and f-strings (f"I have {apples} apples"), and how this same parallel can be seen in many other languages. Yes, it may be a join expression to the compiler, but it's a string to the programmer.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Philipp, you need to stop debating this issue *now*. You need to write a PEP that can go into Python 3.7. Further debate at the current level (a hair-width close to name-calling) is not going to sway anyone. (This actually goes for Chris too -- nothing is obviously going to change Philipp's mind, so you might as well stop debating and save all the onlookers the embarrassment.) -- --Guido van Rossum (python.org/~guido)
Hi Guido, thanks for calling me out. Yikes, I'm terribly sorry that it came over that way! I'll write the RFC. Should I expand the existing one (this would need Chris’ pending changes though) or write a new one? My goals were to sound factual and terse, not to insult anyone. And I don't see the flaws in my phrasing, so it seems I'm still sometimes bad at written communication. @everyone who perceived it as Guido did: It would be really nice if you could pinpoint the phrases and reasons that make it seem that I mean it that way. (In a private mail to me) Best, Philipp Guido van Rossum <guido@python.org> schrieb am Di., 30. Aug. 2016, 18:43:
Philipp, you need to stop debating this issue *now*.
You need to write a PEP that can go into Python 3.7. Further debate at the current level (a hair-width close to name-calling) is not going to sway anyone.
(This actually goes for Chris too -- nothing is obviously going to change Philipp's mind, so you might as well stop debating and save all the onlookers the embarrassment.)
-- --Guido van Rossum (python.org/~guido)
You need to write a new PEP. And it's PEP, not RFC. On Tue, Aug 30, 2016 at 9:56 AM, Philipp A. <flying-sheep@web.de> wrote:
Hi Guido, thanks for calling me out.
Yikes, I'm terribly sorry that it came over that way!
I'll write the RFC. Should I expand the existing one (this would need Chris’ pending changes though) or write a new one?
My goals were to sound factual and terse, not to insult anyone. And I don't see the flaws in my phrasing, so it seems I'm still sometimes bad at written communication.
@everyone who perceived it as Guido did: It would be really nice if you could pinpoint the phrases and reasons that make it seem that I mean it that way. (In a private mail to me)
Best, Philipp
Guido van Rossum <guido@python.org> schrieb am Di., 30. Aug. 2016, 18:43:
Philipp, you need to stop debating this issue *now*.
You need to write a PEP that can go into Python 3.7. Further debate at the current level (a hair-width close to name-calling) is not going to sway anyone.
(This actually goes for Chris too -- nothing is obviously going to change Philipp's mind, so you might as well stop debating and save all the onlookers the embarrassment.)
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Please just call it f-string and move on, we've had the naming debate previously, it's no longer productive. Regarding eventually supporting f'{'x'}', that will have to be a new PEP to extend PEP 498. (I previously thought it would be an incompatibility, but since f'{' is currently invalid, it's not. However it's a huge change conceptually and implementation-wise, and I don't blame Eric if he doesn't want to be responsible for it. So it has to be a new PEP, to be introduced in 3.7 at the earliest. -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 30, 2016 at 11:43:03PM +1000, Chris Angelico wrote:
On Tue, Aug 30, 2016 at 10:56 PM, Philipp A. <flying-sheep@web.de> wrote:
My issue is just that it’s as much of a string as a call of a (string returning) function/method or an expression concatenating strings:
''.join(things) # would you call this a string? '{!r}'.format(x) # or this? it’s basically the same as this “f-string”: f'{x!r}' 'start' + 'end' # or this? it’s a concatenation of two strings, just like f'start{ "end" }'
Yes, an f-string is really a form of expression, not a literal. But prior art has generally been to have similar constructs referred to as "interpolated strings" or similar terms:
*shrug* A misleading name is misleading no matter how many people use it. If <insert the name of a language you dislike here> started calling function calls: func(arg, x, y, z+1) "s-strings" for "source code strings", because you write them as source code, and changed the syntax to func"arg, x, y, z+1" we'd all recognise what a mistake it is to call this a string, string delimiters or not. The *result* of calling it may be a string, but the expression itself is a kind of function call. [nasty thought] Why don't we allow func"*args" as syntactic sugar for str(func(*args))? [/remove tongue from cheek]
Plenty of languages have some such feature, and it's usually considered a type of string.
The result is a type of string. The expression is not.
Notice the close parallels between actual string literals used as format strings ("I have %d apples" % apples) and f-strings (f"I have {apples} apples"),
That's a misleading comparison. The *template string* is "I have %d apples" and that is just a string. The *format operator* is % and the formatting operation or expression is the entire expression. Since f-strings can contain arbitrary expressions, the analogy is not with the template string, but a function (method or operator) call: f"I have {fruit - bananas - oranges} apples" is not equivalent to "I have {} apples", but to "I have {} apples".format(fruit - bananas - oranges) which is clearly a method call that merely returns a string, not a string itself. Consequently, there's no way to delay evaluation of such an f-string, any more than you can delay evaluation of the expression[1]: func(fruit, 2*bananas, 3/oranges) You can't generate a template, and pass it to different environments to be formatted. If you need to evaluate f"I have {apples} apples" in six different contexts, you have to repeat yourself: it is equivalent to a function call, complete with implicit arguments, not a function object, and certainly not equivalent to a template string.
and how this same parallel can be seen in many other languages. Yes, it may be a join expression to the compiler, but it's a string to the programmer.
Only if the programmer is fooled by the mere use of string delimiters. It really isn't a string. It is executable code that returns a string. It is basically a distant cousin to eval(), in disguise. I really wish we had a better name for these things than f-strings :-( [1] Tricks with eval() excluded. -- Steve
On Tue, Aug 30, 2016 at 11:02:25AM +0000, Philipp A. wrote:
Hi Eric,
Very cool of you to get this going!
I hope the outcome is to ban escapes within braced code parts of f-literals for 3.6 and add them “the right way” in 3.7: f'foo{ bar['\n'] }baz'
That looks like you are doing a key lookup on bar: bar = {'\n': 'something'} f'foo{ bar['\n'] }baz' looks like it will return 'foosomethingbaz'. I expect that syntax will confuse an awful lot of people. -- Steve
On Tue, Aug 30, 2016 at 10:30 AM, Steven D'Aprano <steve@pearwood.info> wrote: [...]
That looks like you are doing a key lookup on bar:
bar = {'\n': 'something'} f'foo{ bar['\n'] }baz'
looks like it will return 'foosomethingbaz'. I expect that syntax will confuse an awful lot of people.
Can we please stop debating this? This observation has been made about 100 times by now. -- --Guido van Rossum (python.org/~guido)
On 30 August 2016 at 07:40, Steve Dower <steve.dower@python.org> wrote:
On 29Aug2016 1433, Eric V. Smith wrote:
On 8/29/2016 5:26 PM, Ethan Furman wrote:
Update the PEP, then it's a bugfix. ;)
Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way.
Or, I could stop worrying and typing emails, and instead just get on with it!
I like this approach :)
It would be good to update the PEP to say "No backslash escapes allowed inside braces" and file a bug against 3.6 for allowing it, though :)
But I agree. Release Manager Ned has the final say, but I think this change can comfortably go in during the beta period. (I also disagree that it's a big change - nobody could agree on the 'obvious' behaviour of backslashes anyway, so chances are people would avoid them anyway, and there was strong consensus on advising people to avoid them.)
+1 - the beta deadline is "no new features", rather than "no further changes to features we already added". The beta period wouldn't be very useful if we couldn't make changes to new features based on user feedback :) (e.g. PEP 492's native coroutines needed some fairly major surgery during the 3.5 beta, as the Cython and Tornado folks found that some of the design decisions we'd made in the initial object model were major barriers to interoperability with other event loops and third party coroutine implementations) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8/29/2016 5:33 PM, Eric V. Smith wrote:
On 8/29/2016 5:26 PM, Ethan Furman wrote:
On 08/29/2016 02:16 PM, Eric V. Smith wrote:
I've been looking at this, and I agree it's the best thing to do, for now (and possibly forever).
I'm just not convinced I can get it done before alpha 1.
Isn't the f-string feature already in place?
Yes. It's been in 3.6 for quite a while (maybe a year?).
Update the PEP, then it's a bugfix. ;)
You can do bug fixes during the beta series.
Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way.
Or, I could stop worrying and typing emails, and instead just get on with it!
Eric.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Terry Jan Reedy
On Sat, Aug 20, 2016 at 4:43 AM, Eric V. Smith <eric@trueblade.com> wrote:
Maybe I'm the odd man out, but I really don't care if my editor ever syntax highlights within f-strings. I don't plan on putting anything more complicated than variable names in my f-strings, and I think PEP 8 should recommend something similar.
I'd go further than "variable names", and happily include attribute access, subscripting (item access), etc, including a couple of levels of same: def __repr__(self): return f"{self.__class__.__name__}(foo={self.foo!r}, spam={self.spam!r})" But yes, I wouldn't put arbitrarily complex expressions into f-strings. Whether PEP 8 needs to explicitly say so or not, I would agree with you that it's a bad idea. ChrisA
On 19 August 2016 at 19:43, Eric V. Smith <eric@trueblade.com> wrote:
Maybe I'm the odd man out, but I really don't care if my editor ever syntax highlights within f-strings. I don't plan on putting anything more complicated than variable names in my f-strings, and I think PEP 8 should recommend something similar.
I agree entirely with this. You're definitely not alone. Paul
On Fri, Aug 19, 2016 at 1:43 PM, Eric V. Smith <eric@trueblade.com> wrote:
With your proposal, it's much more difficult to find the end of an f-string. I do not think this is a reasonable requirement.
For example, consider the following: f'a{func({'a{':1,'3}':[{}, ')', '{']})}b'
A regex will not be able to deal with the matching braces needed to find the end of the expression. You need to keep track of the nesting level of parens, braces, brackets, and quotes (at least those, I might have left something off).
The way this currently is written in Python 3.6a4: f"a{func({'a{':1,'3}':[{}, ')', '{']})}b"
It's trivially easy to find the end of the string. It's easy for both humans and the parsers.
It might be harder to find the end of an f-string in one shot, but I think that's the crux of the issue: to a reader/developer, is an f-string conceptually one thing or a compound thing? To me (someone who would like to see f-string expressions appear like normal expressions, without extra quoting, and proper syntax highlighting *always*, just like shell), this argument is essentially the same as trying to use a regex to find a closing bracket or brace or parse HTML. It's only hard (disregarding any underlying impl details) because that view regards f-strings as singular things with only one "end", when in reality an f-string is much much more like a compound expression that just happens to look like a string. If one rejects the notion that an f-string is "one thing", the boundaries can then be defined as either an unescaped closing quote an unescaped opening curly brace. When that boundary is met, the highlighter switches to normal python syntax parsing just like it would have at the real end of the string. It also starts looking for a closing curly brace to denote the start of "another" string. There is a difference however in that f-string "expressions" also support format specifications. These are not proper Python expressions to begin with so they don't have any existing highlights. Not sure what they should look like I really think it should. Please look at python code with f-literals. if
they’re highlighted as strings throughout, you won’t be able to spot which parts are code. if they’re highlighted as code, the escaping rules guarantee that most highlighters can’t correctly highlight python anymore. i think that’s a big issue for readability.
Maybe I'm the odd man out, but I really don't care if my editor ever syntax highlights within f-strings. I don't plan on putting anything more complicated than variable names in my f-strings, and I think PEP 8 should recommend something similar.
If things aren't highlighted properly I can't see them very well. If f-strings look like other strings in my editor I will absolutely gloss over them as normal strings, expecting no magic, until I later realize some other way. Since I spend significant amounts of time reading foreign code I could see this being quite obnoxious. It might be sufficient to highlight the entire f-string a different color, but honestly I don't think an f-string should ever ever... ever misrepresent itself as a "string", because it's not. It's code that outputs a string, in a compact and concise way. Proper syntax highlighting is one of the most important things in my day-to-day development. I loved the idea of f-strings when they were discussed previously, but I haven't used them yet. If they hide what they really doing under the guise of being a string, I personally will use them much less, despite wanting to, and understanding their value. When I look at a string I want to immediately know just how literal it really is.
On Fri, Aug 19, 2016 at 3:11 PM, C Anthony Risinger <anthony@xtfx.me> wrote:
When I look at a string I want to immediately know just how literal it really is.
To further this point, editors today show me \n and \t and friends in a different color, because they are escapes, and this visually tells me the thing going into the string at that point is not what is literally in the code. A raw string does not highlight these because they are no longer escapes, and what you see is what you get. Probably f-strings will be used most in short strings, but they'll also be used for long, heredoc-like triple-quoted strings. It's not going to be fun picking expressions out of that when the wall-of-text contains no visual cues.
I don't think I've ever used a syntax highlighter than changed color of \n in a string. I get the concept, but I haven't suffered for the absence of that. Moreover, although I haven't yet used them, I really doubt I want extra syntax highlighting in f-strings beyond simply the color strings appear as. Well, maybe a uniform distinction for f-string vs. some other kind of string, but nothing internal to the string. YMMV, but that would be my preference in my text editor. Curly braces are perfectly good visual distinction to me. On Aug 19, 2016 1:25 PM, "C Anthony Risinger" <anthony@xtfx.me> wrote:
On Fri, Aug 19, 2016 at 3:11 PM, C Anthony Risinger <anthony@xtfx.me> wrote:
When I look at a string I want to immediately know just how literal it really is.
To further this point, editors today show me \n and \t and friends in a different color, because they are escapes, and this visually tells me the thing going into the string at that point is not what is literally in the code. A raw string does not highlight these because they are no longer escapes, and what you see is what you get.
Probably f-strings will be used most in short strings, but they'll also be used for long, heredoc-like triple-quoted strings. It's not going to be fun picking expressions out of that when the wall-of-text contains no visual cues.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Aug 19, 2016 at 3:39 PM, David Mertz <mertz@gnosis.cx> wrote:
I don't think I've ever used a syntax highlighter than changed color of \n in a string. I get the concept, but I haven't suffered for the absence of that.
Moreover, although I haven't yet used them, I really doubt I want extra syntax highlighting in f-strings beyond simply the color strings appear as. Well, maybe a uniform distinction for f-string vs. some other kind of string, but nothing internal to the string.
YMMV, but that would be my preference in my text editor. Curly braces are perfectly good visual distinction to me.
At least vim does this and so does Sublime Text IIRC. Maybe I spend a lot of time writing shell code too, but I very much appreciate the extra visual cue. The only real point I'm trying to make is that expressions within an f-string are an *escape*. They escape the normal semantics of a string literal and instead do something else for a while. Therefore, the escaped sections should not look like (or need to conform to) the rest of the string and they should not require quoting as if it were still within the string, because I escaped it already!
Ok. My .vimrc is probably different from yours. I'm sure you are right I *could* make that happen. But I haven't so far. On Aug 19, 2016 1:50 PM, "C Anthony Risinger" <anthony@xtfx.me> wrote:
On Fri, Aug 19, 2016 at 3:39 PM, David Mertz <mertz@gnosis.cx> wrote:
I don't think I've ever used a syntax highlighter than changed color of \n in a string. I get the concept, but I haven't suffered for the absence of that.
Moreover, although I haven't yet used them, I really doubt I want extra syntax highlighting in f-strings beyond simply the color strings appear as. Well, maybe a uniform distinction for f-string vs. some other kind of string, but nothing internal to the string.
YMMV, but that would be my preference in my text editor. Curly braces are perfectly good visual distinction to me.
At least vim does this and so does Sublime Text IIRC. Maybe I spend a lot of time writing shell code too, but I very much appreciate the extra visual cue.
The only real point I'm trying to make is that expressions within an f-string are an *escape*. They escape the normal semantics of a string literal and instead do something else for a while. Therefore, the escaped sections should not look like (or need to conform to) the rest of the string and they should not require quoting as if it were still within the string, because I escaped it already!
On 19 August 2016 at 21:50, C Anthony Risinger <anthony@xtfx.me> wrote:
The only real point I'm trying to make is that expressions within an f-string are an *escape*. They escape the normal semantics of a string literal and instead do something else for a while. Therefore, the escaped sections should not look like (or need to conform to) the rest of the string and they should not require quoting as if it were still within the string, because I escaped it already!
So, to me f'{x.partition(' + ')[0]}' reads as a string concatenation. I'm not sure how you'd expect a syntax highlighter to make it look like anything else, to be honest (given that you're arguing *not* to highlight the whole of the content of the f-string the same way). The *real* solution is not to write something like this, instead write f"{x.partition(' + ')[0]}" That makes it easy for *humans* to read. Computers parsing it is irrelevant. Once you do that, the proposal here (that unescaped quotes can be used in an f-string) also becomes irrelevant - this expression parses exactly the same way under both the current code and the proposed approach. And that's generally true - code that is clearly written should, in my mind, work the same way regardless. So the proposal ends up being merely "choose your preference as to which form of badly-written code is a syntax error". So the only relevance of syntax highlighting is how badly it fails when handling badly-written or syntactically incorrect code. And detecting an f-string just like you detect any other string, is *much* better behaved in that situation. Detecting a closing quote is simple, and isn't thrown off by incorrect nesting. If you want the *content* of an f-string to be highlighted as an expression, Vim can do that, it can apply specific syntax when in another syntax group such as an f-string, and I'm sure other editors can do this as well - but you should do that once you've determined where the f-string starts and ends. Paul
On Fri, Aug 19, 2016 at 6:09 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 19 August 2016 at 21:50, C Anthony Risinger <anthony@xtfx.me> wrote:
The only real point I'm trying to make is that expressions within an f-string are an *escape*. They escape the normal semantics of a string literal and instead do something else for a while. Therefore, the escaped sections should not look like (or need to conform to) the rest of the string and they should not require quoting as if it were still within the string, because I escaped it already!
So, to me
f'{x.partition(' + ')[0]}'
reads as a string concatenation. I'm not sure how you'd expect a syntax highlighter to make it look like anything else, to be honest (given that you're arguing *not* to highlight the whole of the content of the f-string the same way).
The two string parts are string-colored and the x.partition bits would look like any other code in the file. It won't look like concatenation at that point. Examples referencing f'{one_expr_and_no_real_string_in_here}' feel somewhat crafted to confuse because the start and end quotes are directly adjacent to the expression. str(...) is the same complexity. Usage in the wild will have plenty of string-stuff on one or both sides, otherwise, why? Shell or Ruby code is probably more representative of how f-strings will be used. I know a couple people have mentioned they won't/don't care about highlighting in an f-string, but I honestly don't know a single person that would prefer this, except maybe one devops guy I know that does everything on old-school green text because why not. I've also spent hours and hours staring at--and sometimes editing--code on barebones servers/containers and I've come to respect the role colors play in my ability to quickly scan and read code.
The *real* solution is not to write something like this, instead write
f"{x.partition(' + ')[0]}"
Why? Why should I have to care what kind of quote I used at the start of the string? I thought I "escaped" the string at the `{` and now my brain has moved on to the expression? Am I still "inside" the string? etc... It's not the highlighting I care about per se, I think we have a small UX failure here. In a quality editor, everything about the {...} will tell me I'm writing a Python expression. It'll be colored like an expression. It'll do fancy completion like an expression. Aw shucks, it *IS* a Python expression! Except for one tiny detail: I'm not allowed to use the quote I use in 95% of all my Python code--without thinking--because I already used it at the string start :-( It's like this weird invisible ruh-roh-still-in-a-string state hangs over you despite everything else suggesting otherwise (highlighting and whatever fanciness helps people output code). The only time I personally use a different quote is when it somehow makes the data more amenable to the task at hand. The data! The literal data! Not the expressions I'm conveniently inlining with the help of f-strings. When I do it's a conscious decision and comes with a reason. Otherwise I'll use one type of quote exclusively (which depends on the lang, but more and more, it's simply doubles). The appeal of f-strings is the rapid inlining of whatever plus string data. "Whatever" is typically more complex than a simple attribute access or variable reference, though not much more complex eg. `object.method(key, "default")`. If I have to water it down for people to find it acceptable (such as creating simpler variables ahead-of-time) I'd probably just keep using .format(...). Because what I have gained with an f-string? The problem I have is the very idea that while inlining expressions I'm still somehow inside the string, and I have to think about that. It's not a *huge* overhead for an experienced, most-handsome developer such as myself, but still falls in that 5% territory (using a quote because I must vs. the one used 95%). Since f-string are fabulous, I want to use them all the time! Alas, now I have to think about flip-flopping quotes. I don't know what it's like to be taught programming but this seems like a possible negative interaction for new people (learning and using [one] quote combined with easy string building). I know it's not *that* big of deal to switch quotes. I believe this simpler implementation out the gate (not a dig! still great!) will come at the cost of introducing a small Python oddity requiring explanation. Not just because it's at odds with other languages, but because it's at odds with what the editor is telling the user (free-form expression). tl;dr, UX is weaker when the editor implies a free-form expression in every possible way, but the writer can't use the quote they always use, and I think strings will be common in f-string expression sections. -- C Anthony
On 08/19/2016 08:57 PM, C Anthony Risinger wrote: [...]
The appeal of f-strings is the rapid inlining of whatever plus string data. "Whatever" is typically more complex than a simple attribute access or variable reference, though not much more complex eg. `object.method(key, "default")`. If I have to water it down for people to find it acceptable (such as creating simpler variables ahead-of-time) I'd probably just keep using .format(...). Because what I have gained with an f-string?
I suspect f-strings are in the same category as lambda -- if it's that complex, use the other tools instead. At this point I don't see this changing. If you want to make any headway you're going to have to do it with a complete alternate implementation, and even then I don't think you have good odds. -- ~Ethan~
On 2016-08-19 20:57, C Anthony Risinger wrote:
In a quality editor, everything about the {...} will tell me I'm writing a Python expression. It'll be colored like an expression. It'll do fancy completion like an expression. Aw shucks, it*IS* a Python expression! Except for one tiny detail: I'm not allowed to use the quote I use in 95% of all my Python code--without thinking--because I already used it at the string start :-( It's like this weird invisible ruh-roh-still-in-a-string state hangs over you despite everything else suggesting otherwise
But it IS inside a string. That's why it's an f-string. The essence of your argument seems to be that you want expressions inside f-strings to act just like expressions outside of f-strings. But there's already a way to do that: just write the expression outside of the f-string. Then you can assign it to a variable, and refer to the variable in the f-string. The whole point of f-strings is that they allow expressions *inside strings*. It doesn't make sense to pretend those expressions are not inside strings. It's true that the string itself "isn't really a string" in the sense that it's put together at runtime rather than being a constant, but again, the point of f-strings is to make things like that writable as strings in source code. If you don't want to write them as strings, you can still concatenate separate string values or use various other solutions. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
C Anthony Risinger writes:
The only time I personally use a different quote is when it somehow makes the data more amenable to the task at hand. The data! The literal data! Not the expressions I'm conveniently inlining with the help of f-strings.
You do *not* know that yet! *Nobody* does. Nobody has yet written an f-string in production code, let alone read thousands and written hundreds. Can you be sure that after you write a couple dozen f-strings you won't find that such "quote context" is carried over naturally from the way you write other strings? (Eg, because "I'm still in a string" is signaled by the highlighting of the surrounding stringish text.) I think the proposed changes to the PEP fall into the "Sometimes never is better than *right* now" category. The arguments I've seen so far are plausible but not founded in experience: it could easily go the other way, and I don't see potential for true disaster.
If I have to water it down for people to find it acceptable (such as creating simpler variables ahead-of-time) I'd probably just keep using .format(...). Because what I have gained with an f-string?
I don't see a problem if you choose not to write f-strings. Would other people using that convention be hard for you to *read*?
Not just because it's at odds with other languages, but because it's at odds with what the editor is telling the user (free-form expression).
There are no editors that will tell you such a thing yet. And if you trust an editor that *does* tell you that it's a free-form expression and use the same kind of quote that delimits the f-string, you won't actually create a syntax error. You're merely subject to the same kind of "problem" that you have if you don't write PEP8 conforming code. Regards,
On 20 August 2016 at 04:57, C Anthony Risinger <anthony@xtfx.me> wrote:
The two string parts are string-colored and the x.partition bits would look like any other code in the file. It won't look like concatenation at that point.
That's entirely subjective and theoretical (unless you've implemented it and reviewed the resulting look of the code). In my (equally subjective and theoretical) opinion it would still look like concatenation, and would confuse me. I made a deliberate point of saying that *to me* it looked like concatenation. YMMV - remember this tangent was started by people stating their opinions. Saying that your opinion differs doesn't invalidate their (my) view.
tl;dr, UX is weaker when the editor implies a free-form expression in every possible way, but the writer can't use the quote they always use, and I think strings will be common in f-string expression sections.
FWIW I would instantly reject any code passed to me for review which used the same quote within an f-string as was used to delimit it, should this proposal be accepted. Also, a lot of code is read on media that doesn't do syntax highlighting - email, books, etc. A construct that needs syntax highlighting to be readable is problematic because of this. Paul
On Fri, Aug 19, 2016, at 19:09, Paul Moore wrote:
So, to me
f'{x.partition(' + ')[0]}'
reads as a string concatenation. I'm not sure how you'd expect a syntax highlighter to make it look like anything else, to be honest
One possible syntax highlighting scheme: - f' and ' are hilighted in pink, along with any plain text content of the string. - - Incidentally, any backslash escapes, not shown here, are highlighted in orange. - { and } are hilighted in blue; along with format specifiers, maybe, or maybe they get another color. - The code inside the expression is highlighted in orange. - Any keywords, builtins, constants, etc, within the expression are highlighted in their usual colors. - - In this example in particular, ' + ' and 0 are highlighted in pink. A pink + is a character within a string, a gray or orange + is an operator. In terms of Vim's basic syntax groups: - C = Constant [pink] - P = PreProc [blue], by precedent as use for the $(...) delimiters in sh) - S = Special [orange], by precedent as use for the content within $(...) in sh, and longstanding near-universal precedent, including in python, for backslash escapes. These would, naturally, have separate non-basic highlight groups, in case a particular user wanted to change one of them. f'foo {x.partition(' + ')[0]:aaa} bar\n' CCCCCCPSSSSSSSSSSSSCCCCCSSCSPPPPPCCCCSSC
(given that you're arguing *not* to highlight the whole of the content of the f-string the same way).
I'm not sure what you mean by "the same way" here, I haven't followed the discussion closely enough to know what statement by whom you're referring to here.
On 20 August 2016 at 05:02, Random832 <random832@fastmail.com> wrote:
On Fri, Aug 19, 2016, at 19:09, Paul Moore wrote:
So, to me
f'{x.partition(' + ')[0]}'
reads as a string concatenation. I'm not sure how you'd expect a syntax highlighter to make it look like anything else, to be honest
One possible syntax highlighting scheme:
Thanks for the detailed explanation and example. Yes, that may well be a reasonable highlighting scheme. I'd still object to reusing single quotes in the example given, though, as it would be confusing if printed, or in email, etc. And as a general principle, "needs syntax highlighting to be readable" is a problem to me. So I stand by my statement that as a style rule, f-strings should be written to work identically regardless of whether this proposal is implemented or not. Paul
On 2016-08-19 13:11, C Anthony Risinger wrote:
It might be harder to find the end of an f-string in one shot, but I think that's the crux of the issue: to a reader/developer, is an f-string conceptually one thing or a compound thing?
To me (someone who would like to see f-string expressions appear like normal expressions, without extra quoting, and proper syntax highlighting *always*, just like shell), this argument is essentially the same as trying to use a regex to find a closing bracket or brace or parse HTML. It's only hard (disregarding any underlying impl details) because that view regards f-strings as singular things with only one "end", when in reality an f-string is much much more like a compound expression that just happens to look like a string.
Personally I think that is a dangerous road to go down. It seems it would lead to the practice of doing all sorts of complicated things inside f-strings, which seems like a bad idea to me. In principle you could write your entire program in an f-string, but that doesn't mean we need to accommodate the sort of syntax highlighting that would facilitate that. To me it seems more prudent to just say that f-strings are (as the name implies) strings, and leave it at that. If I ever get to the point where what I'm doing in the f-string is so complicated that I really need syntax highlighting for it to look good, I'd take that as a sign that I should move some of that code out of the f-string into ordinary expressions. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
participants (20)
-
Brendan Barnwell -
Brett Cannon -
C Anthony Risinger -
Chris Angelico -
David Mertz -
Eric V. Smith -
Ethan Furman -
Franklin? Lee -
Greg Ewing -
Guido van Rossum -
Nick Coghlan -
Paul Moore -
Philipp A. -
Random832 -
Serhiy Storchaka -
Stephen J. Turnbull -
Steve Dower -
Steven D'Aprano -
Terry Reedy -
אלעזר