Escaped braces in format specifiers
The syntax for formatted string literals is given here: https://docs.python.org/3/reference/lexical_analysis.html#f-strings If you were to examine this carefully, you would see that a format_spec (the part within the braces but after the colon) can be empty or it can consist of literal characters and replacement fields. For example: f"result: {value:{width}.{precision}}" In this case '{width}.{precision}' is the format_spec, it consists of two replacement fields, {width} and {precision}, and one literal character, '.'. The definition of literal character includes all characters except braces. Presumably excluding braces makes it easier to distinguish the replacement fields. However, replacement fields are also used in the f_string itself, but in that case escaped braces are explicitly added to the list of valid symbols. I think the fact that one cannot include braces in the format_spec is an unnecessary and undesirable restriction. As an example of where it would be useful to allow braces in the format_spec, consider this simple example: >>> class mydict(dict): ... def __format__(self, template): ... return ''.join( ... template.format(v, k=k, v=v) for k, v in self.items() ... ) >>> accounts = mydict(checking=4256.78, savings=12000, brokerage=24685.5) >>> print('Accounts:\n {0:{{k:>9s}}: ${{v:>9,.2f}}\n }'.format(accounts)) Accounts: checking: $ 4,256.78 savings: $12,000.00 brokerage: $24,685.50 In this example, the format_spec is itself treated as a format string, which allows a recursive specification of the string format. If you try this example in Python3.6 you will find that it works, but it should not because the format_spec contains escaped braces. However, the fact that it works appears be be a happy accident. Things break if the escaped braces are not balanced (thanks to Thomas Jollans for finding this). Furthermore, things break in a different manner when an f-string is used. For example: >>> print(f'Accounts:\n {accounts:{{k:>9s}}: ${{v:>9,.2f}}\n }') File "<fstring>", line 1 ({k:>9s}) ^ SyntaxError: invalid syntax or: >>> print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') File "tryit", line 12, in <module> print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') NameError: name 'k' is not defined Oddly, the f-string still raises the NameError even if both k and v are explicitly defined. At a minimum it would be good to improve the error messages that are produced when escaped braces are included in the format_spec. Most of the error messages that are given, if they are given at all, are misleading. None are as simple as: SyntaxError: escaped braces are not allowed in a format spec. But rather than improve the error messages, I think we should consider simply allowing escaped braces in the format_spec. Doing so enables the neat idea of recursive format strings. But even if you don't like that idea, it would be nice to remove this rather rather odd restriction and would make the behavior of f-strings and the format method more consistent. -Ken
I'm busy at the sprints, so I don't have a lot of time to think about this. However, let me just say that recursive format specs are supported, to a depth of 1.
So first the string is basically expanded to: f'{"test":10}' Then the string is formatted again to produce the final result. That is why the braces must match: they're being used for recursive format specs. There's no mechanism for having braces that aren't inspected by the f-string machinery. Eric On 5/15/18 2:41 AM, Ken Kundert wrote:
On 15 May 2018 at 16:23, Eric V. Smith <eric@trueblade.com> wrote:
https://www.python.org/dev/peps/pep-0536/ also seems worth noting (I don't actually understand the specifics of that PEP myself, just making sure that Ken's aware of its existence if this is an area of interest) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Yes, that’s a good point. Thanks for remembering this PEP. I disagree with much of the PEP and it’s language, such as “These limitations serve no purpose ...”. And I especially object to the fact that it doesn’t take in to account the effect of its proposed changes on other python implementations or on tools like editors, linters, static type checkers, etc., all of which would have to be modified. Not that those are insurmountable issues or mean that we should never improve f-strings, but they do need to be carefully considered and given weight in the decision process. Eric
On Tue, May 15, 2018 at 04:23:26PM -0400, Eric V. Smith wrote:
Eric, You say that recursive format specs are supported to a depth of 1, but it does not seem like true recursion to me. If it were true recursion, wouldn't the escaped braces be interpreted properly inside the format spec, just as they are in the literal portion of the f-string? Is there some reason why escaped braces are not supported in the format specs? Consider the following example: >>> class myclass: ... def __format__(self, fmt_spec): ... return 'fmt_spec: ' + fmt_spec >>> d = myclass() >>> x = 3 >>> print('1: ext={{x}} {d:int={{x}}}'.format(d=d)) 1: ext={x} fmt_spec: int={x} >>> print(f'2: ext={{x}} {d:int={{x}}}') 2: ext={x} fmt_spec: int={3} >>> print(f'3: ext={set([x])} {d:int={set([x])}}') 3: ext={3} fmt_spec: int={3} In each case, the same substring is found in the literal portion (ext) of the format string and in the format spec (int). In case 1, the format string is passed to a format method and the substring is {{x}}. In both cases, the double braces are interpreted as escaped braces, and both are converted to '{x}' in the final printed result. In case 2, the format string is an f string. The substring is still {{x}}. The first instance of {{x}} is found in the literal portion of the format string and the double braces are interpreted as escaped braces. It expands to {x} like in the first case. However, the second instance is found in the format spec, and this case the outer braces are stripped off and what is left over is treated as a Python expression. Here {x} is treated as a set literal containing one value, x. Since x is 3, the value of int is {3}. Case 3 confirms the interpretation of case 2 by simply replacing {x} with an alternate way of specifying a set literal that contains a single value. In this case there are no double braces and so both instances are treated the same, both expand to {3}. So the format method supports escaped braces in both the literal part of the format string and the format spec. f-strings supports escaped braces in the literal part of the format string, but in the format spec the outer level of braces are stripped off and what remains is interpreted as a Python expression. Is there some benefit to this difference? -Ken
On 5/19/2018 1:56 AM, Ken Kundert wrote:
Is the difference you're seeing just in how f-strings and str.format() differ in evaluating expressions? str.format() isn't evaluating "int={x}", which makes sense to me. It never evaluates expressions, by design, it just treats strings as literals. If it did evaluate expressions, there's too great a chance of code being evaluated from user-defined strings. Eric
On Sat, May 19, 2018 at 07:06:41AM -0400, Eric V. Smith wrote:
Eric, My example does highlight the fact that f-strings do evaluate the embedded expressions while str.format() does not, but that is not the point I was trying to make. The substring 'int={{x}}' has two possible interpretations, it is either a string that contains escaped braces and when the escaping is processed becomes 'int={x}', or the outer braces signal an embedded expression, the expression being {x}, a set literal, and when interpolated the substring becomes 'int={3}'. My point was that str.format() uses the first interpretation for both ext and int, but f-strings use the first interpretation for ext and the second interpretation for int. Using the second interpretation for int in the f-string seems inconsistent. Specifically, int and ext are interpreted differently in the f-string, and int is interpreted differently by str.format() and the f-string. This difference does not seem to provide any benefit, so why does it exist? In other words, why aren't the escaped braces supported in the format_spec of f-strings? -Ken
I'm busy at the sprints, so I don't have a lot of time to think about this. However, let me just say that recursive format specs are supported, to a depth of 1.
So first the string is basically expanded to: f'{"test":10}' Then the string is formatted again to produce the final result. That is why the braces must match: they're being used for recursive format specs. There's no mechanism for having braces that aren't inspected by the f-string machinery. Eric On 5/15/18 2:41 AM, Ken Kundert wrote:
On 15 May 2018 at 16:23, Eric V. Smith <eric@trueblade.com> wrote:
https://www.python.org/dev/peps/pep-0536/ also seems worth noting (I don't actually understand the specifics of that PEP myself, just making sure that Ken's aware of its existence if this is an area of interest) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Yes, that’s a good point. Thanks for remembering this PEP. I disagree with much of the PEP and it’s language, such as “These limitations serve no purpose ...”. And I especially object to the fact that it doesn’t take in to account the effect of its proposed changes on other python implementations or on tools like editors, linters, static type checkers, etc., all of which would have to be modified. Not that those are insurmountable issues or mean that we should never improve f-strings, but they do need to be carefully considered and given weight in the decision process. Eric
On Tue, May 15, 2018 at 04:23:26PM -0400, Eric V. Smith wrote:
Eric, You say that recursive format specs are supported to a depth of 1, but it does not seem like true recursion to me. If it were true recursion, wouldn't the escaped braces be interpreted properly inside the format spec, just as they are in the literal portion of the f-string? Is there some reason why escaped braces are not supported in the format specs? Consider the following example: >>> class myclass: ... def __format__(self, fmt_spec): ... return 'fmt_spec: ' + fmt_spec >>> d = myclass() >>> x = 3 >>> print('1: ext={{x}} {d:int={{x}}}'.format(d=d)) 1: ext={x} fmt_spec: int={x} >>> print(f'2: ext={{x}} {d:int={{x}}}') 2: ext={x} fmt_spec: int={3} >>> print(f'3: ext={set([x])} {d:int={set([x])}}') 3: ext={3} fmt_spec: int={3} In each case, the same substring is found in the literal portion (ext) of the format string and in the format spec (int). In case 1, the format string is passed to a format method and the substring is {{x}}. In both cases, the double braces are interpreted as escaped braces, and both are converted to '{x}' in the final printed result. In case 2, the format string is an f string. The substring is still {{x}}. The first instance of {{x}} is found in the literal portion of the format string and the double braces are interpreted as escaped braces. It expands to {x} like in the first case. However, the second instance is found in the format spec, and this case the outer braces are stripped off and what is left over is treated as a Python expression. Here {x} is treated as a set literal containing one value, x. Since x is 3, the value of int is {3}. Case 3 confirms the interpretation of case 2 by simply replacing {x} with an alternate way of specifying a set literal that contains a single value. In this case there are no double braces and so both instances are treated the same, both expand to {3}. So the format method supports escaped braces in both the literal part of the format string and the format spec. f-strings supports escaped braces in the literal part of the format string, but in the format spec the outer level of braces are stripped off and what remains is interpreted as a Python expression. Is there some benefit to this difference? -Ken
On 5/19/2018 1:56 AM, Ken Kundert wrote:
Is the difference you're seeing just in how f-strings and str.format() differ in evaluating expressions? str.format() isn't evaluating "int={x}", which makes sense to me. It never evaluates expressions, by design, it just treats strings as literals. If it did evaluate expressions, there's too great a chance of code being evaluated from user-defined strings. Eric
On Sat, May 19, 2018 at 07:06:41AM -0400, Eric V. Smith wrote:
Eric, My example does highlight the fact that f-strings do evaluate the embedded expressions while str.format() does not, but that is not the point I was trying to make. The substring 'int={{x}}' has two possible interpretations, it is either a string that contains escaped braces and when the escaping is processed becomes 'int={x}', or the outer braces signal an embedded expression, the expression being {x}, a set literal, and when interpolated the substring becomes 'int={3}'. My point was that str.format() uses the first interpretation for both ext and int, but f-strings use the first interpretation for ext and the second interpretation for int. Using the second interpretation for int in the f-string seems inconsistent. Specifically, int and ext are interpreted differently in the f-string, and int is interpreted differently by str.format() and the f-string. This difference does not seem to provide any benefit, so why does it exist? In other words, why aren't the escaped braces supported in the format_spec of f-strings? -Ken
participants (3)
-
Eric V. Smith -
Ken Kundert -
Nick Coghlan