[Python-ideas] Briefer string format

[Steven D'Aprano]

TL;DR:

Please let's just ban implicit concatenation between f strings (a 
runtime function call) and non-f strings. The user should be explicit in 
what they want, using either explicitly escaped braces or the + 
operator. Anything else is going to be surprising.


On Thu, Jul 23, 2015 at 06:57:25PM -0700, Bruce Leban wrote:
> On Thu, Jul 23, 2015 at 7:22 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> 
> >
> > If I had a dollar for everytime somebody on the Internet misused
> > "strawman argument", I would be a rich man.
> 
> 
> You wouldn't get a dollar here. If you want to be strict, a strawman
> argument is misrepresenting an opponent's viewpoint to make it easier to
> refute but it also applies to similar arguments.

Are you saying that any good faith disagreement about people's position 
is a strawman? If not, I don't understand what you mean by "similar 
arguments".

A strawman argument is explicitly a bad-faith argument. Describing my 
argument as a strawman implies bad faith on my part. I don't mind if you 
think my argument is wrong, mistaken or even incoherent, but it is not 
made in bad faith and you should imply that it is without good reason.

Moving on to the feature:

> You stated that "constant
> folding ... *would* change the semantics" *[emphasis added]*.

In context, I said that constant-folding the *explicit* + concatenation 
of f'{a}' + '{b}' to f'{a}{b}' would change the semantics. I'm sorry if 
it was not clear enough that I specifically meant that. I thought that 
the context was enough to show what I meant.

By constant-folding, I mean when the parser/lexer/compiler/whatever (I 
really don't care which) folds expressions like the following:

    'a' + 'b'

to this:

    'ab'

If the parser/whatever does that to mixed f and non-f strings, I think 
that would be harmful, because it would change the semantics:

    f'{a}' + '{b}'

executed at runtime with no constant-folding is not equivalent to the 
folded version:

    f'{a}{b}'

Hence, the peephole optimizer should not do that. I hoped that wouldn't 
be controversial.


[...]
> So the straw here is imagining that the implementer of
> this feature would ignore the accepted rules regarding constant folding and
> then criticizing the implementer for doing that.

I'm taken aback that you seem to think my pointing out the above is a 
criticism of an implementer who doesn't even exist yet! We're still 
discussing what the semantics of f strings should be, and I don't think 
anyone should be offended or threatened by me being explicit about what 
the behaviour should be.

And for the record, it is not unheard of for constant-folding peephole 
optimizers to accidentally, or deliberately, change the sematics of 
code. For example, in D constant-folded 0.1 + 0.2 is not the same as 0.1 
+ 0.2 done at runtime (constant folding is done at single precision 
instead of double precision):

http://stackoverflow.com/questions/6874357/why-0-1-0-2-0-3-in-d

This paper discusses the many pitfalls of optimizing floating point 
code, and mentions that C may change the value of literal expressions 
depending on whether they are done at runtime or not:

    Another effect of this pragma is to change how much the 
    compiler can evaluate at compile time regarding constant 
    initialisations. [...] If it is set to OFF, the compiler can 
    evaluate floating-point constants at compile time, whereas
    if they had been evaluated at runtime, they would have 
    resulted in different values (because of different rounding
    modes) or floating-point exception.

http://arxiv.org/pdf/cs/0701192.pdf


Constant-folding *shouldn't* change the semantics of code, but 
programmers are only human. They make bad design decisions or write 
buggy code the same as all of us.


> (3) The hard case, when you mix f and non-f strings.
> >
> >     f'{spam}' '{eggs}'
> >
> > Notwithstanding raw strings, the behaviour which makes sense to me is
> > that the implicit string concatenation occurs first, followed by format.
> >
> 
> You talk about which happens "first" so let's recast this as an operator
> precedence question. Think of f as a unary operator. Does f bind tighter
> than implicit concatenation? Well, all other string operators like this
> bind more tightly than concatenation.
> f'{spam}' '{eggs}'

I don't think this is correct. Can you give an example? All the examples 
I can come up with show implicit concatenation binding more tightly 
(i.e. it occurs first), e.g.:

py> 'a' 'ba'.replace('a', 'z')
'zbz'

not 'abz'. And of course, you can't implicitly concat to a method call:

py> 'a'.replace('a', 'z') 'ba'
  File "<stdin>", line 1
    'a'.replace('a', 'z') 'ba'
                             ^
SyntaxError: invalid syntax


So I think it would be completely unprecedented if the f pseudo-operator 
bound more tightly than the implicit concatenation.


> > Secondly, it feels that this does the concatenation in the wrong order.
> > Implicit concatenation occurs as early as possible in every other case.
> > But here, we're delaying the concatenation until after the format. So
> > this feels wrong to me.
> >
> 
> Implicit concatenation does NOT happen as early as possible in every case.
> When I write:
> 
>     r'a\n' 'b\n'  ==>  'a\\nb\n'
> 
> the r is applied to the first string *before* the concatenation with the
> second string.

r isn't a function, it's syntax. There's nothing to apply. This is why I 
don't think that the behaviour of mixed raw and cooked strings is a good 
model for mixing f and non-f strings. Both raw and cooked strings are 
lexical features and should be read from left to right, in the order 
that they occur, not function calls which must be delayed until runtime.


[...]
> Imagine that we have another prefix that escapes strings for regex. That is
> e'a+b' ==> 'a\\+b'. This is another function call in disguise, just calling
> re.escape. 

Now you're the one confusing interface with implementation :-) Such an e 
string need not be a function call, it could be a lexical feature like 
raw strings. In fact, I would expect that they should be.

These hypothetical e strings could be a lexical feature, or a runtime 
function, but f *must* be a runtime function since the variables being 
interpolated don't have values to interpolate until runtime. We have no 
choice in the manner, whereas we do have a choice with e strings.

In any case, I don't think it is a productive use of our time to discuss 
a hypothetical e string that neither of us intend to propose.


> Maybe you can't say that concatenation is an optimization but I can (new
> text underlined):
>
> Multiple adjacent string or bytes literals (delimited by whitespace),
> possibly using different quoting conventions, are allowed, and their
> meaning is the same as their concatenation. ... Thus, "hello" 'world' is
> equivalent to "helloworld". This feature can be used to reduce the number
> of backslashes needed, to split long strings conveniently across long
> lines, *to mix formatted and unformatted strings,* or even to add comments
> to parts of strings, for example:
>
> re.compile("[A-Za-z_]"       # letter or underscore
>            "[A-Za-z0-9_]*"   # letter, digit or underscore
>           )
> Note that this feature is defined at the syntactical level, but implemented
> at compile time *as an optimization*.

I don't think that flies. It's *not just an optimization* when it comes 
to f strings. It makes a difference to the semantics.

    f'{spam}' '{eggs}'

being turned into "format first, then concat" has a very different 
meaning to "concat first, then format".

To get the semantics you want, you need a third option:

    escape first, then concat, then format

But there's nothing obvious in the syntax '{eggs}' that tells anyone 
when it will be escaped and when it won't be. You need to be aware of 
the special case "when implicitly concat'ed to f strings, BUT NO OTHER 
TIME, braces in ordinary strings will be escaped".

I dislike special cases. They increase the number of things to memorise 
and lead to surprises.


> *If formatted strings are
> mixed with unformatted strings, they are concatenated at compile time and
> the unformatted parts are escaped so they will not be subject to format
> substitutions.*

That's your opinion for the desirable behaviour.

I don't like it, I don't expect it. The fact that you have to explicitly 
document it shows that it is a special case that doesn't follow from the 
existing behaviour of Python's implicit concatenation rules. I don't 
think we should have such a special case, when there are already at 
least two other ways to get the same effect.

But since my preferred suggestion is unpopular, I'd much rather just ban 
implicit concat'ing of f and non-f strings and avoid the whole argument. 
That's not an onerous burden on the coder:

    result = (f'{this}' + '{that}')

is not that much more difficult to type than:

    result = (f'{this}' '{that}')


and it makes the behaviour clear.


-- 
Steve