[Python-ideas] Draft PEP on string interpolation

[Nick Coghlan]

On 24 August 2015 at 10:35, Eric V. Smith <eric at trueblade.com> wrote:
> On 08/22/2015 09:37 PM, Nick Coghlan wrote:
>> The trick would be to make interpolation lazy *by default* (preserving
>> the triple of the raw template string, the parsed fields, and the
>> expression values), and put the default rendering in the resulting
>> object's *__str__* method.
>
> At this point, I think PEPs 498 and 501 have converged, except for the
> delayed string interpolation object (which I realize is important) and
> how expressions are identified in the strings (which I consider less
> important).
>
> I think the string interpolation object is interesting. It's basically
> what Petr Viktorin and Chris Angelico discussed and suggested here:
> https://mail.python.org/pipermail/python-ideas/2015-August/035303.html.

Aha, I though I'd seen that idea go by in one of the threads, but I
didn't remember where :)

I'll add Petr and Chris to the acknowledgements section in 501.

> My suggestion would be to add both f-strings (PEP 498) and i-strings (as
> they're currently called in PEP 501), but with the exact same syntax to
> identify and evaluate expressions. I don't particularly care what the
> prefixes are. I'd add the plain f-strings first, then i-strings maybe
> later. There are definitely some issues with delayed interpolation we
> need to think about. An f-string would be shorthand for str(i-string).

+1, as this is the point of view I've come to as well.

> I think it's hyperbolic to refers f-strings as a new string formatting
> language. With one small difference (detailed in PEP 498, and with zero
> usage I could find in the stdlib outside of tests), f-strings are a
> strict superset of str.format() strings (but not the arguments to
> .format of course). I think f-strings are no more different from
> str.format strings than PEP 501 i-strings are to string.Template strings.

Yeah, that's a fair criticism of my rhetoric, so I'll stop saying that.

> From what I can tell in the stdlib and in the wild, str.format() has
> hundreds or thousands of times more usage that string.Template. I
> realize that the reasons are not necessarily related to the syntax of
> the replacement strings, but you can't say most people aren't familiar
> with str.format().

Right, and I think we can actually make an example driven decision on
that front by looking at potential *target* formats for template
rendering. After all, one of the interesting discoveries we made in
having both str.__mod__ and str.format available is that %-formatting
is a great way to template str.format strings, and vice-versa, since
the meta-characters don't conflict, so you can minimise the escaping
needed.

For use cases like writing object __repr__ methods, I don't think the
choice of $-substitution or {}-substitution matters - neither $ nor {}
are likely to appear in the desired output (except as part of
interpolated values), so escaping shouldn't be common regardless of
which we choose. (Side note: __repr__ and _str__ implementations are
likely worth highlighting as a good use case for the new syntax!)

I think things get more interesting once we start talking about
interpolation targets other than "human readable text".

For example, one of the neat (/scary, depending on how you feel about
this kind of feature) things I realised in working on the latest draft
of PEP 501 is that you could use it to template *Python code*,
including eagerly bound references to objects in the current scope.
That is:

    a = b + c

could instead be written as:

    a = eval(str(i"$b + $c"))

That's not very interesting if all you do is immediately call eval()
on it, but it's a lot more interesting if you instead want to do
things like extract the AST, dispatch the operation for execution in
another process, etc. For example, you could use this capability to
build eagerly bound closures, which wouldn't see changes in name
bindings, but *would* see state changes in mutable objects.

With $-substitution, that "just works", as $ generally isn't
syntactically significant in Python code - it can only appear inside
strings (and potentially interpolation templates). With
{}-substitution, you'd have to double all the braces for dictionary
displays, dictionary comprehensions and set comprehensions. In example
form:

    data = {k:v for k, v in source}

becomes:

    data = eval(str(i"{k:v for k, v in $source}"))

rather than:

    data = eval(f"{{k:v for k, v in {{source}}}}"))

You hit a similar problem if you're targeting Django or Jinja2
templates, or any content that involves l20n style JavaScript
translation strings: the use of braces for substitution expressions in
the interpolation template conflicts with their use in the target
format.

So far, the only target rendering environments I've come up with where
$-substitution would create a conflict are shell commands and
JavaScript localisation using Mozilla's l20n syntax, and in both of
those, I'd actually *want* the Python lookup to take precedence over
the target environment lookup (and doubling the prefix to "$$" for
target environment lookup seems quite reasonable when you actually do
want to do the name lookup in the target environment).

>> That description is probably as clear as mud, though, so back to the
>> PEP I go! :)
>
> Thanks for PEP 501. Maybe I'll add delayed interpolation to PEP 498!
>
> On a more serious note, I'm thinking of adding i-strings to my f-string
> implementation. I have some ideas that the format_spec (the :.3f stuff)
> could be used by the code that eventually does the string interpolation.
> For example, sql(i-string) might want to interpret this expression using
> __sql__, instead of how str(i-string) would use __format__. Then the
> sql() machinery could look at the format_spec and pass it to the value's
> __sql__ method.

Yeah, that's the key reason PEP 501 is careful to treat them as opaque
strings that it merely transports through to the renderer. The
*default* renderer would expect them to be str.format format
specifiers, but other renderers may either disallow them entirely, or
expect them to do something different.

> For example:
> sql(i'select {date:as_date} from {tablename}'
>
> might call date.__sql__('as_date'), which would know how to cast to the
> write datatype (this happens to me all the time).
>
> This is one reason I'm thinking of ditching !s, !r, and !a, at least for
> the first implementation of PEP 498: they're not needed, and are not
> generally applicable if we add the hooks I'm considering into i-strings.

+1 from me. Given arbitrary expression support, it's both entirely
possible and more explicit to write the builtin calls directly (obj!a,
obj!r, obj!s -> ascii(obj), repr(obj), str(obj))

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia