On 2015-08-21 00:10, Mike Miller wrote:

The ground seems to be settling on the issue, so I have tried my hand at a grand unified pep for string interpolation.

I originally started writing thinking I would fight arbitrary expressions, though agreeing they would be very useful. In my research however, I discovered that they've become an industry standard of sorts. So, I pivoted and started thinking of mitigation strategies to reduce their downsides instead.

There's still plenty to do and details to iron out, I'd appreciate your help. If this PEP doesn't stick I hope fragments of it can be useful for others.

https://bitbucket.org/mixmastamyk/docs/src/default/pep/pep-05XX.rst

(Pls excuse the inline links, I've not moved them to the footer yet.)

In the "Composition with other prefixes" section, I don't like how f'...' uses {} syntax but fb'...' uses % syntax. In the "Environment Access" section, I would've expected f'Home folder: ${HOME}' to do ''.join('Home folder: $', HOME.__format__('')). BTW, in the "Reference Implementation(s)" section, it's "its work", not "it's work".

I found the b'' idea on a recent message here between you and Nick I think, it seemed interesting. It's gone now, as well as the typo, thanks MRAB. The summary is, it is a superset of PEP 498 with i18n integrated, additional background to inform the arbitrary expression decision, and security policy specified for those who will be concerned about it. As for deferring interpolation, I suppose I'd recommend existing N_... syntax until I've had a bit more time to think about it. -Mike On 08/20/2015 05:39 PM, Guido van Rossum wrote:

Can you give a brief discussion of how your version differs from PEP 498? So far I've found:

- lots of language summarizing the discussion following PEPs 498 and 501 - %(name)s in byte strings (which I think is abominable) - t prefix for translated strings - some optional ideas (which I'm skipping for now)

Am I missing something?

Second, do you have a proposal for marking translatable strings that should be extracted by pygettext but not interpolated in the spot where they occur? (This is the N_(...) format from the pygettext docs.)

I'm guessing there's not much else we can do here but use another string prefix. To stay consistent, I'd expect a slight modification to: Nt'Hello {name}.' However, if we were determined to reduce the number of prefixes, an alternative might be to put a flag inside the string. My first thought is of django/jinja templating comments: t'{# deferred=1 #}Hello {name}.' How does that sound? -Mike On 08/20/2015 05:39 PM, Guido van Rossum wrote:

Second, do you have a proposal for marking translatable strings that shouldbe extracted by pygettext but not interpolated in the spot where they occur? (This the N_(...) format from the pygettext docs.)

On 21 August 2015 at 12:52, Guido van Rossum wrote:

Yeah, I think Nick meant that as a way of implementing the "formatting mini-language" for bytes, given that bytes don't have __format__ or format. But using %(name)s for the *syntax* in bytes was never on the table. I think we're better off not supporting this type of string interpolation for bytes at all.

Yeah, I'm OK with doing this as a text-only thing - while printf-style formatting is certainly useful, binary data is still often best approached as a serialisation problem moreso than as an interpolation one. I really like Mike's language survey in his draft, and the main thing I'd highlight in relation to that is that the interpolation syntax used in JavaScript (with the leading "$" for substitution expressions) is essentially the same as that used in PEPs 215, 292 & 501 (with the key difference being to make the braces optional when leaving them out is unambiguous) One key pragmatic benefit of that is that I expect the number of folks needing to context switch between JavaScript code and Python code will vastly outstrip the number of folks context switching between C# and Python. One key compatibility benefit of that particular syntax is that it interoperates much better with the "{{ global_variable }}" substitution used for Mozilla's l20n templating (http://l20n.org/). That makes it more compatible with the similar syntax used for Django and Jinja2 variable substituation, and the "{% %}" syntax used for Django and Jinja2 blocks. However, those latter examples *do* highlight a "What could possibly go wrong?" question we need to ensure we ask, which is how we want to address the likelihood of folks writing things like: myquery = i"SELECT $column FROM $table;" mycommand = i"cat $filename" mypage = i"<html><body>$content</body></html>" It's the opposite of the "interpolating untrusted strings that may contain aribtrary expressions" problem - what happens when the variables being *substituted* are untrusted? It's easy to say "don't do that", but if doing the right thing incurs all the repetition currently involved in calling str.format, we're going to see a *lot* of people doing the wrong thing. At that point, the JavaScript backticks-with-arbitrary-named-callable solution starts looking very attractive: myquery = sql`SELECT $column FROM $table;` mycommand = sh`cat $filename` mypage = html`<html><body>$content</body></html>` At that point, internationalisation could just be: translated = _`This $value and this $other_value are interpolated after translation lookup`

From an implementation perspective, that could be a matter of:

* adding a new "__interpolate__" magic method with a suitable signature * changing the builtin "format" to implement __interpolate__ as str.format * adding an "interpolator" builtin decorator that just did: def interpolator(f): f.__interpolate__ = f.__call__ return f Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 21 August 2015 at 21:06, Nathaniel Smith wrote:

On Aug 20, 2015 23:40, "Nick Coghlan" wrote:

...

[...]

...

myquery = i"SELECT $column FROM $table;" mycommand = i"cat $filename" mypage = i"<html><body>$content</body></html>"

It's the opposite of the "interpolating untrusted strings that may contain aribtrary expressions" problem - what happens when the variables being *substituted* are untrusted? It's easy to say "don't do that", but if doing the right thing incurs all the repetition currently involved in calling str.format, we're going to see a *lot* of people doing the wrong thing. At that point, the JavaScript backticks-with-arbitrary-named-callable solution starts looking very attractive:

myquery = sql`SELECT $column FROM $table;` mycommand = sh`cat $filename` mypage = html`<html><body>$content</body></html>`

Surely if using backticks we would drop the ugly prefix syntax and just make it a function call?

Not really, no, as `obj` already means repr(obj) in Python 2, and we can't silently make it do something else in Python 3 (although we can break it noisily and thus strongly encourage folks to switch to using the builtin instead). The attractiveness of "little bobby tables" [1] vulnerabilities with an interpolation syntax that *doesn't* support custom interpolation engines has switched me from being mildly interested in the idea of good support for SQL, shell command and HTML generation to considering it a necessary capability, though. Cheers, Nick. [1] https://xkcd.com/327/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Aug 20, 2015 at 11:40 PM, Nick Coghlan wrote:

On 21 August 2015 at 12:52, Guido van Rossum wrote: <snip> It's the opposite of the "interpolating untrusted strings that may contain aribtrary expressions" problem - what happens when the variables being *substituted* are untrusted? It's easy to say "don't do that", but if doing the right thing incurs all the repetition currently involved in calling str.format, we're going to see a *lot* of people doing the wrong thing. At that point, the JavaScript backticks-with-arbitrary-named-callable solution starts looking very attractive:

myquery = sql`SELECT $column FROM $table;` mycommand = sh`cat $filename` mypage = html`<html><body>$content</body></html>`

At that point, internationalisation could just be:

translated = _`This $value and this $other_value are interpolated after translation lookup`

The problem with such syntax is that Guido already long ago ruled out using backticks for anything in Python 3: """ No more backticks. Backticks (`) will no longer be used as shorthand for repr -- but that doesn't mean they are available for other uses. Even ignoring the backwards compatibility confusion, the character itself causes too many problems (in some fonts, on some keyboards, when typesetting a book, etc). """ -- https://www.python.org/dev/peps/pep-3099/ Regards, Chris -- http://chrisrebert.com

On 22 August 2015 at 05:52, Mike Miller wrote:

Yes, we were discussing these custom prefixes in Yuri's thread yesterday, but Guido dropped a big -1 there. However, you Eric and Nick make some compelling arguments in favor of them; they do solve several of our outstanding issues.

Would he be able to be persuaded to change his mind?

It's also worth reiterating my concept of using "!" to introducing the arbitrary "magic happens here" prefixes. That is, you'd write them like this: myquery = !sql"SELECT $column FROM $table;" mycommand = !sh"cat $filename" mypage = !html"<html><body>$content</body></html>" I'd previously suggested a syntax along those lines for full compile time AST manipulation where the compiler also had to be made aware of the prefix names somehow, but I think the proposals that have evolved around f-strings make it possible to instead resolve the named reference at runtime, while still having the compiler handle the subexpression extraction and evaluation. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2015-08-22 00:21, Mike Miller wrote:

The more I think about it, trying to push str.format() (or similar) syntax into i18n is just too much. We'll need a different prefix anyway because of the compile/runtime differences so why not stick with str.Template formatting?

It fits the use case perfectly, and requires little additional work.

Trying to get consistency with str.format() syntax creates security, policy, docs, and tools requirements. Instead, we could accept a small inconsistency (in the grand scheme of things) in order to use the right tool for the job::

# powerful formatting f'Folder {folder}'

# simple translation t'Hello $name.' # or i'', _'', etc dt'Hello # deferred translation

I'd like to update my draft to reflect this change, unless anyone has objections.

f-strings use {} syntax, so I'd prefer t-string to use {} syntax too, unless you're happy to explain it to all those users who'll be asking why f-strings use {} but t-strings use $. (I might even be one of them! :-))

On 08/21/2015 04:33 PM, MRAB wrote:

f-strings use {} syntax, so I'd prefer t-string to use {} syntax too, unless you're happy to explain it to all those users who'll be asking why f-strings use {} but t-strings use $. (I might even be one of them! :-))

Hi, the reason is above, it fits the use case perfectly. Most devs (that know nothing about i18n) will not know it exists. However, imagine the opposite situation... We'll have to explain to non-technical translators how to use a new syntax, which they'll sometimes make mistakes with. We'll have to explain and lecture devs and translators frequently not to use format specifiers and arbitrary expressions, etc. We'll have to document policy and write tools to make sure advanced features don't get used in translating. The checks may not be done at compile time, so those who don't use linters won't be helped. All to avoid $,${} in favor of {}. Certainly either strategy could be done, but one is easier, doesn't put i18n needs at a disadvantage, and won't affect the typical developer. I agree neither choice is perfect. -Mike

On Aug 21, 2015 5:59 PM, "Nick Coghlan" wrote:

On 22 August 2015 at 05:52, Mike Miller wrote:

...

Yes, we were discussing these custom prefixes in Yuri's thread

yesterday,

...

but Guido dropped a big -1 there. However, you Eric and Nick make some compelling arguments in favor of them; they do solve several of our outstanding issues.

Would he be able to be persuaded to change his mind?

It's also worth reiterating my concept of using "!" to introducing the arbitrary "magic happens here" prefixes. That is, you'd write them like this:

myquery = !sql"SELECT $column FROM $table;" mycommand = !sh"cat $filename" mypage = !html"<html><body>$content</body></html>"

I'd previously suggested a syntax along those lines for full compile time AST manipulation where the compiler also had to be made aware of the prefix names somehow, but I think the proposals that have evolved around f-strings make it possible to instead resolve the named reference at runtime, while still having the compiler handle the subexpression extraction and evaluation.

So, str subclasses with _repr_sql_ functions that sometimes serialize and translate differently based on ~threadlocals for SQL variant, lang, charset ; and a new syntax for str.format(**globals()+locals())?

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Hi, I'm not sure that's the case any more, after reading the threads here this week there are numerous difficulties with trying to reconcile both use cases, and didn't get the feeling anyone has an elegant solution to them. We could implement f'' and (i'', aka t'') using either syntax of course, parsing variables from the string, but choosing translation with str.format() seems to cause several more issues than (string.Template() and a bit of inconsistency does). Which syntax would you rather have for translation? (Knowing that you might give a different answer for standard interpolation.) -Mike On 08/21/2015 06:38 PM, Barry Warsaw wrote:

On Aug 21, 2015, at 04:21 PM, Mike Miller wrote:

...

The more I think about it, trying to push str.format() (or similar) syntax into i18n is just too much. We'll need a different prefix anyway because of the compile/runtime differences so why not stick with str.Template formatting?

It fits the use case perfectly, and requires little additional work.

The main annoyance with string.Template based approaches is the same as str.format() -- the requirement to use sys._getframe() to access the interpolation values. I think this was one of the main reasons to propose new syntax, since the compiler can parse the interpolation string and arrange for the values to be composed into a substitution dictionary without having to do ugly locals/globals references. I think this is also Guido's main gripe about the current function-based implementations.

I still wish we could solve this more limited problem, but I don't see a way around that without adding syntax, and if you're going to do that, then I think most people want to go down the whole PEP 498/501 road.

Cheers, -Barry

On Sat, Aug 22, 2015 at 3:09 AM, Nick Coghlan wrote:

...

Hi,

I'm not sure that's the case any more, after reading the threads here

On 22 August 2015 at 15:52, Mike Miller wrote: this

...

week there are numerous difficulties with trying to reconcile both use cases, and didn't get the feeling anyone has an elegant solution to them.

We could implement f'' and (i'', aka t'') using either syntax of course, parsing variables from the string, but choosing translation with str.format() seems to cause several more issues than (string.Template() and a bit of inconsistency does).

Which syntax would you rather have for translation? (Knowing that you might give a different answer for standard interpolation.)

I just pushed a major rewrite of PEP 501 based on the discussions since the initial version of that and PEP 498 went online: https://www.python.org/dev/peps/pep-0501/

It switches to using a magic method and explicitly named interpolator in interpolation expressions, with "!str" being the interpolator reference for default string formatting. From a motivation perspective, while i18n remains a consideration, more easily addressing the risk of code injection attacks against naive use of string interpolation when generating database queries, shell commands or HTML pages now provides a stronger motivation making the interpolation semantics extensible.

Writing a custom interpolator (including for i18n) becomes as simple as doing:

@interpolator def my_interpolator(raw_template, parsed_fields, field_values): ...

While using it then looks like:

result = !my_interpolator "This has $values $mixed into it"

(Similar to yield, it is proposed that interpolation expressions would require parentheses when embedded inside a larger expression)

1. That's an entirely different proposal, you're just reusing the PEP number. 2. Have I died and gone to Perl? -- --Guido van Rossum (python.org/~guido)

On Sat, Aug 22, 2015 at 1:36 PM, Nick Coghlan wrote:

On 23 August 2015 at 02:16, Guido van Rossum wrote:

...

On Sat, Aug 22, 2015 at 3:09 AM, Nick Coghlan wrote:

...

(Similar to yield, it is proposed that interpolation expressions would require parentheses when embedded inside a larger expression)

1. That's an entirely different proposal, you're just reusing the PEP number.

It's aiming to solve the same basic problem though, which is the aspect I consider most important when tackling a design question. The discussions following the posting of my first draft highlighted some real limitations of my original design both at a semantic level and at a motivational level, so I changed it in place rather than introducing yet another PEP on the same topic (Mike Miller's draft PEP was an excellent synthesis, but there's no way he could account for the fact that 501 was still only a first draft).

Yeah, it's not unheard of for PEP authors to pivot after listening to feedback. :-) OTOH this topic is rich enough that I have no problem spending a few more PEP numbers on it. If Mike asks for a PEP number I am not going to withhold it.

...

2. Have I died and gone to Perl?

That's my question in relation to PEP 498 - it seems to introduce lots of line noise for people to learn to read for little to no benefit (my perspective is heavily influenced by the fact that most of the code I write myself these days consists of network API calls + logging messages + UI template rendering, with only very occasional direct calls to str.format that use anything more complicated than "{}" or "{!r}" as the substitution field).

As a result, I'd be a lot more comfortable with PEP 498 if it had more examples of potential practical use cases, akin to the examples section from PEP 343 for context managers.

Since you accept "!r", you must be asking about the motivation for including ":spec", right? That's inherited from PEP 3101. For myself, I know that the most common use of format specs is to limit the number of digits printed for floating point numbers, e.g. t0 = time.time() chop_onions(n) t1 = time.time() print("Chopped %d onions in %.3f seconds." % (n, t1-t0)) Or, using PEP 3101, print("Chopped {} onions in {:.3f} seconds.".format(n, t1-t0)) Using the PEP 498 I can write this as print("Chopped {n} onions in {t1-t0:.3f} seconds.") But in PEP 498 without :spec, I'd have to find some other way of formatting t1-t0, and none of the alternatives look pretty. (Anything that requires introducing a temporary variable feels particularly ugly to me.)

While the second draft of PEP 501 is even more line-noisy than PEP 498 due to the use of both "!" and "$", it at least generalises the underlying semantics of compiler-assisted interpolation to apply to additional use cases like logging, i18n (including compatibility with Mozilla's l20n syntax), safe SQL interpolation, safe shell command interpolation, HTML template rendering, etc.

That's perhaps a bit *too* ambitious. The claim of "safety" for PEP 498 is simple -- it does not provide a way for a dynamically generated string to access values in the current scope (and it does this by not supporting dynamically generated strings). For most domains you mention, safety is much more complex, and in fact mostly orthogonal -- code injection attacks rely on the value of the interpolated variables, so PEP 498's "safety" does not help at all. I18n safety may be the exception -- the scenario is an untrustworthy translator who adds an interpolation that references a variable whose content is deemed sensitive, perhaps a database key.

For the third draft, I'll take another pass at the surface syntax - I like the currently proposed semantics, but agree the current spelling is overly sigil heavy.

Good luck. -- --Guido van Rossum (python.org/~guido)

On Sat, Aug 22, 2015 at 6:37 PM, Nick Coghlan wrote:

On 23 August 2015 at 08:50, Guido van Rossum wrote:

...

OTOH this topic is rich enough that I have no problem spending a few more PEP numbers on it. If Mike asks for a PEP number I am not going to withhold it.

Aye, agreed - at the very least, we want to preserve his survey of interpolation in other languages, as I found that to be an incredibly valuable contribution.

For that he should just update the Wikipedia page on the topic.Or maybe write the PEP, and then update Wikipedia, using the PEP as the [needed citation]. :-)

...

...

...

2. Have I died and gone to Perl?

That's my question in relation to PEP 498 - it seems to introduce lots of line noise for people to learn to read for little to no benefit (my perspective is heavily influenced by the fact that most of the code I write myself these days consists of network API calls + logging messages + UI template rendering, with only very occasional direct calls to str.format that use anything more complicated than "{}" or "{!r}" as the substitution field).

As a result, I'd be a lot more comfortable with PEP 498 if it had more examples of potential practical use cases, akin to the examples section from PEP 343 for context managers.

Since you accept "!r", you must be asking about the motivation for including ":spec", right?

Sorry, I wasn't clear - PEP 501 also retains the field formatting capabilities, and is hence strictly "noisier" than PEP 498 (especially the ! prefix version of the syntax). It's just that it solves enough *other* problems for it to seem worth the cost to me.

Wow. "PEP 498 seems to introduce a lot of line noise" was a rather broken way to say that...

When the benefit is "str.format is prettier, all other forms of interpolation remain repetitively verbose",

Who says that, and what does it mean?

it seems a very invasive change just to replace:

print("Chopped {} onions in {:.3f} seconds.".format(n, t1-t0))

with:

print(f"Chopped {n} onions in {t1-t0:.3f} seconds.")

But only people who are politically correct about it use str.format(). Everyone else (and the logging module :-) still uses %.

...

...

While the second draft of PEP 501 is even more line-noisy than PEP 498 due to the use of both "!" and "$", it at least generalises the underlying semantics of compiler-assisted interpolation to apply to additional use cases like logging, i18n (including compatibility with Mozilla's l20n syntax), safe SQL interpolation, safe shell command interpolation, HTML template rendering, etc.

That's perhaps a bit *too* ambitious. The claim of "safety" for PEP 498 is simple -- it does not provide a way for a dynamically generated string to access values in the current scope (and it does this by not supporting dynamically generated strings). For most domains you mention, safety is much more complex, and in fact mostly orthogonal -- code injection attacks rely on the value of the interpolated variables, so PEP 498's "safety" does not help at all.

Right, but that's where I came to the conclusion that the lack of arbitrary interpolation support ends up making PEP 498 actively dangerous, as string interpolation based substitution ends up being so much prettier than doing things right. Compare:

os.system(f"echo {filename}") subprocess.call(f"echo {filename}") subprocess.call(["echo", filename])

Even in that simple case, the two unsafe approaches are much nicer to read, and as the command line gets more complex, the safe version gets harder and harder to read relative to the unsafe ones.

That reasoning is perverse, and feels disingenuous.

With the latest PEP 501 draft (which switched the proposed syntax and semantics to behave more like a traditional binary operator), we could make invoking a subprocess *safely* look like:

subprocess.call $"echo $filename"

Which reminds me of your one-time attempts to make call parentheses optional, so we could have print be a function and yet be able to write print x, y

However, I'm now coming full circle back to the idea of making this a string prefix, so that would instead look like:

subprocess.call($"echo $filename")

The trick would be to make interpolation lazy *by default* (preserving the triple of the raw template string, the parsed fields, and the expression values), and put the default rendering in the resulting object's *__str__* method.

That's a clever idea. But I expect it will make interpolation much less convenient, because every recipient will have to call str(). The elegance of PEP 498 is that the recipient doesn't have to do or know anything special, because the result is *just* a string object.

That description is probably as clear as mud, though, so back to the PEP I go! :)

I recommend taking a break first. Or maybe sample the recent activity in datetime-sig instead. :-) -- --Guido van Rossum (python.org/~guido)

On 23 August 2015 at 14:09, Guido van Rossum wrote:

On Sat, Aug 22, 2015 at 6:37 PM, Nick Coghlan wrote:

...

Right, but that's where I came to the conclusion that the lack of arbitrary interpolation support ends up making PEP 498 actively dangerous, as string interpolation based substitution ends up being so much prettier than doing things right. Compare:

os.system(f"echo {filename}") subprocess.call(f"echo {filename}") subprocess.call(["echo", filename])

Even in that simple case, the two unsafe approaches are much nicer to read, and as the command line gets more complex, the safe version gets harder and harder to read relative to the unsafe ones.

That reasoning is perverse, and feels disingenuous.

Yeah, professional paranoia produces a weird way of looking at the world :) The key change in my thinking relative to a couple of years ago has been that it's no longer the things that throw surprising exceptions that cause me the most concern, but rather those that *appear* to work, but are actually hiding a dangerous latent defect. These are the situations where a developer (or reviewer) has to "just know" that the apparently obvious way to do something is actually problematic, and that's where we get security vulnerabilities. Having the preferred interpolation syntax produce a non-string object by default provides us with the opportunity to consider on an interface by interface basis whether we want to: * require callers to prerender interpolation templates (the default) * implicitly render interpolation templates with the default renderer (by calling str on the input, which many APIs do already) * define and use a custom renderer for interpolation templates This wouldn't prevent folks from doing the wrong thing - os.system(str(i"echo $filename")) is just as dangerous from a code injection perspective as os.system(f"echo {filename}"). The difference lies in the appearance of the *fixed* code, where subprocess.call(i"echo $filename") would be just as readable as the os.system version, while f-strings don't help with any case that requires a custom renderer in order to do the right thing. That way, when a security linter picks up a problematic call like os.system(str(i"echo $filename")), the solution it suggests can be just as easy to read as the original.

Which reminds me of your one-time attempts to make call parentheses optional, so we could have print be a function and yet be able to write

print x, y

Yeah, that comparison occurred to me as well. It's one of the reasons I kept looking for a way to do custom interpolation using a normal function call instead of needing a new binary operator :)

...

However, I'm now coming full circle back to the idea of making this a string prefix, so that would instead look like:

subprocess.call($"echo $filename")

The trick would be to make interpolation lazy *by default* (preserving the triple of the raw template string, the parsed fields, and the expression values), and put the default rendering in the resulting object's *__str__* method.

That's a clever idea. But I expect it will make interpolation much less convenient, because every recipient will have to call str(). The elegance of PEP 498 is that the recipient doesn't have to do or know anything special, because the result is *just* a string object.

Right, although eager rendering with i-strings just involves calling "str" at the point of definition. Another alternative would be to combine the two ideas, and have i-strings be an implementation detail of f-strings, with f"echo $filename" being a highly optimised version of str(i"echo $filename") that avoids the need for a builtin name lookup (modulo whichever substitution field syntax you eventually choose).

...

That description is probably as clear as mud, though, so back to the PEP I go! :)

I recommend taking a break first.

Aye, having got PEP 501 back to a place where *I* like it again, I'll leave it alone for a while. The pace of iteration this weekend was because I kept discovering aspects I didn't like myself, and coming up with related improvements.

Or maybe sample the recent activity in datetime-sig instead. :-)

Minstrels (singing): Brave Sir Robin ran away, bravely ran away away , when danger reared its ugly head, he bravely turned his tail and fled... ;) Cheers, Nick. P.S. For folks not familiar with that last reference: http://www.montypython.net/scripts/bravesir.php :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 8/23/2015 12:09 AM, Nick Coghlan wrote:

Indeed, after working through this latest change, I ended up back where I started from a syntactic perspective, with a proposal for i(nterpolated)-strings rather than f(ormatted)-strings: https://www.python.org/dev/peps/pep-0501/

As I understand the two proposals, the essential difference, glossing over surface syntax, is this. Compiling f'<template>' would parse the template to an inaccessible structure of existing type (tuple?) and process it at runtime with unreplaceable code returning a string with interpolations. Compiling i'<templat>', in your latest revision, would parse the template to an accessible structure of a new class. The new class would have default code (in .__repr__) equivalent in result to the f code. But additional methods or functions could return other strings (or even non-strings). (Being able to access the structure for debugging purposes might be helpful.) Is this basically it? -- Terry Jan Reedy

On 08/23/2015 08:23 PM, Terry Reedy wrote:

On 8/23/2015 12:09 AM, Nick Coghlan wrote:

...

Indeed, after working through this latest change, I ended up back where I started from a syntactic perspective, with a proposal for i(nterpolated)-strings rather than f(ormatted)-strings: https://www.python.org/dev/peps/pep-0501/

As I understand the two proposals, the essential difference, glossing over surface syntax, is this. Compiling f'<template>' would parse the template to an inaccessible structure of existing type (tuple?) and process it at runtime with unreplaceable code returning a string with interpolations. Compiling i'<templat>', in your latest revision, would parse the template to an accessible structure of a new class. The new class would have default code (in .__repr__) equivalent in result to the f code. But additional methods or functions could return other strings (or even non-strings). (Being able to access the structure for debugging purposes might be helpful.) Is this basically it?

I think so. I just posted a longer version of this to python-ideas. My current thinking is to add both f-strings and i-strings. The f-string version, which would be much more common, would build in the str() call around the i-string. Eric.

I'm feeling pretty good about f-strings. They're pretty much a proven concept, combining .format() strings in Python, and expression interpolation in other languages. But for i-strings, I think it would be good if we could gather more actual experience using them. Every potential use case brought up for these so far (translation, html/shell/sql quoting) feels like there's a lot of work needing to be done to see if the idea is actually viable there. It would be a shame if we added all the (considerable!) machinery for i-strings and all we got was yet another way to do it (https://xkcd.com/927/), without killing at least one competing approach (similar to the way .format() has failed to replace %). It's tough to envision how we could gather more experience with i-strings *without* building them into the language, but I'm really hesitant to add them without more experience. (This is the "new on the job market" paradox. :-) Maybe they could be emulated using a function call that uses sys._getframe() under the covers? Or maybe it's possible to cook up an experiment using other syntax hooks? E.g. the coding hack used in pyxl ( https://github.com/dropbox/pyxl).[1] Some specific thoughts: - In HTML, there are multiple different ways that stuff needs to be quoted, depending on context, e.g. as element text, or as an attribute value, or inside <script></script>. My (limited) experience with pyxl at Dropbox also suggests that html often is constructed programmatically in multiple stages, so it's important to be able to include already-interpolated html fragments into another html block. - In SQL the evaluation of $N is often built into the SQL parser. - Honestly, subprocess.call(i'echo $filename') looks like it's referencing an environment variable, not a variable in the Python code. [1] I am not endorsing pyxl -- its use is currently controversial at Dropbox. But its "coding: pyxl" hack is easily adapted for other syntax experiments (e.g. https://github.com/JukkaL/mypy/tree/master/mypy/codec). -- --Guido van Rossum (python.org/~guido)

On Sun, Aug 23, 2015 at 08:35:17PM -0400, Eric V. Smith wrote:

I think the string interpolation object is interesting. It's basically what Petr Viktorin and Chris Angelico discussed and suggested here: https://mail.python.org/pipermail/python-ideas/2015-August/035303.html.

Are you sure that's the right URL? It seems only barely relevant to me. It has Chris replying to Petr, but it's a vague suggestion of a "quantum string interpolation" (Chris' words) with no details. He asks: "How hard would this be to implement? Something that isn't a string, retains all the necessary information, and then collapses to a string when someone looks at it?" I looked ahead a dozen or two posts, and can't see any further discussion. Have I missed something? -- Steve

On 24 August 2015 at 10:35, Eric V. Smith wrote:

On 08/22/2015 09:37 PM, Nick Coghlan wrote:

...

The trick would be to make interpolation lazy *by default* (preserving the triple of the raw template string, the parsed fields, and the expression values), and put the default rendering in the resulting object's *__str__* method.

At this point, I think PEPs 498 and 501 have converged, except for the delayed string interpolation object (which I realize is important) and how expressions are identified in the strings (which I consider less important).

I think the string interpolation object is interesting. It's basically what Petr Viktorin and Chris Angelico discussed and suggested here: https://mail.python.org/pipermail/python-ideas/2015-August/035303.html.

Aha, I though I'd seen that idea go by in one of the threads, but I didn't remember where :) I'll add Petr and Chris to the acknowledgements section in 501.

My suggestion would be to add both f-strings (PEP 498) and i-strings (as they're currently called in PEP 501), but with the exact same syntax to identify and evaluate expressions. I don't particularly care what the prefixes are. I'd add the plain f-strings first, then i-strings maybe later. There are definitely some issues with delayed interpolation we need to think about. An f-string would be shorthand for str(i-string).

+1, as this is the point of view I've come to as well.

I think it's hyperbolic to refers f-strings as a new string formatting language. With one small difference (detailed in PEP 498, and with zero usage I could find in the stdlib outside of tests), f-strings are a strict superset of str.format() strings (but not the arguments to .format of course). I think f-strings are no more different from str.format strings than PEP 501 i-strings are to string.Template strings.

Yeah, that's a fair criticism of my rhetoric, so I'll stop saying that.

From what I can tell in the stdlib and in the wild, str.format() has hundreds or thousands of times more usage that string.Template. I realize that the reasons are not necessarily related to the syntax of the replacement strings, but you can't say most people aren't familiar with str.format().

Right, and I think we can actually make an example driven decision on that front by looking at potential *target* formats for template rendering. After all, one of the interesting discoveries we made in having both str.__mod__ and str.format available is that %-formatting is a great way to template str.format strings, and vice-versa, since the meta-characters don't conflict, so you can minimise the escaping needed. For use cases like writing object __repr__ methods, I don't think the choice of $-substitution or {}-substitution matters - neither $ nor {} are likely to appear in the desired output (except as part of interpolated values), so escaping shouldn't be common regardless of which we choose. (Side note: __repr__ and _str__ implementations are likely worth highlighting as a good use case for the new syntax!) I think things get more interesting once we start talking about interpolation targets other than "human readable text". For example, one of the neat (/scary, depending on how you feel about this kind of feature) things I realised in working on the latest draft of PEP 501 is that you could use it to template *Python code*, including eagerly bound references to objects in the current scope. That is: a = b + c could instead be written as: a = eval(str(i"$b + $c")) That's not very interesting if all you do is immediately call eval() on it, but it's a lot more interesting if you instead want to do things like extract the AST, dispatch the operation for execution in another process, etc. For example, you could use this capability to build eagerly bound closures, which wouldn't see changes in name bindings, but *would* see state changes in mutable objects. With $-substitution, that "just works", as $ generally isn't syntactically significant in Python code - it can only appear inside strings (and potentially interpolation templates). With {}-substitution, you'd have to double all the braces for dictionary displays, dictionary comprehensions and set comprehensions. In example form: data = {k:v for k, v in source} becomes: data = eval(str(i"{k:v for k, v in $source}")) rather than: data = eval(f"{{k:v for k, v in {{source}}}}")) You hit a similar problem if you're targeting Django or Jinja2 templates, or any content that involves l20n style JavaScript translation strings: the use of braces for substitution expressions in the interpolation template conflicts with their use in the target format. So far, the only target rendering environments I've come up with where $-substitution would create a conflict are shell commands and JavaScript localisation using Mozilla's l20n syntax, and in both of those, I'd actually *want* the Python lookup to take precedence over the target environment lookup (and doubling the prefix to "$$" for target environment lookup seems quite reasonable when you actually do want to do the name lookup in the target environment).

...

That description is probably as clear as mud, though, so back to the PEP I go! :)

Thanks for PEP 501. Maybe I'll add delayed interpolation to PEP 498!

On a more serious note, I'm thinking of adding i-strings to my f-string implementation. I have some ideas that the format_spec (the :.3f stuff) could be used by the code that eventually does the string interpolation. For example, sql(i-string) might want to interpret this expression using __sql__, instead of how str(i-string) would use __format__. Then the sql() machinery could look at the format_spec and pass it to the value's __sql__ method.

Yeah, that's the key reason PEP 501 is careful to treat them as opaque strings that it merely transports through to the renderer. The *default* renderer would expect them to be str.format format specifiers, but other renderers may either disallow them entirely, or expect them to do something different.

For example: sql(i'select {date:as_date} from {tablename}'

might call date.__sql__('as_date'), which would know how to cast to the write datatype (this happens to me all the time).

This is one reason I'm thinking of ditching !s, !r, and !a, at least for the first implementation of PEP 498: they're not needed, and are not generally applicable if we add the hooks I'm considering into i-strings.

+1 from me. Given arbitrary expression support, it's both entirely possible and more explicit to write the builtin calls directly (obj!a, obj!r, obj!s -> ascii(obj), repr(obj), str(obj)) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Aug 23, 2015 at 8:41 PM, Nick Coghlan wrote:

On 24 August 2015 at 10:35, Eric V. Smith wrote:

...

On 08/22/2015 09:37 PM, Nick Coghlan wrote:

...

The trick would be to make interpolation lazy *by default* (preserving the triple of the raw template string, the parsed fields, and the expression values), and put the default rendering in the resulting object's *__str__* method.

At this point, I think PEPs 498 and 501 have converged, except for the delayed string interpolation object (which I realize is important) and how expressions are identified in the strings (which I consider less important).

I think the string interpolation object is interesting. It's basically what Petr Viktorin and Chris Angelico discussed and suggested here: https://mail.python.org/pipermail/python-ideas/2015-August/035303.html.

Aha, I though I'd seen that idea go by in one of the threads, but I didn't remember where :)

I'll add Petr and Chris to the acknowledgements section in 501.

...

My suggestion would be to add both f-strings (PEP 498) and i-strings (as they're currently called in PEP 501), but with the exact same syntax to identify and evaluate expressions. I don't particularly care what the prefixes are. I'd add the plain f-strings first, then i-strings maybe later. There are definitely some issues with delayed interpolation we need to think about. An f-string would be shorthand for str(i-string).

+1, as this is the point of view I've come to as well.

...

I think it's hyperbolic to refers f-strings as a new string formatting language. With one small difference (detailed in PEP 498, and with zero usage I could find in the stdlib outside of tests), f-strings are a strict superset of str.format() strings (but not the arguments to .format of course). I think f-strings are no more different from str.format strings than PEP 501 i-strings are to string.Template strings.

Yeah, that's a fair criticism of my rhetoric, so I'll stop saying that.

...

From what I can tell in the stdlib and in the wild, str.format() has hundreds or thousands of times more usage that string.Template. I realize that the reasons are not necessarily related to the syntax of the replacement strings, but you can't say most people aren't familiar with str.format().

Right, and I think we can actually make an example driven decision on that front by looking at potential *target* formats for template rendering. After all, one of the interesting discoveries we made in having both str.__mod__ and str.format available is that %-formatting is a great way to template str.format strings, and vice-versa, since the meta-characters don't conflict, so you can minimise the escaping needed.

For use cases like writing object __repr__ methods, I don't think the choice of $-substitution or {}-substitution matters - neither $ nor {} are likely to appear in the desired output (except as part of interpolated values), so escaping shouldn't be common regardless of which we choose. (Side note: __repr__ and _str__ implementations are likely worth highlighting as a good use case for the new syntax!)

I think things get more interesting once we start talking about interpolation targets other than "human readable text".

For example, one of the neat (/scary, depending on how you feel about this kind of feature) things I realised in working on the latest draft of PEP 501 is that you could use it to template *Python code*, including eagerly bound references to objects in the current scope. That is:

a = b + c

could instead be written as:

a = eval(str(i"$b + $c"))

That's not very interesting if all you do is immediately call eval() on it, but it's a lot more interesting if you instead want to do things like extract the AST, dispatch the operation for execution in another process, etc. For example, you could use this capability to build eagerly bound closures, which wouldn't see changes in name bindings, but *would* see state changes in mutable objects.

With $-substitution, that "just works", as $ generally isn't syntactically significant in Python code - it can only appear inside strings (and potentially interpolation templates). With {}-substitution, you'd have to double all the braces for dictionary displays, dictionary comprehensions and set comprehensions. In example form:

data = {k:v for k, v in source}

becomes:

data = eval(str(i"{k:v for k, v in $source}"))

rather than:

data = eval(f"{{k:v for k, v in {{source}}}}"))

You hit a similar problem if you're targeting Django or Jinja2 templates, or any content that involves l20n style JavaScript translation strings: the use of braces for substitution expressions in the interpolation template conflicts with their use in the target format.

So far, the only target rendering environments I've come up with where $-substitution would create a conflict are shell commands and JavaScript localisation using Mozilla's l20n syntax, and in both of those, I'd actually *want* the Python lookup to take precedence over the target environment lookup (and doubling the prefix to "$$" for target environment lookup seems quite reasonable when you actually do want to do the name lookup in the target environment).

...

...

That description is probably as clear as mud, though, so back to the PEP I go! :)

Thanks for PEP 501. Maybe I'll add delayed interpolation to PEP 498!

On a more serious note, I'm thinking of adding i-strings to my f-string implementation. I have some ideas that the format_spec (the :.3f stuff) could be used by the code that eventually does the string interpolation. For example, sql(i-string) might want to interpret this expression using __sql__, instead of how str(i-string) would use __format__. Then the sql() machinery could look at the format_spec and pass it to the value's __sql__ method.

Yeah, that's the key reason PEP 501 is careful to treat them as opaque strings that it merely transports through to the renderer. The *default* renderer would expect them to be str.format format specifiers, but other renderers may either disallow them entirely, or expect them to do something different.

...

For example: sql(i'select {date:as_date} from {tablename}'

might call date.__sql__('as_date'), which would know how to cast to the write datatype (this happens to me all the time).

This is one reason I'm thinking of ditching !s, !r, and !a, at least for the first implementation of PEP 498: they're not needed, and are not generally applicable if we add the hooks I'm considering into i-strings.

+1 from me. Given arbitrary expression support, it's both entirely possible and more explicit to write the builtin calls directly (obj!a, obj!r, obj!s -> ascii(obj), repr(obj), str(obj))

IIUC, to do this with SQL,

sql(i'select {date:as_date} from {tablename}'

needs to be ['select ', unescaped(date, 'as_date'), 'from ', unescaped(tablename)] so that e.g. sql_92(), sql_2011() would know that 'select ' is presumably implicitly escaped * https://en.wikipedia.org/wiki/SQL#Interoperability_and_standardization * http://docs.sqlalchemy.org/en/rel_1_0/dialects/ * https://docs.djangoproject.com/en/1.7/ref/models/queries/#f-expressions "Django F-Expressions"

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Aug 24, 2015 at 3:24 AM, Steven D'Aprano wrote:

On Sun, Aug 23, 2015 at 08:35:17PM -0400, Eric V. Smith wrote:

...

I think the string interpolation object is interesting. It's basically what Petr Viktorin and Chris Angelico discussed and suggested here: https://mail.python.org/pipermail/python-ideas/2015-August/035303.html.

Are you sure that's the right URL? It seems only barely relevant to me. It has Chris replying to Petr, but it's a vague suggestion of a "quantum string interpolation" (Chris' words) with no details. He asks:

"How hard would this be to implement? Something that isn't a string, retains all the necessary information, and then collapses to a string when someone looks at it?"

I looked ahead a dozen or two posts, and can't see any further discussion. Have I missed something?

Actually, it's I who missed something – replied from a phone, and sent the reply to Chris only instead of to the list. And that killed further discussion, it seems. My answer was:

Not too hard, but getting the exact semantics right could be tricky. It's probably something the language/stdlib should enable, rather than having it in the stdlib itself.

This seems roughly in line with what Guido was saying earlier. (Am I misrepresenting your words, Guido?) I thought a bit about what's bothering me with this idea, and I realized I just don't like that "quantum effect" – collapsing when something looks at a value. All the parts up to that point sound OK, it's the str() that seems too magical to me. We could require a more explicit function, not just str(), to format the string:

...

...

t0=1; t1=2; n=3 template = i"Peeled {n} onions in {t1-t0:.2f}s" str(template) types.InterpolationTemplate(template="Peeled {n} onions in {t1-t0:.2f}s", fields=(('Peeled', 0, 'n', '', ''), ...), values=(3, 1)) format_template(template) # (or make it a method?) 'Peeled 3 onions in 1s'

This no longer feels "too magic" to me, and it would allow some experimentation before (if ever) InterpolationTemplate grows a more convenient str(). Compared to f-strings, all this is doing is exposing the intermediate structure. (What the "i" really stands for is "internal".) Now f-strings would be just i-strings with a default formatter applied. And, InterpolationTemplate should only allow attribute access (i.e. it shouldn't be structseq). That way the internal structure can be changed later, and the "old" attributes can be synthetized on access.

On 24 August 2015 at 18:51, Mike Miller wrote:

On 08/23/2015 06:41 PM, Nick Coghlan wrote:

...

You hit a similar problem if you're targeting Django or Jinja2 templates, or any content that involves l20n style JavaScript translation strings: the use of braces for substitution expressions in

Hi, this part I don't get, maybe because it's so late here. Why create Django/Jinja2/i20n templates inside Python code using another templating language (whether Template or .format)?

Those kind of templates should be in dedicated text files, no?

Think of meta-templating tools like cookie-cutter or DevAssistant (or the project wizards in an IDE) - for those kinds of tools, "source file formats" are actually output formats. Once you look at enough different parts of the software development pipeline you find that pretty much *every* input format is an output format for some other tool :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 24 August 2015 at 11:41, Nick Coghlan wrote:

That's not very interesting if all you do is immediately call eval() on it, but it's a lot more interesting if you instead want to do things like extract the AST, dispatch the operation for execution in another process, etc. For example, you could use this capability to build eagerly bound closures, which wouldn't see changes in name bindings, but *would* see state changes in mutable objects.

Offering a nice early binding syntax is a question I've been pondering for years (cf. PEPs 403 and 3150), so I'm intrigued by this question of whether or not f-strings and i-strings might be able to deliver those in a way that's more attractive than the current options. This idea doesn't necessarily need deferred interpolation, so I'll use the current PEP 498 f-string prefix and substitution expression syntax. Consider the following function definition: def defer(expr): return eval("lambda: (" + expr + ")") We can use this today as a strange way of writing a lambda expression: >>> f = defer("42") >>> f >>> f() 42 There's no reason to do that, of course - you'd just use an actual lambda expression instead. However, f-strings will make it possible for folks to write code like this: callables = [defer(f"{i}") for i in range(10)] "{i}" in that example isn't a one-element set, it's a substitution expression that interpolates "str(i)" into the formatted string, which is then evaluated by "defer" as if the template contained the literal value of "i" at the time of interpolation, rather than being a lazy reference to a closure variable. (If you were to get appropriately creative with exec, you could even use a trick like this to define multiline lambdas) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Aug 24, 2015 at 08:14:21PM +1000, Nick Coghlan wrote:

This idea doesn't necessarily need deferred interpolation, so I'll use the current PEP 498 f-string prefix and substitution expression syntax. Consider the following function definition:

def defer(expr): return eval("lambda: (" + expr + ")")

We can use this today as a strange way of writing a lambda expression:

>>> f = defer("42") >>> f >>> f() 42

There's no reason to do that, of course - you'd just use an actual lambda expression instead.

There's a problem with the idea of using eval to defer objects -- it relies on your object having an eval'able representation. Try to defer() the following list L: L = [] L.append(L) But putting that aside...

However, f-strings will make it possible for folks to write code like this:

callables = [defer(f"{i}") for i in range(10)]

How is that different from this? callables = [defer(str(i)) for i in range(10)] If they are not the same, then what would this return? [func() for func in callables] I expect it to give [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. Am I wrong?

"{i}" in that example isn't a one-element set, it's a substitution expression that interpolates "str(i)" into the formatted string,

I understand that f-strings are evaluated at the time of, um, their evaluation, so this would be equivalent to: callables = [defer("0"), defer("1"), defer("2", ... defer("9")]

which is then evaluated by "defer" as if the template contained the literal value of "i" at the time of interpolation, rather than being a lazy reference to a closure variable.

I'm completely lost. How would you get a closure variable here? I mean, I know how to get a closure in general terms, e.g.: [(lambda : i) for i in range(10)] but I'm not seeing where you would get a closure *specifically* in this situation with your defer function. -- Steve

On Mon, Aug 24, 2015 at 2:41 PM, Eric V. Smith wrote:

On 08/24/2015 07:35 AM, Paul Moore wrote:

...

I'm once again losing the thread of all the variations being proposed.

As a reality check, is the expectation that something like the following will still be possible:

print(f"Iteration {n}: Duration {end-start} seconds")

Yes, that's the PEP 498 proposal. I think (and this is just my opinion) that if we do something more complicated, like the delayed interpolation of i-strings, that we'd still keep f-strings.

And further, while internally we may rewrite f-strings to use the i-string infrastructure, to the user they'd still look like the same f-strings.

...

Explicit str() calls or temporary variables or anything like that are no improvement over the current options. Of course they may offer more advanced features, but let's not lose the 80% case for the sake of the 20% (that's actually more like 95-5, to be honest).

Agreed.

Indeed. On the other hand, let's make reasonably sure that next year we won't need yet another syntax for the 20%.

On 08/23/2015 09:13 PM, Guido van Rossum wrote:

But for i-strings, I think it would be good if we could gather more actual experience using them. Every potential use case brought up for these so far (translation, html/shell/sql quoting) feels like there's a lot of work needing to be done to see if the idea is actually viable there. It would be a shame if we added all the (considerable!) machinery for i-strings and all we got was yet another way to do it (https://xkcd.com/927/), without killing at least one competing approach (similar to the way .format() has failed to replace %).

It's tough to envision how we could gather more experience with i-strings *without* building them into the language, but I'm really hesitant to add them without more experience. (This is the "new on the job market" paradox. :-) Maybe they could be emulated using a function call that uses sys._getframe() under the covers? Or maybe it's possible to cook up an experiment using other syntax hooks? E.g. the coding hack used in pyxl (https://github.com/dropbox/pyxl).[1]

I hope you don't mind that I borrowed the keys to the time machine. I'm using the implementation of _string.formatter_parser() that I added for implementing string.Formatter: ---8<--------------------------------------------- import sys import _string class i: def __init__(self, s): self.s = s locals = sys._getframe(1).f_locals globals = sys._getframe(1).f_globals self.values = {} # evaluate the expressions for literal, expr, format_spec, conversion in \ _string.formatter_parser(self.s): if expr: value = eval(expr, locals, globals) self.values[expr] = value def __str__(self): result = [] for literal, expr, format_spec, conversion in \ _string.formatter_parser(self.s): result.append(literal) if expr: value = self.values[expr] result.append(value.__format__(format_spec)) return ''.join(result) ---8<--------------------------------------------- So now, instead of i"x={x}", we say i("x={x}"). Let's use it with str:

...

...

x = i('Version in caps {sys.version[0:7].upper()}') x <__main__.i object at 0x7f1653311e90> str(x) 'Version in caps 3.6.0A0'

Cool. Now let's whip up a simple i18n example:

...

...

def gettext(s): ... # Our complicated string lookup ... if s == 'My name is {name}, my dog is {dog}': ... return 'Mi pero es {dog}, y mi nombre es {name}' ... return s ... def _(istring): ... result = [] ... # do the gettext lookup ... s = gettext(istring.s) ... # use the values from our original istring, ... # but the literals and ordering from our ... # looked-up string ... for literal, expr, format_spec, conversion in \ ... _string.formatter_parser(s): ... result.append(literal) ... if expr is not None: ... result.append(istring.values[expr]) ... return ''.join(result) ... name = 'Eric' dog = 'Misty' x = i('My name is {name}, my dog is {dog}') str(x) 'My name is Eric, my dog is Misty' _(x) 'Mi pero es Misty, y mi nombre es Eric'

That should be enough to play with i-strings in logging, sql, xml, etc. Several things should be addressed: hiding the call to _string.formatter_parse inside the 'i' class, for example. And of course don't use sys._getframe. But the ideas are all there. I can't swear that _string.formatter_parser will parse all known expressions, since that's not what it was designed to do. It will likely fail with expressions that contain strings and braces, for example. I haven't really checked. But hey, what do you want for free? With a slight tweak, this code even works with 2.7: replace "_string.formatter_parser" with "str._formatter_parser". Unfortunately, 2.7 will then only support very simple expressions. Oh, well. Enjoy! Eric.

On Aug 23 2015, Nick Coghlan wrote:

On 23 August 2015 at 11:37, Nick Coghlan wrote:

...

However, I'm now coming full circle back to the idea of making this a string prefix, so that would instead look like:

subprocess.call($"echo $filename")

The trick would be to make interpolation lazy *by default* (preserving the triple of the raw template string, the parsed fields, and the expression values), and put the default rendering in the resulting object's *__str__* method.

Indeed, after working through this latest change, I ended up back where I started from a syntactic perspective, with a proposal for i(nterpolated)-strings rather than f(ormatted)-strings: https://www.python.org/dev/peps/pep-0501/

With appropriate modifications to subprocess.call, the proposal would then enable us to write a *safe* shell command interpolation as:

subprocess.call(i"echo $filename")

I like the idea, but *please* stop using this example. It's just terrible. Firstly, subprocess.call defaults to shell=False, so this wouldn't even work. Secondly, subprocess.call('echo', filename') looks orders of magnitude cleaner. Thirdly, your i-string wouldn't even know how to quote because it doesn't know what shell you are using. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

On Aug 21, 2015, at 10:52 PM, Mike Miller wrote:

Which syntax would you rather have for translation? (Knowing that you might give a different answer for standard interpolation.)

For i18n, $-strings (aka PEP 292, string.Template) is by far the best choice. Translators are very familiar with the syntax, having used it now for many years (and not just in a Python context), and it's very difficult for non-technical folks to get wrong. I don't see any advantages to springing yet another i18n interpolation syntax on translators, and I definitely don't see the advantage of introducing a *second* i18n syntax to translators of Python programs. If that means PEP 498/501 isn't appropriate for Python i18n, so be it. What we have now works, even if its implementation requires the use of some frowned-upon APIs, and the use of function syntax for marking and invocation. Cheers, -Barry

On Aug 24, 2015 12:39 PM, "Guido van Rossum" wrote:

(...), and Nick can focus on motivational examples from html/sql/shell code injection for PEP 501 (but only if he can live with the PEP 498 surface syntax for interpolation).

f('select {date} from {tablename}') ~= ['select ', UnescapedStr(date), 'from ', UnescapedStr(tablename)] * UnescapedUntranslatedSoencodedStr * _repr_shell * quote or not? * _repr_html * charset, encoding * _repr_sql * WHERE x LIKE '%\%%'

-- --Guido van Rossum (python.org/~guido)

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Ok thanks, I know someone out there is probably using templating to make templating templates. But, we're getting out into the wilderness here. The original use cases were shell scripts and "whipping up a quick string", which I'd argue are more important. Cheers, -Mike On 08/24/2015 02:48 AM, Nick Coghlan wrote:

On 24 August 2015 at 18:51, Mike Miller wrote

...

Hi, this part I don't get, maybe because it's so late here. Why create Django/Jinja2/i20n templates inside Python code using another templating language (whether Template or .format)?

Those kind of templates should be in dedicated text files, no?

Think of meta-templating tools like cookie-cutter or DevAssistant (or the project wizards in an IDE) - for those kinds of tools, "source file formats" are actually output formats. Once you look at enough different parts of the software development pipeline you find that pretty much *every* input format is an output format for some other tool :)

Cheers, Nick.

On Aug 24, 2015, at 10:38 AM, Guido van Rossum wrote:

I cannot accept $ interpolation in the language definition. I also don't want PEP 498 and 501 to use different interpolation syntaxes. So to me, this means that i18n is off the table as a motivation for PEP 501 (it never was on the table for 498), and Nick can focus on motivational examples from html/sql/shell code injection for PEP 501 (but only if he can live with the PEP 498 surface syntax for interpolation).

I agree with this. Ignoring i18n, str.format() syntax is greatly preferred over old-school %-syntax IMO, so focusing 498/501 on being compatible with the former makes a lot of sense. Hopefully we can continue to make %-syntax obsolete, deprecated, or at least disfavored. Cheers, -Barry

On Mon, Aug 24, 2015 at 10:38 AM, Guido van Rossum wrote:

I cannot accept $ interpolation in the language definition. I also don't want PEP 498 and 501 to use different interpolation syntaxes. So to me, this means that i18n is off the table as a motivation for PEP 501 (it never was on the table for 498), and Nick can focus on motivational examples from html/sql/shell code injection for PEP 501 (but only if he can live with the PEP 498 surface syntax for interpolation).

From the early part of this discussion [1], I had the impression that the goal was that eventually string interpolation would be on by default for all strings, with PEP 498 intended as an intermediate step towards that goal. Is that still true, or is the plan now that interpolated strings will always require an explicit marker (like 'f')?

I ask because if they *do* require an explicit marker, then obviously the best thing is for the syntax to match that of .format. But, if this will be enabled for all strings in Python 3.something, then it seems like we should be careful now to make sure that the syntax is clearly distinct from that used for .format ("${...}" or "\{...}" or ...), because anything else creates nasty compatibility problems for people trying to write format template strings that work on both old and new Pythons. (This is also assuming that f-string interpolation and the eventual plain-old-string interpolation will use the same syntax, but that seems like a highly desirable property to me..) -n [1] http://thread.gmane.org/gmane.comp.python.ideas/34980 -- Nathaniel J. Smith -- http://vorpus.org

On Aug 24 2015, Mike Miller wrote:

Also, 2) a bit of responsibility is pushed to stdlib/pypi. In a handful of sensitive places, the object is checked beforehand and escaped when needed:

def os_system(command): # imagine os.system, subprocess, dbapi, etc. if isinstance(command, estr): command = command.escape(shlex.quote) # each chooses its own rules do_something(command)

This means a billion lines of code using e-strings won't have to care about them, only a handful of places. What is easiest to type is now safe as well:

os.system(e'cat {filename}') # sleep easy

*shudder*. After years of efforts to get people not to do this, you want to change course by 180 degrees and start telling people this is ok if they add an additional single character in front of the string? This sounds like very bad idea to me for many reasons: - People will forget to type the 'e', and things will appear to work but buggy. - People will forget that they need the 'e' (and the same thing will happen, further reinforcing the thought that the e is not required) - People will be confused because other languages don't have the 'e' (hmm. how do I do this in Perl? I guess I'll just drop the 'e'... *check*, works, great!) - People will assume that their my_custom_system() call also special-cases e strings and escape them (which it won't). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

On Mon, Aug 24, 2015 at 1:44 PM, Nathaniel Smith wrote:

From the early part of this discussion [1], I had the impression that the goal was that eventually string interpolation would be on by default for all strings, with PEP 498 intended as an intermediate step towards that goal. Is that still true, or is the plan now that interpolated strings will always require an explicit marker (like 'f')?

That was not received well, so I think it's dead.

I ask because if they *do* require an explicit marker, then obviously the best thing is for the syntax to match that of .format. But, if this will be enabled for all strings in Python 3.something, then it seems like we should be careful now to make sure that the syntax is clearly distinct from that used for .format ("${...}" or "\{...}" or ...), because anything else creates nasty compatibility problems for people trying to write format template strings that work on both old and new Pythons.

Good point.

(This is also assuming that f-string interpolation and the eventual plain-old-string interpolation will use the same syntax, but that seems like a highly desirable property to me..)

-n

[1] http://thread.gmane.org/gmane.comp.python.ideas/34980

-- Nathaniel J. Smith -- http://vorpus.org

-- --Guido van Rossum (python.org/~guido)

On 24 August 2015 at 22:28, Nikolaus Rath wrote:

...

os.system(e'cat {filename}') # sleep easy

*shudder*. After years of efforts to get people not to do this, you want to change course by 180 degrees and start telling people this is ok if they add an additional single character in front of the string?

This sounds like very bad idea to me for many reasons:

- People will forget to type the 'e', and things will appear to work but buggy. - People will forget that they need the 'e' (and the same thing will happen, further reinforcing the thought that the e is not required) - People will be confused because other languages don't have the 'e' (hmm. how do I do this in Perl? I guess I'll just drop the 'e'... *check*, works, great!) - People will assume that their my_custom_system() call also special-cases e strings and escape them (which it won't).

Agreed. In a convenience library where it's absolutely clear that a shell is involved (something like sarge or invoke) this is OK, but not in the stdlib as the "official" way to call external programs. Also: - People will fail to understand the difference between e'...' and f'...' and will use the wrong one when using os.system, and things will work correctly but with security vulnerabilities. - Teaching Python will be complicated by needing to explain why both f'...' and e'...' exist, and what the difference is. Trying to do that for beginners without baffling them with discussions of security vulnerabilities will be challenging... Paul

On 08/24/2015 02:54 PM, Paul Moore wrote:

Agreed. In a convenience library where it's absolutely clear that a shell is involved (something like sarge or invoke) this is OK, but not in the stdlib as the "official" way to call external programs.

Don't focus on os.system(), it could be any function, and not particularly relevant, nor do I recommend this line as the official way. Remember Nick Coghlan's statement that the "easy way should be the right way"? That's what this is trying to accomplish.

- People will fail to understand the difference between e'...' and f'...' and will use the wrong one when using os.system, and things will work correctly but with security vulnerabilities.

I don't recommend e'' and f'', only e'' at this moment. -Mike

In the given example it uses shlex.quote on each variable: https://docs.python.org/dev/library/shlex.html#shlex.quote Btw, no one has to use this form, it simply helps when someone does. -Mike

On Mon, Aug 24, 2015 at 2:28 PM, Nikolaus Rath wrote:

On Aug 24 2015, Mike Miller wrote:

...

Also, 2) a bit of responsibility is pushed to stdlib/pypi. In a handful of sensitive places, the object is checked beforehand and escaped when needed:

def os_system(command): # imagine os.system, subprocess, dbapi, etc. if isinstance(command, estr): command = command.escape(shlex.quote) # each chooses its own rules do_something(command)

This means a billion lines of code using e-strings won't have to care about them, only a handful of places. What is easiest to type is now safe as well:

os.system(e'cat {filename}') # sleep easy

*shudder*. After years of efforts to get people not to do this, you want to change course by 180 degrees and start telling people this is ok if they add an additional single character in front of the string?

The problem is that despite years of effort trying to get people not to do things like this, it's still the case that if you look at, say, MITRE's ranked list of the "top 25 most dangerous software errors": https://cwe.mitre.org/top25/index.html then numbers #1, #2, and #4 are improper quoting. (#3 is buffer overflows.) Or if you look at the OWASP consensus list on the most critical web application security risks ("based on 8 datasets from 7 firms that specialize in application security, including 4 consulting companies and 3 tool/SaaS vendors (1 static, 1 dynamic, and 1 with both). This data spans over 500,000 vulnerabilities..."), then numbers #1 and #3 are improper quoting: https://www.owasp.org/index.php/Top_10_2013-Top_10 I mean, it's great that the rise of languages like Python that have easy range-checked string manipulation has knocked buffer overflows out of the #1 spot, but... :-) Guido is right that the nice thing about classic string interpolation is that its use in many languages gives us tons of data about how it works in practice. But one of the things that data tells us is that it actually causes a lot of problems! Do we actually want to continue the status quo, where one set of people keep designing languages features to make it easier and easier to slap strings together, and then another set of people spend increasing amounts of energy trying to educate all the users about why they shouldn't actually use those features? It wouldn't be the end of the world (that's why we call it "the status quo" ;-)), and trying to design something new and better is always difficult and risky, but this seems like a good moment to think very hard about whether there's a better way. (And possibly about whether that better way is something we could put up on PyPI now while the 3.6 freeze is still a year out...) -n -- Nathaniel J. Smith -- http://vorpus.org

On 08/24/2015 03:37 PM, Nathaniel Smith wrote:

- If I concatenate two e-string objects, or an e-string and a regular string, or interpolate an e-string into an e-string, then what happens?

In the example url I just posted, concatenation renders each string before concatenation, the returns a regular string with both concatenated. If interp into interp ((boggle)), when the passed one gets formated, the formatting operation will render it. Good test case.

- How problematic will it be that an e-string pins all the interpolated objects in memory for its lifetime?

It will be an object holding a raw template string, and a number of variables. In normal usage I don't suspect it to be a problem. -Mike

Eric V. Smith wrote:

An f-string would be shorthand for str(i-string).

If I understand correctly, the point of i-strings would be to make it easy to do things like sql argument interpolation the right way. But if sql(f-string) is still legal (as it seems like it would have to be for quite a while to come, for backwards compatibility) then the wrong way is still just as easy as the right way, and no less obvious (what do the letters "f" and "i" have to do with sql?). So it seems to me that having both f-strings and i-strings will just add a lot of complication and confusion without really helping anything. -- Greg

On Aug 24 2015, Mike Miller wrote:

On 08/24/2015 02:28 PM, Nikolaus Rath wrote:

...

*shudder*. After years of efforts to get people not to do this, you want to change course by 180 degrees and start telling people this is ok if they add an additional single character in front of the string?

This sounds like very bad idea to me for many reasons:

- People will forget to type the 'e', and things will appear to work but buggy. - People will forget that they need the 'e' (and the same thing will happen, further reinforcing the thought that the e is not required) - People will be confused because other languages don't have the 'e' (hmm. how do I do this in Perl? I guess I'll just drop the 'e'... *check*, works, great!) - People will assume that their my_custom_system() call also special-cases e strings and escape them (which it won't).

No, since the variables will not be replaced, therefore the command-line won't work.

How is that compatible with your statement that

This means a billion lines of code using e-strings won't have to care about them, only a handful of places.

Either str(estr) performs interpolation (so billions of lines of code don't have to change, and my custom system()-like call get's an interpolated string as well until I change it to be estr-aware), or it does not (and billions of lines of code will break when they unexpectedly get an estr instead of a str). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

On 08/24/2015 07:05 PM, Nikolaus Rath wrote:

How is that compatible with your statement that

...

This means a billion lines of code using e-strings won't have to care about them, only a handful of places.

Either str(estr) performs interpolation (so billions of lines of code don't have to change, and my custom system()-like call get's an interpolated string as well until I change it to be estr-aware), or it does not (and billions of lines of code will break when they unexpectedly get an estr instead of a str).

Not sure I understand... your system_like() call already accepts strings that could be formatted? The estr adds a protection (by escaping variables) that didn't exist in the past. It is not removing any protections or best practices. It is therefore safer than the f-string version, but you read additional protection as more dangerous, perhaps because someone in the future might get lazy. Is that right? But, people are already lazy (in a manner...), so it looks like a small win to me. By "don't have to care" I don't mean we throw out best practices, only that doing the right thing (rephrased as, not doing the wrong thing) becomes easier, as Nick C. taught is a good idea in his PEP. Any future docs certainly won't be shouting, "do this with os.system!!! It's safe now!!" They will still direct to subprocess.call(). In fact I'm sorry I mentioned os.system at all, it's just a few hours ago someone chewed out Nick C. for using subprocess.call() in his examples. ;) -Mike

On 24 August 2015 at 22:00, Steven D'Aprano wrote:

I mean, I know how to get a closure in general terms, e.g.:

[(lambda : i) for i in range(10)]

but I'm not seeing where you would get a closure *specifically* in this situation with your defer function.

I was wrong when I though you could do this trick with f-strings - you need the delayed interpolation offered by PEP 501's i-strings in order to access the original objects directly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 08/24/2015 06:37 PM, Nathaniel Smith wrote:

On Mon, Aug 24, 2015 at 1:57 PM, Mike Miller wrote:

...

Hi, here's my latest idea, riffing on other's latest this weekend.

Let's call these e-strings (for expression), as it's easier to refer to the letter of the proposals than three digit numbers.

So, an e-string looks like an f-string, though at compile-time, it is converted to an object instead (like i-string):

print(e'Hello {friend}, filename: {filename}.') # converts to ==>

print(estr('Hello {friend}, filename: {filename}.', friend=friend, filename=filename))

An estr is a subclass of str, therefore able to do the nice things a string can do. Rendering is deferred, and it also has a raw member, escape(), and translate() methods:

class estr(str): # init: saves self.raw, args, kwargs for later # methods, ops render it # def escape(self, escape_func): # handles escaping # def translate(self, template, safe=True): # optional i18n support

To make it as simple as possible to use by end-developers, it 1) doesn't require str() to be run explicitly, it renders itself when needed via its various methods and operators. Look for .raw, if you need the original.

This is a really interesting idea.

You could potentially re-use PyUnicode_READY to do the default rendering.

I doubt you could get this to work, although feel free to prove me wrong. I think you'll end up with the same decision Pathlib made (PEP 428): don't derive from str.

Some things to think about:

- If I concatenate two e-string objects, or an e-string and a regular string, or interpolate an e-string into an e-string, then what happens?

- How problematic will it be that an e-string pins all the interpolated objects in memory for its lifetime?

Well, it seems to work for logging, but those don't tend to stay around very long. But this is one of the reasons to play with a sample implementation, to understand these sorts of issues. Eric.

On 08/25/2015 08:02 AM, Nikolaus Rath wrote:

No, but it muddles the water as to what is good and what is bad practice. 'rm {file}' has always been bad practice, but with e-strings e'rm {file}' may or may not be bad practice, depending what you do with it.

It would be bad practice since the function is deprecated, or just discouraged. But, are you implying that the escaping could be bypassed? Would that be possible? -Mike

On Aug 25 2015, Mike Miller wrote:

On 08/25/2015 08:02 AM, Nikolaus Rath wrote:

...

No, but it muddles the water as to what is good and what is bad practice. 'rm {file}' has always been bad practice, but with e-strings e'rm {file}' may or may not be bad practice, depending what you do with it.

It would be bad practice since the function is deprecated, or just discouraged.

What function?

But, are you implying that the escaping could be bypassed? Would that be possible?

According to you, yes. Just look at your example: | def os_system(command): # imagine os.system, subprocess, dbapi, etc. | if isinstance(command, estr): | command = command.escape(shlex.quote) # each chooses its own rules | do_something(command) So any function that doesn't special-case estr will "bypass" the escaping and pass it do it's version of the do_something() function without quoting. Best, -Rikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

TL;DR: (Version 2, hopefully more clear) Let's discuss whether to make "doing the right thing as easy as doing the wrong thing" a desired goal for string interpolation. Details -- we could: 1) Automatically escape potentially dangerous input variables to sensitive functions, or 2) Make developers do it the hard way, making them completely responsible for safety, and always responsible. (Knowing that often they don't). 3) Some combination of the two. A trivial implementation of 1) is below. Instead of rendering the string immediately, it is deferred until use, with template and parameters stashed inside an object, allowing the receiver to specify escaping/quoting rules. --------------------------------- Let's call these e-strings (for expression), as it's easier to refer to the letter of the proposals than three digit numbers. So, an e-string looks like an f-string, though at compile-time, it is converted to an object instead (like an i-string): print(e'Hello {friend}, filename: {filename}.') # converts to ==> print(estr('Hello {friend}, filename: {filename}.', friend=friend, filename=filename)) An estr is a subclass of str, therefore able to do the nice things a string can do. Rendering is deferred until the variable is used, and it also has a .raw member, escape(), and translate() methods: class estr(str): # init: saves self.raw, args, kwargs for later # methods, ops render it # def escape(self, escape_func): # handles escaping # def translate(self, template, safe=True): # optional i18n support To make it as simple as possible to use by end-developers, it: 1) Doesn't require str() to be run explicitly, it renders itself when needed via its various methods and operators. Look for .raw, if you need the original. Also, 2) A bit of responsibility is pushed to stdlib/pypi. In a handful of sensitive places, the object is checked beforehand and escaped when needed: # imagine html, db, subprocess input etc. def sensitive_func_that_escapes(input): if isinstance(input, estr): input = input.escape(shlex.quote) # each chooses its own rules do_something(input) This means numerous callers using e-strings won't have to do explicit escaping, only a handful of callee libraries will--which is common with database apis, for example. What is easiest to type is now safe as well:: sensitive_func_that_escapes_input(e'user input: {input}') # sleep easy This could enable the safety and features we'd like, without burdening the everyday user. I've created a sample script to demonstrate at: https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_example.py Here is the output: # consider: e'Hello {friend}, filename: {filename}.' friend: 'John' filename: "somefile; rm -rf ~ 'foo' <html>" original: Hello {friend}, filename: {filename}. w/ print(): Hello John, filename: somefile; rm -rf ~ 'foo' <html>. shell escape: Hello John, filename: 'somefile; rm -rf ~ '"'"'foo'"'"' <html>'. html escape: Hello John, filename: somefile; rm -rf ~ 'foo' <html>. sql escape: Hello "John", filename: "somefile; rm -rf ~ 'foo' <html>". logger DEBUG Hello John, filename: somefile; rm -rf ~ 'foo' <html>. upper+encode: b"HELLO JOHN, FILENAME: SOMEFILE; RM -RF ~ 'FOO' <HTML>." translated?: Hola John, archivo: somefile; rm -rf ~ 'foo' <html>. Is this automatic escaping desired? Or should we continue to make the end-developer fully responsible for escaping input? -Mike

On 25 August 2015 at 20:06, Mike Miller wrote:

This means numerous callers using e-strings won't have to do explicit escaping, only a handful of callee libraries will--which is common with database apis, for example. What is easiest to type is now safe as well::

sensitive_func_that_escapes_input(e'user input: {input}') # sleep easy

OK. The issue here is that if the user mistakenly calls a function that *doesn't* escape its input, expecting that it will, there will be a silent vulnerability. The problem isn't what I thought it was, using the wriong type of string, it's more about using the wrong function. Of course, having two functions one of which is e-string aware and safe, and one of which isn't, and is unsafe, is a pretty bad API. Or is it? Developers will for quite a long time have to deal with providing compatibility for versions of Python with and without e-strings. Consider pyinvoke - invoke.run() runs shell commands, much like os.system. Suppose version X of pyinvoke adds e-string support. If I write a program using e-strings and invoke.run, it's safe for people with pyinvoke version X installed, but unsafe if my users have version X-1 installed. That's a pretty nasty bug. I honestly have no idea how significant this risk is. But it's something that should be considered when claiming that the proposal makes it "hard to do the wrong thing". Paul.

On 25 August 2015 at 23:36, Mike Miller wrote:

The e-string object and .escape(escape_function) method could still be useful for manual use though, do you agree?

I'm not sure. The principle of having something like that makes sense (more than just sense, it's highly useful), but DB-api functions have been more or less doing that for years with the cursor("select * from foo where bar = ?") approach. I'm not clear how much advantage new syntax gives. I'll have to actually read the proposal in more detail to really say. Paul

On 08/24/2015 09:42 PM, Eric V. Smith wrote:

...

On Aug 24, 2015, at 10:23 PM, Ron Adam wrote:

...

On 08/24/2015 06:45 PM, Mike Miller wrote:

...

...

...

> - How problematic will it be that an e-string pins all > the interpolated objects in memory for its lifetime?

It will be an object holding a raw template string, and a number of variables. In normal usage I don't suspect it to be a problem.

If an objects __str__ method could have an optional fmt='spec' argument, then an estring, could just hold strings, and not the object references. That also prevent surprises if the object is mutated between the time it's estring is created and when the estring is used as a string. For that matter it prevents an estring from printing one way at one time, and another at another time.

I don't know if the fomatting can be split like this... Where an object is formatted to a string representation, and then that is formatted to a field specification. The later being things like width, fill, right, center, and left. These are independent of the object and belong to the string. Things like nubmer of places and sign or to use leading or trailing zeros is part of the object being converted to a string.

It's not possible. For examples, look at all of the number format options. How would you implement hex conversions? Or datetime %A?

I'm not sure which part you are referring to.. But I think adding an optional argument to __str__ methods is probably out. As to splitting the format spec, I think it would be possible, but It may not be needed. I still think early evaluation is a must here. The issue I have with the late evaluation is shown in your current example of logging. If the time which may be from an actual time() function rather than a fixed time is not evaluated until the logged list is printed at the end of the run, all the times will be set to when it's printed rather than when the logged even happened. Another similar reason is the evaluated expression is sensitive to what object is in the name at the time it is evaluated. If it's evaluated later, the object from the name look up may be something entirely unexpected because that name may have been reused during each iteration of a loop. So all the logged entries that refer to that name will give the last value rather than the value at the time the event was logged. Here's a slightly reworked version to compare to. Hope this is helpful, Ron import sys import _string def interleave(*iters): result = [] for items in zip(*iters): for item in items: result.append(item) return result # i-string class i: def __init__(self, s): self.s = s locals = sys._getframe(1).f_locals globals = sys._getframe(1).f_globals self.literals = [] self.values = [] # Evaluate the expressions now, and remember them. # This freezes the value at execution time. for literal, expr, format_spec, conversion in \ _string.formatter_parser(self.s): self.literals.append(literal) if expr: value = eval(expr, locals, globals) self.values.append(value.__format__(format_spec)) else: self.values.append('') def __str__(self): return ''.join(interleave(self.literals, self.values)) # f-string def f(s): return str(i(s)) # logging def log(istring, echo=True): logged = 'log:' + str(istring) print(logged) return logged # test if __name__ == '__main__': x = i('Version in caps {sys.version.upper()!r}') print(str(x)) name = 'Eric' dog = 'Fido' s = f('My name is {name}, my dog is {dog}') print(repr(s)) assert repr(s) == "'My name is Eric, my dog is Fido'" assert type(s) == str import datetime def func(value): return i('called func with "{value:10}"') logline = 'as of {now:%Y-%m-%d} the value is {400+1:#06x}' now = datetime.datetime(2015, 8, 10, 12, 13, 15) logged = log(i(logline), echo=True) assert logged == "log:as of 2015-08-10 the value is 0x0191" now = datetime.datetime(2015, 8, 11, 12, 13, 15) logged = log(i(logline), echo=True) assert logged == "log:as of 2015-08-11 the value is 0x0191" logged = log(i('{func(42)}')) assert logged == 'log:called func with " 42"' import re delimiter = '+' trailing_re = re.escape(r'\S+') regex = i(r'{delimiter}\d+{delimiter}{trailing_re}') print(regex) assert str(regex) == r"+\d++\\S\+"

On 8/25/2015 10:20 PM, Ron Adam wrote:

On 08/24/2015 09:42 PM, Eric V. Smith wrote:

...

...

On Aug 24, 2015, at 10:23 PM, Ron Adam wrote:

...

On 08/24/2015 06:45 PM, Mike Miller wrote:

...

...

>> - How problematic will it be that an e-string pins all >> the interpolated objects in memory for its lifetime?

It will be an object holding a raw template string, and a number of variables. In normal usage I don't suspect it to be a problem.

If an objects __str__ method could have an optional fmt='spec' argument, then an estring, could just hold strings, and not the object references. That also prevent surprises if the object is mutated between the time it's estring is created and when the estring is used as a string. For that matter it prevents an estring from printing one way at one time, and another at another time.

I don't know if the fomatting can be split like this... Where an object is formatted to a string representation, and then that is formatted to a field specification. The later being things like width, fill, right, center, and left. These are independent of the object and belong to the string. Things like nubmer of places and sign or to use leading or trailing zeros is part of the object being converted to a string.

...

It's not possible. For examples, look at all of the number format options. How would you implement hex conversions? Or datetime %A?

I'm not sure which part you are referring to.. But I think adding an optional argument to __str__ methods is probably out.

The part that's not possible is to have the format_spec always be interpreted on a string ojbect, even if the format_spec refers to a different type (such as datetime).

As to splitting the format spec, I think it would be possible, but It may not be needed.

I still think early evaluation is a must here. The issue I have with the late evaluation is shown in your current example of logging. If the time which may be from an actual time() function rather than a fixed time is not evaluated until the logged list is printed at the end of the run, all the times will be set to when it's printed rather than when the logged even happened.

There are two things being evaluated: the expressions (the things inside the {}'s), and the value of the i-string (or whatever it's called here, I've lost track). The expressions would be evaluated immediately, when the i-string is created. This is identical to what would happen if, instead of being in an i-string, the expressions were written in Python code. The value of the i-string would be evaluated later, such as when str() or log() or whatever evaluated the contents of the string. This is what my example on bitbucket does. See i.__init__ for eval(), where the expressions are evaluated. Then later, i.join() actually evaluates the content of the string. Note that evaluating the i-string need not result in a string as the result. See the regex example. The 'i' class needs better support for this, but it's doable. Adding that is on my list of things to do, once I have a better API thought out.

Another similar reason is the evaluated expression is sensitive to what object is in the name at the time it is evaluated. If it's evaluated later, the object from the name look up may be something entirely unexpected because that name may have been reused during each iteration of a loop. So all the logged entries that refer to that name will give the last value rather than the value at the time the event was logged.

Sure. Currently: logging.info('the time is %s', datetime.datetime.now()) Evaluates the current time immediately, but builds up the string later. That's equivalent to what this would do in my bitbucket log.py example: msg = i("the time is {datetime.datetime.now()}") log.log(msg) Also, see test_i in simple.py, again on bitbucket. It shows that changing the values after an i-string is created has no effect on the contents of the i-string. This would be different if the values were mutable, of course. I'll add a test for that to show what I mean. I think your example below is a functional subset of what I have on bitbucket. The only real distinction is that I can do substitutions from a different string, using the expressions that were originally evaluated when the i-string was constructed. This is needed for the i18n case. I realize i18n might never use this, but it's a useful thought experiment in any case. Eric.

Here's a slightly reworked version to compare to.

Hope this is helpful, Ron

import sys import _string

def interleave(*iters): result = [] for items in zip(*iters): for item in items: result.append(item) return result

# i-string class i: def __init__(self, s): self.s = s locals = sys._getframe(1).f_locals globals = sys._getframe(1).f_globals self.literals = [] self.values = [] # Evaluate the expressions now, and remember them. # This freezes the value at execution time. for literal, expr, format_spec, conversion in \ _string.formatter_parser(self.s): self.literals.append(literal) if expr: value = eval(expr, locals, globals) self.values.append(value.__format__(format_spec)) else: self.values.append('')

def __str__(self): return ''.join(interleave(self.literals, self.values))

# f-string def f(s): return str(i(s))

# logging def log(istring, echo=True): logged = 'log:' + str(istring) print(logged) return logged

# test

if __name__ == '__main__':

x = i('Version in caps {sys.version.upper()!r}') print(str(x))

name = 'Eric' dog = 'Fido' s = f('My name is {name}, my dog is {dog}') print(repr(s)) assert repr(s) == "'My name is Eric, my dog is Fido'" assert type(s) == str

import datetime def func(value): return i('called func with "{value:10}"')

logline = 'as of {now:%Y-%m-%d} the value is {400+1:#06x}' now = datetime.datetime(2015, 8, 10, 12, 13, 15) logged = log(i(logline), echo=True) assert logged == "log:as of 2015-08-10 the value is 0x0191"

now = datetime.datetime(2015, 8, 11, 12, 13, 15) logged = log(i(logline), echo=True) assert logged == "log:as of 2015-08-11 the value is 0x0191"

logged = log(i('{func(42)}')) assert logged == 'log:called func with " 42"'

import re delimiter = '+' trailing_re = re.escape(r'\S+') regex = i(r'{delimiter}\d+{delimiter}{trailing_re}') print(regex) assert str(regex) == r"+\d++\\S\+"

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 08/26/2015 10:51 AM, Ron Adam wrote:

On 08/26/2015 07:56 AM, Eric V. Smith wrote:

...

I think your example below is a functional subset of what I have on bitbucket. The only real distinction is that I can do substitutions from a different string, using the expressions that were originally evaluated when the i-string was constructed. This is needed for the i18n case. I realize i18n might never use this, but it's a useful thought experiment in any case.

In my example... the literal and value parts of the strings are stored as strings in two different lists, so you can still apply an i18n translator to just the literal parts, or to the value parts, or to both. It just needs another method. If it's done as a property it could be spelled...

s = 'string' i'This {s} will be translated'._

I still think the i18n case is off the table, per Barry. But in any event, you can't translate the literals in pieces. I think you need to design something that works with gettext. Since the part of my design that allows this is just an optional parameter to my i.join() method, there's not much cost. I do scan the string again, but that would likely be optimized away in a C version.

A nice improvement to that would be to add a literal quote ability to the format language.

i'This {"string":Q} will be translated'.+

That would just work, without the :Q. Expressions cannot be translated, and "string" is an expression.

It allows marking parts of a string to not translate without needing to set it an external (to the string) variable as the example above does. Adding a raw quote option, RQ, would help in the cases of html and regular expressions. (as your's does), but it seems this would be a good addition to the format language so it would work with regular strings too.

I don't have time to test yours this morning, but What happens in this case?

x = [1] ix = i('{x}') x = [2] # Mutates i-string content? print(str(ix))

Does this print "[1]" or "[2]"?

I added a similar test this morning. My code produces "[2]". I can't imagine a design that could produce a different result, but follow the "delayed evaluation of the string" model. Eric.

On Aug 26 2015, Nick Coghlan wrote:

I'm also going to switch to using completely made up API names, since folks otherwise anchor on "but that's not the way that API currently works" without accounting for the fact that APIs can be updated to dispatch to different behaviours based on the types of their arguments :)

If you "update" subprocess.call (I assume this is one of the examples you have in mind) to perform proper escaping and calling a shell when receiving a X-string, the caller now needs to check if he's actually using the right version of the module. Before: subprocess.call(['rm', file]) after: if subprocess.__version__ < something: subprocess.call(['rm', file]) else: subprocess.call(sh'rm {file}') is that really an improvement? In practice you'd probably declare the dependency in setup.py instead, but this just makes it more likely to go out-of-sync, or to be completely lost when code is being cargo-culted. Or are you proposing that sh'rm {file}' wouldn't actually behave like a str, so str(sh'rm {file}') would fail? I guess that would work, but it seems that would have other implications - aren't we talking about *string* interpolation here? If the result isn't even behaving like a str, this seems like a misnomer. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

On 08/26/2015 10:06 AM, Eric V. Smith wrote:

...

A nice improvement to that would be to add a literal quote ability to

...

the format language.

i'This {"string":Q} will be translated'.+ That would just work, without the :Q. Expressions cannot be translated, and "string" is an expression.

Not quite.. the {"string":Q} would include the quotes, while the expression {"string"} would not include the quotes. Cheers, Ron

On 08/26/2015 09:41 PM, Eric V. Smith wrote:

On 8/26/2015 9:08 PM, Ron Adam wrote:

...

...

If you store a string instead of the value, then mutating the object won't effect the i-string. Also you don't get held references to objects that may be more expensive than a string.

I think these points need to be in the PEP. Well, it's Nick's PEP, so you'll have to convince him.

Here I'll talk about my ideas on i-strings, which I've been implementing on that bitbucket repo I've posted. Although I believe they're consistent with where Nick is taking PEP 501.

As I've said before, it's not possible for an i-string to convert all of its expressions to text when the i-string is first constructed. The entire point of delaying the interpolation until some point after the object is constructed is that you don't know how the string conversion is going to be done.

Take this i-string: i'value: {value}'

How would you convert value to a string before you know how it's being converted, or even if it's being converted to a string? What if you use a conversion function that converts the i-string to a list, containing the values of the expressions? Or maybe your converter is going to call repr() on each expression. If you convert to a string first, you've destroyed information that the converter needs.

n = 10 s = 'text' x = i'{n}:{s}'

to_list(x) -> [10, ':', 'text'] to_repr(x) -> '10:"text"'

And this doesn't even take into account the format_specs or conversions, which only have meaning to the conversion function.

Sure it does, you can access the format spec and apply it manually to each item or not. Is there another choice? Depending on how you want to make the values and specs visible. def to_repr(istr): return ''.join(repr(item.format(spec)) for item, spec in istr.items()) I think an actual repr of an i-string may look like this... repr(x) #-> i'{10}:{"text"}' Or maybe... "i'{10}:{\"text\"}'", so it can be used with eval. What concerns me is how much memory it could take to keep object references arround. Considder a logging situation that logs thousands of items. Each i-string could contains references to several objects. And possibly each of those objects contains references to more objects of which memory would have been released hours ago if it weren't for the i-strings. Oops.. my computer is now disc caching so bad it will take days to finish the process it is logging. Meanwhile, it can't process any new input. If one of the use cases is logging, then this is a realistic possibility. I do recognize the added flexibility that keeping the references offers, but I'm not sure it's needed. Cheers, Ron

On 8/27/2015 2:13 AM, Ron Adam wrote:

On 08/26/2015 09:41 PM, Eric V. Smith wrote:

...

On 8/26/2015 9:08 PM, Ron Adam wrote:

...

...

If you store a string instead of the value, then mutating the object won't effect the i-string. Also you don't get held references to objects that may be more expensive than a string.

I think these points need to be in the PEP. Well, it's Nick's PEP, so you'll have to convince him.

Here I'll talk about my ideas on i-strings, which I've been implementing on that bitbucket repo I've posted. Although I believe they're consistent with where Nick is taking PEP 501.

As I've said before, it's not possible for an i-string to convert all of its expressions to text when the i-string is first constructed. The entire point of delaying the interpolation until some point after the object is constructed is that you don't know how the string conversion is going to be done.

...

Take this i-string: i'value: {value}'

How would you convert value to a string before you know how it's being converted, or even if it's being converted to a string? What if you use a conversion function that converts the i-string to a list, containing the values of the expressions? Or maybe your converter is going to call repr() on each expression. If you convert to a string first, you've destroyed information that the converter needs.

n = 10 s = 'text' x = i'{n}:{s}'

to_list(x) -> [10, ':', 'text'] to_repr(x) -> '10:"text"'

And this doesn't even take into account the format_specs or conversions, which only have meaning to the conversion function.

Sure it does, you can access the format spec and apply it manually to each item or not. Is there another choice?

You're not reading what I'm writing. Using your proposal of immediately converting to strings, how would you write the version of "to_list" whose output I show above?

Depending on how you want to make the values and specs visible.

def to_repr(istr): return ''.join(repr(item.format(spec)) for item, spec in istr.items())

I think an actual repr of an i-string may look like this...

repr(x) #-> i'{10}:{"text"}'

Or maybe... "i'{10}:{\"text\"}'", so it can be used with eval.

Again, that's not what I'm talking about. How would you write the "to_repr" function whose output I show above?

What concerns me is how much memory it could take to keep object references arround. Considder a logging situation that logs thousands of items. Each i-string could contains references to several objects. And possibly each of those objects contains references to more objects of which memory would have been released hours ago if it weren't for the i-strings. Oops.. my computer is now disc caching so bad it will take days to finish the process it is logging. Meanwhile, it can't process any new input.

If one of the use cases is logging, then this is a realistic possibility.

Logging is already passed the object references. This is how logging is called today: logging.info('the values are %d and %f', an_int, get_a_float()) As you can see, it's passed a string and some objects. That's what an i-string is! But with a nicer syntax and a more flexible way to convert objects to strings. If logging were instead passed an i-string: logging.info(i'the values are {an_int} and {get_a_float()}') and if logging were changed so that where it currently builds a string using "msg = str(msg), msg = msg % self.args" [1], it instead said: if (isinstance(msg, types.InterpolationTemplate)): msg = str(msg) else: msg = str(msg) % self.args then there would be zero change in the memory usage of the logging module [2]. Anyway, that's my last input on the subject. You can either follow the code in my bitbucket repo and show how you'd implement its use cases with your approach, or we can just wait for Nick to update the PEP. Eric. [1]: https://hg.python.org/cpython/file/tip/Lib/logging/__init__.py#l328 [2]: Sadly, it's not quite so simple since logging has a pluggable setLogRecordFactory architecture. But the point on memory usage stands.

I do recognize the added flexibility that keeping the references offers, but I'm not sure it's needed.

Cheers, Ron

On 28 August 2015 at 04:21, Ron Adam wrote:

On 08/27/2015 04:17 AM, Eric V. Smith wrote:

...

...

...

n = 10

...

...

s = 'text' x = i'{n}:{s}'

to_list(x) -> [10, ':', 'text'] to_repr(x) -> '10:"text"'

And this doesn't even take into account the format_specs or conversions, which only have meaning to the conversion function.

...

Sure it does, you can access the format spec and apply it manually to each item or not. Is there another choice?

You're not reading what I'm writing. Using your proposal of immediately converting to strings, how would you write the version of "to_list" whose output I show above?

...

...

Depending on how you want to make the values and specs visible.

def to_repr(istr): return ''.join(repr(item.format(spec)) for item, spec in istr.items())

Well, what you have above is applying repr to the values but not the literal parts. It's doable, but that should really be part of the format spec rather than applying a function from outside, or it should be part of the expression in the i-strings.

i'{repr(n)}:{repr(s}}' i'{n!r}:{s!r}'

Yes, this topic is getting too drawn out. Possibly I'm not seeing a finer point in your examples. I think it will sort it self out as the implementation progress's, so I'm not too worried about it.

The key with i-strings is that they introduce the possibility of replacing additional elements in the rendering pipeline: the field interpolator, and the overall renderer. With f-strings, there's only one field interpolator: the format() builtin, which receives both the value to be interpolated (eagerly calculated at the point where the f-string appears in the code) and the format string. Each substitution field is then replaced with the result of "format(field_value, field_spec)". With f-strings, there's also only one overall renderer: "".join. The literal text elements and the substituted fields are combined back together through string concatenation. The *whole point* of i-strings is to make not just the format() call replaceable, but also the overall process whereby the literal elements, the values of the substitution expressions, and the format specifiers for those expressions are rendered into an output object. Guido's not convinced yet that it makes sense to expose that capability to end users, and I think that skepticism is fair. However, if types.InterpolationTemplate is developed as an implementation detail of f-strings (which is the option Eric has been exploring), then we can create those at runtime from normal strings, and see how useful they might be for cases where ''.join() and format() aren't the best choice of rendering primitives. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia