On 08/27/2015 06:36 AM, Cody Piersall wrote:

Is the Draft PEP column of the table supposed to have both "Immediate Render" and "Deferred Render" as "Yes"? I'm hoping that's a typo, otherwise I don't understand what it means at all.

Yes, it supports both. By storing all inputs to the object, it can be rendered again, with optional changes such as overriding a value, or escaping the input. -Mike

On 08/27/2015 09:19 AM, Cody Piersall wrote:

When you say immediate, do you mean that it only takes a call to str()? Or is there some way to have e'this is an e-string' evaluate to a string without doing anything else?

There is no need to call str() manually with e'', the .rendered member is returned by default: >>> print(estr('Hello {friend}.')) 'Hello John' Here is the example implementation: https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py -Mike

On 08/27/2015 10:27 AM, Eric V. Smith wrote:

The problem with this auto-rendering is that the format_spec and

Hmm, I believe this is a design choice, one that should be made depending on whether this use-case is important and/or common. The estr provides for this situation instead by allowing for additional renderings if/when needed, but doesn't require str() in the common case. This is the first I've seen of directives passed inside the format spec, I'll add it to the comparison table. -Mike

On 08/27/2015 02:30 PM, Andrew Barnert via Python-ideas wrote:

On Aug 27, 2015, at 10:27, Eric V. Smith <eric@trueblade.com> wrote:

...

...

On 08/27/2015 11:22 AM, Mike Miller wrote:

...

On 08/27/2015 06:36 AM, Cody Piersall wrote:

Is the Draft PEP column of the table supposed to have both "Immediate Render" and "Deferred Render" as "Yes"? I'm hoping that's a typo, otherwise I don't understand what it means at all.

Yes, it supports both. By storing all inputs to the object, it can be rendered again, with optional changes such as overriding a value, or escaping the input.

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

Imagine that ":raw" is interpreted by to_html() to mean that the string does not get html escaped.

With PEP 501, the format_spec and conversion are opaque to the i-string machinery, and are only interpreted by the custom interpolation function (here, to_html()).

With str.format, it's the type of value that decides how to interpret the format spec. If you leave it up to the consumer of the i-string (the interpolation function) instead of the value's type, how do you handle things like numeric formats and datetime formats and so on? Would I need to do something like this:

to_html(e'<p>{str(e"{value:05}"):raw}</p>')

Or would to_html (and every other custom interpolator) have to take things like :raw05 and parse out a format spec to pass to value.__format__?

Your interpolator would need to decide. It might never call value.__format__. It might invent some other protocol, like __html_escape__(fmt_spec). Or, it might bake-in knowledge of how to convert whatever types it cares about. Or another good choice would be to use the singledispatch module. In fact, I like singledispatch so much that I'm going to have to use it in an example.

Maybe what you really want is !raw rather than :raw. If there is no conversion, __format__ gets called and the result passed around as part of the i-string object; if there is one, the value, conversion, and format spec get passed instead (and then to_html could decide that conversion 'raw' means to call format(value, format_spec) and then not escape the result).

You could do that in addition or in place of format_spec. Except currently conversions are only allowed to be a single character, but I don't see any reason not to relax that. The take away is that the PEP 501 i-string machinery applies zero significance to format_spec and conversion. It just parses them out of the template string. It's left up to the interpolator to apply some meaning to them.

Although that's pretty different from how the standard conversions work (call repr or ascii on the value, then format the resulting string with the format spec). So maybe replacement fields need another subpart separate from both the conversion and the format spec that's only used by custom interpolators?

There's nothing from stopping you from doing this. You could decide that your format_spec, for some interpolator, is composed of "part1^part2", and do something based on part1 and part2. See: https://bitbucket.org/ericvsmith/istring/src/d92e47c96609eed44ed57b7d3c1932b... for how my i-string str() interpolator applies format_spec and conversion. Another interpolator could do something different (for example, https://bitbucket.org/ericvsmith/istring/src/d92e47c96609eed44ed57b7d3c1932b... for regex escaping). Eric.

On Thu, Aug 27, 2015, at 13:27, Eric V. Smith wrote:

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

I feel like this would be better done with something like {value!raw}.

On Fri, Aug 28, 2015, at 03:52, Paul Moore wrote:

On 28 August 2015 at 02:55, <random832@fastmail.us> wrote:

...

On Thu, Aug 27, 2015, at 13:27, Eric V. Smith wrote:

...

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

I feel like this would be better done with something like {value!raw}.

While I appreciate that people are still in a design phase with this, I'd like to point out that in the end, people will need to teach and remember this stuff.

Currently ! introduces a conversion, which is one of r, s, or a (and is rarely used except for the occasional !r). Whereas : introduces a format spec, which is a mini-language for describing how to format the value and is specific to the type of the value.

Yes, but the format spec mini-language belongs to the type of the value. Depending on what value is, "raw" could _already_ have a meaning.

The "raw" thing above feels like neither of those things. It feels more like a conversion, but if so then conversions are currently single letters, and language-defined.

Well, at the time I posted that I thought we were moving away from things being language-defined, because my mind was still on the "user- defined string prefixes" proposal from a while back. Them currently being single letters isn't really a compelling argument.

I'd also expect (for that reason) that any conversions would be valid in any type of formatting (str.format, f-strings, e-strings, whatever).

And what if the type of value expects to be able to process a format specifier of "raw", rather than it being used by to_html for its own purpose? The advantage of conversion specifiers is that they're currently a closed set. ---- One thing that I don't think *either* version successfully expresses is that while in many cases (including the to_html example) we want a string, that won't always be the case. If we have a syntax for inserting something as, e.g., a SQL parameter, it should be able to accept a double, but I'm not convinced it shouldn't _also_ be able to describe putting the string result of converting the double with ".05d" as a varchar.

On Aug 27, 2015, at 11:27, Guido van Rossum <guido@python.org> wrote:

I also don't understand the claim that no str(estr) is necessary to render the result -- the estr implementation given at https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py has a __str__ method that renders the .rendered attribute, but without the str() call the type of 'a' in the above examples would not be str, and various things that operate on strings (e.g. regular expression searches) would not work. A solution might be to make estr a subclass of str, but nothing in the PEP suggests that you have even considered this problem. (The only hint I can find is the comment "more magic-methods to be implemented here, to improve str compatibility" in your demo implementation, but without subclassing str this is not enough.)

Even subclassing str doesn't really help, because there's plenty of code (including, I believe, regex searches) that just looks at the raw string storage that gets created at str.__new__ and can never be mutated or replaced later. So, anything that's delayed-rendered is not a str, or at least it's not the right str. (I know someone earlier in the discussion suggested that at the C level you could replace PyUnicode_READY with a function that, if it's an estr, first calls self.__str__ and then initializes the string storage to the result and then does the normal READY stuff, but I don't think that actually works, does it?)

On 08/27/2015 11:27 AM, Guido van Rossum wrote:

specified. E.g. I couldn't figure out what code should be produced by:

a = e"Sliced {n} onions in {t1-t0:.3f} seconds."

Generalizing from the only example in the specification, this would become:

a = est("Sliced {n} onions in {t1-t0:.3f} seconds", n=n, t1-t0=t1-t0)

Yes, the demo does not currently handle arbitrary expressions, only .format() is implemented. The PEP is relying on much of the PEP 498 implementation (which I don't have at hand), so that part is underspecified. I hope to reconcile the details if/when the larger design is chosen. For now, I will add the ability to pass a context dictionary at init as well as keywords to prepare for further implementation.

I also don't understand the claim that no str(estr) is necessary to render the result -- the estr implementation given at https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py has a __str__ method that renders the .rendered attribute, but without the str() call

Yes, no explicit call is necessary. When you do things like print(e'') or e''.upper() or '' + e'', you'll get the rendered string without str(). That could be more clear, yes, and maybe not substantially different than i''.

A solution might be to make estr a subclass of str, but nothing in the PEP suggests that

I originally subclassed it from str, but others on the list advised against it. Which is preferred? Under what name is the string in a string object actually held (if any)? -Mike

On Thu, Aug 27, 2015 at 1:38 PM, Mike Miller <python-ideas@mgmiller.net> wrote:

I've addressed those questions, updated the demo script, and added your examples in a separate examples section. I'll write more after lunch.

Looking through your latest commit ( https://bitbucket.org/mixmastamyk/docs/commits/760274613d8c306cd688385fbbcf2...) I think you've painted yourself into an impossible corner, and haven't thought through the consequences in all cases enough. Apparently my Socratic questions didn't help enough, so I feel compelled to give you the answers. :-) The interpreter can't and shouldn't be passing in the values of all the variables involved in an expression. It should only be passing in the final evaluated result of each "slot" in the e-string. For my second example (with three rand() calls) it should pass the three different random values returned by the three rand() calls into the estr() constructor, e.g.: b = estr("Three random numbers: {rand()}, {rand()}, {rand()}.", rand(), rand(), rand()) Your entire formatting machinery should be rewritten using positional values instead of keyword args. Regarding subclassing str, there are indeed many problems with that (e.g. what is the type of str(...)+estr(...)), but without it, you will never be able to claim that str() calls are never needed, because quite a few built-in operations and stdlib modules in Python *require* that their arguments are str subclasses (or they treat str subclasses different than other classes). An important example is the re module.

There is the question of how much detail to copy from PEP 498, I wonder if it will change any further?

Undoubtedly PEP 498 will evolve. You're better off not depending on it directly (despite being an alternative or variant). -- --Guido van Rossum (python.org/~guido)

Hi, I was able to get this done tonight. There's still simplifications to be done, perhaps keeping the string fragments in one piece? Also, a lot of things need to be passed to the constructor to avoid parsing the template twice. Positional arguments are working however, and we're back to inheriting from str. It is now rendering into the "real" string, and everything seems to work without the magic methods. ;) Thanks, -Mike On 08/27/2015 02:02 PM, Guido van Rossum wrote:

On 08/28/2015 08:11 AM, Guido van Rossum wrote:

Thanks, this looks much better, if we ever want to go in this direction. (Though I think you may want to separate the field names into two parts, the expression text and the format spec.)

It does currently separate the expression texts from the format specs internally, it felt excessive to pass them in separately in the constructor, as I'm sort of complaining about below. But, the end-developer won't see this typically, so it could be done before or after I suppose. Anyone have a preference?

Can you work with the team at peps@python.org <mailto:peps@python.org> to get a PEP number for this?

Yes, nice to have a third alternative to choose from. -Mike

On 08/27/2015 06:36 AM, Cody Piersall wrote:

Is the Draft PEP column of the table supposed to have both "Immediate Render" and "Deferred Render" as "Yes"? I'm hoping that's a typo, otherwise I don't understand what it means at all.

Yes, it supports both. By storing all inputs to the object, it can be rendered again, with optional changes such as overriding a value, or escaping the input. -Mike

On 08/27/2015 09:19 AM, Cody Piersall wrote:

When you say immediate, do you mean that it only takes a call to str()? Or is there some way to have e'this is an e-string' evaluate to a string without doing anything else?

There is no need to call str() manually with e'', the .rendered member is returned by default: >>> print(estr('Hello {friend}.')) 'Hello John' Here is the example implementation: https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py -Mike

On 08/27/2015 10:27 AM, Eric V. Smith wrote:

The problem with this auto-rendering is that the format_spec and

Hmm, I believe this is a design choice, one that should be made depending on whether this use-case is important and/or common. The estr provides for this situation instead by allowing for additional renderings if/when needed, but doesn't require str() in the common case. This is the first I've seen of directives passed inside the format spec, I'll add it to the comparison table. -Mike

On 08/27/2015 02:30 PM, Andrew Barnert via Python-ideas wrote:

On Aug 27, 2015, at 10:27, Eric V. Smith <eric@trueblade.com> wrote:

...

...

On 08/27/2015 11:22 AM, Mike Miller wrote:

...

On 08/27/2015 06:36 AM, Cody Piersall wrote:

Is the Draft PEP column of the table supposed to have both "Immediate Render" and "Deferred Render" as "Yes"? I'm hoping that's a typo, otherwise I don't understand what it means at all.

Yes, it supports both. By storing all inputs to the object, it can be rendered again, with optional changes such as overriding a value, or escaping the input.

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

Imagine that ":raw" is interpreted by to_html() to mean that the string does not get html escaped.

With PEP 501, the format_spec and conversion are opaque to the i-string machinery, and are only interpreted by the custom interpolation function (here, to_html()).

With str.format, it's the type of value that decides how to interpret the format spec. If you leave it up to the consumer of the i-string (the interpolation function) instead of the value's type, how do you handle things like numeric formats and datetime formats and so on? Would I need to do something like this:

to_html(e'<p>{str(e"{value:05}"):raw}</p>')

Or would to_html (and every other custom interpolator) have to take things like :raw05 and parse out a format spec to pass to value.__format__?

Your interpolator would need to decide. It might never call value.__format__. It might invent some other protocol, like __html_escape__(fmt_spec). Or, it might bake-in knowledge of how to convert whatever types it cares about. Or another good choice would be to use the singledispatch module. In fact, I like singledispatch so much that I'm going to have to use it in an example.

Maybe what you really want is !raw rather than :raw. If there is no conversion, __format__ gets called and the result passed around as part of the i-string object; if there is one, the value, conversion, and format spec get passed instead (and then to_html could decide that conversion 'raw' means to call format(value, format_spec) and then not escape the result).

You could do that in addition or in place of format_spec. Except currently conversions are only allowed to be a single character, but I don't see any reason not to relax that. The take away is that the PEP 501 i-string machinery applies zero significance to format_spec and conversion. It just parses them out of the template string. It's left up to the interpolator to apply some meaning to them.

Although that's pretty different from how the standard conversions work (call repr or ascii on the value, then format the resulting string with the format spec). So maybe replacement fields need another subpart separate from both the conversion and the format spec that's only used by custom interpolators?

There's nothing from stopping you from doing this. You could decide that your format_spec, for some interpolator, is composed of "part1^part2", and do something based on part1 and part2. See: https://bitbucket.org/ericvsmith/istring/src/d92e47c96609eed44ed57b7d3c1932b... for how my i-string str() interpolator applies format_spec and conversion. Another interpolator could do something different (for example, https://bitbucket.org/ericvsmith/istring/src/d92e47c96609eed44ed57b7d3c1932b... for regex escaping). Eric.

On Thu, Aug 27, 2015, at 13:27, Eric V. Smith wrote:

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

I feel like this would be better done with something like {value!raw}.

On Fri, Aug 28, 2015, at 03:52, Paul Moore wrote:

On 28 August 2015 at 02:55, <random832@fastmail.us> wrote:

...

On Thu, Aug 27, 2015, at 13:27, Eric V. Smith wrote:

...

The problem with this auto-rendering is that the format_spec and conversion character have to make sense to __format__. For example, you couldn't do this, if value were a string:

to_html(e'<p>{value:raw}</p>')

I feel like this would be better done with something like {value!raw}.

While I appreciate that people are still in a design phase with this, I'd like to point out that in the end, people will need to teach and remember this stuff.

Currently ! introduces a conversion, which is one of r, s, or a (and is rarely used except for the occasional !r). Whereas : introduces a format spec, which is a mini-language for describing how to format the value and is specific to the type of the value.

Yes, but the format spec mini-language belongs to the type of the value. Depending on what value is, "raw" could _already_ have a meaning.

The "raw" thing above feels like neither of those things. It feels more like a conversion, but if so then conversions are currently single letters, and language-defined.

Well, at the time I posted that I thought we were moving away from things being language-defined, because my mind was still on the "user- defined string prefixes" proposal from a while back. Them currently being single letters isn't really a compelling argument.

I'd also expect (for that reason) that any conversions would be valid in any type of formatting (str.format, f-strings, e-strings, whatever).

And what if the type of value expects to be able to process a format specifier of "raw", rather than it being used by to_html for its own purpose? The advantage of conversion specifiers is that they're currently a closed set. ---- One thing that I don't think *either* version successfully expresses is that while in many cases (including the to_html example) we want a string, that won't always be the case. If we have a syntax for inserting something as, e.g., a SQL parameter, it should be able to accept a double, but I'm not convinced it shouldn't _also_ be able to describe putting the string result of converting the double with ".05d" as a varchar.

On Aug 27, 2015, at 11:27, Guido van Rossum <guido@python.org> wrote:

I also don't understand the claim that no str(estr) is necessary to render the result -- the estr implementation given at https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py has a __str__ method that renders the .rendered attribute, but without the str() call the type of 'a' in the above examples would not be str, and various things that operate on strings (e.g. regular expression searches) would not work. A solution might be to make estr a subclass of str, but nothing in the PEP suggests that you have even considered this problem. (The only hint I can find is the comment "more magic-methods to be implemented here, to improve str compatibility" in your demo implementation, but without subclassing str this is not enough.)

Even subclassing str doesn't really help, because there's plenty of code (including, I believe, regex searches) that just looks at the raw string storage that gets created at str.__new__ and can never be mutated or replaced later. So, anything that's delayed-rendered is not a str, or at least it's not the right str. (I know someone earlier in the discussion suggested that at the C level you could replace PyUnicode_READY with a function that, if it's an estr, first calls self.__str__ and then initializes the string storage to the result and then does the normal READY stuff, but I don't think that actually works, does it?)

On 08/27/2015 11:27 AM, Guido van Rossum wrote:

specified. E.g. I couldn't figure out what code should be produced by:

a = e"Sliced {n} onions in {t1-t0:.3f} seconds."

Generalizing from the only example in the specification, this would become:

a = est("Sliced {n} onions in {t1-t0:.3f} seconds", n=n, t1-t0=t1-t0)

Yes, the demo does not currently handle arbitrary expressions, only .format() is implemented. The PEP is relying on much of the PEP 498 implementation (which I don't have at hand), so that part is underspecified. I hope to reconcile the details if/when the larger design is chosen. For now, I will add the ability to pass a context dictionary at init as well as keywords to prepare for further implementation.

I also don't understand the claim that no str(estr) is necessary to render the result -- the estr implementation given at https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py has a __str__ method that renders the .rendered attribute, but without the str() call

Yes, no explicit call is necessary. When you do things like print(e'') or e''.upper() or '' + e'', you'll get the rendered string without str(). That could be more clear, yes, and maybe not substantially different than i''.

A solution might be to make estr a subclass of str, but nothing in the PEP suggests that

I originally subclassed it from str, but others on the list advised against it. Which is preferred? Under what name is the string in a string object actually held (if any)? -Mike

On Thu, Aug 27, 2015 at 1:38 PM, Mike Miller <python-ideas@mgmiller.net> wrote:

I've addressed those questions, updated the demo script, and added your examples in a separate examples section. I'll write more after lunch.

Looking through your latest commit ( https://bitbucket.org/mixmastamyk/docs/commits/760274613d8c306cd688385fbbcf2...) I think you've painted yourself into an impossible corner, and haven't thought through the consequences in all cases enough. Apparently my Socratic questions didn't help enough, so I feel compelled to give you the answers. :-) The interpreter can't and shouldn't be passing in the values of all the variables involved in an expression. It should only be passing in the final evaluated result of each "slot" in the e-string. For my second example (with three rand() calls) it should pass the three different random values returned by the three rand() calls into the estr() constructor, e.g.: b = estr("Three random numbers: {rand()}, {rand()}, {rand()}.", rand(), rand(), rand()) Your entire formatting machinery should be rewritten using positional values instead of keyword args. Regarding subclassing str, there are indeed many problems with that (e.g. what is the type of str(...)+estr(...)), but without it, you will never be able to claim that str() calls are never needed, because quite a few built-in operations and stdlib modules in Python *require* that their arguments are str subclasses (or they treat str subclasses different than other classes). An important example is the re module.

There is the question of how much detail to copy from PEP 498, I wonder if it will change any further?

Undoubtedly PEP 498 will evolve. You're better off not depending on it directly (despite being an alternative or variant). -- --Guido van Rossum (python.org/~guido)

Hi, I was able to get this done tonight. There's still simplifications to be done, perhaps keeping the string fragments in one piece? Also, a lot of things need to be passed to the constructor to avoid parsing the template twice. Positional arguments are working however, and we're back to inheriting from str. It is now rendering into the "real" string, and everything seems to work without the magic methods. ;) Thanks, -Mike On 08/27/2015 02:02 PM, Guido van Rossum wrote:

On 08/28/2015 08:11 AM, Guido van Rossum wrote:

Thanks, this looks much better, if we ever want to go in this direction. (Though I think you may want to separate the field names into two parts, the expression text and the format spec.)

It does currently separate the expression texts from the format specs internally, it felt excessive to pass them in separately in the constructor, as I'm sort of complaining about below. But, the end-developer won't see this typically, so it could be done before or after I suppose. Anyone have a preference?

Can you work with the team at peps@python.org <mailto:peps@python.org> to get a PEP number for this?

Yes, nice to have a third alternative to choose from. -Mike