Am Do., 10. Juni 2021 um 14:42 Uhr schrieb Rob Cliffe via Python-ideas < python-ideas@python.org>:

I am concerned that we have too many string formatting methods: % formatting .format() f-strings string templates (are there others I've missed?). And now we have another proposed one. It's all getting a bit much. And IMO a turnoff for people learning Python (which one should I use, and why?). Best wishes Rob Cliffe

I can feel with you. There are some new things in Python which make it harder to learn. I personally don't see a need for the walrus operator. But on the other hand you can't compare Template Literals with normal string handling. This really helps developers to avoid cross-site-scripting attacks by enabling a secure escaping of all strings which are not explicitly marked as safe. Template Literals only make sense if you want to escape values like in HTML, XML or SQL templates. Regards, Thomas

Well, I share Rob's concern but I do see the point of this:

...

Template Literals only make sense if you want to escape values like in HTML, XML or SQL templates.

Maybe they should have a name to better reflect this intended use case? Safe Templates, Escaped Strings, I don't know, I'm terrible at naming things but I do think this would be a nice modification.

...

Thank you Felipe for looking into this. Yes, you are right. A better

Am Do., 10. Juni 2021 um 15:48 Uhr schrieb Felipe Rodrigues < felipe@felipevr.com>: heading might make sense. I just copied the name (Template Literals) from JavaScript. I am not good at naming either. Maybe someone else has an idea how to name this? Regards, Thomas

There may well be use cases for this, but one thing struck me. From the PEP: "Template Literals provide an easy way to access the local and global variables (like f-strings), so that passing a dictionary to the Template is not necessary." This seems to be crossing the line between "data" and "code" -- that's a line that can get pretty fuzzy in Python, but it's still a distinction I find helpful to think about. f-strings provide an easy way to build strings with stuff stored in code: local variables. This makes them very helpful for things like building Exception messages and the like, or showing the results of computation. But for the most part, populating a template is likely to be done from data, rather than code -- results of a database query, or what have you. So this kind of template building is usually very well suited to passing a dictionary around, rather than using the local namespace. Even if you are using Python objects to model your data (e.g. dataclasses and the like) -- you still have: a) one object to pass in to your template builder and/or b) an easy way to make a dict out of the object to pass into a template. I could be missing something, but I just don't see the benefits of having f-string like access to local variables in this context. -CHB On Thu, Jun 10, 2021 at 8:56 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

Thomas Güttler writes:

...

This really helps developers to avoid cross-site-scripting attacks by enabling a secure escaping of all strings which are not explicitly marked as safe.

Frameworks can already do this by unconditionally applying a function like conditional_escape to all evaluated template variables. (If that's too drastic for your taste, there could be a pragma %conditional_escape_everything to turn it on.) Why don't they? If it's not "they just didn't think of it", and there's a real reason, why doesn't that reason apply to your template literals?

Note that str has no "safe" attribute, and attributes defined by a framework are not known to Python. You need to explain how you can Python-evaluate an expression to a str as your template literal does, and still preserve the "safe" mark that I presume is an attribute of a class defined by the framework.

I guess the problem of accessing the framework's attribute can be solved by delegating that to the __format__ method of the framework type, and maybe preserving it can be handled by having that __format__ method return a subclass of str.

But this reintroduces a strong possibility of programmer error, because any function that constructs and returns a new str will strip the "safe" mark. This happens *before* the __format__ method can be invoked -- str's __format__ does not check for a safe mark -- so it's a real problem. This might dramatically reduce the utility of these template literals because it's simply not safe to allow the full range of expressions that f-strings allow. (That could be a YAGNI, but you need to explain and if possible document that.) Also, this means that frameworks can no longer just inherit from str: they need to reimplement literally every method that returns str, or prohibit its use in templates.

Note that 'is_literal' is not the same as "safe". Based on the example, this is intentional: is_literal simply means that this isn't the value of an expression, simplifying implementation of the internal function that evaluates the template string to a TemplateLiteral. But this means that the function joining a TemplateLiteral needs to consider both is_literal (which is safe but apparently unmarked) and the 'safe' attribute. This seems more complicated than it needs to be.

TemplateLiteral is not a good name for that type. The backtick construct is a literal (except it's really not ;-), the TemplateLiteral is a constructed value. TemplateValue or TemplateTokenSequence or something like that might be a better name. In any case it's a little confusing that both the syntax and the value are called "literal". It's not impossible to understand, but at least for me I have to think "is this the syntax or is this an object?" every time I see it.

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WHOPY2... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Fri, Jun 11, 2021 at 2:37 AM Thomas Güttler wrote:

If you don't create HTML with Python daily, then you might not feel the pain.

If you create many HTML strings daily, then you will be typing `foo=foo,

bar=bar` (to pass the variables into the template) over and over again. My goal is to have a super reduced syntax, so that developers need only a few characters to create save (properly escaped) html with Python.

This is exactly why I oppose this. I, like many, many other developers, simply do not do that. Like basically not ever. In the very rare cases I do, f-strings are more than enough. This is trying to turn Python into PHP, a language dedicated to HTML processing. We simply do not need dedicated syntax that is ONLY relevant to making HTML strings, saving a small number of characters for that one single task. ... and no, there isn't any other non-fanciful usage. It is specifically harmful to performance in SQL queries, so is an anti-pattern. Maybe, conceivably, once in a blue moon, someone will come across some other domain where the new syntax could be useful (LaTeX templating?), but far, far, less than to make it worth changing the language and imposing the burden on 10s or 100s of millions of people who use Python. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.

Thomas Güttler writes:

I don't understand what you mean with "pragma %conditional_escape_everything". Could you please elaborate?

"Pragma" just means it's a way to switch on conditional_escape for all template variable accesses for the rest of the file. The main point is that Django already can conditional_escape all the template variable accesses if it wants to, but it doesn't. It can provide a way for the template author to conditional_escape all the template variable accesses if the author wants to, but apparently it doesn't do that either. I have to wonder why not.

That "safe" attribute is outside the scope of the PEP.

But it's not outside the scope of the PEP, because by using conditional_escape in the template_literal_to_safestring() example, you used the assumption that it's accessible. But that's only true if you pass "safe strings" straight through to the tokens in the TemplateLiteral. If you don't, conditional_escape in your example would escape a safe string. I don't see how you can guarantee that it won't if you allow full f-string syntax.

In Django this is solved via mark_safe(), conditional_escape() and fomat_html().

This isn't an explanation though. The question is if you are implementing a web framework that depends on returning something that is not a str in the tokens, how do you do that and still retain the power of f-string format expression syntax? Or are you proposing that a template literal be restricted to plain variables -- no expressions and maybe no format specs that involve constructing strs -- in the {}?

Please provide an example how to simplify `template_literal_to_safestring()` of the draft: https://github.com/guettli/peps/blob/master/pep-9999.rst#specification

That's not my problem. I do not want this PEP, but if you want to push it to completion, I want to be sure that it works well for the proposed applications. For the reasons above, so far I don't see how it can be a big win for the web templating application. I'm not clear on how it's a win for the logging application, either. Note that as proposed in your PEP, template literal syntax would be forbidden by Google for the same reasons that f-strings are: (1) the logging implementation can theoretically collect the patterns from .template as queryable strs, but that seems to be just wasted effort because they are just strs and not usable as TemplateLiterals, and (2) it always gets rendered (to TemplateLiteral), which is also wasted effort if the logging function has no logs to send it to. PEP 501 is intended to address both issues. Note that in this example (a slight extension of your example): while (theres_work_to_do_matey): if load > max_load: logging.warn(f'Load is too high: {load}') the f-string works because it is evaluated when 'warn' is called. The logging problem that PEP 501 addresses is this: load_warning = f'Load is too high: {load}' while (theres_work_to_do_matey): if load > max_load: logging.warn(load_warning) Here the f-string does *not* work because it's evaluated at the binding to 'load_warning', not at 'warn' time. Template literal syntax as in your PEP also fails, for the same reason. This works: load_warning = 'Load is too high: {load}' while (theres_work_to_do_matey): if load > max_load: logging.warn(load_warning.format(load=load)) but for some reason ;-) people hate that, and Google recommends load_warning = 'Load is too high: {load}' while (theres_work_to_do_matey): if load > max_load: logging.warn(load_warning, load=load) if the logging package supports it, which is still kinda ugly. But a PEP 501 i-string "just works" nicely: load_warning = i'Load is too high: {load}' while (theres_work_to_do_matey): if load > max_load: logging.warn(load_warning) (This assumes a future version of logging.warn that calls str() on the first argument if it is an InterpolationTemplate.)

He/she will see: `<h1>Hello {name}</h1>` is the way to create a HTML fragment and he/she will do so.

That's not interesting, though. The interesting question is what is an author of a consumer of TemplateLiteral going to do. I'll guarantee you she/he will see this fragment = ''.join([x for x, _ in template_literal]) somewhere. I hope she/he doesn't copy it! The problem here is that frameworks like Django, or even templating languages like Jinja, are quite heavy. People looking for more lightweight approaches are likely to grab on to template literals and implement a stripped-down "vocabulary" directly as Python functions and methods. If this is going to be implemented, I want this to work naturally for them, and not be more restricted (especially not if the restrictions are "don't do this or you'll be sorry" conventions like Google's Logging Style Guide!) than f-strings are. I'm NOT saying it won't "just work" in any of the applications suggested: I haven't done, and am not going to do, the necessary work to show it doesn't. I'm "just asking questions". But if I didn't ask them, eventually somebody will, and as PEP proponent you probably need to answer them. If you don't, I believe the risk that the PEP will be rejected increases. Steve

Hi Nick and all other Python ideas friends, yes, you are right. There is not much difference between PEP-501 or my proposal. One argument why I would like to prefer backticks: Some IDEs detect that you want to use a f-string automatically: You type: name = 'Peter' print('Hello {name... and the IDE automatically adds the missing "f" in front of the string: name = 'Peter' print(f'Hello {name... This is a handy feature (of PyCharm), which would not work reliably if there are two different prefixes. ------- You mentioned these things: eager rendering: I think deferred rendering would increase the complexity a lot. And I think it is not needed. runtime value interpolation: It is up to the receiver of types.InterpolationTemplate to handle the data structure. dedicated templating libraries: One temp after the other. I think HTML and SQL libraries would adapt as soon as the foundation is available. I would be happy if PEP-501 would come true. Regards, Thomas Güttler Am So., 27. Juni 2021 um 06:45 Uhr schrieb Nick Coghlan

:

Stephen J. Turnbull wrote:

...

But a PEP 501 i-string "just works" nicely: load_warning = i'Load is too high: {load}' while (theres_work_to_do_matey): if load > max_load: logging.warn(load_warning) (This assumes a future version of logging.warn that calls str() on the first argument if it is an InterpolationTemplate.)

A "Why this rather than PEP 501's interpolation templates?" is the main thing I was looking for in the PEP and I didn't find it.

If the proposal is just a variant on PEP 501 with the syntax changed from i"template" to \`template\` and the template type name changed from InterpolationTemplate to TemplateLiteral, it doesn't need to be a new PEP, I can just explicitly reject those spelling options in PEP 501. (The reasons for that PEP's deferral unfortunately still hold, though - eager rendering, runtime value interpolation, and dedicated templating libraries together cover enough cases that the motivation for introducing the semantic complexity of yet another templating option gets weakened dramatically).

If the differences between the proposals run deeper than that, then the proposed new PEP needs to spell them out. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XSGJO3... Code of Conduct: http://python.org/psf/codeofconduct/

Am Fr., 2. Juli 2021 um 12:06 Uhr schrieb Nick Coghlan :

On Fri, 2 Jul 2021, 5:12 pm Thomas Güttler, wrote:

...

Hi Nick and all other Python ideas friends,

yes, you are right. There is not much difference between PEP-501 or my proposal.

One argument why I would like to prefer backticks:

Some IDEs detect that you want to use a f-string automatically:

You type:

name = 'Peter' print('Hello {name...

and the IDE automatically adds the missing "f" in front of the string:

name = 'Peter' print(f'Hello {name...

This is a handy feature (of PyCharm), which would not work reliably if there are two different prefixes.

-------

You mentioned these things:

eager rendering: I think deferred rendering would increase the complexity a lot. And I think it is not needed.

Eager rendering is f-strings. Any templating proposal necessarily involves a delayed rendering step, when the template is combined with the interpolated values.

runtime value interpolation: It is up to the receiver of

...

types.InterpolationTemplate to handle the data structure.

I really meant runtime template parsing here (i.e. str.format).

dedicated templating libraries: One temp after the other. I think HTML and

...

SQL libraries would adapt as soon as the foundation is available.

The existence of i-strings likely wouldn't change the syntax of jinja2 templates, Django templates, SQL Alchemy, pandas, etc.

I would be happy if PEP-501 would come true.

...

So would I, but I still don't have a compelling answer to the "but it's yet another subtly different way to do it" objection.

Today I read the replies to this thread again. Of course there where some "-1" replies, but overall there was positive feedback. Today I played with template literals from Javascript: [image: image.png] I like it. Nick, would you be open to adapting to the JS syntax for PEP-501? I propose `...{var} ...` This means backticks, but without the dollar sign. I would make it like in JS: The string in backticks can span several lines. So what is the next step now? Regards, Thomas

On 5 Jul 2021, at 08:07, Thomas Güttler wrote:

This means backticks, but without the dollar sign.

In bash the backtick was so often a problem that $(cmd) was added. Having removes the grit-on-Tim's-screen backtick in python 3 I would not like to see it return with its issue of being confused with single-quote. Barry

On Tue, Jul 6, 2021 at 6:10 AM Jim Baker wrote:

On Mon, Jul 5, 2021, 12:56 PM Barry Scott wrote:

...

On 5 Jul 2021, at 08:07, Thomas Güttler wrote:

This means backticks, but without the dollar sign.

In bash the backtick was so often a problem that $(cmd) was added.

Having removes the grit-on-Tim's-screen backtick in python 3 I would not like to see it return with its issue of being confused with single-quote.

One mitigation is that the backtick should always require a tag as a prefix. So seeing something like

elem = html`<p>Some item: {value}</p>`

is hopefully fairly obvious what's going on - it's not just going to be mixed up with a single quote. Uses like log(f`foo`) should be hopefully discouraged, in the same way that we don't use l (that's the lower-case letter L if you're not reading this email with the numeric codepoints) as a variable, but we are happy enough to write something like limit = 42 - it's clear in the context.

Question: what's the advantage of this magic syntax over something much simpler: elem = html(i'<p>Some item: {value}</p>') That avoids the backtick problem because it actually IS an apostrophe. Or a double quote. Or triple quotes, whatever you want to use. It's the exact format already used for other special string literal types, including f-strings. ChrisA

FWIW, we could make f-strings properly nest too, like you are proposing for backticks. It's just that we'd have to change the lexer. But it would not be any harder than would be for backticks (since it would be the same algorithm), nor would it be backward incompatible. So this is not an argument for backticks. Separately, should there be a way to *delay* evaluation of the templated expressions (like we explored in our private little prototype last year)? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On 6/07/21 9:56 am, Jim Baker wrote:

d = deferred_tag"Some expr: {:(x*2)}"

All that is happening here is that this being wrapped in a lambda, which captures any scope lexically as usual.

Is there reason to think this will be a common enough requirement to justify having an abbreviated syntax for a parameterless lambda that's only available in template expressions? -- Greg

On Mon, Jul 5, 2021 at 5:05 PM Chris Angelico wrote:

On Tue, Jul 6, 2021 at 9:39 AM Greg Ewing wrote:

...

On 6/07/21 9:56 am, Jim Baker wrote:

...

d = deferred_tag"Some expr: {:(x*2)}"

All that is happening here is that this being wrapped in a lambda,

which

...

...

captures any scope lexically as usual.

Is there reason to think this will be a common enough requirement to justify having an abbreviated syntax for a parameterless lambda that's only available in template expressions?

If it's just being wrapped in a lambda function, probably nothing, but I think that that would be *very* surprising behaviour. People will expect that the expressions' values will be collected at the point you hit the interpolated string, not later when it gets used.

It would be surprising indeed if this was the *default* behavior, that's why you have to specially mark it using {:...} in Jim's proposal. IIUC the idea is that the entire template becomes effectively a lambda. It's useful if the evaluation is relatively expensive and may never be needed, e.g. for logging at a level that is off in production. We can debate whether it's better to mark individual substitutions with something like {:...} or whether we should mark the template as a whole (obviously the marking must be understandable by the parser). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Tue, 6 Jul 2021, 7:56 am Jim Baker, wrote:

On Mon, Jul 5, 2021, 2:40 PM Guido van Rossum wrote:

...

FWIW, we could make f-strings properly nest too, like you are proposing for backticks. It's just that we'd have to change the lexer. But it would not be any harder than would be for backticks (since it would be the same algorithm), nor would it be backward incompatible. So this is not an argument for backticks.

Good point. At some point, I was probably thinking of backticks without a tag, since JS supports this for their f-string like scenario. but if we always require a tag - so long as it's not a prefix already in use (b, f, r, fr, hopefully not forgetting as I type this email in a parking lot...) - then it can be disambiguated using standard quotes.

There's a deferred PEP proposing a resolution to the f-string nesting limitations: https://www.python.org/dev/peps/pep-0536/#motivation

...

Separately, should there be a way to *delay* evaluation of the templated expressions (like we explored in our private little prototype last year)?

I think so, but probably with an explicit marker on *each* deferred expression. I'm in favor of Julia's expression quote, which generally needs to be enclosed in parentheses, but possibly not needed in expression braces (at the risk of looking like a standalone format spec). https://docs.julialang.org/en/v1/manual/metaprogramming/

So this would like x = 42 d = deferred_tag"Some expr: {:(x*2)}"

All that is happening here is that this being wrapped in a lambda, which captures any scope lexically as usual. Then per that experiment you mentioned, it's possible to use that scope using fairly standard - or at least portable to other Python implementations - metaprogramming, including the deferred evaluation of the lambda. (No frame walking required!)

Other syntax could work for deferring.

It reminds me of the old simple implicit lambda proposal: https://www.python.org/dev/peps/pep-0312/ The old proposal has more ambiguities to avoid now due to type hinting syntax but a parentheses-required version could work: "(:call_result)" Cheers, Nick.

On 6/07/21 8:39 am, Guido van Rossum wrote:

FWIW, we could make f-strings properly nest  too... So this is not an argument for backticks.

An argument might be that single and double quotes currently do not nest and never have done, so having them start to nest but only in the context of f-strings could be confusing. Whereas backticks are currently unused, and they used to nest back when they were used, so giving them a new nesting usage will be unsurprising to those with long enough memories. -- Greg

On Thu, Jun 10, 2021 at 11:50 PM Thomas Güttler wrote:

Am Do., 10. Juni 2021 um 15:33 Uhr schrieb David Mertz :

...

Strong -1

As others noted in prior discussion, even if this existed, it works be an anti-pattern for SQL. So basically, it's just baking in an HTML-only template language into the language syntax.

Python already had excellent HTML templating in libraries. The fact Django has a function with a long name just suggests importing it with a shorter name.

...

Python has many uses having nothing to do with web pages. This could make sense for PHP (does anyone use that still?). It's a poor for for Python.

PHP is very productive. The majority of sites are written in PHP.

The only thing which PHP has not: Respect.

It does not get respect since it is too easy.

Too easy to do things badly, that's why PHP lacks respect. It's not about how easy it is to quickly whip up a web site; it's about how easy it is to quickly whip up a really really bad web site, and how the design of the language encourages practices that tend towards that.

Python is getting pushed into the backend during the last years, while web frontend is mostly done in Javascript.

I think this is the wrong direction.

Together with tools like htmx (fragments over the wire), I would rather choose PHP than Javascript.

What's the advantage of htmx? When I want to build a good interactive web site, my general pattern is a back end with a well-defined API, and a front end in JavaScript that makes use of this API. That API is usually going to be based on either a RESTful (or roughly REST-like) JSON transactional system, or something like websockets, again carrying JSON payloads. HTML is the realm of the display, not the back end.

PHP is great for creating HTML. I know that the easy solution does not get much respect, but maybe you understand that making it easy to create HTML would be a major benefit for Python.

I actually like the idea, but this justification seems *extremely* weak. ChrisA

On Thu, Jun 10, 2021 at 4:07 PM Chris Angelico wrote:

What's the advantage of htmx? When I want to build a good interactive web site, my general pattern is a back end with a well-defined API, and a front end in JavaScript that makes use of this API. That API is usually going to be based on either a RESTful (or roughly REST-like) JSON transactional system, or something like websockets, again carrying JSON payloads. HTML is the realm of the display, not the back end.

I use unpoly but the deal is the same: backend HTML rendering lets us leverage all sorts of meta programing, ie. add a field to a form and you don't have to change your presentation layer: the Form instance will automatically render a form field with all the necessary validation, ie. in case of a username field which must be unique in the database. -- ∞

Something I don't understand is whether there is anything about this proposed feature that can't be accomplished with a simple function. IIUC, the proposal turns this: foo = "spam & eggs" `Here, have some {foo}.` ...into something like this (I am making up a more readable repr): TemplateLiteral( template = " Here, have some {foo}.", tokens = (("Here, have some ", True), ("spam & eggs", False)), ) What is it about this task that cannot be accomplished with a class and function? Skeleton might look like: @dataclass class TemplateLiteral: template: str tokens: Sequence(Tuple(str, bool)) = field(init = False) def templify(s: str) -> TemplateLiteral: ... And use it like this:

...

...

templify("Here, have some {foo}.") TemplateLiteral(template = " Here, have some {foo}.", tokens = (("Here, have some ", True), ("spam & eggs", False)))

What is it about this task that requires it being handled at the language level...? --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Thu, Jun 10, 2021 at 9:36 AM David Mertz wrote:

Strong -1

As others noted in prior discussion, even if this existed, it works be an anti-pattern for SQL. So basically, it's just baking in an HTML-only template language into the language syntax.

Python already had excellent HTML templating in libraries. The fact Django has a function with a long name just suggests importing it with a shorter name.

Python has many uses having nothing to do with web pages. This could make sense for PHP (does anyone use that still?). It's a poor for for Python.

On Thu, Jun 10, 2021, 2:31 AM Thomas Güttler wrote:

...

Thank you Guido, Chris, Matt and Richard for your feedback to my last email.

Here is an updated version called "Template Literals".

I am looking for a core developer who can sponsor this PEP.

Please speak up if you want to help me.

Regards, Thomas Güttler

Source and updates: Pre-PEP 9999 https://github.com/guettli/peps/blob/master/pep-9999.rst

PEP: 9999 Title: Template Literals Author: Thomas Güttler <info at thomas-guettler.de> Sponsor: TODO Status: Draft Type: Standards Track Content-Type: text/x-rst http:///dev/peps/pep-0012 Created: 08-Jun-2021 Python-Version: TODO Post-History: 08-Jun-2021 ------------------------------

Contents

- Abstract <#m_2818986526677889823_m_-138379983175526697_abstract> - Motivation <#m_2818986526677889823_m_-138379983175526697_motivation> - Rationale <#m_2818986526677889823_m_-138379983175526697_rationale> - Specification <#m_2818986526677889823_m_-138379983175526697_specification> - Security Implications <#m_2818986526677889823_m_-138379983175526697_security-implications> - Reference Implementation <#m_2818986526677889823_m_-138379983175526697_reference-implementation> - Alternative Ideas <#m_2818986526677889823_m_-138379983175526697_alternative-ideas> - Rejected Ideas <#m_2818986526677889823_m_-138379983175526697_rejected-ideas> - Open Issues <#m_2818986526677889823_m_-138379983175526697_open-issues> - References <#m_2818986526677889823_m_-138379983175526697_references> - Copyright <#m_2818986526677889823_m_-138379983175526697_copyright>

Abstract <#m_2818986526677889823_m_-138379983175526697_id6>

This PEP adds Template Literals to Python.

To avoid code injection like XSS or SQL-injection Template Literals can help you to write save Python code.

Template Literals provide an easy way to access the local and global variables (like f-strings), so that passing a dictionary to the Template is not necessary. Motivation <#m_2818986526677889823_m_-138379983175526697_id7>

In the context of web development Python can do more than providing REST APIs via http. With the trend to Server-Side-Rendering, we face a fundamental question:

How to create HTML with Python?

If you use the FrOW pattern (HTML fragments over the wire) [1] <#m_2818986526677889823_m_-138379983175526697_frow>, then you will be writing small methods returning small HTML fragments.

As a developer I want to pass escaped data into template literals to be as simple as possible. Rationale <#m_2818986526677889823_m_-138379983175526697_id8>

Imagine you want to create a small HTML fragment in Python, and return it as HTTP-Response:

HttpResponse(f''' <h1>Hi {name}</h1> Your messages: {messages}''')

The problem in above example is, that no escaping gets done.

In above example "name" and "messages" should be treated differently.

The variable "name" should get escaped. For example if the name is "Mary & Bob", the result should be "Mary & Bob".

The variable "messages" contains HTML which is already escaped. It should not be escaped again.

Most frameworks have a way to do this conditional escaping.

For example Django uses conditional_escape() https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.condition... [2] <#m_2818986526677889823_m_-138379983175526697_id2>

With the help of conditional_escape() the above problem could be solved like this:

HttpResponse(f''' <h1>Hi {conditional_escape(name)}</h1> Your messages: {conditional_escape(messages)}''')

This solution has two drawbacks:

1. It is too verbose. Typing "conditional_escape(...)" again and again is cumbersome. 2. If a conditional_escape() gets forgotten Cross-site scripting attacks could be possible, since malicious users could inject HTML.

Specification <#m_2818986526677889823_m_-138379983175526697_id9>

Template Literals use backticks (like JavaScript Template Literals https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l... [3] <#m_2818986526677889823_m_-138379983175526697_id4>)

Example:

name = 'Mary & Bob' messages = `<ul><li>message1</li><li>message2</li></ul>` return HttpResponse(` <h1>Hi {name}</h1> Your messages: {messages}

Today: {datetime.date.today()}`)

Expressions within curly braces get handled liked in f-strings (PEP-498).

The Template Literal creates an instance of the new class types.TemplateLiteral.

types.TemplateLiteral has two attributes:

- template: The raw string inside the backticks. - tokens: A list of tuples: (value, is_literal).

For above example this would mean:

template = ''' <h1>Hi {name}</h1> Your messages: {messages}

Today: {datetime.date.today()}'''

tokens = [ ('\n <h1>Hi ', True), ('Mary & Bob', False), ('</h1>\n Your messages: ', True), (, False), ('\n\n Today: ', True), (, False) ]

It is outside this PEP how a consumer of TemplateLiteral handles this data structure.

For example the Django web framework could transform a TemplateLiteral to a SafeString like this:

def template_literal_to_safestring(template_literal): return mark_safe( ''.join( [ conditional_escape(value) if not is_literal else value for (value, is_literal) in template_literal.tokens ] ) )

This PEP is not related or constraint to the Django framework. It is even not related to HTML. It can be used for any kind of templating. Security Implications <#m_2818986526677889823_m_-138379983175526697_id10>

Template Literals can execute arbitrary code (like f-strings).

Template Literals get created by Python developers, not by users. If you want to make templates available for users (for example if you develop a CMS), then please use a different solution. Reference Implementation <#m_2818986526677889823_m_-138379983175526697_id11>

TODO Alternative Ideas <#m_2818986526677889823_m_-138379983175526697_id12>

Instead of backticks for example t'...' could be used. Rejected Ideas <#m_2818986526677889823_m_-138379983175526697_id13>

TODO Open Issues <#m_2818986526677889823_m_-138379983175526697_id14>

TODO References <#m_2818986526677889823_m_-138379983175526697_id15> [1] <#m_2818986526677889823_m_-138379983175526697_id1> FrOW, "HTML Fragments Over the Wire". Frameworks like Unpoly, Hotwire or htmx. [2] <#m_2818986526677889823_m_-138379983175526697_id3> https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.condition... [3] <#m_2818986526677889823_m_-138379983175526697_id5> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l... Copyright <#m_2818986526677889823_m_-138379983175526697_id16>

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LYAC7J... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6MMGTZ... Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Jun 10, 2021 at 7:10 AM Chris Angelico wrote:

This proposal is basically for a way to take an f-string-like construct and, instead of calling format() on each of the values and joining them together into a string, you do something else with it. Or from a language perspective, you package it all up and hand it to a custom function.

So it's basically an f-string minus the final step - which is why PEP 501 described f-strings in terms of interpolated strings.

Unevaluated f-strings is a nice way to think about this functionality. Another use-case that comes to mind is logging. The Google Python style guide says not to use f-strings, for example, because it wants to be able to collect the unexpanded pattern strings and not waste time rendering unlogged messages: https://google.github.io/styleguide/pyguide.html#3101-logging

Thomas Güttler writes:

Am Fr., 11. Juni 2021 um 03:17 Uhr schrieb Stephan Hoyer :

...

Unevaluated f-strings is a nice way to think about this functionality.

But they're not "unevaluated" in a lot of important ways. A better term might be "pre-assembled". :-)

...

Another use-case that comes to mind is logging. The Google Python style guide says not to use f-strings, for example, because it wants to be able to collect the unexpanded pattern strings and not waste time rendering unlogged messages: https://google.github.io/styleguide/pyguide.html#3101-logging

Thank you Stephan for this feedback. I added Logging to the draft: https://github.com/guettli/peps/blob/master/pep-9999.rst#logging

If this does solve the logging problem, it would be a killer app (but that still might not be enough, given that PEP 501 couldn't clear the bar and logging was a leading application for that PEP). Perhaps you should focus on that rather than the HTML escaping problem. Steve

I think this idea is promising but instead of doing it by adding new syntax and a totally different object, why not attach a __templates__ dunder member to every string but only OPTIONALLY populate it when a string is formatted? For every regular string it would just be None:

...

...

"".__template__

But if you create a string using an f string, you can provide a directive, as part of the format specification mini language, to store it:

...

...

s = f"{foo!t}"

...or a directive at the front of the string to store all the f string arguments:

...

...

s = ft"{foo}"

This would save the values marked for storage as a tuple of Template arguments in the __templates__ member for later. It may also be desirable for the Template object to store the name foo and the position in the string the foo object was formatted. So:

...

...

foo = "spam" s = f"{foo!t}"

Now instead of being None, s.__template__ would store an object recording the name foo, what was passed to foo, and the position:

...

...

s.__templates___ ( Template (name="foo", value="spam", position=(0,4)), )

You could do the same with positional f string arguments, but using an integer as the name. An html or log function would later use, or discard, the template information as it sees fit. On Fri, Jun 11, 2021, 6:23 AM Thomas Güttler wrote:

Am Fr., 11. Juni 2021 um 11:10 Uhr schrieb Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp>:

...

Thomas Güttler writes:

...

Am Fr., 11. Juni 2021 um 03:17 Uhr schrieb Stephan Hoyer < shoyer@gmail.com>:

...

...

Unevaluated f-strings is a nice way to think about this functionality.

But they're not "unevaluated" in a lot of important ways. A better term might be "pre-assembled". :-)

...

...

Another use-case that comes to mind is logging. The Google Python style guide says not to use f-strings, for example, because it wants to be able to collect the unexpanded pattern strings and not waste time rendering unlogged messages: https://google.github.io/styleguide/pyguide.html#3101-logging

Thank you Stephan for this feedback. I added Logging to the draft: https://github.com/guettli/peps/blob/master/pep-9999.rst#logging

If this does solve the logging problem, it would be a killer app (but that still might not be enough, given that PEP 501 couldn't clear the bar and logging was a leading application for that PEP). Perhaps you should focus on that rather than the HTML escaping problem.

Hi Stephen,

I can't solve a problem which I don't know. Logging plays no big role in my development tasks any more. I use checks, metrics and sentry. In the past logging was important for me, but this changed.

But I am sure that the proposal can be used to improve logging.

Regards, Thomas

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YEYP2P... Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Jun 11, 2021 at 10:12 AM Thomas Güttler wrote:

Am Fr., 11. Juni 2021 um 14:51 Uhr schrieb Ricky Teachey < ricky@teachey.org>:

...

I think this idea is promising but instead of doing it by adding new syntax and a totally different object, why not attach a __templates__ dunder member to every string but only OPTIONALLY populate it when a string is formatted?

I am very happy that you think this is promising.

Changing the interface of every string in Python feels much too big for me. I don't dare to think about it. I have concerns that it could have negative impacts on parts which I am not aware of.

I'm not a python internals expert, but i am imagining it wouldn't be a very significant performance hog for anyone not interested in the feature... seems like all that would be added is the equivalent of a __slot__ to the str class object, and a reference to None inside it. so the memory footprint of every string object would not have to grow. the class just grows a new dunder. the templated string objects themselves would have to have a bigger memory footprint of course. but the user/developer is choosing to do that by using templated strings. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

On Thu, Jun 10, 2021 at 10:12 AM Stestagg wrote:

On Thu, Jun 10, 2021 at 2:58 PM Ricky Teachey wrote:

...

Something I don't understand is whether there is anything about this proposed feature that can't be accomplished with a simple function...

...

.... And use it like this:

...

...

...

templify("Here, have some {foo}.") TemplateLiteral(template = " Here, have some {foo}.", tokens = (("Here, have some ", True), ("spam & eggs", False)))

What is it about this task that requires it being handled at the language level...?

AIUI, that is hard to do without language support, because the function templify can't easily access the value of 'foo' because the binding only exists in the caller's scope.

So for a simple function to be used, you'd have to pass in the placeholder values explicitly as separate arguments, which somewhat defeats the point.

Steve

Got it now. Should have been able to figure that out on my own thanks for explaining! Makes me wonder if stepping back and thinking, instead, about a way to automatically (optionally?) put the current calling context dictionary(ies) somewhere that a called function can get at it/them, so it can do what it wants with the context. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

Supporting JavaScript-style template literals, specifically tagged templates, could be generally useful. If we look at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l..., it's a really simple idea: The tag function is called with the following arguments: the strings from the template (the raw string is also available), plus any evaluated expressions referenced by that string. This idea can be equivalently expressed in a number of ways in Python. So this might be: ``` from some_html_templating_library import html_template_function as h num = 42 constraint = "A number > 40" html_text = h"<b>{num}</b> complies with the constraint <i>{constraint}</i>" ``` which results in `html_text` being set to `h(template, 42, "A number > 40")`, where `template` is some object that contains at the very least the original raw string as well as the non-expression strings. Notably such tagged template literals - so just like JavaScript tagged templates - are not restricted to HTML or even to returning strings, but instead could be used for a variety of additional purposes, that I won't work out this moment: * Logging strings which don't have to format their expressions - so similar to current logging when not using f-strings, but with the easy interface of f-strings, including desired formatting of expressions in simple logger scenarios * Internationalization of strings * SQL queries with sanitization of expressions (to avoid "Bobby Tables" injection) * Object literals - something like D"10.2" instead of the somewhat longer D("10.2"), assuming for the latter `from decimal import Decimal as D` (this is a pretty weak argument on the face of it, but I can see the possibility of being more restrictive in the use of monkey patching namespaces with such literals to avoid unnecessary function call overhead...) * Code (or AST) construction, following the quasiquote idea from Lisp * Latex and other minilanguages (is Latex properly a minilanguage? ;) ) For some of these use cases, ideally we don't use `sys._getframe` if this were to be implemented - assuming f-string style expressions, it's perfectly feasible to use a lambda to capture any variables as desired using lexical scoping instead of the dynamic scoping of `sys._getframe` AND we need deferred evaluation of the expression. Some shorthand for the lambda keyword could be nice however. I should mention that when I looked at some similar ideas last summer ( https://github.com/jimbaker/fl-string-pep/, useful for some experiments on lexical scoping), in part with Guido, I was thinking that an implicit lambda should be used for all such expressions in tagged template literals. But my feeling now is that this would not be so Pythonic - we explicitly mark such usages now, and we should continue to do so. Just need to find the right character sequence for such shorthand! On Sat, Jun 12, 2021 at 5:51 AM Jeff Allen wrote:

On 10/06/2021 15:46, Ricky Teachey wrote:

Makes me wonder if stepping back and thinking, instead, about a way to automatically (optionally?) put the current calling context dictionary(ies) somewhere that a called function can get at it/them, so it can do what it wants with the context.

I think that would violate the expectation that a function call means the same thing in different contexts, hence f-strings work by supplying values at the "call site" not variables. I know you will find exceptions (like import context), but they are not as all-consuming as snapping the entire namespace. If you really want to, there's sys._getframe.

Jeff _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DUEB3H... Code of Conduct: http://python.org/psf/codeofconduct/

On 6/13/2021 7:24 PM, Chris Angelico wrote:

On Mon, Jun 14, 2021 at 8:58 AM Jim Baker wrote:

...

* Logging strings which don't have to format their expressions - so similar to current logging when not using f-strings, but with the easy interface of f-strings, including desired formatting of expressions in simple logger scenarios

The only benefit would be the string formatting; the expressions have to be evaluated still. Is that worth it?

Personally, I always just log with f-strings and skip the delayed formatting step. I suspect (but have run no benchmarks) that it's faster than delayed formatting, since you skip the whole "parse the string and look for %'s" step. In my case, most of my logging calls actually get logged, anyway.

...

* Internationalization of strings Of dubious value, since the strings have to be externalized. MAYBE useful but would need something complicated where the original form is a code construct, but the i18n library can get the source code and use it as a lookup key to find the translated form. Seems pretty messy.

I don't think this is of dubious value. It might require some tooling support.

...

* SQL queries with sanitization of expressions (to avoid "Bobby Tables" injection) You shouldn't be sanitizing your expressions, you should be parameterizing your query. That doesn't mean this is of *no* value, but it does make it look a lot less like a string literal.

Agreed.

...

* Object literals - something like D"10.2" instead of the somewhat longer D("10.2"), assuming for the latter `from decimal import Decimal as D` (this is a pretty weak argument on the face of it, but I can see the possibility of being more restrictive in the use of monkey patching namespaces with such literals to avoid unnecessary function call overhead...)

Of value only if it gets an extremely compact syntax. Other use-cases could be better served by a syntax like html(i".....") but this one only works if it's that kind of notation.

...

* Code (or AST) construction, following the quasiquote idea from Lisp Not sure how this would work, but given that I don't use Lisp enough to be familiar with the use-cases, I won't argue this one.

...

* Latex and other minilanguages (is Latex properly a minilanguage? ;) ) Explain?

I'd love to have some sort of generic interpolation construct, but the use-cases all seem really weak...

Ditto. I do plan to turn my attention to it, however. Eric

My bad, I had missed nesting template literals support in the PEP, it makes it interesting then!

Am Mo., 14. Juni 2021 um 11:12 Uhr schrieb J. Pic :

On Thu, Jun 10, 2021 at 8:34 AM Thomas Güttler wrote:

...

This solution has two drawbacks:

1. It is too verbose. Typing "conditional_escape(...)" again and again is cumbersome.

from django import conditional_espace as esc

f''' <h1>Hi {esc(name)}</h1> Your messages: {esc(messages)} '''

...

1. If a conditional_escape() gets forgotten Cross-site scripting attacks could be possible, since malicious users could inject HTML.

This is specific to Django and other frameworks out there which accept

anything as user input by default, that's an anti-pattern which OWASP recommends against because obviously it opens a wide range of attack vectors, absolutely no security audit would ever validate the default validation of a CharField or a TextField.

You are right. Validating user input is an important topic. But it is a different topic, which should be discussed at a different time.

Another problem I see with this proposal is how do you actually use safe HTML in variables?

msgs = [f'<li>{msg}</li>' for msg in messages] f''' <h1>Hi {name}</h1> Your messages: <ul>{msgs}</ul> '''

Will output:

<h1>Hi Your name</h1> Your messages: <ul><li>Your message</li></ul>

Instead of what we would want in this situation:

<h1>Hi Your name</h1> Your messages: <ul><li>Your message</li></ul>

Otherwise good idea, it's an issue we have, even though the first immediate fix needed is Django's default input validation which is just openbar.

Thank you for your feedback James. The "magic" is done in conditional_escape(): https://github.com/django/django/blob/824981b2dc61a76a59d0e470bed6e61626a44c... I updated the PEP so that it contains a hyperlink to the github repo. Regards, Thomas