Custom-Strings: Combine f-strings and condtional_escape()
Hi,
I have a crazy idea how to extend Python for web development.
What do you think? What are the next steps?
Original: https://github.com/guettli/python-custom-strings
Python Custom Strings
Python Custom Strings want to combine two great things:
- Python's f-strings https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals - Django's conditional_escape() https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.conditional_escape
Python Custom Strings is a proposal to enhance Python, to give developers new ways to create escaped strings.
The Python Custom Strings proposal is not related to Django or HTML. This text uses HTML and Django just as an example. https://github.com/guettli/python-custom-strings/blob/main/README.md#motivation Motivation
I could create HTML in Python with the help of Django's format_html() https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.format_html like this:
html = format_html(''' <h1>Hi {username}</h1> Your messages: {messages}''', username=username, messages=messages )
With "username" being a simple string that gets quoted. For example Mary & Bob will get Mary & Bob.
With "messages" being a previously escaped string. For example <ul><li>line1</li><li>line2</li></ul>. It does not get escaped, since it is already escaped.
This "magic" detection whether escaping should be done or not gets handled by conditional_escape() https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.html.conditional_escape . https://github.com/guettli/python-custom-strings/blob/main/README.md#goal Goal
I would like to shorten the above code to something similar to this:
html = h''' <h1>Hi {username}</h1> Your messages: {messages}'''
The developer should not need to type these line:
username=username, messages=messages
These lines are meaningless and distracting. Avoiding them reduces the cognitive load, especially for the reader of the code. https://github.com/guettli/python-custom-strings/blob/main/README.md#customization Customization
To make this customizable, the user needs to define the methods which should get used.
In the above example the h prefix of this snippet:
html = h''' <h1>Hi {username}</h1> Your messages: {messages}'''
Should be executed like
html = mark_safe(''' <h1>Hi {username}</h1> Your messages: {messages}''', username=conditional_escape(username), messages=conditional_escape(message))
Python does not know about these methods mark_safe() and conditional_escape().
So we need a way to define that. https://github.com/guettli/python-custom-strings/blob/main/README.md#defining-custom-stringsDefining custom strings
Somehow, there needs to be a definition of the custom strings. Up to now I open concerning "how to define custom strings?".
The syntax could look like this:
register_custom_string('h', pre_return, pre_insert)
With pre_return() and pre_insert() being methods which get executed to create the custom string.
In the above example you would use this:
register_custom_string('h', mark_safe, conditional_escape)
This is just a first idea. I guess there are better ways to define both methods. Please speak up if you have a better idea.
The letter h is just an example. The developer should be able to choose his prefered character. Replacing the already used letteres like r, b, u is not possible and results in an exception.
TODO: What should happen if register_custom_string() gets called twice for the same character? https://github.com/guettli/python-custom-strings/blob/main/README.md#what-is-inside-curly-bracesWhat is inside curly braces?
In the above example, we just use a variable name. But having more is better.
Custom strings are meant for developers, so arbitrary code execution is fine.
Arbitrary code in curly braces should be allowed:
html = h'''...{my_object.my_method(some_arg)}...'''
https://github.com/guettli/python-custom-strings/blob/main/README.md#why-not-just-this-hackWhy not just this hack?
Instead of h'...' you could use this hack to get the desired implementation:
def h(html): """ Django's format_html() on steroids """ def replacer(match): call_frame = sys._getframe(3) return conditional_escape( eval(match.group(1), call_frame.f_globals, call_frame.f_locals)) return mark_safe(re.sub(r'{(.*?)}', replacer, html))
The above implementation has one big drawback:
IDEs and linters don't know that variables get used inside the custom string.
This means IDEs and linters think variables (or imports) don't get used and act accordingly.
Nevertheless it makes no sense to type myvar=mvar again and again, just to make IDEs/linters happy. https://github.com/guettli/python-custom-strings/blob/main/README.md#concerns Concerns
"Custom strings will downgrade Python to the level of PHP".
I think Separation of Concerns makes sense. But sometimes Locality of Behaviour https://htmx.org/essays/locality-of-behaviour/ makes more sense.
It really depends on the context. Sometimes it makes sense to have external templates, sometimes it makes senes to have inline templates.
Don't judge this proposal just because your context never requires external templates. https://github.com/guettli/python-custom-strings/blob/main/README.md#background Background
If you use the html-fragments-over-the-wire approach to web development (for example with htmx https://htmx.org/), then you create many small methods returning small html snippets.
If you have small methods, then keeping the HTML inside one file together with your Python logic simplifies the development process. (See Locality of Behaviour (LoB) https://htmx.org/essays/locality-of-behaviour/)
I guess a lot of people won't like this. Nevertheless some people like to mix Python and HTML. Those people who don't like this, still can create HTML in the way they like it. https://github.com/guettli/python-custom-strings/blob/main/README.md#pep PEP?
Do you think this could make it to a PEP? Please let me know: Just create an issue here at Github.
On Fri, Jun 4, 2021 at 10:02 PM Thomas Güttler info@thomas-guettler.de wrote:
Hi,
I have a crazy idea how to extend Python for web development.
What do you think? What are the next steps?
[chomp details]
Something like this - but more generic - has been proposed already:
https://www.python.org/dev/peps/pep-0501/
There've been a few variants on the idea, but I think PEP 501 is the best summary.
If you want to press forward with this, I think that would be the best starting point.
ChrisA
Interpolation templates were recently brought up here ( https://mail.python.org/archives/list/python-ideas@python.org/thread/5AW73IC...), and Guido mentioned that in his opinion the SC would be unlikely to reconsider PEP 501 in its current state. I trust that he has a much better insight into this sort of thing than the rest of us, so he's likely right, but if I'm honest I wasn't able to find any of the actual reasons why PEP 501 was deferred in the Rationale or Discussion sections of the PEP.
If anyone has any insight into the actual reasons why PEP 501 might be considered unworkable in its current state, I'd be really curious to find out what they are. It's hard to argue in favor of a feature if you don't know what the actual arguments against it are.
The rationale for it at least seems quite solid to me, as it provides powerful templating and injection-safety functionality. It's not a purely hypothetical feature either, as it has been quite successful in other languages like C#.
If the success of f-strings over the last five years is anything to go by, people also seem to overwhelmingly prefer this style of string formatting compared to `str.format` or `%` formatting. I literally can't think of a single modern Python codebase I've seen lately where f-strings weren't the main method of string-formatting employed.
Is it something in the implementation then? Or something else?
On Fri, Jun 4, 2021 at 1:37 PM Chris Angelico rosuav@gmail.com wrote:
On Fri, Jun 4, 2021 at 10:02 PM Thomas Güttler info@thomas-guettler.de wrote:
Hi,
I have a crazy idea how to extend Python for web development.
What do you think? What are the next steps?
[chomp details]
Something like this - but more generic - has been proposed already:
https://www.python.org/dev/peps/pep-0501/
There've been a few variants on the idea, but I think PEP 501 is the best summary.
If you want to press forward with this, I think that would be the best starting point.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WZHFKM... Code of Conduct: http://python.org/psf/codeofconduct/
PEP 501 is unlikely to be accepted *as is*. But it’s still a good starting point.
Personally I would look for inspiration towards JavaScript template literals ( https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l...), combined with f-string-like interpolation.
On Fri, Jun 4, 2021 at 07:24 Matt del Valle matthewgdv@gmail.com wrote:
Interpolation templates were recently brought up here ( https://mail.python.org/archives/list/python-ideas@python.org/thread/5AW73IC...), and Guido mentioned that in his opinion the SC would be unlikely to reconsider PEP 501 in its current state. I trust that he has a much better insight into this sort of thing than the rest of us, so he's likely right, but if I'm honest I wasn't able to find any of the actual reasons why PEP 501 was deferred in the Rationale or Discussion sections of the PEP.
If anyone has any insight into the actual reasons why PEP 501 might be considered unworkable in its current state, I'd be really curious to find out what they are. It's hard to argue in favor of a feature if you don't know what the actual arguments against it are.
The rationale for it at least seems quite solid to me, as it provides powerful templating and injection-safety functionality. It's not a purely hypothetical feature either, as it has been quite successful in other languages like C#.
If the success of f-strings over the last five years is anything to go by, people also seem to overwhelmingly prefer this style of string formatting compared to `str.format` or `%` formatting. I literally can't think of a single modern Python codebase I've seen lately where f-strings weren't the main method of string-formatting employed.
Is it something in the implementation then? Or something else?
On Fri, Jun 4, 2021 at 1:37 PM Chris Angelico rosuav@gmail.com wrote:
On Fri, Jun 4, 2021 at 10:02 PM Thomas Güttler info@thomas-guettler.de wrote:
Hi,
I have a crazy idea how to extend Python for web development.
What do you think? What are the next steps?
[chomp details]
Something like this - but more generic - has been proposed already:
https://www.python.org/dev/peps/pep-0501/
There've been a few variants on the idea, but I think PEP 501 is the best summary.
If you want to press forward with this, I think that would be the best starting point.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WZHFKM... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PWZALQ... Code of Conduct: http://python.org/psf/codeofconduct/
Am Fr., 4. Juni 2021 um 19:20 Uhr schrieb Guido van Rossum <guido@python.org
:
PEP 501 is unlikely to be accepted *as is*. But it’s still a good starting point.
OK, before starting a new project, it maybe makes sense to finish the first.
Why not reject PEP-501 officially? Maybe with a reason. Then we know what's wrong with it.
After that is done, we can think about a new start.
What do you think?
Personally I would look for inspiration towards JavaScript template literals (
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l...), combined with f-string-like interpolation.
I personally really would like to avoid the dollar sign which gets used in JS template literals.
Regards, Thomas Güttler
On Fri, Jun 4, 2021 at 3:08 PM Thomas Güttler info@thomas-guettler.de wrote:
Am Fr., 4. Juni 2021 um 19:20 Uhr schrieb Guido van Rossum < guido@python.org>:
PEP 501 is unlikely to be accepted *as is*. But it’s still a good starting point.
OK, before starting a new project, it maybe makes sense to finish the first.
Why not reject PEP-501 officially? Maybe with a reason. Then we know what's wrong with it.
You'd have to ask its author. Most likely he'll just withdraw the PEP completely without telling you anything more.
In short, I don't think you're going to get much out of trying to get anyone to say anything "official" about PEP 501.
If you really want to know more, you could probably dig up the discussions around that PEP and learn from that.
After that is done, we can think about a new start.
There's no need to wait for that.
What do you think?
Personally I would look for inspiration towards JavaScript template literals (
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l...), combined with f-string-like interpolation.
I personally really would like to avoid the dollar sign which gets used in JS template literals.
Well of course. The Python equivalent would use the exact same interpolation syntax as f-strings (and before it, str.format()).
Tagged templates in JavaScript look interesting[1]
foo`...${bar} ...`
Where `foo` is an arbitrary method provided by the user.
But somehow I think general purpose string interpolation (PEP501) is enough.
At least my use-case: Get something like this format_html() but with the power of f-strings, PEP501 would be enough.
What do you think?
Regards, Thomas
[1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l...
Am Fr., 4. Juni 2021 um 23:39 Uhr schrieb Guido van Rossum <guido@python.org
:
On Fri, Jun 4, 2021 at 3:08 PM Thomas Güttler info@thomas-guettler.de wrote:
Am Fr., 4. Juni 2021 um 19:20 Uhr schrieb Guido van Rossum < guido@python.org>:
PEP 501 is unlikely to be accepted *as is*. But it’s still a good starting point.
OK, before starting a new project, it maybe makes sense to finish the first.
Why not reject PEP-501 officially? Maybe with a reason. Then we know what's wrong with it.
You'd have to ask its author. Most likely he'll just withdraw the PEP completely without telling you anything more.
In short, I don't think you're going to get much out of trying to get anyone to say anything "official" about PEP 501.
If you really want to know more, you could probably dig up the discussions around that PEP and learn from that.
After that is done, we can think about a new start.
There's no need to wait for that.
What do you think?
Personally I would look for inspiration towards JavaScript template literals (
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_l...), combined with f-string-like interpolation.
I personally really would like to avoid the dollar sign which gets used in JS template literals.
Well of course. The Python equivalent would use the exact same interpolation syntax as f-strings (and before it, str.format()).
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/
Thank you for this hint. PEP-501 looks very good.
But how to get this dream come true?
Regards, Thomas
Am Fr., 4. Juni 2021 um 14:39 Uhr schrieb Chris Angelico rosuav@gmail.com:
On Fri, Jun 4, 2021 at 10:02 PM Thomas Güttler info@thomas-guettler.de wrote:
Hi,
I have a crazy idea how to extend Python for web development.
What do you think? What are the next steps?
[chomp details]
Something like this - but more generic - has been proposed already:
https://www.python.org/dev/peps/pep-0501/
There've been a few variants on the idea, but I think PEP 501 is the best summary.
If you want to press forward with this, I think that would be the best starting point.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WZHFKM... Code of Conduct: http://python.org/psf/codeofconduct/
I think one fundamental problem is that you can’t detect programmatically if a string needs to be escaped or not.
For instance, the sequence & might be an already escaped & or it might be the text to tell you how to escape an ampersand and needs to be converted to &amp;
Promoting sloppy coding by handling the most common cases is the way to it easier to create security vulnerabilities.
Maybe if escaped strings had a different type than unescaped strings, but then you need a bunch of different types for different rules for escaping.
Am Fr., 4. Juni 2021 um 15:42 Uhr schrieb Richard Damon < Richard@damon-family.org>:
I think one fundamental problem is that you can’t detect programmatically if a string needs to be escaped or not.
For instance, the sequence & might be an already escaped & or it might be the text to tell you how to escape an ampersand and needs to be converted to &amp;
That's a problem which Python can't solve. This is up to the consumer of InterpolationTemplate.
In the django world, it is solved via conditional_escape(): https://github.com/django/django/blob/3.2.4/django/utils/html.py#L92
Regards, Thomas
Promoting sloppy coding by handling the most common cases is the way to it easier to create security vulnerabilities.
Maybe if escaped strings had a different type than unescaped strings, but then you need a bunch of different types for different rules for escaping.
participants (5)
-
Chris Angelico -
Guido van Rossum -
Matt del Valle -
Richard Damon -
Thomas Güttler