Re: [Python-ideas] [Python-Dev] PEP-498: Literal String Formatting
On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner@gmail.com> wrote:
... I'm now -1000 on this.
~"Make it hard to do wrong; or easy to do correctly"
... Here are these, (which should also not be used for porting shell scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions
So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing. * #1 most prevalent security vulnerability: *1**CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')* * ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy * #2 most prevalent security vulnerability: *2**CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')* * Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0) - [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well)
On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
[off list]
On 12 August 2015 at 01:28, Wes Turner <wes.turner@gmail.com> wrote:
On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner@gmail.com> wrote:
On Aug 11, 2015 10:10 AM, "Alexander Walters" <
wrote:
This may seam like a simplistic solution to i18n, but why not just
add a
method to string objects (assuming we implement f-strings) that just returns the original, unprocessed string. If the string was not an f-string, it just returns self. The gettext module can be modified, I think
tritium-list@sdamon.com> trivially,
to use the method instead of the string directly.
Is this a horrible idea?
- [ ] review all string interpolation (for "injection") * [ ] review every '%' * [ ] review every ".format()" * [ ] review every f-string (AND LOCALS AND GLOBALS) * every os.system, os.exec*, subprocess.Popen * every unclosed tag * every unescaped control character
This would create work we don't need.
Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ refs a global list of reserved words.
Wes, we're not mind readers - I know you're trying to be concise to save people time when reading, but these bullet-point-only posts are *harder* to read than if you wrote out a full explanation of what you meant. With this cryptic form, we have to try to guess the missing pieces, which is slower and less certain than having them already written out in the post.
~"This is another way to make it easier to do the wrong thing; where a better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less likely to increase the ocurrence of CWE TOP 25 #1 and #2"
printf is often dangerous and wrng because things aren't escaped (or scope is not controlled, or things are mutable)
~"Make it hard to do; or easy to do the right way"
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Wes: Your objection is noted. Thanks. Eric. On 08/11/2015 02:22 PM, Wes Turner wrote:
On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner@gmail.com <mailto:wes.turner@gmail.com>> wrote:
... I'm now -1000 on this.
~"Make it hard to do wrong; or easy to do correctly"
... Here are these, (which should also not be used for porting shell scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions
So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing.
* #1 most prevalent security vulnerability: *1* *CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')*
* ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy
* #2 most prevalent security vulnerability: *2* *CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')*
* Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0)
- [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well)
On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner@gmail.com <mailto:wes.turner@gmail.com>> wrote:
On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
[off list]
On 12 August 2015 at 01:28, Wes Turner <wes.turner@gmail.com <mailto:wes.turner@gmail.com>> wrote: > > On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner@gmail.com <mailto:wes.turner@gmail.com>> wrote: >> >> >> On Aug 11, 2015 10:10 AM, "Alexander Walters" <tritium-list@sdamon.com <mailto:tritium-list@sdamon.com>> >> wrote: >> > >> > This may seam like a simplistic solution to i18n, but why not just add a >> > method to string objects (assuming we implement f-strings) that just returns >> > the original, unprocessed string. If the string was not an f-string, it >> > just returns self. The gettext module can be modified, I think trivially, >> > to use the method instead of the string directly. >> > >> > Is this a horrible idea? > > - [ ] review all string interpolation (for "injection") > * [ ] review every '%' > * [ ] review every ".format()" > * [ ] review every f-string (AND LOCALS AND GLOBALS) > * every os.system, os.exec*, subprocess.Popen > * every unclosed tag > * every unescaped control character > > This would create work we don't need. > > Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ refs > a global list of reserved words.
Wes, we're not mind readers - I know you're trying to be concise to save people time when reading, but these bullet-point-only posts are *harder* to read than if you wrote out a full explanation of what you meant. With this cryptic form, we have to try to guess the missing pieces, which is slower and less certain than having them already written out in the post.
~"This is another way to make it easier to do the wrong thing; where a better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less likely to increase the ocurrence of CWE TOP 25 #1 and #2"
printf is often dangerous and wrng because things aren't escaped (or scope is not controlled, or things are mutable)
~"Make it hard to do; or easy to do the right way"
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Isn't it already like this? It's no harder than: Popen('%s a.c' % cc, shell=True) Heck, I used to do that when I started programming (I hadn't yet learned about injection stuff). If someone is uneducated about injection, they *will do it anyway*. The introduction of format strings (f-strings sounds like a certain word to me...) wouldn't make it any easier, really. On August 11, 2015 1:22:06 PM CDT, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner@gmail.com> wrote:
... I'm now -1000 on this.
~"Make it hard to do wrong; or easy to do correctly"
... Here are these, (which should also not be used for porting shell scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions
So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing.
* #1 most prevalent security vulnerability: *1**CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')*
* ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy
* #2 most prevalent security vulnerability: *2**CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')*
* Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0)
- [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well)
On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner@gmail.com>
wrote:
On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
[off list]
On 12 August 2015 at 01:28, Wes Turner <wes.turner@gmail.com>
On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner@gmail.com>
On Aug 11, 2015 10:10 AM, "Alexander Walters" <
wrote: tritium-list@sdamon.com>
wrote: > > This may seam like a simplistic solution to i18n, but why not just add a > method to string objects (assuming we implement f-strings)
just returns
> the original, unprocessed string. If the string was not an f-string, it > just returns self. The gettext module can be modified, I
wrote: that think
trivially,
> to use the method instead of the string directly. > > Is this a horrible idea?
- [ ] review all string interpolation (for "injection") * [ ] review every '%' * [ ] review every ".format()" * [ ] review every f-string (AND LOCALS AND GLOBALS) * every os.system, os.exec*, subprocess.Popen * every unclosed tag * every unescaped control character
This would create work we don't need.
Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ refs a global list of reserved words.
Wes, we're not mind readers - I know you're trying to be concise to save people time when reading, but these bullet-point-only posts are *harder* to read than if you wrote out a full explanation of what you meant. With this cryptic form, we have to try to guess the missing pieces, which is slower and less certain than having them already written out in the post.
~"This is another way to make it easier to do the wrong thing; where a better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less likely to increase the ocurrence of CWE TOP 25 #1 and #2"
printf is often dangerous and wrng because things aren't escaped (or scope is not controlled, or things are mutable)
~"Make it hard to do; or easy to do the right way"
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
------------------------------------------------------------------------
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
On Tue, Aug 11, 2015 at 1:37 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:
Isn't it already like this? It's no harder than:
Popen('%s a.c' % cc, shell=True)
Heck, I used to do that when I started programming (I hadn't yet learned about injection stuff).
If someone is uneducated about injection, they *will do it anyway*. The introduction of format strings (f-strings sounds like a certain word to me...) wouldn't make it any easier, really.
Well, exactly. So I/we must grep for shell=True, %, .format(, .format_globals(**kwargs), and f" or f' and update static analysis tools (to essentially re-AST string.Template with merge(globals, locals, kwargs))
On August 11, 2015 1:22:06 PM CDT, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner@gmail.com> wrote:
... I'm now -1000 on this.
~"Make it hard to do wrong; or easy to do correctly"
... Here are these, (which should also not be used for porting shell scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions
So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing.
* #1 most prevalent security vulnerability: *1**CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')*
* ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy
* #2 most prevalent security vulnerability: *2**CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')*
* Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0)
- [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well)
On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
[off list]
On 12 August 2015 at 01:28, Wes Turner <wes.turner@gmail.com> wrote:
On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner@gmail.com> wrote: > > > On Aug 11, 2015 10:10 AM, "Alexander Walters" <
> wrote: > > > > This may seam like a simplistic solution to i18n, but why not just add a > > method to string objects (assuming we implement f-strings) that just returns > > the original, unprocessed string. If the string was not an f-string, it > > just returns self. The gettext module can be modified, I think
tritium-list@sdamon.com> trivially,
> > to use the method instead of the string directly. > > > > Is this a horrible idea?
- [ ] review all string interpolation (for "injection") * [ ] review every '%' * [ ] review every ".format()" * [ ] review every f-string (AND LOCALS AND GLOBALS) * every os.system, os.exec*, subprocess.Popen * every unclosed tag * every unescaped control character
This would create work we don't need.
Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ refs a global list of reserved words.
Wes, we're not mind readers - I know you're trying to be concise to save people time when reading, but these bullet-point-only posts are *harder* to read than if you wrote out a full explanation of what you meant. With this cryptic form, we have to try to guess the missing pieces, which is slower and less certain than having them already written out in the post.
~"This is another way to make it easier to do the wrong thing; where a better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less likely to increase the ocurrence of CWE TOP 25 #1 and #2"
printf is often dangerous and wrng because things aren't escaped (or scope is not controlled, or things are mutable)
~"Make it hard to do; or easy to do the right way"
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
------------------------------
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
I would rather think of this as an opportunity to help avoid injection vectors. if there was a separate.. . interpolation provider .. then something like os.system('dosomething {a} {b} {c}'.format(...)) could be written as ( !cmd here being a special type of f-string that does command line escaping, borrowing syntax from another thread a few days ago..) os.sytem(!cmd'dosomething {a} {b} {c}') This is both shorter and more resilient to injections. Essentially it feels like you annotate a string as "this will be executed on the command line" and the interpolation adapts. this would make doing the right thing the same as doing the easy thing and this would be good overall, no? I don't know about you, but i dont know by heart how to escape arbitrary user input and deal with all of the corner cases. yes, you can do this more safely with Popen.. but that is quite a bit more effort. also often times there is no such alternative or it is very unweildy (sql land this happens more often)
On Tue, Aug 11, 2015 at 2:25 PM, Joonas Liik <liik.joonas@gmail.com> wrote:
I would rather think of this as an opportunity to help avoid injection vectors.
you get an "F" grade/letter/mark every time you build an f-string without defining what the user-supplied input and destination outputs could/would be.
if there was a separate.. . interpolation provider .. then something like
os.system('dosomething {a} {b} {c}'.format(...))
could be written as ( !cmd here being a special type of f-string that does command line escaping, borrowing syntax from another thread a few days ago..)
os.sytem(!cmd'dosomething {a} {b} {c}')
sarge.run('do something {0} {1} {2}', a, b, c) is currently supported (and could/should be stdlib IMHO) https://sarge.readthedocs.org/en/latest/overview.html#why-not-just-use-subpr... . * (again, sorry) this adds ~subprocess compat to sarge: https://bitbucket.org/vinay.sajip/sarge/pull-requests/1/enh-add-call-check_c... ("ENH: Add call, check_call, check_output, CalledProcessError, expect_returncode")
This is both shorter and more resilient to injections. Essentially it feels like you annotate a string as "this will be executed on the command line" and the interpolation adapts.
this would make doing the right thing the same as doing the easy thing and this would be good overall, no? I don't know about you, but i dont know by heart how to escape arbitrary user input and deal with all of the corner cases.
So, IPython/Jupyter understands _repr_html_ (_repr_*_) methods, IDK why we couldn't have e.g. _repr_shell_path_, _repr_shell_cmdarg_, _repr_sql_sqlite_reserved_keywords_. Representing things for an output format which is expressed as a string but has control characters in order to separate data and code.
yes, you can do this more safely with Popen.. but that is quite a bit more effort. also often times there is no such alternative or it is very unweildy (sql land this happens more often)
POSIX exec accepts a tuple (and does not parse ';' or '--').
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Aug 11, 2015 at 2:35 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 2:25 PM, Joonas Liik <liik.joonas@gmail.com> wrote:
I would rather think of this as an opportunity to help avoid injection vectors.
you get an "F" grade/letter/mark every time you build an f-string without defining what the user-supplied input and destination outputs could/would be.
A configuration object (passable as e.g. format(**conf)) more explicitly defines the scope (as variables that need to be - [ ] escaped - [ ] encoded - [ ] translated - [ ] concatenated - [ ] mutated or not mutated - [ ] formatted In an ordered idempotent sequence. lookup = partial[kwargs, locals, globals] merged = merge(globals, locals, kwargs) .formatg(**kwargs) .format(lookup(kwargs)) .formatl(**kwargs) uno = trans("one}") f"abc {uno}" ft"abc {uno}" eetcmf"abc {uno}"
if there was a separate.. . interpolation provider .. then something like
os.system('dosomething {a} {b} {c}'.format(...))
could be written as ( !cmd here being a special type of f-string that does command line escaping, borrowing syntax from another thread a few days ago..)
os.sytem(!cmd'dosomething {a} {b} {c}')
sarge.run('do something {0} {1} {2}', a, b, c) is currently supported (and could/should be stdlib IMHO) https://sarge.readthedocs.org/en/latest/overview.html#why-not-just-use-subpr... .
* (again, sorry) this adds ~subprocess compat to sarge: https://bitbucket.org/vinay.sajip/sarge/pull-requests/1/enh-add-call-check_c... ("ENH: Add call, check_call, check_output, CalledProcessError, expect_returncode")
This is both shorter and more resilient to injections. Essentially it feels like you annotate a string as "this will be executed on the command line" and the interpolation adapts.
this would make doing the right thing the same as doing the easy thing and this would be good overall, no? I don't know about you, but i dont know by heart how to escape arbitrary user input and deal with all of the corner cases.
So, IPython/Jupyter understands _repr_html_ (_repr_*_) methods, IDK why we couldn't have e.g. _repr_shell_path_, _repr_shell_cmdarg_, _repr_sql_sqlite_reserved_keywords_.
Representing things for an output format which is expressed as a string but has control characters in order to separate data and code.
yes, you can do this more safely with Popen.. but that is quite a bit more effort. also often times there is no such alternative or it is very unweildy (sql land this happens more often)
POSIX exec accepts a tuple (and does not parse ';' or '--').
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Wes, I don't know you, but your contributions to this thread are adding more noise than light. I am not the only one who is exasperated at many of your posts. Please stop. -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 11, 2015 at 1:22 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:52 PM, Wes Turner <wes.turner@gmail.com> wrote:
... I'm now -1000 on this.
~"Make it hard to do wrong; or easy to do correctly"
... Here are these, (which should also not be used for porting shell scripts to python): http://jinja.pocoo.org/docs/dev/templates/#expressions
So, again, I am -1000 on (both of these PEPs) because they are just another way of making it too easy to do the wrong thing.
* #1 most prevalent security vulnerability: *1**CWE-89 <http://cwe.mitre.org/data/definitions/89.html>: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')*
* ORM with parametrization, quoting, escaping and lists of reserved words * SQLAlchemy
* #2 most prevalent security vulnerability: *2**CWE-78 <http://cwe.mitre.org/data/definitions/78.html>: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')*
* Command preparation library (which builds a tuple() for exec) * Sarge, subprocess.call(shell=False=0)
- [ ] DOC: (Something like this COULD/SHOULD be in the % and str.format docs as well)
Maybe it would be helpful to think of string concatenation more in terms of compiling a template for serializable DOM(html,js,brython)/doctree(docutils,sphinx)/jinja nodes which have types (Path, CommandOption/Arg, [Tag, Attr]) and appropriate quoting, escaping, encoding, **and translation** rules according to a given output context. # because this is what could just not be: [os.system(f'echo "{cmd}") for cmd in cmds] os.system(f'echo2 '{cmd}') What is the target output format for this string concatenation, most of the time?
On Tue, Aug 11, 2015 at 12:48 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Aug 11, 2015 at 12:08 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
[off list]
On 12 August 2015 at 01:28, Wes Turner <wes.turner@gmail.com> wrote:
On Aug 11, 2015 10:19 AM, "Wes Turner" <wes.turner@gmail.com> wrote:
On Aug 11, 2015 10:10 AM, "Alexander Walters" <
wrote: > > This may seam like a simplistic solution to i18n, but why not just add a > method to string objects (assuming we implement f-strings) that just returns > the original, unprocessed string. If the string was not an f-string, it > just returns self. The gettext module can be modified, I think
tritium-list@sdamon.com> trivially,
> to use the method instead of the string directly. > > Is this a horrible idea?
- [ ] review all string interpolation (for "injection") * [ ] review every '%' * [ ] review every ".format()" * [ ] review every f-string (AND LOCALS AND GLOBALS) * every os.system, os.exec*, subprocess.Popen * every unclosed tag * every unescaped control character
This would create work we don't need.
Solution: __str_shell_ escapes, adds slashes, and quotes. __str__SQL__ refs a global list of reserved words.
Wes, we're not mind readers - I know you're trying to be concise to save people time when reading, but these bullet-point-only posts are *harder* to read than if you wrote out a full explanation of what you meant. With this cryptic form, we have to try to guess the missing pieces, which is slower and less certain than having them already written out in the post.
~"This is another way to make it easier to do the wrong thing; where a better solution (AND/OR DOCS ON ALL STRING INTERPOLATION) would be less likely to increase the ocurrence of CWE TOP 25 #1 and #2"
printf is often dangerous and wrng because things aren't escaped (or scope is not controlled, or things are mutable)
~"Make it hard to do; or easy to do the right way"
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Eric V. Smith -
Guido van Rossum -
Joonas Liik -
Ryan Gonzalez -
Wes Turner