String interpolation: environment variables, command substitution
One of the remaining questions for the string interpolation subject is whether to allow for easy access to environment variables and output-capture of external processes (aka command-substitution) as bash does. While incredibly useful in use-cases such as shell-script replacements, the functionality is perceived to be, if not dangerous. More so than arbitrary expressions? Given that we are talking about string literals and not input, I'm not sure, so am looking for feedback. The idea is not unheard of in Python, there was a module that captured process output called commands in the old days, which was superseded at some point by subprocess.check_output() I believe. Here is some example syntax modeled on bash, though placed inside .format braces. Note both start with $ as the signal:: >>> x'Home folder: {$HOME}' # environment 'Home folder: /home/nobody' >>> x'Files: {$(/bin/ls .)}' # capture output 'foo foo1 foo2' For safety, command substitution should return output in a way analogous to the modern equivalent:: subprocess.check_output(['/bin/ls', '.'], shell=False).decode(encoding) -Mike
On 26 August 2015 at 19:20, Mike Miller
One of the remaining questions for the string interpolation subject is whether to allow for easy access to environment variables and output-capture of external processes (aka command-substitution) as bash does.
While incredibly useful in use-cases such as shell-script replacements, the functionality is perceived to be, if not dangerous. More so than arbitrary expressions? Given that we are talking about string literals and not input, I'm not sure, so am looking for feedback.
The idea is not unheard of in Python, there was a module that captured process output called commands in the old days, which was superseded at some point by subprocess.check_output() I believe.
This seems like a really good idea for an external module but I see no reason why it is important enough to deserve built in syntax in the core language. Note that cross platform issues are going to be a major issue: * $HOME is "the wrong way to do it" - Windows has no HOME env variable. The right way is os.path.expanduser("~") * $(/bin/ls) is the wrong way - os.listdir(".") is the right way (windows has no ls command) Making it easy to write platform specific code when it's not needed is as much of an antipattern as making it easy to write insecure code IMO. Paul
Hi, The use case is for shell-script replacements, which can be but are often not typically cross-platform. Let's try different examples: >>> x'version: {$(/usr/bin/xdpyinfo -version)}' # capture 'version: xdpyinfo 1.3.1' >>> x'display: {$DISPLAY}' # env 'display: :0.0' -Mike On 08/26/2015 11:58 AM, Paul Moore wrote:
Note that cross platform issues are going to be a major issue
On 26 August 2015 at 20:16, Mike Miller
The use case is for shell-script replacements, which can be but are often not typically cross-platform. Let's try different examples:
>>> x'version: {$(/usr/bin/xdpyinfo -version)}' # capture 'version: xdpyinfo 1.3.1'
>>> x'display: {$DISPLAY}' # env 'display: :0.0'
This is not an important enough use case to warrant language support, IMO. If you want something like this, there are a lot of tools already available on PyPI: https://pypi.python.org/pypi/invoke/0.10.1 if you want to run sets of command lines, grouped together as "tasks" https://pypi.python.org/pypi/sarge/0.1.4 if you're looking for a shell-like syntax (with cross-platform support for constructs like &&, || etc) http://plumbum.readthedocs.org/en/latest/ if you want access to shell commands from within Python code and probably a host of others. Honestly, this is starting to feel like Perl. Sorry, but I don't like this proposal at all. Paul
Understood, Btw, subprocess.check_output() is already in the standard library. This idea was about further simplifying it even further than that, making Python a contender for shell-scripting. -Mike On 08/26/2015 12:54 PM, Paul Moore wrote:
On Wed, Aug 26, 2015 at 1:02 PM, Mike Miller
Btw, subprocess.check_output() is already in the standard library. This idea was about further simplifying it even further than that, making Python a contender for shell-scripting.
No. This is an outright bad idea. Before you know it people are calling out to bash for tasks like removing a file or finding out the current directory, yet claiming to know Python on their resume. -- --Guido van Rossum (python.org/~guido)
On Aug 26, 2015, at 12:16, Mike Miller
Hi,
The use case is for shell-script replacements, which can be but are often not typically cross-platform. Let's try different examples:
x'version: {$(/usr/bin/xdpyinfo -version)}' # capture 'version: xdpyinfo 1.3.1'
People who expect this to work will likely expect to be able to break the results into separates "arguments", with the bash quoting rules. People already ask on StackOverflow why things like this don't work: subprocess.call('files=$(ls)', shell=True) subprocess.call('cp $files %s' % dest, shell=True) No matter how many places they try to put the quotes, or braces, or where they add extern, still nothing gets copied. With your change, they can fix it like this: files = x'{$(ls)}' subprocess.call(x'cp {files} {dest}') ... and now it seems to work, except that it's actually not copying any files with spaces in the name. They may not even notice, which is bad. But if they do, no matter where you add the quotes or braces, there's no way to fix it. A Python string value is not a bash array value that stringifies itself in different ways depending on the quoting context. The right answer is still to actually use the shell by cramming it into one line, to get the output as a list of lines and insert each line as a quoted argument, or, best of all, to just use listdir and shutil in the first place instead of trying to translate from Bash to Python one word at a time, which works about as well as one-abstract word one-abstract instance that manner Japanese from English to translating.
On 08/26/2015 02:20 PM, Mike Miller wrote:
One of the remaining questions for the string interpolation subject is whether to allow for easy access to environment variables and output-capture of external processes (aka command-substitution) as bash does.
While incredibly useful in use-cases such as shell-script replacements, the functionality is perceived to be, if not dangerous. More so than arbitrary expressions? Given that we are talking about string literals and not input, I'm not sure, so am looking for feedback.
The idea is not unheard of in Python, there was a module that captured process output called commands in the old days, which was superseded at some point by subprocess.check_output() I believe.
Here is some example syntax modeled on bash, though placed inside .format braces. Note both start with $ as the signal::
>>> x'Home folder: {$HOME}' # environment 'Home folder: /home/nobody'
>>> x'Files: {$(/bin/ls .)}' # capture output 'foo foo1 foo2'
-1000 for any language syntax that allows access to environment variables or shell output. That said, with PEP-498 you can do:
import os f'HOME={os.environ["HOME"]}' 'HOME=/home/eric'
Which is about as easy as I'd like to make this. Eric.
Hold on, it took f'' strings a while to grow on you, give it a few minutes. ;) I'd like Python to be competitive with other (shell) scripting languages, and appeals to purity stand in the way of that. Sometimes practical is just darn useful. We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages. -Mike On 08/26/2015 12:21 PM, Eric V. Smith wrote:
Which is about as easy as I'd like to make this.
On 08/26/2015 03:53 PM, Mike Miller wrote:
Hold on, it took f'' strings a while to grow on you, give it a few minutes. ;)
I'd like Python to be competitive with other (shell) scripting languages, and appeals to purity stand in the way of that. Sometimes practical is just darn useful.
We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages.
Python is never going to be bash.
env=os.environ.get f'HOME={env("HOME")}' 'HOME=/home/eric'
That's good enough. Eric.
On Aug 26, 2015 3:01 PM, "Eric V. Smith"
On 08/26/2015 03:53 PM, Mike Miller wrote:
Hold on, it took f'' strings a while to grow on you, give it a few minutes. ;)
I'd like Python to be competitive with other (shell) scripting languages, and appeals to purity stand in the way of that. Sometimes practical is just darn useful.
We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages.
Python is never going to be bash.
env=os.environ.get f'HOME={env("HOME")}' 'HOME=/home/eric'
How is this wrong/another way to do the wrong thing? * $HOME may contain quotes, single quotes, newlines (which subprocess.call interprets as separate commands), semicolons
That's good enough.
No, tuples (as exec specifies) are good enough. This is the wrong kind of lazy.
Eric.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Aug 26, 2015 at 12:53:47PM -0700, Mike Miller wrote:
Hold on, it took f'' strings a while to grow on you, give it a few minutes. ;)
I'd like Python to be competitive with other (shell) scripting languages,
Why? If you want a shell language, there are many existing shell languages that are far more compact/terse/unreadable/convenient/"easy to use (wrongly)" than Python will ever be, even with f-strings. Even Perl is not a shell language. Python already makes a good scripting language. It just requires more typing and more thought, which encourages writing correct code rather than "easy to type" code which may not be correct. Can we get away from the harmful meme that being "easier" is necessarily always better? That way of thinking leads to PHP.
and appeals to purity stand in the way of that. Sometimes practical is just darn useful.
[snark] We've thrown away the rest of the Zen with these f-strings, so why not throw away that one too? *fractional-wink* I don't think there is any "practical beats purity" argument to be made here. It's not like it is hard to get access to environment variables.
We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages.
Perhaps less icing on the cake and more the straw that breaks the camel's back? -- Steve
On 2015-08-26 12:53, Mike Miller wrote:
We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages.
There's no comparison between "arbitrary expressions" and "new syntax for shell shortcuts". "Arbitrary expressions" are arbitrary Python expressions, so that just means being able to do what you can already do in Python, with Python syntax, in Python strings. This includes being able to access environment variables, since you can already do that with a Python expression. These shell shortcuts are just a way to open a back door that would bring all of shell syntax into Python, and add new complications to Python's own syntax as well. I see quick-and-dirty shell scripting as pretty small potatoes in the scheme of things Python can be used for; it's not worth changing the language in any significant way to a accommodate that. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Wed, Aug 26, 2015 at 6:55 PM, Brendan Barnwell
On 2015-08-26 12:53, Mike Miller wrote:
We've already acquiesced to arbitrary expressions, so this is a small further step, icing on the cake, no? I believe Guido mentioned something about "half-measures" in one of his messages.
There's no comparison between "arbitrary expressions" and "new syntax for shell shortcuts". "Arbitrary expressions" are arbitrary Python expressions, so that just means being able to do what you can already do in Python, with Python syntax, in Python strings. This includes being able to access environment variables, since you can already do that with a Python expression. These shell shortcuts are just a way to open a back door that would bring all of shell syntax into Python, and add new complications to Python's own syntax as well. I see quick-and-dirty shell scripting as pretty small potatoes in the scheme of things Python can be used for; it's not worth changing the language in any significant way to a accommodate that.
I am more and more beginning to believe that Mike is just playing an elaborate prank on us, seeing how far he can go with this before people start noticing the proposal has jumped the shark. -- --Guido van Rossum (python.org/~guido)
Yes, another way of stating that I'm getting documentation on design decisions. -Mike On 08/26/2015 06:59 PM, Guido van Rossum wrote:
I am more and more beginning to believe that Mike is just playing an elaborate prank on us, seeing how far he can go with this before people start noticing the proposal has jumped the shark.
-- --Guido van Rossum (python.org/~guido http://python.org/~guido)
On 2015-08-26 11:20, Mike Miller wrote:
One of the remaining questions for the string interpolation subject is whether to allow for easy access to environment variables and output-capture of external processes (aka command-substitution) as bash does.
You can already do this with the existing proposals by interpolating an expression whose value is an environment variable (e.g., 'My home is {os.environ["HOME"]}') or whatever other data you want to interpolate. There's no reason to add special syntax for this. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
True, though less readable I think. If we're going to go as far as arbitrary expressions, let's discuss making very common scripting tasks easier. -Mike On 08/26/2015 12:27 PM, Brendan Barnwell wrote:
You can already do this with the existing proposals by interpolating an expression whose value is an environment variable (e.g., 'My home is {os.environ["HOME"]}') or whatever other data you want to interpolate. There's no reason to add special syntax for this.
On 27 August 2015 at 05:44, Mike Miller
True, though less readable I think. If we're going to go as far as arbitrary expressions, let's discuss making very common scripting tasks easier.
There's already a way to make common scripting tasks easy: use the preferred shell for your preferred platform. That said, if anyone really wants to advance the state of the art in Python's "embedded shell scripting" capabilities, then I'd highly recommend exploring Julia's capabilities in that area and seeing how to produce a comparable system using runtime processing of strings in Python: http://julia.readthedocs.org/en/latest/manual/running-external-programs/ Combining a system like that with f-strings (and/or i-strings) would then allow ready interpolation of Python variables into command lines using either *nix or Windows appropriate syntax. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (9)
-
Andrew Barnert -
Brendan Barnwell -
Eric V. Smith -
Guido van Rossum -
Mike Miller -
Nick Coghlan -
Paul Moore -
Steven D'Aprano -
Wes Turner