PEP: Idiomatic API for shell scripting
Python is often used a shell scripting language due to it being more robust than traditional Unix shells for managing complex programs, but it is significantly more verbose than Bash & friends for doing common shell scripting tasks such as writing pipelines and redirection . I propose adding an idiomatic shell scripting API that uses Python operator overloading in a way that allows invoking commands in a Bash-like way. A fully working implementation (which I've been using in my personal scripts) can be found at https://github.com/tarruda/python-ush/. Here I will describe the basics of the API, more details in the github README.rst (which is also a doctest for the project). Here's how one can create commands: >>> from shell import Shell >>> sh = Shell() >>> cat = sh('cat') >>> ls = sh('ls') >>> sort = sh('sort') # or something like this: >>> cat, ls, sort = sh('cat', 'ls', 'sort') The Shell class is the entry point of the API and it is used as a factory for Command objects. For convenience we can add a builtin Shell instance which is a also module from where commands can be quickly imported from (idea taken from sh.py http://amoffat.github.io/sh/): >>> from shell.sh import cat, ls, sort We can construct more complex commands by calling it with certain arguments: >>> ls('-l', '-a', env={'LANG': 'C.UTF-8'}) And we can invoke the command by calling it without arguments: >>> ls() # or >>> ls('-l', '-a', env={'LANG': 'C.UTF-8'})() Invoking the command returns a tuple with the status code: >>> ls() (0,) The are more ways to invoke a command other than calling it without arguments. One such example is by iterating through it, which automatically takes care of piping the output back to python: >>> files = [] >>> for line in ls: ... files.append(line) Pipelines can be easily created with the `|` operator: >>> ls = ls('--hide=*.txt', env={'LANG': 'C.UTF-8'}) >>> sort = sort('--reverse') >>> ls | sort ls --hide=*.txt (env={'LANG': 'C.UTF-8'}) | sort --reverse Everything that can be done with Command objects, can also be done with Pipeline objects: >>> (ls | sort)() (0, 0) # single commands return a tuple to make the return value compatible with Pipelines >>> list(ls | sort) [u'tests', u'setup.cfg', u'pytest.ini', u'bin', u'README.rst'] We also use the Pipeline syntax for redirecting input/output. # piping to a filename string redirects output to the file >>> (ls | sort | '.stdout')() (0, 0) >>> str(cat('.stdout')) 'tests\nsetup.cfg\npytest.ini\nbin\nREADME.rst\n' # piping from a filename string redirects input from the file >>> str('setup.cfg' | cat) '[metadata]\ndescription-file = README.rst\n\n[bdist_wheel]\nuniversal=1\n' This is the basic idea. The current implementation only supports the classic subprocess API, but I can probably make it compatible with asyncio (to create async shells which spawns commands compatible with `await`). Thoughts? Is it worth writing a PEP for this? Best regards, Thiago.
Something very similar to this had been done many times in third party libraries. None of those have become hugely popular, including none quite compelling me personally to do more than try them out as toys. It's too bad, in a way, since I love bash and use it all the time when it seems easier than Python. Your library is 100% certainly not going to become part of the standard library unless it can first become popular as a third party tool. If it does become popular that way, I wouldn't put it's odds of eventual soon to the standard library as higher than 5-10%. But that doesn't really matter, widespread use his 'pip install' world be a good thing in itself. On Sat, Aug 10, 2019, 7:52 AM Thiago Padilha <tpadilha84@gmail.com> wrote:
On Sat, Aug 10, 2019 at 6:53 AM David Mertz <mertz@gnosis.cx> wrote:
Something very similar to this had been done many times in third party libraries.
Check out Xonch for example: https://xon.sh/ -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
In various different ways, also: https://pypi.org/project/bash/ https://pypi.org/project/python-pipe/ https://pypi.org/project/pypette/ https://pypi.org/project/shell/ Just a few I've seen before. Xonsh is pretty cool also. On Sat, Aug 10, 2019, 9:55 PM Christopher Barker <pythonchb@gmail.com> wrote:
I actually gave a talk along these lines at the Chicago Python (ChiPy) meetup last week: slides https://docs.google.com/presentation/d/1v5z4f-FQkS-bQYE-Xv5SvA6cyaTiqlxs2w2C... Part of the argument was about using pure standard library so a self-contained script/repo could run anywhere Python is in order to (e.g.) bootstrap other testing environments and/or work within restricted ones, just like your average shell script. A gigantic step up from there is using *anything* not in the stdlib because you may need to copy a bunch of files (venv creation), and download executable things from *the Internet* (horrific to some). On Sat, Aug 10, 2019 at 11:03 PM David Mertz <mertz@gnosis.cx> wrote:
Nick Timkovich wrote:
Nice presentation. I've adapted the examples in the "how to parent" section to illustrate the potential gains in expressiveness when using ush: Example 1: >>> import subprocess | >>> from ush.sh import echo, git | >>> result = subprocess.run( | >>> str(echo(b'asdf') | git('rev-parse', ... ["git", "rev-parse", "HEAD"], | ... 'HEAD', cwd="/home/nick/Code/toss", ... cwd="/home/nick/Code/toss", | ... env={"A":"B"})) ... env={"A": "B", **os.environ}, | ... input=b"asdf", | ... capture_output=True, | ... ) | Example 2 (with different way of passing env/cwd to children, useful for spawning multiple commands with the same env/cwd): >>> import os | >>> from ush.sh import chdir, setenv, git | >>> os.chdir("/home/nick") | >>> with chdir("/home/nick"), setenv({"A":"B"}): >>> os.execvpe("git", | ... git("rev-parse", "HEAD")() ... args=["git", "rev-parse", "HEAD"], | ... env={"A": "B", **os.environ} | ... ) | Note that chdir/setenv context managers don't pollute anything in the host python process. The temporary changes in the env/cwd are managed by the Shell instance, which is not only a factory for Command objects but also stores default options values. That makes it possible to have multiple shell instances in separate threads running commands in completely different environments (though this will be more useful once I add asyncio support). There's a few features I haven't documented yet (chdir/setenv are examples). My goal was create a full shell scripting DSL right into python, while also: - Not using the shell=True under the hoods, which ends up making the python script depend on the underlying shell. - Not relying on native code while also creating full OS-level pipelines. Some alternatives like http://amoffat.github.io/sh/ implement piping by manually transfering data between processes. ush creates OS pipes and lets it handle data transfer, which is how shells like bash do it. - Being cross platform (windows/unix supported) - Supporting python2/3 (Though python2 support is less of a priority now).
One of the reasons I implemented ush in one file without dependencies was to make it simple bootstrapping it into any project. It is as easy as adding something like this to the top of your script: import pathlib script_dir = pathlib.Path(__file__).parent ush_path = script_dir / 'ush.py' if not ush_path.exists(): print('downloading ush.py') from urllib.request import urlretrieve urlretrieve( 'https://raw.githubusercontent.com/tarruda/python-ush/master/ush.py', str(ush_path)) (script_dir / '__init__.py').touch(exist_ok=True) from ush.sh import git, grep # import whatever commands you need Though it would be much better to have a library like ush built into standard library ;)
Nick Timkovich wrote:
In my experience (though I am curious about yours!) of environments where this sort of constraint applies, it applies just as strongly to the use of a version of Python that's newer than the one the system came with. That means that adding something to the stdlib isn't a route to getting to use it particularly soon in such environments, either. Availability in environments that are conservative about new or additional software is just inherently a long-term project. As a thought experiment, imagine there were consensus today on adding some library of this flavor to the stdlib, and the work rapidly got done and got merged. It'd be released in Python 3.9, circa early 2021. That in turn would be picked up in the next Ubuntu LTS in 2022... and the next RHEL/CentOS perhaps in 2024, and its derivatives like Amazon Linux sometime after that. When would you predict it might be "just always there" in the environments you work with? In the parts of the industry I'm most familiar with, in fact, the lag from publication to practical availability is much shorter through PyPI than through the stdlib: it's common to run the system Python from an old Ubuntu release that's pushing the limit of its 5-year supported lifespan, but for better and worse to cheerfully run the latest anything that any developer at the company wants to pull in from PyPI. so a self-contained script/repo could run anywhere Python is in order to
(e.g.) bootstrap other testing environments and/or work within restricted ones
For specifically this side of the constraints, though, I think there is hope for a technical solution! Doesn't help if you really can't pull in new code, but it does make bootstrapping far simpler at a technical level. The PyOxidizer project: https://github.com/indygreg/PyOxidizer makes it possible to bundle up (a) CPython including (a') the stdlib together with (b) some arbitrary set of Python libraries of your choice, all as a single statically-linked executable file. Then you can distribute that one file to other machines, and run it there, without needing to create temporary files or download stuff from the Internet or anything like that. In particular, you can have the executable behave just like `python` -- it runs scripts, offers a REPL, etc. -- except that it comes with not only the usual stdlib but also the extra libraries you bundled into it. So that's one thing I intend to do for Pysh, the library I described in a sibling thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/DR6I2PY... You'll be able to write scripts with e.g. `#!/usr/bin/env python-pysh`, and then `python-pysh` is a single binary file which you just need to stick in `/usr/local/bin/` or `~/.local/bin/`; and in those scripts `import pysh` will work just as smoothly as `import subprocess`. (Or you can ignore that and get the library for your usual Python interpreter in the usual ways.) Greg On Mon, Aug 12, 2019 at 11:49 AM Nick Timkovich <prometheus235@gmail.com> wrote:
Something very similar to this had been done many times in third party libraries. None of those have become hugely popular, including none quite compelling me personally to do more than try them out as toys. It's too bad, in a way, since I love bash and use it all the time when it seems easier than Python. Your library is 100% certainly not going to become part of the standard library unless it can first become popular as a third party tool. If it does become popular that way, I wouldn't put it's odds of eventual soon to the standard library as higher than 5-10%. But that doesn't really matter, widespread use his 'pip install' world be a good thing in itself. On Sat, Aug 10, 2019, 7:52 AM Thiago Padilha <tpadilha84@gmail.com> wrote:
On Sat, Aug 10, 2019 at 6:53 AM David Mertz <mertz@gnosis.cx> wrote:
Something very similar to this had been done many times in third party libraries.
Check out Xonch for example: https://xon.sh/ -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
In various different ways, also: https://pypi.org/project/bash/ https://pypi.org/project/python-pipe/ https://pypi.org/project/pypette/ https://pypi.org/project/shell/ Just a few I've seen before. Xonsh is pretty cool also. On Sat, Aug 10, 2019, 9:55 PM Christopher Barker <pythonchb@gmail.com> wrote:
I actually gave a talk along these lines at the Chicago Python (ChiPy) meetup last week: slides https://docs.google.com/presentation/d/1v5z4f-FQkS-bQYE-Xv5SvA6cyaTiqlxs2w2C... Part of the argument was about using pure standard library so a self-contained script/repo could run anywhere Python is in order to (e.g.) bootstrap other testing environments and/or work within restricted ones, just like your average shell script. A gigantic step up from there is using *anything* not in the stdlib because you may need to copy a bunch of files (venv creation), and download executable things from *the Internet* (horrific to some). On Sat, Aug 10, 2019 at 11:03 PM David Mertz <mertz@gnosis.cx> wrote:
Nick Timkovich wrote:
Nice presentation. I've adapted the examples in the "how to parent" section to illustrate the potential gains in expressiveness when using ush: Example 1: >>> import subprocess | >>> from ush.sh import echo, git | >>> result = subprocess.run( | >>> str(echo(b'asdf') | git('rev-parse', ... ["git", "rev-parse", "HEAD"], | ... 'HEAD', cwd="/home/nick/Code/toss", ... cwd="/home/nick/Code/toss", | ... env={"A":"B"})) ... env={"A": "B", **os.environ}, | ... input=b"asdf", | ... capture_output=True, | ... ) | Example 2 (with different way of passing env/cwd to children, useful for spawning multiple commands with the same env/cwd): >>> import os | >>> from ush.sh import chdir, setenv, git | >>> os.chdir("/home/nick") | >>> with chdir("/home/nick"), setenv({"A":"B"}): >>> os.execvpe("git", | ... git("rev-parse", "HEAD")() ... args=["git", "rev-parse", "HEAD"], | ... env={"A": "B", **os.environ} | ... ) | Note that chdir/setenv context managers don't pollute anything in the host python process. The temporary changes in the env/cwd are managed by the Shell instance, which is not only a factory for Command objects but also stores default options values. That makes it possible to have multiple shell instances in separate threads running commands in completely different environments (though this will be more useful once I add asyncio support). There's a few features I haven't documented yet (chdir/setenv are examples). My goal was create a full shell scripting DSL right into python, while also: - Not using the shell=True under the hoods, which ends up making the python script depend on the underlying shell. - Not relying on native code while also creating full OS-level pipelines. Some alternatives like http://amoffat.github.io/sh/ implement piping by manually transfering data between processes. ush creates OS pipes and lets it handle data transfer, which is how shells like bash do it. - Being cross platform (windows/unix supported) - Supporting python2/3 (Though python2 support is less of a priority now).
One of the reasons I implemented ush in one file without dependencies was to make it simple bootstrapping it into any project. It is as easy as adding something like this to the top of your script: import pathlib script_dir = pathlib.Path(__file__).parent ush_path = script_dir / 'ush.py' if not ush_path.exists(): print('downloading ush.py') from urllib.request import urlretrieve urlretrieve( 'https://raw.githubusercontent.com/tarruda/python-ush/master/ush.py', str(ush_path)) (script_dir / '__init__.py').touch(exist_ok=True) from ush.sh import git, grep # import whatever commands you need Though it would be much better to have a library like ush built into standard library ;)
Nick Timkovich wrote:
In my experience (though I am curious about yours!) of environments where this sort of constraint applies, it applies just as strongly to the use of a version of Python that's newer than the one the system came with. That means that adding something to the stdlib isn't a route to getting to use it particularly soon in such environments, either. Availability in environments that are conservative about new or additional software is just inherently a long-term project. As a thought experiment, imagine there were consensus today on adding some library of this flavor to the stdlib, and the work rapidly got done and got merged. It'd be released in Python 3.9, circa early 2021. That in turn would be picked up in the next Ubuntu LTS in 2022... and the next RHEL/CentOS perhaps in 2024, and its derivatives like Amazon Linux sometime after that. When would you predict it might be "just always there" in the environments you work with? In the parts of the industry I'm most familiar with, in fact, the lag from publication to practical availability is much shorter through PyPI than through the stdlib: it's common to run the system Python from an old Ubuntu release that's pushing the limit of its 5-year supported lifespan, but for better and worse to cheerfully run the latest anything that any developer at the company wants to pull in from PyPI. so a self-contained script/repo could run anywhere Python is in order to
(e.g.) bootstrap other testing environments and/or work within restricted ones
For specifically this side of the constraints, though, I think there is hope for a technical solution! Doesn't help if you really can't pull in new code, but it does make bootstrapping far simpler at a technical level. The PyOxidizer project: https://github.com/indygreg/PyOxidizer makes it possible to bundle up (a) CPython including (a') the stdlib together with (b) some arbitrary set of Python libraries of your choice, all as a single statically-linked executable file. Then you can distribute that one file to other machines, and run it there, without needing to create temporary files or download stuff from the Internet or anything like that. In particular, you can have the executable behave just like `python` -- it runs scripts, offers a REPL, etc. -- except that it comes with not only the usual stdlib but also the extra libraries you bundled into it. So that's one thing I intend to do for Pysh, the library I described in a sibling thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/DR6I2PY... You'll be able to write scripts with e.g. `#!/usr/bin/env python-pysh`, and then `python-pysh` is a single binary file which you just need to stick in `/usr/local/bin/` or `~/.local/bin/`; and in those scripts `import pysh` will work just as smoothly as `import subprocess`. (Or you can ignore that and get the library for your usual Python interpreter in the usual ways.) Greg On Mon, Aug 12, 2019 at 11:49 AM Nick Timkovich <prometheus235@gmail.com> wrote:
participants (6)
-
Christopher Barker -
David Mertz -
Greg Price -
Nick Timkovich -
Thiago Arruda -
Thiago Padilha