[Python-ideas] Interest in seeing sh.py in the stdlib

[Andrew Moffat]

On Mon, Oct 22, 2012 at 9:52 AM, Jasper St. Pierre <jstpierre at mecheye.net>wrote:

> On Sat, Oct 20, 2012 at 8:33 PM, Andrew Moffat
> <andrew.robert.moffat at gmail.com> wrote:
> > Hi,
> >
> > I'm the author of sh.py, an intuitive interface for launching
> subprocesses
> > in Linux and OSX http://amoffat.github.com/sh/.  It has been maintained
> on
> > github https://github.com/amoffat/sh for about 10 months and currently
> has
> > about 25k installs, according to pythonpackages.com
> > (http://pythonpackages.com/package/sh,
> > http://pythonpackages.com/package/pbs)
> >
> > Andy Grover maintains the Fedora rpm for sh.py
> > http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=94247  and Nick
> > Moffit has submitted an older version of sh.py (which was called pbs) to
> be
> > included in Debian distros
> >
> http://pkgs.org/debian-wheezy/debian-main-i386/python-pbs_0.95-1_all.deb.html
> >
> > I'm interested in making sh.py more accessible to help bring Python
> forward
> > in the area of shell scripting, so I'm interested in seeing if sh would
> be
> > suitable for the standard library.  Is there any other interest in
> something
> > like this?
>
> I'm not one for the sugar. Seems like you're stuffing the Python
> syntax where it doesn't quite belong, as evidenced by the many escape
> hatches. Basic query of things not covered in the documentation:
>
> If I import a non-existant program, will it give me back a function
> that will fail or raise an ImportError?
>
> How do I run a program with a - in the name? You say you replace -
> with _, but thatdoesn't specify what happens in the edge case of "if I
> have google-chrome and google_chrome, which one wins? What about
> /usr/bin/google-chrome and /usr/local/bin/google_chrome"? That is,
> will it exhaust the PATH before trying fallbacks replacements or will
> it check all replacements at once?
>
> If I have a program that's not on PATH, what do I do? I can manipulate
> the PATH environment variable, but am I guaranteed that will work? Are
> you going to double fork forever to guarantee that environment? Can I
> build a custom prefix, like p =
> sh.MagicPrefix(path="/opt/android_devtools/bin"), and have that work
> like the regular sh module? p.gcc("whatever") ? Even with the
> existence of a regular gcc in the path?
>
> I wonder what happens if you do from sh import *.
>
> Does it block execution before continuing? How can I do parallel
> execution of four subprocesses, and get notified when all four are
> done? (Seems like this might be a thing for a Future as well, even in
> the absence of any scheduler or event loop).
>
> Are newcomers going to be confused by this? What happens if I try and
> do something like sh.ls("-l -a")? Will you use the POSIX shell parsing
> algorithm, pass it to bash, or pass it as one parameter? Will some
> form of injection attack be mitigated by this design?
>
>
> If you see this magic syntax as your one unique feature, I'd propose
> that you add it to the subprocess module, and improve the standard
> subprocess module's interface to cope with the new feature.
>
> But I don't see this as a worthwhile thing to have. -1 on the thing.
>
> > Thanks
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
>
> --
>   Jasper
>



Hi Jasper, thanks for your questions

If I import a non-existant program, will it give me back a function
> that will fail or raise an ImportError?


Yes, an exception will be raised

 How do I run a program with a - in the name? You say you replace -
> with _, but thatdoesn't specify what happens in the edge case of "if I
> have google-chrome and google_chrome, which one wins? What about
> /usr/bin/google-chrome and /usr/local/bin/google_chrome"? That is,
> will it exhaust the PATH before trying fallbacks replacements or will
> it check all replacements at once?


The full PATH will be exhausted for the exact command, as typed, before any
kind of "-" replacement is exercised.  There hasn't been much concern about
this because most people who want to call commands with special characters
prefer to use the Command class (e.g. chrome =
Command("/usr/bin/google-chrome")), so the documentation makes a note of
this on this issue.


If I have a program that's not on PATH, what do I do? I can manipulate
> the PATH environment variable, but am I guaranteed that will work? Are
> you going to double fork forever to guarantee that environment? Can I
> build a custom prefix, like p =
> sh.MagicPrefix(path="/opt/android_devtools/bin"), and have that work
> like the regular sh module? p.gcc("whatever") ? Even with the
> existence of a regular gcc in the path?


You could manipulate the PATH, but a better way would be to use the Command
class, which can take a full path of a command.  The returned object can be
used just like other commands.

I wonder what happens if you do from sh import *.


"ImportError: Cannot import * from sh. Please import sh or import programs
individually."  Commands are lazy resolved on sh anyways, so loading from
all would be undefined.

Does it block execution before continuing? How can I do parallel
> execution of four subprocesses, and get notified when all four are
> done? (Seems like this might be a thing for a Future as well, even in
> the absence of any scheduler or event loop).


Commands may be run in the background, like this:

job1 = sh.tar("-zc", "-f", "archive-name.tar.gz", "/some/directory",
_bg=True)
job2 = sh.tar(..., _bg=True)
job3 = sh.tar(..., _bg=True)

job1.wait()
job2.wait()
job3.wait()


Are newcomers going to be confused by this? What happens if I try and
> do something like sh.ls("-l -a")? Will you use the POSIX shell parsing
> algorithm, pass it to bash, or pass it as one parameter? Will some
> form of injection attack be mitigated by this design?


Bash--nor any shell-- is called into play. Sh.py doesn't do any argument
parsing either.  Arguments are passed into commands exactly as they're sent
through sh.py.  Newcomers have loved it so far, and after seeing some
examples, there's been minimal confusion about how to use it.

My thoughts about the magical-ness of sh.py..  I typically don't support
very magical or dynamically resolving modules.  I can be a little
apprehensive of ORMs for this reason... I like to know how my code behaves
explicitly.  I think clever/magic can be confusing for people and
inexplicit, and it's important to know more or less what's going on under
the hood.  But I also think that sh.py scratches an itch that a less
dynamic approach fails to reach.  My goal for sh.py has been to make
writing system scripts as easy for Python as it is for Bash.  People who
write Bash scripts do so for a few reasons, one being that a shell script
is pretty portable for *nix systems, and also because it's very easy to
call programs with arguments, feed them input, and parse their output.  But
the shortcomings of shell scripts are how obfuscated
and unnecessarily difficult they are to accomplish generic programming
tasks.  This is somewhere where Python excels.  But unfortunately, until
sh.py, I have not found any tool that lets you call commands nearly as
easily as Bash, that didn't rely on Bash.  Subprocess is painful.  Other
modules are extremely verbose.  Sh.py, yes, uses a dynamic lookup mechanism
(but it should be noted that you don't have to rely on it *at all* if you
don't like it), but it does so with a very specific intention, and that is
to make Python more suited and commonplace for writing system shell scripts.

My push here to see if there is support is because I believe if sh.py could
enter the stdlib, and therefore become more ubiquitous on Linux and OSX,
that more shell-style scripts could be written in Python, more new users
would be comfortable in using Python, and Bash scripts could go the way of
the dodo :)


Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121022/bf4ecfb9/attachment.html>