shutil.runret and shutil.runout
Hello, subprocess is low level, cryptic, does too much, with poor usability, i.e. "don't make me think" is not about it. I don't know about you, but I can hardly write any subprocess call without spending at least 5-10 meditating over the documentation. So, I propose two high level KISS functions for shell utils (shutil) module: runret(command) - run command through shell, return ret code runout(command) - run command through shell, return output To avoid subprocess story (that makes Python too complicated) I deliberately limit the scope to: - executing from shell only - return one thing at a time I hope that this covers 80% of what _users_ need to execute commands from Python. If somebody needs more - there is `subprocess`. But if your own scripts are mostly outside these 80% - feel free to provide your user story and arguments, why this should be done in shutil and not in subprocess. Open questions: - security quoting for 'command' -- anatoly t.
On Fri, Feb 24, 2012 at 8:52 PM, anatoly techtonik <techtonik@gmail.com> wrote:
Hi Anatoly, I believe you'll find the simple convenience methods you are requesting already exist, in the form of subprocess.call(), subprocess.check_call() and subprocess.check_output(). The documentation has also been updated to emphasise these convenience functions over the Popen swiss army knife. If you do "pip install shell-command" you can also access the shell_call(), shell_check_call() and shell_output() functions I currently plan to include in subprocess for 3.3. (I'm not sure which versions of Python that module currently supports though - 2.7 and 3.2, IIRC). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Feb 24, 2012 at 1:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't find the names of these functions more intuitive than Popen(). I also think they far from being simple, because (in the order of appearance): 1. they require try/catch 2. docs still refer Popen, which IS complicated 3. contain shell FUD 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff http://docs.python.org/library/subprocess.html#subprocess.check_call My verdict - these fail to be simple, and require the same low-level system knowledge as Popen() for confident use.
Don't you find strange that shell utils module don't have any functions for the main shell function - command execution? In game development current state of subprocess bloat is called "featurecrepping" and the "scope definition" is a method to cope with this disease. -- anatoly t.
On Fri, Feb 24, 2012 at 9:12 PM, anatoly techtonik <techtonik@gmail.com> wrote:
They may still end up in shutil. I haven't really decided which location I like better. However, if you (or anyone else) wants to see Python's innate capabilities improve in this area (and they really are subpar compared to Perl 5, for example), your best bet is to download my Shell Command module and give me feedback on any problems you find with it via the BitBucket issue tracker. http://shell-command.readthedocs.org Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Feb 24, 2012 at 11:11 PM, Tshepang Lekhonkhobe <tshepang@gmail.com> wrote:
Just curious: If put in the stdlib, will the above-mentioned module bring CPython shell handling to Perl 5 level?
Closer, but it's hard to match backticks and implicit interpolation for convenience (neither of which is going to happen in Python). However, the trade-off is that you get things like the ability to create pre-defined commands and easier invocation of shlex.quote when appropriate, along with exceptions for some errors that would otherwise pass silently. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Anyone wanting to use Python as a system shell should look at IPython rather than the standard Python interactive interpreter. http://ipython.org/ipython-doc/dev/interactive/shell.html -- Steven
On 2012-02-24, at 12:12 , anatoly techtonik wrote:
1. they require try/catch
No.
2. docs still refer Popen, which IS complicated
True.
3. contain shell FUD
No, they contain warnings, against shell injection security risks. Warnings are not FUD, it's not trying to sell some sort of alternative it's just warning that `shell=True` is dangerous on untrusted input.
4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
http://docs.python.org/library/subprocess.html#subprocess.check_call
On the one hand, these notes are a bit clumsy. On the other hand, piping is a pretty fundamental concept of shell execution, I see nothing wrong about saying that these functions *can't* be involved in pipes. In fact stating it upfront looks sensible.
What "shell utils" module? Subprocess has exactly that in `call` and its variants. And "shutil" does not bill itself as a "shell utils" module right now, its description is "High-level file operations".
shutil.runret() - by definition has shell=True
Great, so your recommendation is to be completely insecure by default?
That's a high-level _user_ function. When user runs command in shell he sees both. So, this 'shell util' is an analogue.
That makes no sense, when users invoke shell commands programmatically (which is what these APIs are about), they expect two semantically different reporting streams to be split, not to be merged, indistinguishable and unusable as a default. Dropping stderr on the ground may be an acceptable default but munging stdout and stderr is not.
The main purpose of this function is to be useful from Python console
Then I'm not sure it belongs in subprocess or shutil, and users with that need should probably be driven towards iPython which provides extensive means of calling into the system shell in interactive sessions[0]. bpython may also provide such facilities. It *may* belong in the interactive interpreter's own namespace.
As opposed to `check_output(command)`?
This assertion demonstrably does not match reality, shell injections (the very reason for this warning) would not exist if this were the case. [0] http://ipython.org/ipython-doc/rel-0.12/interactive/reference.html#system-sh...
On Fri, Feb 24, 2012 at 2:50 PM, Masklinn <masklinn@masklinn.net> wrote:
Quote from the docs: "Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError." http://docs.python.org/library/subprocess.html#subprocess.check_call
Warnings would be o.k. if they provided at least some guidelines where shell=True can be useful and where do you need to use Popen (or escaping). Without positive examples, and a little research to show attack vectors (so that users can analyse if they are applicable in their specific case) it is FUD IMO.
The point is that it makes things more complicated than necessary. As a system programmer I feel confident about all this stuff, but users struggle to get it and they blame Python for complexity, and I have to agree. We can change that with high level API. The API that will automatically provide a rolling buffer for output if required to avoid locks (for the missing info as a drawback), and remove headache about "what to do about that?".
Not "by default" - only if it is impossible to make shutil.run*() functions more secure. They only make sense with shell=True, so my recommendation is to analyse security implications and *let* users make their grounded choice. Not frighten them, but making them think about security. The difference. User friendly docs for shutil.run*() docs should be structured as following: 1. you are free to use these functions 2. but know that they are insecure 3. in these cases: 3.1 3.2 3.3 4. if you think these cases won't apply to your project, then feel free to use, otherwise look at subprocess Of course, if some cases 3.1-3.3 have workarounds, they should be mentioned.
Conflict point: Do users care about stdout/stderr when they invoke shell commands? Do users care about stdout/stderr when they use Python syntax for invoking shell commands? These functions is no a syntax sugar for developers (as the aforementioned "alternatives" from subprocess modules are). They are helper for users. If you're a developer, who cares about pipes and needs programmatic acces - there is already a low level subprocess API with developer's defaults. If we speak about users: The standard shell console behaviour is to output both streams to the screen. That means that if I want to process this output, I don't know if it comes from stderr or stdout. So, if I want to process the output - I use Python to do this. If I know what I need the output from stderr only, I specify this explicitly. That's my default user story.
I think it is a good idea to unify interface across interactive mode in Python. Hopefully shutil.copy and friends are already good enough so that they don't have reasons to reimplement them (and users to learn new commands).
As opposed to check_output(command, *, stdin=None, stdout=None, stderr=None, shell=True)
It is not assertion, it is a wannabe for shutil documentation to clarify shell injections problems to the level that allow users to make a reasonable choice, so if the user is "using shell execution functions he already realizes that will happen if his input parameters are not validated properly". -- anatoly t.
On Fri, 24 Feb 2012 15:00:25 +0200 anatoly techtonik <techtonik@gmail.com> wrote:
Quote from the docs: subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False) Run the command described by args. Wait for command to complete, then return the returncode attribute. No documented exceptions raised, so no need for try/catch.
You mean something like (quoting from the docs): Warning Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input: <example removed> <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On 2012-02-24, at 14:00 , anatoly techtonik wrote:
Yes. If you want to run commands you just do. try/except are only needed if you call commands which may fail and want to handle them without quitting the whole interpreter. And for your stated use case of interactively calling those functions, there is no need whatsoever for try/catch. And `subprocess.call` returns the status code, no exception ever thrown.
http://docs.python.org/library/subprocess.html#frequently-used-arguments
How?
As a system programmer I feel confident about all this stuff
You feel confident about something which does not work, without warning?
Oh? Because this:
states precisely that the function would be insecure by default, and would have caveat warnings in the docs. Which is the correct approach to security… never as far as I know.
Considering IPython uses syntactic extentions (a "!" prefix) and does not require any importing effort currently, I doubt that's going to happen.
You may want to look up the definition of that word, I did not remove any context, you asserted people using shell-exec functions are aware of the risks. Which is, as, factually wrong.
Not sufficient when the default behavior is unsafe (and broken), as numerous users *will* discover the function through third parties and may never come close to the caveats they *should* know for the default usage of the function.
Am 24.02.2012 12:12, schrieb anatoly techtonik:
And therefore they need to be completely replaced by something incompatible and in another module? Sorry, Anatoly, this is not how Python development happens. We usually work incrementally, improving on what we have rather than throwing all out the door. I think this is what rubs most people wrong about your posts: you invariably propose radical changes that invalidate all previous work in the related area. That's something apart from your style of expression, which was discussed recently. So here's some constructive advice: your point 1 was shown invalid. The points 2-4 are "merely" documentation related: how about you think about how to improve these docs to be less confusing? Georg
On Fri, Feb 24, 2012 at 11:52 AM, anatoly techtonik <techtonik@gmail.com>wrote:
mmm you are describing subprocess.call() here... I don't see how this new command makes thing better, besides shell=True.
runout(command) - run command through shell, return output
what is 'output' ? the stderr ? the stdout ? a merge of both ? what about subprocess.check_output() ?
To avoid subprocess story (that makes Python too complicated)
I seems to me that the only complication here is shell=True, which seems ok to me to have it at False for security reasons. Cheers Tarek -- Tarek Ziadé | http://ziade.org
On Fri, Feb 24, 2012 at 2:10 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
shutil.runret() - by definition has shell=True
runout(command) - run command through shell, return output
what is 'output' ? the stderr ? the stdout ? a merge of both ?
That's a high-level _user_ function. When user runs command in shell he sees both. So, this 'shell util' is an analogue. If you have you own user scripts that require stdout or stderr separately, I am free to discuss the cases. The main purpose of this function is to be useful from Python console, so the interface should be very simple to remember from the first try. Like runout(command, ret='stdout|stderr|both'). No universal PIPEs.
what about subprocess.check_output() ?
See my reply above.
It won't be 'shell util' function anymore. If you're using shell execution functions, you already realize that will happen if your input parameters are not validated properly. Isolating calls that require shell execution in shutil module will also simplify security analysis for 3rd party libraries. -- anatoly t.
On Fri, 24 Feb 2012 12:10:08 +0100 Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The stated purpose of the new functions is to allow people to run shell commands without thinking about them. That's a bad idea (isn't most programming without thinking about it?). The first problem is that it's a great way to add data injection vulnerabilities to your application. It's also a good way to introduce bugs in your application when asked to (for instance) process user-provided file names. -1 <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 2:25 PM, Mike Meyer <mwm@mired.org> wrote:
The proposal doesn't took into account security implications, so your -1 is premature. I agree with your point that users should think about *security* when they run commands. But they should not think about how tons of different ways to execute their command and different combinations on different operating systems, *and* security implications about this. This is *the main point* that make subprocess module a failure, and a basis (main reason) of this proposal. If users choose to trade security over simplicity, they should know what the risks are, and what to do if they want to avoid them. So I completely support the idea of shutil docs containing a user friendly explanation of how to exploit and how to protect (i.e. use subprocess) from the flaws provided by this method of execution - if they need to protect. Python is not a Java - it should give users a choice of simple API when they don't need security, and let this choice of shooting themselves in the foot be explicit.. and simple. -- anatoly t.
On Fri, 24 Feb 2012 13:46:12 +0200 anatoly techtonik <techtonik@gmail.com> wrote:
Failing to take into account security implications means the -1 isn't premature, it's mandatory!
This sounds like a documentation issue, not a code issue. In fact, checking the shutil docs (via pydoc) turns up: shutil - Utility functions for copying and archiving files and directory trees. Clearly, running commands is *not* part of this functionality, so these new functions don't belong there.
So now look at use cases. The "simple" method you propose is *only* safe to use on a very small set of constant strings. If any of the values in the string are supplied by the user in any way, you can't use it. If any of the arguments contain shell meta-characters, you either have to quote them or not use your method. Since you're explicitly proposing passing the command to the shell, the programmer doesn't even know which characters are meta-characters when they write the code. This means these functions - as proposed - are more attractive nuisances than useful utilities. Oddly enough, I read the Julia docs on external commands between my first answer and your reply, and their solution is both as simple as what you want, and safe. This inspired a counter proposal: How about adding your new function to subprocess, except instead of passing them to the shell, they use shlex to parse them, then call Popen with the appropriate arguments? shlex might need some work for this. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer <mwm@mired.org> wrote:
http://shell-command.readthedocs.org
Unless someone uncovers a major design flaw in the next few months, at least ShellCommand, shell_call, shell_check_call and shell_output are likely to make an appearance in subprocess for 3.3. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 24 Feb 2012 22:19:07 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
That says: This module aims to take over where subprocess leaves off, providing convenient, low-level access to the system shell, that automatically handles filenames and paths containing whitespace, as well as protecting naive code from shell injection vulnerabilities. That's a backwards approach to security. Rather than allowing anything and turning off what you know isn't safe, you should disallow everything and turn on what you know is safe. So rather than trying to make the strings you pass to the shell safe, you should parse them yourself and avoid calling the shell at all. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 10:59 PM, Mike Meyer <mwm@mired.org> wrote:
Yes, that's why these are *separate functions* (each with "shell" in the name to make the shell's involvement rather hard to miss). Any application (rather than system administration script) that calls them with user provided data should immediately fail a security audit. The new APIs are intended specifically for system administrators that want the *system shell*, not a language level "cross platform" reinvention of it (and when it comes to shells, "cross platform" generally means, "POSIX even if you're on Windows, because we're not interesting in trying to reproduce Microsoft's idiosyncratic way of doing things"). The automatic quoting feature is mainly there to handle spaces in filenames - providing poorly structured programs with some minimal defence against shell injections is really just a bonus (although I admit I wasn't thinking about it that way when I wrote the current docs). As things stand, Python is a lousy language for system administration tasks - the standard APIs are either *very* low level (os.system()) or they're written almost entirely from the point of view of an application programmer (subprocess). Even when I *am* the administrator writing automation scripts for my own use, the subprocess library still keeps getting in the way, telling me it isn't safe to access my own shell. Normally, Python is pretty good about striking a sensible balance between "safe defaults" and "consenting adults", but it currently fails badly on this particular point. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Feb 25, 2012 at 12:11:57AM +1000, Nick Coghlan wrote:
I disagree with this analysis. Python, with its fantastic subprocess module, is the only language I really trust for system administration tasks. Most languages provide "shell=True" as the default, making them extremely frustrating for system administration. Every time I choose to write a shell script instead of using Python, the lack of robustness makes me eventually regret it (and then rewrite in Python with subprocess). Setting "shell=True" (or equivalent) seems really convenient in the short term, but in the long term, scripts behave erratically and are vulnerable to attacks. The subprocess module (with "shell=False") is a wonderful balance between "safe defaults" and "consenting adults". -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Sun, Feb 26, 2012 at 12:03 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
For applications where correctness in all circumstances is the dominant criterion? Sure. For throwaway scripts, though, most of the Linux sysadmins I know just use shell scripts or Perl. For the devops (and deployment automation in general) crowd, there's no real Python-based competitor to Chef and Puppet (both Ruby based) (my understanding is that the Python-based Fabric doesn't play in *quite* the same space as the other two). As things currently stand, Python deliberately makes it hard to say "I want my individual commands to be shell commands, but I also want Python's superior flow control constructs to decide which shell commands to run". For an application, that's a good thing. For personal automation, it's not. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Feb 26, 2012 at 08:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
Personally I find Python just find for all kinds of automation, including bash/Perl replacement. Yes, some things may be a few characters more to type than in Perl, but I'm happy to have all the other Python features and libraries in my arsenal. Sysadmins use what they learned, and it also depends on culture. Some places do use Python for sysadmin stuff too. The Chef/Puppet/Fabric example is a good one to support this point - Ruby, like Python is also more a dev language than a sysadmin language, and yet Chef & Puppet are written in Ruby and not Perl. Eli
On Sun, Feb 26, 2012 at 6:53 PM, Eli Bendersky <eliben@gmail.com> wrote:
For the key operation I'm talking about here, though, Ruby works the same way Perl does: it supports shell command execution via backtick quoted strings with implicit string interpolation. Is it really that hard to admit that there are some tasks that other languages are currently just plain better for than Python, and perhaps we can learn something useful from that? (And no, I'm not suggesting we adopt backtick command execution or implicit string interpolation. A convenience API that combines shell invocation, explicit string interpolation and whitespace and shell metacharacter quoting, though, *that* I support). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 26 Feb 2012 21:46:33 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Does Ruby also have something like Perl's -t/-T options and supporting functions?
The key word is "perhaps". There are some things other languages are better at than Python, and Python is the better off for it. I think that "supporting code injection attacks" is one such feature.
I'm only willing to support it if it's at least as safe as Perl. Meaning that either 1) It doesn't really invoke the shell, but handles provides those features explicitly, or 2) it throws errors if passed tainted strings. On the other hand, my support (or lack of it) isn't worth very much. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan writes:
For throwaway scripts, though, most of the Linux sysadmins I know just use shell scripts
Sure, but it's really hard to beat *sh plus GNU readline for brevity in using recent history to create a script. At some point, we "just don't want to go there." As for the Perl arm of your disjunction, do those sysadmins use Python for anything? There's a lot of history in the Linux sysadmin community favoring Perl. (Although the l33t Perlmonger I know is a Ruby hacker now....)
No, there isn't, but creating one could be rather hard, as Puppet and Chef both make heavy use of Ruby features conducive to writing DSLs. Note that although Fabric plays in a distinct space, its implementation looks like Chef, far more so than Puppet (ie, you write Fabric configs in Python, and Chef configs in a (domain-specific extension of) Ruby, while Puppet is a restricted DSL). One of the Puppet rationales for using Puppet rather than Chef is telling here: 3. Choice of configuration languages The language which Puppet uses to configure servers is designed specifically for the task: it is a domain language optimised for the task of describing and linking resources such as users and files. Chef uses an extension of the Ruby language. Ruby is a good general-purpose programming language, but it is not designed for configuration management - and learning Ruby is a lot harder than learning Puppet’s language. Some people think that Chef’s lack of a special-purpose language is an advantage. “You get the power of Ruby for free,” they argue. Unfortunately, there are many things about Ruby which aren’t so intuitive, especially for beginners, and there is a large and complex syntax that has to be mastered. -- http://bitfieldconsulting.com/puppet-vs-chef That applies equally well to "DSL"s that are extensions of (function calls in) Python. Making it easier to write DSLs in Python has come up many times, and so far the answer has always been "if you want to write a DSL in Python, write a DSL in Python; but you can't, and won't soon be able to, run it directly in the Python interpreter." DSLs have been done; there's configparser for one, argparse and ancestors, and things like gitosis. But it's hard to see Python beating Ruby at that game.
I don't think that's ever been my motivation for writing a script in Python. Really, is Python's for loop so much better than bash's? For me, it's data structures: something where my sed fu isn't enough, or the content has to persist longer than into the next pipe. And quoting. Shell quoting is such a pain, especially if there's an ssh remote command in there somewhere. This is not to say I'm opposed to making it easier to use Python as a command shell in principle, but I have to wonder whether it can be done as easily as all that, and without sacrificing some of the things we've insisted on in past discussions. On the other hand, for things where avoiding shell makes sense, Python is one of my tools of choice (the other being Emacs Lisp, where I want integration with my editor and don't much care about performance).
On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer <mwm@mired.org> wrote:
That *is* rather nice, although they never get around to actually explaining *how* to capture the output from the child processes (http://julialang.org/manual/running-external-programs/, for anyone else that's interested). It should definitely be possible to implement something along those lines as a third party library on top of subprocess (although it would be a lot more complicated than Shell Command is). Kenneth Reitz (author of "requests") has also spent some time tinkering with subprocess invocation API design concepts: https://github.com/kennethreitz/envoy Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 February 2012 12:41, Nick Coghlan <ncoghlan@gmail.com> wrote:
Vinay Sanjip extended this with "sarge" (available on PyPI, IIRC). One key advantage of sarge for me is that it handles piping and redirection in a cross-platfom manner, rather than just deferring to the shell. (I think envoy does this too, but it's not very reliable on WIndows from what I recall of my brief experiments). Paul.
On Sat, Feb 25, 2012 at 12:32 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Ah, I knew I'd seen a more polished version of that somewhere - Vinay posted about it a while back. As I see it, the two complement each other fairly nicely: shell_command is for direct access to the system shell. Appropriate when you're writing platform specific administration scripts. sarge is for cross platform scripting support. I'm actually not sure what this is useful for (since the default Windows shell has different spellings for so many basic commands and different syntax for environment variable expansion, it seems easier to just use the *actual* cross platform abstractions in the os module instead), but apparently it's good for something (or Vinay wouldn't have taken the time to write it). Of course, since it's just a convenience wrapper around Popen, ShellCommand does let you get pretty cute:
(I didn't actually do a great deal in ShellCommand to enable that - it's just a matter of passing all the keyword args through to subprocess.Popen) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
24.02.12 14:41, Nick Coghlan написав(ла):
Yes, I want this in Python: readall(cmd('cut -d: -f3 $file', file='/etc/passwd') | cmd('sort -n') | cmd('tail -n5')) or cmd('cut', '-d:', '-f3', '/etc/passwd').pipe('sort', '-n').pipe('tail', '-n5').readlines() or something similar.
https://github.com/JuliaLang/julia/blob/10aabddc3834223568a87721149d05765e7e... See readall and each_line.
Serhiy Storchaka writes:
But you can already do sorted([l.split(":")[2] for l in open('/etc/passwd')])[-5:] (and I don't really care whether you were being ironic or not; either way that one-liner is an answer<wink/>). Actually, I wrote that off the top of my head and it almost worked. The problem I ran into is that I'm on a Mac, and there was a bunch of cruft comments (which don't contain any colons) in the beginning of the file. So I got a list index out of range when accessing the split line. In this case, cut | sort | tail would produce the expected output. But cut | sort | head would just produce garbage (the leading comments in sorted order). So the failure modes differ. It might be useful for people used to shell failure modes.
I did recently see "pyp" touted as a Python-like sed/awk. I guess this stuff always comes down to what you're used to. To me it is insane to be still using Perl yet I prefer perl regex over posix anyday :) On Feb 26, 2012 10:03 PM, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
On Fri, Feb 24, 2012 at 9:46 PM, anatoly techtonik <techtonik@gmail.com> wrote:
This is *the main point* that make subprocess module a failure, and a basis (main reason) of this proposal.
Anatoly, this is the exact kind of blanket statement that pisses people off and makes them stop listening to you. The subprocess module is not a failure by any means. Safely invoking subprocesses is a *hard problem*. Other languages make the choice "guarding against shell injections is a problem for the user to deal with" and allow them by default in their subprocess invocation interfaces. They also make the choice that the risk of data leakage through user provided format strings is something for the developer to worry about and allow implicit string interpolation. Python doesn't allow either of those as a *deliberate design choice*. The current behaviour isn't an accident, or due to neglect, or because we're stupid. Instead, we default to the more secure, less convenient options, and allow people to explicitly request the insecure behaviour if they either: 1. don't care; or 2. do care, but also know it isn't actually a problem for their use case. This is a *good thing* if you're an application programmer - secure defaults lets you conduct security audits by looking specifically for cases where the safety checks have been bypassed. However, it mostly sucks if you're wanting to use Python for system administration (or similar) tasks where the shell is an essential tool rather than a security risk and there's no untrusted data that comes anywhere near your script. I'll repeat my suggestion: if you want to do something *constructive* about this, get Shell Command from PyPI and start using it, as it aims to address both the shell invocation and the string interpolation aspects of this issue. If you find problems, report them on the module's issue tracker (although I'll point out in advance that STDERR being separate from STDOUT by default is *deliberate*. If people want them merged they can include a redirection in their shell command. Otherwise STDERR needs to remain mapped to the same stream as it is in the parent process so that tools like getpass() will still work in an invoked shell command). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 24/02/2012 11:52, anatoly techtonik a écrit :
runret(command) - run command through shell, return ret code runout(command) - run command through shell, return output
Hi, Brevity is nice, but I had no idea what either of these functions is supposed to do before reading these descriptions. The names could be more explicit. (By the way, I agree with other issues raised in this thread. This was only my first impression.) Regards, -- Simon Sapin
On Fri, Feb 24, 2012 at 8:52 PM, anatoly techtonik <techtonik@gmail.com> wrote:
Hi Anatoly, I believe you'll find the simple convenience methods you are requesting already exist, in the form of subprocess.call(), subprocess.check_call() and subprocess.check_output(). The documentation has also been updated to emphasise these convenience functions over the Popen swiss army knife. If you do "pip install shell-command" you can also access the shell_call(), shell_check_call() and shell_output() functions I currently plan to include in subprocess for 3.3. (I'm not sure which versions of Python that module currently supports though - 2.7 and 3.2, IIRC). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Feb 24, 2012 at 1:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't find the names of these functions more intuitive than Popen(). I also think they far from being simple, because (in the order of appearance): 1. they require try/catch 2. docs still refer Popen, which IS complicated 3. contain shell FUD 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff http://docs.python.org/library/subprocess.html#subprocess.check_call My verdict - these fail to be simple, and require the same low-level system knowledge as Popen() for confident use.
Don't you find strange that shell utils module don't have any functions for the main shell function - command execution? In game development current state of subprocess bloat is called "featurecrepping" and the "scope definition" is a method to cope with this disease. -- anatoly t.
On Fri, Feb 24, 2012 at 9:12 PM, anatoly techtonik <techtonik@gmail.com> wrote:
They may still end up in shutil. I haven't really decided which location I like better. However, if you (or anyone else) wants to see Python's innate capabilities improve in this area (and they really are subpar compared to Perl 5, for example), your best bet is to download my Shell Command module and give me feedback on any problems you find with it via the BitBucket issue tracker. http://shell-command.readthedocs.org Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Feb 24, 2012 at 11:11 PM, Tshepang Lekhonkhobe <tshepang@gmail.com> wrote:
Just curious: If put in the stdlib, will the above-mentioned module bring CPython shell handling to Perl 5 level?
Closer, but it's hard to match backticks and implicit interpolation for convenience (neither of which is going to happen in Python). However, the trade-off is that you get things like the ability to create pre-defined commands and easier invocation of shlex.quote when appropriate, along with exceptions for some errors that would otherwise pass silently. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Anyone wanting to use Python as a system shell should look at IPython rather than the standard Python interactive interpreter. http://ipython.org/ipython-doc/dev/interactive/shell.html -- Steven
On 2012-02-24, at 12:12 , anatoly techtonik wrote:
1. they require try/catch
No.
2. docs still refer Popen, which IS complicated
True.
3. contain shell FUD
No, they contain warnings, against shell injection security risks. Warnings are not FUD, it's not trying to sell some sort of alternative it's just warning that `shell=True` is dangerous on untrusted input.
4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
http://docs.python.org/library/subprocess.html#subprocess.check_call
On the one hand, these notes are a bit clumsy. On the other hand, piping is a pretty fundamental concept of shell execution, I see nothing wrong about saying that these functions *can't* be involved in pipes. In fact stating it upfront looks sensible.
What "shell utils" module? Subprocess has exactly that in `call` and its variants. And "shutil" does not bill itself as a "shell utils" module right now, its description is "High-level file operations".
shutil.runret() - by definition has shell=True
Great, so your recommendation is to be completely insecure by default?
That's a high-level _user_ function. When user runs command in shell he sees both. So, this 'shell util' is an analogue.
That makes no sense, when users invoke shell commands programmatically (which is what these APIs are about), they expect two semantically different reporting streams to be split, not to be merged, indistinguishable and unusable as a default. Dropping stderr on the ground may be an acceptable default but munging stdout and stderr is not.
The main purpose of this function is to be useful from Python console
Then I'm not sure it belongs in subprocess or shutil, and users with that need should probably be driven towards iPython which provides extensive means of calling into the system shell in interactive sessions[0]. bpython may also provide such facilities. It *may* belong in the interactive interpreter's own namespace.
As opposed to `check_output(command)`?
This assertion demonstrably does not match reality, shell injections (the very reason for this warning) would not exist if this were the case. [0] http://ipython.org/ipython-doc/rel-0.12/interactive/reference.html#system-sh...
On Fri, Feb 24, 2012 at 2:50 PM, Masklinn <masklinn@masklinn.net> wrote:
Quote from the docs: "Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError." http://docs.python.org/library/subprocess.html#subprocess.check_call
Warnings would be o.k. if they provided at least some guidelines where shell=True can be useful and where do you need to use Popen (or escaping). Without positive examples, and a little research to show attack vectors (so that users can analyse if they are applicable in their specific case) it is FUD IMO.
The point is that it makes things more complicated than necessary. As a system programmer I feel confident about all this stuff, but users struggle to get it and they blame Python for complexity, and I have to agree. We can change that with high level API. The API that will automatically provide a rolling buffer for output if required to avoid locks (for the missing info as a drawback), and remove headache about "what to do about that?".
Not "by default" - only if it is impossible to make shutil.run*() functions more secure. They only make sense with shell=True, so my recommendation is to analyse security implications and *let* users make their grounded choice. Not frighten them, but making them think about security. The difference. User friendly docs for shutil.run*() docs should be structured as following: 1. you are free to use these functions 2. but know that they are insecure 3. in these cases: 3.1 3.2 3.3 4. if you think these cases won't apply to your project, then feel free to use, otherwise look at subprocess Of course, if some cases 3.1-3.3 have workarounds, they should be mentioned.
Conflict point: Do users care about stdout/stderr when they invoke shell commands? Do users care about stdout/stderr when they use Python syntax for invoking shell commands? These functions is no a syntax sugar for developers (as the aforementioned "alternatives" from subprocess modules are). They are helper for users. If you're a developer, who cares about pipes and needs programmatic acces - there is already a low level subprocess API with developer's defaults. If we speak about users: The standard shell console behaviour is to output both streams to the screen. That means that if I want to process this output, I don't know if it comes from stderr or stdout. So, if I want to process the output - I use Python to do this. If I know what I need the output from stderr only, I specify this explicitly. That's my default user story.
I think it is a good idea to unify interface across interactive mode in Python. Hopefully shutil.copy and friends are already good enough so that they don't have reasons to reimplement them (and users to learn new commands).
As opposed to check_output(command, *, stdin=None, stdout=None, stderr=None, shell=True)
It is not assertion, it is a wannabe for shutil documentation to clarify shell injections problems to the level that allow users to make a reasonable choice, so if the user is "using shell execution functions he already realizes that will happen if his input parameters are not validated properly". -- anatoly t.
On Fri, 24 Feb 2012 15:00:25 +0200 anatoly techtonik <techtonik@gmail.com> wrote:
Quote from the docs: subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False) Run the command described by args. Wait for command to complete, then return the returncode attribute. No documented exceptions raised, so no need for try/catch.
You mean something like (quoting from the docs): Warning Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input: <example removed> <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On 2012-02-24, at 14:00 , anatoly techtonik wrote:
Yes. If you want to run commands you just do. try/except are only needed if you call commands which may fail and want to handle them without quitting the whole interpreter. And for your stated use case of interactively calling those functions, there is no need whatsoever for try/catch. And `subprocess.call` returns the status code, no exception ever thrown.
http://docs.python.org/library/subprocess.html#frequently-used-arguments
How?
As a system programmer I feel confident about all this stuff
You feel confident about something which does not work, without warning?
Oh? Because this:
states precisely that the function would be insecure by default, and would have caveat warnings in the docs. Which is the correct approach to security… never as far as I know.
Considering IPython uses syntactic extentions (a "!" prefix) and does not require any importing effort currently, I doubt that's going to happen.
You may want to look up the definition of that word, I did not remove any context, you asserted people using shell-exec functions are aware of the risks. Which is, as, factually wrong.
Not sufficient when the default behavior is unsafe (and broken), as numerous users *will* discover the function through third parties and may never come close to the caveats they *should* know for the default usage of the function.
Am 24.02.2012 12:12, schrieb anatoly techtonik:
And therefore they need to be completely replaced by something incompatible and in another module? Sorry, Anatoly, this is not how Python development happens. We usually work incrementally, improving on what we have rather than throwing all out the door. I think this is what rubs most people wrong about your posts: you invariably propose radical changes that invalidate all previous work in the related area. That's something apart from your style of expression, which was discussed recently. So here's some constructive advice: your point 1 was shown invalid. The points 2-4 are "merely" documentation related: how about you think about how to improve these docs to be less confusing? Georg
On Fri, Feb 24, 2012 at 11:52 AM, anatoly techtonik <techtonik@gmail.com>wrote:
mmm you are describing subprocess.call() here... I don't see how this new command makes thing better, besides shell=True.
runout(command) - run command through shell, return output
what is 'output' ? the stderr ? the stdout ? a merge of both ? what about subprocess.check_output() ?
To avoid subprocess story (that makes Python too complicated)
I seems to me that the only complication here is shell=True, which seems ok to me to have it at False for security reasons. Cheers Tarek -- Tarek Ziadé | http://ziade.org
On Fri, Feb 24, 2012 at 2:10 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
shutil.runret() - by definition has shell=True
runout(command) - run command through shell, return output
what is 'output' ? the stderr ? the stdout ? a merge of both ?
That's a high-level _user_ function. When user runs command in shell he sees both. So, this 'shell util' is an analogue. If you have you own user scripts that require stdout or stderr separately, I am free to discuss the cases. The main purpose of this function is to be useful from Python console, so the interface should be very simple to remember from the first try. Like runout(command, ret='stdout|stderr|both'). No universal PIPEs.
what about subprocess.check_output() ?
See my reply above.
It won't be 'shell util' function anymore. If you're using shell execution functions, you already realize that will happen if your input parameters are not validated properly. Isolating calls that require shell execution in shutil module will also simplify security analysis for 3rd party libraries. -- anatoly t.
On Fri, 24 Feb 2012 12:10:08 +0100 Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The stated purpose of the new functions is to allow people to run shell commands without thinking about them. That's a bad idea (isn't most programming without thinking about it?). The first problem is that it's a great way to add data injection vulnerabilities to your application. It's also a good way to introduce bugs in your application when asked to (for instance) process user-provided file names. -1 <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 2:25 PM, Mike Meyer <mwm@mired.org> wrote:
The proposal doesn't took into account security implications, so your -1 is premature. I agree with your point that users should think about *security* when they run commands. But they should not think about how tons of different ways to execute their command and different combinations on different operating systems, *and* security implications about this. This is *the main point* that make subprocess module a failure, and a basis (main reason) of this proposal. If users choose to trade security over simplicity, they should know what the risks are, and what to do if they want to avoid them. So I completely support the idea of shutil docs containing a user friendly explanation of how to exploit and how to protect (i.e. use subprocess) from the flaws provided by this method of execution - if they need to protect. Python is not a Java - it should give users a choice of simple API when they don't need security, and let this choice of shooting themselves in the foot be explicit.. and simple. -- anatoly t.
On Fri, 24 Feb 2012 13:46:12 +0200 anatoly techtonik <techtonik@gmail.com> wrote:
Failing to take into account security implications means the -1 isn't premature, it's mandatory!
This sounds like a documentation issue, not a code issue. In fact, checking the shutil docs (via pydoc) turns up: shutil - Utility functions for copying and archiving files and directory trees. Clearly, running commands is *not* part of this functionality, so these new functions don't belong there.
So now look at use cases. The "simple" method you propose is *only* safe to use on a very small set of constant strings. If any of the values in the string are supplied by the user in any way, you can't use it. If any of the arguments contain shell meta-characters, you either have to quote them or not use your method. Since you're explicitly proposing passing the command to the shell, the programmer doesn't even know which characters are meta-characters when they write the code. This means these functions - as proposed - are more attractive nuisances than useful utilities. Oddly enough, I read the Julia docs on external commands between my first answer and your reply, and their solution is both as simple as what you want, and safe. This inspired a counter proposal: How about adding your new function to subprocess, except instead of passing them to the shell, they use shlex to parse them, then call Popen with the appropriate arguments? shlex might need some work for this. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer <mwm@mired.org> wrote:
http://shell-command.readthedocs.org
Unless someone uncovers a major design flaw in the next few months, at least ShellCommand, shell_call, shell_check_call and shell_output are likely to make an appearance in subprocess for 3.3. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 24 Feb 2012 22:19:07 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
That says: This module aims to take over where subprocess leaves off, providing convenient, low-level access to the system shell, that automatically handles filenames and paths containing whitespace, as well as protecting naive code from shell injection vulnerabilities. That's a backwards approach to security. Rather than allowing anything and turning off what you know isn't safe, you should disallow everything and turn on what you know is safe. So rather than trying to make the strings you pass to the shell safe, you should parse them yourself and avoid calling the shell at all. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
On Fri, Feb 24, 2012 at 10:59 PM, Mike Meyer <mwm@mired.org> wrote:
Yes, that's why these are *separate functions* (each with "shell" in the name to make the shell's involvement rather hard to miss). Any application (rather than system administration script) that calls them with user provided data should immediately fail a security audit. The new APIs are intended specifically for system administrators that want the *system shell*, not a language level "cross platform" reinvention of it (and when it comes to shells, "cross platform" generally means, "POSIX even if you're on Windows, because we're not interesting in trying to reproduce Microsoft's idiosyncratic way of doing things"). The automatic quoting feature is mainly there to handle spaces in filenames - providing poorly structured programs with some minimal defence against shell injections is really just a bonus (although I admit I wasn't thinking about it that way when I wrote the current docs). As things stand, Python is a lousy language for system administration tasks - the standard APIs are either *very* low level (os.system()) or they're written almost entirely from the point of view of an application programmer (subprocess). Even when I *am* the administrator writing automation scripts for my own use, the subprocess library still keeps getting in the way, telling me it isn't safe to access my own shell. Normally, Python is pretty good about striking a sensible balance between "safe defaults" and "consenting adults", but it currently fails badly on this particular point. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Feb 25, 2012 at 12:11:57AM +1000, Nick Coghlan wrote:
I disagree with this analysis. Python, with its fantastic subprocess module, is the only language I really trust for system administration tasks. Most languages provide "shell=True" as the default, making them extremely frustrating for system administration. Every time I choose to write a shell script instead of using Python, the lack of robustness makes me eventually regret it (and then rewrite in Python with subprocess). Setting "shell=True" (or equivalent) seems really convenient in the short term, but in the long term, scripts behave erratically and are vulnerable to attacks. The subprocess module (with "shell=False") is a wonderful balance between "safe defaults" and "consenting adults". -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Sun, Feb 26, 2012 at 12:03 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
For applications where correctness in all circumstances is the dominant criterion? Sure. For throwaway scripts, though, most of the Linux sysadmins I know just use shell scripts or Perl. For the devops (and deployment automation in general) crowd, there's no real Python-based competitor to Chef and Puppet (both Ruby based) (my understanding is that the Python-based Fabric doesn't play in *quite* the same space as the other two). As things currently stand, Python deliberately makes it hard to say "I want my individual commands to be shell commands, but I also want Python's superior flow control constructs to decide which shell commands to run". For an application, that's a good thing. For personal automation, it's not. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Feb 26, 2012 at 08:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
Personally I find Python just find for all kinds of automation, including bash/Perl replacement. Yes, some things may be a few characters more to type than in Perl, but I'm happy to have all the other Python features and libraries in my arsenal. Sysadmins use what they learned, and it also depends on culture. Some places do use Python for sysadmin stuff too. The Chef/Puppet/Fabric example is a good one to support this point - Ruby, like Python is also more a dev language than a sysadmin language, and yet Chef & Puppet are written in Ruby and not Perl. Eli
On Sun, Feb 26, 2012 at 6:53 PM, Eli Bendersky <eliben@gmail.com> wrote:
For the key operation I'm talking about here, though, Ruby works the same way Perl does: it supports shell command execution via backtick quoted strings with implicit string interpolation. Is it really that hard to admit that there are some tasks that other languages are currently just plain better for than Python, and perhaps we can learn something useful from that? (And no, I'm not suggesting we adopt backtick command execution or implicit string interpolation. A convenience API that combines shell invocation, explicit string interpolation and whitespace and shell metacharacter quoting, though, *that* I support). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 26 Feb 2012 21:46:33 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Does Ruby also have something like Perl's -t/-T options and supporting functions?
The key word is "perhaps". There are some things other languages are better at than Python, and Python is the better off for it. I think that "supporting code injection attacks" is one such feature.
I'm only willing to support it if it's at least as safe as Perl. Meaning that either 1) It doesn't really invoke the shell, but handles provides those features explicitly, or 2) it throws errors if passed tainted strings. On the other hand, my support (or lack of it) isn't worth very much. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan writes:
For throwaway scripts, though, most of the Linux sysadmins I know just use shell scripts
Sure, but it's really hard to beat *sh plus GNU readline for brevity in using recent history to create a script. At some point, we "just don't want to go there." As for the Perl arm of your disjunction, do those sysadmins use Python for anything? There's a lot of history in the Linux sysadmin community favoring Perl. (Although the l33t Perlmonger I know is a Ruby hacker now....)
No, there isn't, but creating one could be rather hard, as Puppet and Chef both make heavy use of Ruby features conducive to writing DSLs. Note that although Fabric plays in a distinct space, its implementation looks like Chef, far more so than Puppet (ie, you write Fabric configs in Python, and Chef configs in a (domain-specific extension of) Ruby, while Puppet is a restricted DSL). One of the Puppet rationales for using Puppet rather than Chef is telling here: 3. Choice of configuration languages The language which Puppet uses to configure servers is designed specifically for the task: it is a domain language optimised for the task of describing and linking resources such as users and files. Chef uses an extension of the Ruby language. Ruby is a good general-purpose programming language, but it is not designed for configuration management - and learning Ruby is a lot harder than learning Puppet’s language. Some people think that Chef’s lack of a special-purpose language is an advantage. “You get the power of Ruby for free,” they argue. Unfortunately, there are many things about Ruby which aren’t so intuitive, especially for beginners, and there is a large and complex syntax that has to be mastered. -- http://bitfieldconsulting.com/puppet-vs-chef That applies equally well to "DSL"s that are extensions of (function calls in) Python. Making it easier to write DSLs in Python has come up many times, and so far the answer has always been "if you want to write a DSL in Python, write a DSL in Python; but you can't, and won't soon be able to, run it directly in the Python interpreter." DSLs have been done; there's configparser for one, argparse and ancestors, and things like gitosis. But it's hard to see Python beating Ruby at that game.
I don't think that's ever been my motivation for writing a script in Python. Really, is Python's for loop so much better than bash's? For me, it's data structures: something where my sed fu isn't enough, or the content has to persist longer than into the next pipe. And quoting. Shell quoting is such a pain, especially if there's an ssh remote command in there somewhere. This is not to say I'm opposed to making it easier to use Python as a command shell in principle, but I have to wonder whether it can be done as easily as all that, and without sacrificing some of the things we've insisted on in past discussions. On the other hand, for things where avoiding shell makes sense, Python is one of my tools of choice (the other being Emacs Lisp, where I want integration with my editor and don't much care about performance).
On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer <mwm@mired.org> wrote:
That *is* rather nice, although they never get around to actually explaining *how* to capture the output from the child processes (http://julialang.org/manual/running-external-programs/, for anyone else that's interested). It should definitely be possible to implement something along those lines as a third party library on top of subprocess (although it would be a lot more complicated than Shell Command is). Kenneth Reitz (author of "requests") has also spent some time tinkering with subprocess invocation API design concepts: https://github.com/kennethreitz/envoy Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 February 2012 12:41, Nick Coghlan <ncoghlan@gmail.com> wrote:
Vinay Sanjip extended this with "sarge" (available on PyPI, IIRC). One key advantage of sarge for me is that it handles piping and redirection in a cross-platfom manner, rather than just deferring to the shell. (I think envoy does this too, but it's not very reliable on WIndows from what I recall of my brief experiments). Paul.
On Sat, Feb 25, 2012 at 12:32 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Ah, I knew I'd seen a more polished version of that somewhere - Vinay posted about it a while back. As I see it, the two complement each other fairly nicely: shell_command is for direct access to the system shell. Appropriate when you're writing platform specific administration scripts. sarge is for cross platform scripting support. I'm actually not sure what this is useful for (since the default Windows shell has different spellings for so many basic commands and different syntax for environment variable expansion, it seems easier to just use the *actual* cross platform abstractions in the os module instead), but apparently it's good for something (or Vinay wouldn't have taken the time to write it). Of course, since it's just a convenience wrapper around Popen, ShellCommand does let you get pretty cute:
(I didn't actually do a great deal in ShellCommand to enable that - it's just a matter of passing all the keyword args through to subprocess.Popen) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
24.02.12 14:41, Nick Coghlan написав(ла):
Yes, I want this in Python: readall(cmd('cut -d: -f3 $file', file='/etc/passwd') | cmd('sort -n') | cmd('tail -n5')) or cmd('cut', '-d:', '-f3', '/etc/passwd').pipe('sort', '-n').pipe('tail', '-n5').readlines() or something similar.
https://github.com/JuliaLang/julia/blob/10aabddc3834223568a87721149d05765e7e... See readall and each_line.
Serhiy Storchaka writes:
But you can already do sorted([l.split(":")[2] for l in open('/etc/passwd')])[-5:] (and I don't really care whether you were being ironic or not; either way that one-liner is an answer<wink/>). Actually, I wrote that off the top of my head and it almost worked. The problem I ran into is that I'm on a Mac, and there was a bunch of cruft comments (which don't contain any colons) in the beginning of the file. So I got a list index out of range when accessing the split line. In this case, cut | sort | tail would produce the expected output. But cut | sort | head would just produce garbage (the leading comments in sorted order). So the failure modes differ. It might be useful for people used to shell failure modes.
I did recently see "pyp" touted as a Python-like sed/awk. I guess this stuff always comes down to what you're used to. To me it is insane to be still using Perl yet I prefer perl regex over posix anyday :) On Feb 26, 2012 10:03 PM, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
On Fri, Feb 24, 2012 at 9:46 PM, anatoly techtonik <techtonik@gmail.com> wrote:
This is *the main point* that make subprocess module a failure, and a basis (main reason) of this proposal.
Anatoly, this is the exact kind of blanket statement that pisses people off and makes them stop listening to you. The subprocess module is not a failure by any means. Safely invoking subprocesses is a *hard problem*. Other languages make the choice "guarding against shell injections is a problem for the user to deal with" and allow them by default in their subprocess invocation interfaces. They also make the choice that the risk of data leakage through user provided format strings is something for the developer to worry about and allow implicit string interpolation. Python doesn't allow either of those as a *deliberate design choice*. The current behaviour isn't an accident, or due to neglect, or because we're stupid. Instead, we default to the more secure, less convenient options, and allow people to explicitly request the insecure behaviour if they either: 1. don't care; or 2. do care, but also know it isn't actually a problem for their use case. This is a *good thing* if you're an application programmer - secure defaults lets you conduct security audits by looking specifically for cases where the safety checks have been bypassed. However, it mostly sucks if you're wanting to use Python for system administration (or similar) tasks where the shell is an essential tool rather than a security risk and there's no untrusted data that comes anywhere near your script. I'll repeat my suggestion: if you want to do something *constructive* about this, get Shell Command from PyPI and start using it, as it aims to address both the shell invocation and the string interpolation aspects of this issue. If you find problems, report them on the module's issue tracker (although I'll point out in advance that STDERR being separate from STDOUT by default is *deliberate*. If people want them merged they can include a redirection in their shell command. Otherwise STDERR needs to remain mapped to the same stream as it is in the parent process so that tools like getpass() will still work in an invoked shell command). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 24/02/2012 11:52, anatoly techtonik a écrit :
runret(command) - run command through shell, return ret code runout(command) - run command through shell, return output
Hi, Brevity is nice, but I had no idea what either of these functions is supposed to do before reading these descriptions. The names could be more explicit. (By the way, I agree with other issues raised in this thread. This was only my first impression.) Regards, -- Simon Sapin
participants (16)
-
anatoly techtonik -
Andrew McNabb -
Dirkjan Ochtman -
Eli Bendersky -
Georg Brandl -
Masklinn -
Matt Joiner -
Mike Meyer -
Nick Coghlan -
Paul Moore -
Serhiy Storchaka -
Simon Sapin -
Stephen J. Turnbull -
Steven D'Aprano -
Tarek Ziadé -
Tshepang Lekhonkhobe