[Python-ideas] easier subprocess piping and redirection

[Timo Kluck]

Dear python enthousiasts,

While replacing some of my bash scripts by python scripts, I found the
following useful. I wrote a small utility module that allows piping
input and output in a very similar way to how you do it in bash. The
major advantage over just using shell=True is that this does not
expose us to the latter's security risks and quoting hazards.

The major advantage of using a pipe instead of just manually feeding
the output of one process to the other is that with a pipe, it doesn't
all have to fit into memory at once. However, the Python way of doing
that is rather cumbersome:

    >>> from subprocess import Popen, PIPE
    >>> p1 = Popen(['echo', 'a\nb'], stdout=PIPE)
    >>> p2 = Popen([head], '-n1'], stdin=p1.stdout, stdout=PIPE)
    >>> p2.communicate()
    ('a\n', None)

I've been able to replace this by:

    >>> from pipeutils import call
    >>> (call('echo', 'a\nb') | call('head', '-n1')).output()
    'a\n'

And similarly, I can do direct file redirects like this:

    >>> (call('echo', 'Hello world!') > 'test.txt').do()
    0
    >>> (call('cat') < 'test.txt').output()
    'Hello world!\n'

I think that this is a lot more concise and readable. As far as I'm
concerned, this was the only reason I would ever use bash scripts
instead of python scripts.

I would love to hear opinions on this. How would people like this as a library?

The source is online at https://bitbucket.org/tkluck/pipeutils

Best, Timo