One line command line filter

[Peter Otten]

Jon Redgrave wrote:

> It seems unreasonably hard to write simple one-line unix command line
> filters in python:
> 
> eg: ls | python -c "<something>  print x.upper()"
> 
> to get at sys.stdin or similar needs an import, which makes a
> subsequent for-loop illegal.
> python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError
> 
> Am I missing something obvious?
> 
> The best I've come up with is to use sitecustomize.py to add to
> __builtin__
> def stdin():
>   import sys
>   for x in sys.stdin():
>     if not x: return
>     yield x.strip()
> import __builtin__
> __builtin__.stdin = stdin
> 
> This allows
> ls | python -c "for x in stdin(): print x.upper()"
> 
> Is there a better solution - if not is this worth a PEP?

$ touch alpha beta gamma omega
$ ls | python -c 'import sys; sys.stdout.writelines(s.upper() for s in 
sys.stdin if s.startswith(("a", "o")))'
ALPHA
OMEGA

If you are doing this a lot, why don't you write a helper script and invoke 
that? 

$ ls | pyfilter.py -f '"m" in line' -s 'lines = (line + line[::-1] for line 
in map(str.strip, lines))' -s'import re' -p 're.compile(r"(([a-
z])\2)").sub(lambda m: m.group(1).upper(), line)'
alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

This relies on the convention that a single line of input is accessible as 
"line" and the complete input is called "lines". Of course the same can be 
done with python -c ..., -- and it is even more readable:

$ ls | python -c 'import re, sys
for line in sys.stdin:
>     line = line.strip()
>     line = line + line[::-1]
>     print re.compile(r"(([a-z])\2)").sub(lambda m: m.group(1).upper(), 
line)
> '
alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

> Is there a better solution - if not is this worth a PEP?

The python interpreter could accept multiple -c arguments, but to see how 
this will be received a post on python-ideas should be sufficient.

For the sake of completeness here's the script I used to produce the example 
above:

$ cat pyfilter.py
#!/usr/bin/env python
import sys

def main():
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-s", "--setup",
        action="append", default=[],
        dest="setups", metavar="SETUP")
    parser.add_argument(
        "-f", "--filter",
        action="append", default=[],
        dest="filters", metavar="FILTER")
    parser.add_argument("-p", "--print", dest="format")

    args = parser.parse_args()
    lines = sys.stdin
    for setup in args.setups:
        exec setup
    for line in lines:
        line = line.rstrip("\n")
        for filter in args.filters:
            if not eval(filter):
                continue
        if args.format:
            line = eval(args.format)
        try:
            print line
        except IOError as e:
            if e.errno == 32: # broken pipe
                break
            raise
if __name__ == "__main__":
    main()