[Tutor] feedback on writing pipelines in python

[Alan Gauld]

On 21/03/12 18:03, Abhishek Pratap wrote:

> 1. stitch pipelines : I want python to act as a glue allowing me to run
> various linux shell based programs. If needed wait for a program to
> finish and then move on, logs if required

Python by default reads/writes to stdin/stdout so you can do stuff like:

$ foo.py < fred.txt | bar.py | baz.py > joe.txt

You can also control that explicitly by importing the sys module and 
reassigning sys.stdin/sys.stdout or even reading and writing explicitly 
from those file like objects.

You can also use the subprocess module to execute other programs (either 
Python or any other ind of executable file) in a sub procvess or a 
separate shell and get full control over the pipes, stdin, stdout and 
stderr. Of course that flexibility brings a price and the full power of 
subprocess.Popen is a little complex but there are lots of examples in 
the documentation.

> 2. run the same pipeline but on a local grid if required. (SGE flavor
> mainly)

I confess I have no idea what you mean by a local grid, or an
SGE flavor thereof...

Somebody else will need to address that one!

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/