Open a command pipe for reading

[Stefan Schwarzer]

Hi Rodrick,

On 2010-08-17 18:40, Rodrick Brown wrote:
> I have a fairly large file 1-2GB in size that I need to
> process line by line but I first need to convert the file
> to text using a 3rd party tool that prints the records
> also line by line.
> 
> I've tried using Popen to do this with no luck. I'm trying
> to simulate 
> 
> /bin/foo myfile.dat 

Is foo the 3rd-party conversion tool you've mentioned?

It would be good to have a bit more context, e. g. an actual
snippet from your code.

> And as the records are being printed do some calculations. 
> 
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout 
> 
> for data in pipe.readlines():
>     print data,
> 
> This operation blocks forever I'm guessing it's trying to
> process the entire file at once. 

If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is

    for line in pipe:
        print line,

which reads and prints the file contents line by line.

Stefan