Open a command pipe for reading

[Thomas Jollans]

On Tuesday 17 August 2010, it occurred to Rodrick Brown to exclaim:
> I have a fairly large file 1-2GB in size that I need to process line by
> line but I first need to convert the file to text using a 3rd party tool
> that prints the records also line by line.
> 
> I've tried using Popen to do this with no luck. I'm trying to simulate
> 
> /bin/foo myfile.dat
> 
> And as the records are being printed do some calculations.
> 
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
> 
> for data in pipe.readlines():
>     print data,
> 
> This operation blocks forever I'm guessing it's trying to process the
> entire file at once.

Yes. It is. That's what you're telling it to do: file.readline returns a list 
of all the lines in the file.

What you want to do is iterate over the stream, as in:

for line in pipe:
    process(line)

Also, there's probably no need to use shell=True.



> 
> Sent from my iPhone 4.

Is that a fact? This is so interesting.
Phones these days. Almost as annoyingly obnoxious as gmx and yahoo mail.