Open a command pipe for reading

[Rodrick Brown]

I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.

I've tried using Popen to do this with no luck. I'm trying to simulate 

/bin/foo myfile.dat 

And as the records are being printed do some calculations. 

pipe = Popen(exttool,shell=True,stdout=PIPE).stdout 

for data in pipe.readlines():
    print data,

This operation blocks forever I'm guessing it's trying to process the entire file at once. 

Sent from my iPhone 4.