Open a command pipe for reading

[Chris Rebert]

On Tue, Aug 17, 2010 at 9:40 AM, Rodrick Brown <rodrick.brown at gmail.com> wrote:
> I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.
>
> I've tried using Popen to do this with no luck. I'm trying to simulate
>
> /bin/foo myfile.dat
>
> And as the records are being printed do some calculations.
>
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

I'd strongly suggest trying to avoid shell=True.

> for data in pipe.readlines():

for data in pipe:

>    print data,
>
> This operation blocks forever I'm guessing it's trying to process the entire file at once.

Indeed, that's how readlines() works, so don't use it.
http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects

Cheers,
Chris
--
http://blog.rebertia.com