Open a command pipe for reading
thomas at jollybox.de
Tue Aug 17 22:05:24 CEST 2010
On Tuesday 17 August 2010, it occurred to Rodrick Brown to exclaim:
> I have a fairly large file 1-2GB in size that I need to process line by
> line but I first need to convert the file to text using a 3rd party tool
> that prints the records also line by line.
> I've tried using Popen to do this with no luck. I'm trying to simulate
> /bin/foo myfile.dat
> And as the records are being printed do some calculations.
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
> for data in pipe.readlines():
> print data,
> This operation blocks forever I'm guessing it's trying to process the
> entire file at once.
Yes. It is. That's what you're telling it to do: file.readline returns a list
of all the lines in the file.
What you want to do is iterate over the stream, as in:
for line in pipe:
Also, there's probably no need to use shell=True.
> Sent from my iPhone 4.
Is that a fact? This is so interesting.
Phones these days. Almost as annoyingly obnoxious as gmx and yahoo mail.
More information about the Python-list