Open a command pipe for reading
sschwarzer at sschwarzer.net
Mon Aug 23 10:15:56 CEST 2010
On 2010-08-17 18:40, Rodrick Brown wrote:
> I have a fairly large file 1-2GB in size that I need to
> process line by line but I first need to convert the file
> to text using a 3rd party tool that prints the records
> also line by line.
> I've tried using Popen to do this with no luck. I'm trying
> to simulate
> /bin/foo myfile.dat
Is foo the 3rd-party conversion tool you've mentioned?
It would be good to have a bit more context, e. g. an actual
snippet from your code.
> And as the records are being printed do some calculations.
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
> for data in pipe.readlines():
> print data,
> This operation blocks forever I'm guessing it's trying to
> process the entire file at once.
If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is
for line in pipe:
which reads and prints the file contents line by line.
More information about the Python-list