Open a command pipe for reading
clp2 at rebertia.com
Tue Aug 17 21:57:43 CEST 2010
On Tue, Aug 17, 2010 at 9:40 AM, Rodrick Brown <rodrick.brown at gmail.com> wrote:
> I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.
> I've tried using Popen to do this with no luck. I'm trying to simulate
> /bin/foo myfile.dat
> And as the records are being printed do some calculations.
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
I'd strongly suggest trying to avoid shell=True.
> for data in pipe.readlines():
for data in pipe:
> print data,
> This operation blocks forever I'm guessing it's trying to process the entire file at once.
Indeed, that's how readlines() works, so don't use it.
More information about the Python-list