Open a command pipe for reading

Chris Rebert clp2 at
Tue Aug 17 21:57:43 CEST 2010

On Tue, Aug 17, 2010 at 9:40 AM, Rodrick Brown <rodrick.brown at> wrote:
> I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.
> I've tried using Popen to do this with no luck. I'm trying to simulate
> /bin/foo myfile.dat
> And as the records are being printed do some calculations.
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

I'd strongly suggest trying to avoid shell=True.

> for data in pipe.readlines():

for data in pipe:

>    print data,
> This operation blocks forever I'm guessing it's trying to process the entire file at once.

Indeed, that's how readlines() works, so don't use it.


More information about the Python-list mailing list