How to read large amounts of output via popen

[Gabriel Genellina]

En Fri, 06 Aug 2010 06:06:29 -0300, loial <jldunn2000 at gmail.com> escribió:

> I need to read a large amount of data that is being returned in
> standard output by a shell script I am calling.
>
> (I think the script should really be writing to a file but I have no
> control over that)
>
> Currently I have the following code. It seeems to work, however I
> suspect this may not work with large amounts of standard output.
>
> What is the best way to read a large amount of data from standard
> output and write to a file?
>
> Here is my code.
>
> process=subprocess.Popen(['myscript', 'param1'],
> shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
>
> cmdoutput=process.communicate()
>
> myfile = open('/home/john/myoutputfile','w')
>
> myfile.write(cmdoutput[0])
>
> myfile.close()


If all you do with the process' output is to write it to the output file,  
you can avoid the intermediate step:


myfile = open('/home/john/myoutputfile','w')
myerror = open('/home/john/myerrorfile','w')
process=subprocess.Popen(['myscript', 'param1'],
shell=False,stdout=myfile,stderr=myerror)
process.wait()

(untested)

-- 
Gabriel Genellina