subprocess pipe question
rmorgan466 at gmail.com
Sat Feb 26 06:30:15 CET 2011
The results look right! I did a rather large test and the checksum passed.
I will hold off any speed ups as you suggested.
On Wed, Feb 23, 2011 at 8:37 PM, Rob Williscroft <rtw at rtw.me.uk> wrote:
> Rita wrote in
> news:AANLkTi=88dCPM_kqRs2g620OBsNXz0maJUbfWPEmEtnL at mail.gmail.com in
> [Top post relocated]
> > On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft <rtw at rtw.me.uk>
> > wrote:
> >> Rita wrote in
> >> news:AANLkTi=w95gXoSc1TKt2BntGjqys1cbmDnoJHOKQ4YBA at mail.gmail.com in
> >> gmane.comp.python.general:
> >> >
> >> > When using wait() it works a bit better but not consistent
> >> > def run(cmd):
> >> > p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
> >> > rc=p.wait()
> >> > print rc
> >> > return p.stdout
> >> >
> >> > When the output of cmd is a small ascii file it works perfectly
> >> > fine, but when the file is large (more than 2MB) the process just
> >> > waits for ever (I am guessing its blocking?).
> >> Your OS has supplied a pipe buffer of 2MB, its full and the prossess
> >> is waiting until you read something from the pipe (i.e.
> >> p.stdout.read()).
> >> > When I use the communicate call
> >> > it works perfectly but my process is consuming way too much memory.
> >> >
> >> > Is there a better way to get my return code consistently
> >> > efficiently and not take up so much memory?
> >> If you don't need the processes output then don't use the PIPE
> >> argument. If you do you will need to read from p.stdout until the
> >> process is complete, then get the return code:
> >> while True:
> >> buf = p.stdout.read( 1024 )
> >> # do somthing with buf
> >> if len( buf ) < 1024:
> >> break
> >> rc = p.wait()
> > For extra points is there a way to speed up the p.stdout.read(bufsize)
> > ?
> Its unlikely that is the problem, most likely you are reading all of the
> 2MB OS pipe buffer and then having to wait for the programme you are
> calling to produce some more output.
> Before trying to speed things up be sure to test with real work being
> done to the output you recieve.
> If its still slow and you have multiple cores/processors then you will
> need to put you read loop in a thread to speed it up.
--- Get your facts first, then you can distort them as you please.--
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list