subprocess pipe question

Rita rmorgan466 at gmail.com
Sat Feb 26 00:30:15 EST 2011


The results look right! I did a rather large test and the checksum passed.

I will hold off any speed ups as you suggested.

On Wed, Feb 23, 2011 at 8:37 PM, Rob Williscroft <rtw at rtw.me.uk> wrote:

> Rita wrote in
> news:AANLkTi=88dCPM_kqRs2g620OBsNXz0maJUbfWPEmEtnL at mail.gmail.com in
> gmane.comp.python.general:
>
> [Top post relocated]
>
> > On Tue, Feb 22, 2011 at 7:57 PM, Rob Williscroft <rtw at rtw.me.uk>
> > wrote:
> >
> >> Rita wrote in
> >> news:AANLkTi=w95gXoSc1TKt2BntGjqys1cbmDnoJHOKQ4YBA at mail.gmail.com in
> >> gmane.comp.python.general:
> >>
> >> >
> >> > When using wait() it works a bit better but not consistent
> >> > def run(cmd):
> >> >   p=subprocess.Popen(cmd,stdout=subprocess.PIPE)
> >> >   rc=p.wait()
> >> >   print rc
> >> >   return p.stdout
> >> >
> >>
> >> > When the output of cmd is a small ascii file it works perfectly
> >> > fine, but when the file is large (more than 2MB) the process just
> >> > waits for ever (I am guessing its blocking?).
> >>
> >> Your OS has supplied a pipe buffer of 2MB, its full and the prossess
> >> is waiting until you read something from the pipe (i.e.
> >> p.stdout.read()).
> >>
> >> >            When I use the communicate call
> >> > it works perfectly but my process is consuming way too much memory.
> >> >
> >> > Is there a better way to get my return code consistently
> >> > efficiently and not take up so much memory?
> >>
> >> If you don't need the processes output then don't use the PIPE
> >> argument. If you do you will need to read from p.stdout until the
> >> process is complete, then get the return code:
> >>
> >> while True:
> >>        buf = p.stdout.read( 1024 )
> >>        # do somthing with buf
> >>        if len( buf ) < 1024:
> >>                break
> >>
> >> rc = p.wait()
>
> >
> > For extra points is there a way to speed up the p.stdout.read(bufsize)
> > ?
>
> Its unlikely that is the problem, most likely you are reading all of the
> 2MB OS pipe buffer and then having to wait for the programme you are
> calling to produce some more output.
>
> Before trying to speed things up be sure to test with real work being
> done to the output you recieve.
>
> If its still slow and you have multiple cores/processors then you will
> need to put you read loop in a thread to speed it up.
>
> --
> Rob.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
--- Get your facts first, then you can distort them as you please.--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110226/bdc8fdcc/attachment.html>


More information about the Python-list mailing list