Question about unreasonable slowness

[allenjo5 at mail.northgrum.com]

Leif K-Brooks wrote:
> allenjo5 at mail.northgrum.com wrote:
> > i = 0
> > while (i < 20):
> >   i = i + 1
>
> for i in xrange(20):
>
> >   (shellIn, shellOut) = os.popen4("/bin/sh -c ':'")  # for testing, the
> > spawned shell does nothing
> >   print 'next'
> > #  for line in shellOut:
> > #       print line
> >
> > On my system (AIX 5.1 if it matters, with Python 2.4.3), this simple
> > loop spawning 20 subshells takes .75 sec.  Ok, that's reasonable.  Now,
> > if I uncomment the two commented lines, which loop over the empty
> > shellOut array, the progam now takes 11 secs.   That slowdown seems
> > very hard to believe.  Why should it slow down so much?
>
> The key fact here is that shellOut isn't an array; it's a living,
> breathing file object. If you don't iterate over it, you can run all 20
> shell processes in parallel if necessary; but if you do iterate over it,
>    you're waiting for sh's stdout pipe to reach EOF, which effectively
> means you can only run one process at a time.

Aha!  I now notice that with the second loop commented out, I see many
python processes running for a little while after the main program
ends.   So that confirms what you stated.

> On my system (OS X 10.4 with Python 2.5 installed), your code runs in
> .187 secs with the loop commented out, and in .268 secs otherwise. But I
> guess AIX's sh is slower than OS X's.

Ok, I built Python 2.5 (same AIX 5.1 machine).  With the "for line in
shellOut" loop in, it now takes "only" 7 secs instead of the 11 secs in
python 2.4.3.   So, that's better, but still unreasonably slow.  And to
answer another's question, I'm using the ksh builtin 'time' command to
time the overall script.

BTW, I don't think the AIX /bin/sh (actually ksh) is inherently slow.
This naively translated pure shell version of my python test script
completes in .1 secs:

i=1
while ((i<20))
do ((i+=1))
   print next
   print "$shellIn" | /bin/sh -c ':' |
           while read line
           do print $line
           done
done

Has anyone tried this on a true unix box (AIX, HPUX, Solaris, Linux)?
It seems to be functioning differently (and faster) on Windows and OS X
 (though I guess at its heard, OS X is essentially unix).

John.