Question about unreasonable slowness

[James Antill]

On Fri, 17 Nov 2006 12:39:16 -0800, allenjo5 wrote:

>> allenjo5 at mail.northgrum.com:
>> > Ok, I built Python 2.5 (same AIX 5.1 machine).  With the "for line in
>> > shellOut" loop in, it now takes "only" 7 secs instead of the 11 secs in
>> > python 2.4.3.   So, that's better, but still unreasonably slow.  And to
>> > answer another's question, I'm using the ksh builtin 'time' command to
>> > time the overall script.
>> >
>> > BTW, I don't think the AIX /bin/sh (actually ksh) is inherently slow.
>> > This naively translated pure shell version of my python test script
>> > completes in .1 secs:
>> >
>> > i=1
>> > while ((i<20))
>> > do ((i+=1))
>> >    print next
>> >    print "$shellIn" | /bin/sh -c ':' |
>> >            while read line
>> >            do print $line
>> >            done
>> > done
>>
> Yeah, apparently this is an AIX specific issue - perhaps the python
> implementation of popen4() needs to do something special for AIX?

 This seems likely a more general issue, rather than just a python issue
(although the huge speed up from moving to 2.5.x). A
couple of things I'd try:

1. Split the spawn/IO apart, twenty procs. should be fine.
2. Try making the pipe buffer size bigger (optional third argument to
os.popen4).
3. Note that you might well be spawning three processes, and
are definitely doing two shells. Any shell init. slowness is going to be
none fun. Use an array to run /bin/true, and time that.

-- 
James Antill -- james at and.org
http://www.and.org/and-httpd/ -- $2,000 security guarantee
http://www.and.org/vstr/