Question about unreasonable slowness

[allenjo5 at mail.northgrum.com]

James Antill wrote:
> On Fri, 17 Nov 2006 12:39:16 -0800, allenjo5 wrote:
>
> >> allenjo5 at mail.northgrum.com:
> >> > Ok, I built Python 2.5 (same AIX 5.1 machine).  With the "for line in
> >> > shellOut" loop in, it now takes "only" 7 secs instead of the 11 secs in
> >> > python 2.4.3.   So, that's better, but still unreasonably slow.  And to
> >> > answer another's question, I'm using the ksh builtin 'time' command to
> >> > time the overall script.
> >> >
> >> > BTW, I don't think the AIX /bin/sh (actually ksh) is inherently slow.
> >> > This naively translated pure shell version of my python test script
> >> > completes in .1 secs:
> >> >
> >> > i=1
> >> > while ((i<20))
> >> > do ((i+=1))
> >> >    print next
> >> >    print "$shellIn" | /bin/sh -c ':' |
> >> >            while read line
> >> >            do print $line
> >> >            done
> >> > done
> >>
> > Yeah, apparently this is an AIX specific issue - perhaps the python
> > implementation of popen4() needs to do something special for AIX?
>
>  This seems likely a more general issue, rather than just a python issue
> (although the huge speed up from moving to 2.5.x). A
> couple of things I'd try:

With help from c.u.aix, I've discovered the problem.  Python (in
popen2.py) is attempting to close filedescriptors 3 through 32767
before running the /bin/sh.   This is because os.sysconf('SC_OPEN_MAX')
is returning 32767.   So far, it looks like SC_OPEN_MAX is being set
correctly to 4 in posixmodule.c, and indeed, os.sysconf_names seems to
also have SC_OPEN_MAX set to 4:

python -c 'import os;  print os.sysconf_names'

...
'SC_XOPEN_XCU_VERSION': 109, 'SC_OPEN_MAX': 4, 'SC_PRIORITIZED_IO': 91,
...

In fact, none of the values that sysconf_names has set for the various
constants are being returned by os.sysconf().  For example, the 2
others I just listed:

$ ./python -c 'import os; print os.sysconf("SC_XOPEN_XCU_VERSION")'
4

$ ./python -c 'import os; print os.sysconf("SC_PRIORITIZED_IO")'
-1

This makes no sense to me... unless there is some memory alignment or
endian issue going on here?