Python 2.2.1 and select()

[Francesco Bochicchio]

Il Mon, 24 Mar 2008 17:58:42 -0400, Derek Martin ha scritto:

> Hi kids!
> 
> I've got some code that uses select.select() to capture all the output
> of a subprocess (both stdout and stderr, see below).  This code works as
> expected on a variety of Fedora systems running Python > 2.4.0, but on a
> Debian Sarge system running Python 2.2.1 it's a no-go.  I'm thinking
> this is a bug in that particular version of Python, but I'd like to have
> confirmation if anyone can provide it.
> 
> The behavior I see is this:  the call to select() returns: [<file
> corresponding to sub-proc's STDOUT>] [] []
> 
> If and only if the total amount of output is greater than the specified
> buffer size, then reading on this file hangs indefinitely. For what it's
> worth, the program whose output I need to capture with this generates
> about 17k of output to STDERR, and about 1k of output to STDOUT, at
> essentially random intervals.  But I also ran it with a test shell
> script that generates roughly the same amount of output to each file
> object, alternating between STDOUT and STDERR, with the same results.
> 
> Yes, I'm aware that this version of Python is quite old, but I don't
> have a great deal of control over that (though if this is indeed a
> python bug, as opposed to a problem with my implementation, it might
> provide some leverage to get it upgraded)...  Thanks in advance for any
> help you can provide.  The code in question (quite short) follows:
> 
> def capture(cmd):
>     buffsize = 8192
>     inlist = []
>     inbuf = ""
>     errbuf = ""
> 
>     io = popen2.Popen3(cmd, True, buffsize) inlist.append(io.fromchild)
>     inlist.append(io.childerr)
>     while True:
>         ins, outs, excepts = select.select(inlist, [], []) for i in ins:
>             x = i.read()
>             if not x:
>                 inlist.remove(i)
>             else:
>                 if i == io.fromchild:
>                     inbuf += x
>                 if i == io.childerr:
>                     errbuf += x
>         if not inlist:
>             break
>     if io.wait():
>         raise FailedExitStatus, errbuf
>     return (inbuf, errbuf)
> 
> If anyone would like, I could also provide a shell script and a main
> program one could use to test this function...

>From yor description, it would seem that two events occurs:

- there are actual data to read, but in amount less than bufsize.
- the subsequent read waits (for wathever reason) until a full buffer can 
be read, and therefore lock your program.

Try specifying bufsize=1 or doing read(1). If my guess is correct, you 
should not see the problem. I'm not sure that either is a good solution
for you, since both have performance issues.

Anyway, I doubt that the python library does more than wrapping the 
system call, so if there is a bug it is probably in the software layers
under python.

Ciao
----
FB