os.popen3 hangs in Windows XP SP1, SP2. Python 2.5 & 2.4. Consistent test case.

[Pierre Rouleau]

Hi all,

I have a consistent test case where os.popen3() hangs in Windows.  The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.

I stumbled on a case where a piece of code works in some occasions and
hangs consistently given different data. It performs exactly the same
on 3 different computers running Windows:

- computer 1: XP, SP1, Python 2.4.3
- computer 2: XP, SP2, Python 2.4.3
- computer 3: XP, SP2, Python 2.5

I know that a bug
(http://sourceforge.net/tracker/index.php?func=detail&aid=527783&group_id=5470&atid=105470)

related to popen3 hanging was opened and closed as 'not-a-bug'.
However I think that the test case I have is not a application problem
(a deadlock) and is a true problem with popen3() under Windows (but I
could be wrong.)

What I have is:

-  a test script: tpopen.py.
   - It uses nosetests.py to execute unit tests: it runs:
       - test_roman.py, a test script for :
          - roman.py,  a modified version of Mark Pilgrims's roman.py
module, I used to test nosetests.


#--------------
The test case I have is a short program that invokes nosetests to run a
test_roman.py:

- The script topen.py is executed on the command line with::

                 topen test_roman

- topen.py executes

        stdin, stdout, stderr = os.popen3('nosetests --nocapture
test_roman')

- The program runs properly if it then executes the following code
right after the call to os.popen3()::

        stderr_lines = list(stderr)
        stdin.close()
        stderr.close()
        pgm_exit_code = stdout.close() or 0

- The program hangs when it tries to read stdout (**but only ** when it
is running nosetests for the test_roman.py). The code following the
call to os.popen3() is::

       for line in stdout:
            sys.stdout.write(line)  # prints almost all nosetests
stdout lines except for some at the end!
                                               # then it HANGS.
        stderr_lines = list(stderr)
        stdin.close()
        stderr.close()
        pgm_exit_code = stdout.close() or 0


Note that it also hangs if topen.py executes:

          stdin, stdout, stderr = os.popen3('nosetests test_roman')

Or if I modify the print statements inside test_roman.py, the unit test
scrip executed by nosetests.py.
I also tried changing the order of the closing of the stream (like
doing stdin.close() right after the popen3() call).  No impact.

Note that popen3() call does not hang if topen runs other unit test
scripts i have.

#---------------------

QUESTIONS:
- Can any one see a problem with the order of execution that could
cause a deadlock here (topen.py code posted below)?
- Should I post a new Python bug, providing the complete test code or,
re-open the above mentioned bug?




# tpopen.py
---------------------------------------------------------------

import sys
import os

#
-----------------------------------------------------------------------------
# p o p e n 3 ( )         -- Execute a popen3 command --
# ^^^^^^^^^^^^^^^
#
def popen3(command,
            stdout_parser=None,
            stderr_parser=None,
            default_stdout_value=None,
            default_stderr_value=None) :

    pgm_exit_code = 0
    stdout_value, stderr_value = default_stdout_value,
default_stderr_value
    stdin, stdout, stderr = os.popen3(command)
    stdin.close()
    if stdout_parser:
        stdout_value = stdout_parser(stdout)
    if stderr_parser:
        stderr_value = stderr_parser(stderr)
    stderr.close()
    pgm_exit_code = stdout.close() or 0
    return (stdout_value, stderr_value, pgm_exit_code)


#
-----------------------------------------------------------------------------

cmd1 = 'nosetests ' + sys.argv[1]
cmd2 = 'nosetests --nocapture ' + sys.argv[1]

def print_stream(stream):
    for line in stream:
        sys.stdout.write(line)

print '--------- CMD 1 , no stdout print : does not hang --------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, None, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code


print '--------- CMD 2 : HANGS after printing almost all stdout -'
stdout, stderr_lines, pgm_exit_code = popen3(cmd2, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code


print '--------- CMD 1 , stdout print : HANGS -------------------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code

#
-----------------------------------------------------------------------------

--

Pierre Rouleau