[Python-bugs-list] [ python-Bugs-566037 ] Popen exectuion blocking w/threads

noreply@sourceforge.net noreply@sourceforge.net
Wed, 24 Jul 2002 09:31:28 -0700


Bugs item #566037, was opened at 2002-06-07 17:07
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=566037&group_id=5470

Category: Threads
Group: Python 2.2.1
Status: Open
Resolution: None
Priority: 3
Submitted By: Martin Stoufer (mstoufer)
Assigned to: Nobody/Anonymous (nobody)
Summary: Popen exectuion blocking w/threads

Initial Comment:
The following unit test hangs after the first two lines
of output. This is wholly reproducible (for us) under
2.2.1 and totaly unreproducible under 2.1. We have both
interpreters installed on a RH7.2 PC with linkings
against the apparent same /lib/libthread.so.0

import os, threading,time, sys
def read_output(self, cmd, timeout):
        print "run: %s" % cmd
        (inf,outf) = os.popen4(cmd)
        print "started! out_fd=%d" % outf.fileno()
        while 1:
            line = outf.readline()
            if line == "": break
            print len(line),line
            sys.stdout.flush()
        return
if __name__ == '__main__': 
    thr =
threading.Thread(target=read_output,args=(1,"ping -c 5
www.mit.edu",0))
    thr.start()
    print "Started thread"
    while 1: time.sleep(1)

mstoufer@dpsslx05(3)>ldd /usr/local/bin/python2.{1,2}
/usr/local/bin/python2.1:
	libpthread.so.0 => /lib/libpthread.so.0 (0x40029000)
	libdl.so.2 => /lib/libdl.so.2 (0x40040000)
	libutil.so.1 => /lib/libutil.so.1 (0x40044000)
	libstdc++-libc6.1-2.so.3 =>
/usr/local/lib/libstdc++-libc6.1-2.so.3 (0x40047000)
	libm.so.6 => /lib/libm.so.6 (0x4008f000)
	libc.so.6 => /lib/libc.so.6 (0x400b1000)
	/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
/usr/local/bin/python2.2:
	libdl.so.2 => /lib/libdl.so.2 (0x40029000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x4002d000)
	libutil.so.1 => /lib/libutil.so.1 (0x40044000)
	libm.so.6 => /lib/libm.so.6 (0x40047000)
	libc.so.6 => /lib/libc.so.6 (0x4006a000)
	/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

----------------------------------------------------------------------

Comment By: Martin Stoufer (mcstoufer)
Date: 2002-07-24 09:31

Message:
Logged In: YES 
user_id=243169

I've found that using the generic os.popen() call and just 
reading everything back to be a somewhat equitable fix. The 
returned object is file like and the .close() call on it at 
the end seems to clean up house pretty well. Before, with a 
Popen4() object, I was getting a lot of <defunct> processes. 
Now they all go away nicely. I'd be willing to follow up in 
person with anyone here, MCStoufer@lbl.gov


----------------------------------------------------------------------

Comment By: Zoran Bosnjak (zoranbosnjak)
Date: 2002-07-24 06:28

Message:
Logged In: YES 
user_id=583469

I have the same problem with python2.2 (the child 
process does not terminate on SIGTERM - it only 
terminates on SIGKILL). The same program works 
(terminates) fine with python2.1.1 (all other libs are the 
same). Here is my simple testfile: 
#!/usr/bin/env python2.2 
 
import os, time, signal, threading, string 
from popen2 import Popen3, Popen4 
 
pid = 0 
 
def run(): 
    global pid 
    f = Popen4('ping localhost') 
    pid = f.pid 
    while 1: 
        s = f.fromchild.readline() 
        if not s: break 
 
t = threading.Thread(target=run) 
t.start() 
time.sleep(10) 
print 'timeout' 
os.kill(pid, signal.SIGTERM) 
t.join() 
print 'bye' 
 

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-06-11 21:16

Message:
Logged In: YES 
user_id=6380

Anything is possible. I suggest asking for help in a Linux
threading guru forum.

----------------------------------------------------------------------

Comment By: Martin Stoufer (mcstoufer)
Date: 2002-06-11 21:10

Message:
Logged In: YES 
user_id=243169

This same issue arises when any subprocess is declared that makes 
more than 1 network read/write. From the system side, the Ping process 
is simply sleeping; we feel that the process is looking for the pipe to 
empty so it can finish it's write to stdout.

Replacing cmd with 'ping -c 1  www.mit.edu' works everytime. 
Replacing cmd with 'echo\'Hello. world\'' works as well.

We have beeen using a therading model similar to this under 2.1 and it 
has been quite robust in its execution. Is it possible that this is a 
compile-time issue with the thread linking?


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-06-10 12:39

Message:
Logged In: YES 
user_id=6380

Mixing forks and threads is often asking for trouble...

Does this also hang with another command that produces more
than two lines of output, or is it limited to ping?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-06-09 01:32

Message:
Logged In: YES 
user_id=21627

I can't reproduce this on a SuSE system, either (which has
glibc 2.2.5), so I'm tempted to declare it a glibc 2.1 bug.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2002-06-07 22:42

Message:
Logged In: YES 
user_id=14198

Just noting that current CVS Python works fine on Windows,
but also hangs on RH7.  My Linux debuggings skills are such
that I can offer no further help ;)

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=566037&group_id=5470