[ python-Bugs-1404925 ] subprocess.Popen inside thread locks the thread in some case

[SourceForge.net]

Bugs item #1404925, was opened at 2006-01-13 16:05
Message generated for change (Comment added) made by ragnark
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1404925&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Threads
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Toon Verstraelen (tovrstra)
Assigned to: Peter Ã
strand (astrand)
Summary: subprocess.Popen inside thread locks the thread in some case

Initial Comment:
The bug can be verified with the example program
attached to this bugs. Two files are attached:
my_thread.py and main.py

When main.py is executed the thread hangs on
subprocess.Popen, while my_thread.py, which is imported
by main.py runs without problems. In fact both should
do exactly the same since main.py doesn nothing but
importing my_thread.

My python version:

Python 2.4.2 (#1, Oct 17 2005, 09:05:20)
[GCC 3.3.6 (Gentoo 3.3.6, ssp-3.3.6-1.0, pie-8.7.8)] on
linux2

If any more info is required, 

----------------------------------------------------------------------

Comment By: Ragnar Kjørstad (ragnark)
Date: 2006-05-19 10:37

Message:
Logged In: YES 
user_id=10550

I too have verified that the testprograms (with an
additional sleep) cause a deadlock, and that moving the
import statement out of os._execvp solves the problem.


However, there is a more generic problem. If fork() is
executed while any locks are held by other threads than the
one executing fork, it will deadlock the new process.

E.g, if you have a process with 2 threads. The first one is
doing work that requires locking. This could be imports, but
also anything else that uses other locks.

The second thread does a fork. If this is accidently done
while thread 1 held the lock, it will deadlock if the new
process needs the same lock.

Now, the most common use of fork is of course to do exec
right afterwards, in which case it doesn't really matter,
but it's not the only use of fork. 

See the rationalie in the manpage of pthread_atfork for
details.
(http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html)



----------------------------------------------------------------------

Comment By: Yair Chuchem (bigorilla)
Date: 2006-05-09 12:30

Message:
Logged In: YES 
user_id=232571

this thread explains why this happens and how to solve it:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/5fae8a453c95ae89/42f5c9f9215dbb1e#42f5c9f9215dbb1e
I implemented the fix locally at mine:
at os.py, _execvpe, first line:
remove the "from errno import ...",
add a global "import errno" at the start
and change stuff imported from errno from "X" to "errno.X"


----------------------------------------------------------------------

Comment By: Ralf Schmitt (titty)
Date: 2006-01-13 17:41

Message:
Logged In: YES 
user_id=17929

glibc version 2.3.5.
If I add a time.sleep(1) at the end of my_thread.py I can
also reproduce this error reliably on ubuntu.


----------------------------------------------------------------------

Comment By: Toon Verstraelen (tovrstra)
Date: 2006-01-13 16:58

Message:
Logged In: YES 
user_id=970944

What version of glibc do you have on your ubuntu system? I
have glibc-2.3.5-r2, where r2 stands for second patch
revision from the gentoo distribution. In my case I can not
interrupt 'python main.py'. I have to use ctrl-z and then
'kill %1'.

----------------------------------------------------------------------

Comment By: Ralf Schmitt (titty)
Date: 2006-01-13 16:30

Message:
Logged In: YES 
user_id=17929

I cannot reproduce this error on ubuntu linux using python
2.4.2.
However, I am able to reproduce it using python 2.4.2
running on FreeBSD 4.11.

Stopping the program with ctrl-c results in the following
output:

ralf at stronzo:~/tmp$ python main.py 
before Popen
^CException in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/threading.py", line 442, in
__bootstrap
    self.run()
  File "/home/ralf/tmp/my_thread.py", line 7, in run
    sp = Popen(["ls", "-al"], stdout=PIPE, stderr=STDOUT)
  File "/usr/local/lib/python2.4/subprocess.py", line 542,
in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.4/subprocess.py", line 970,
in _execute_child
    data = os.read(errpipe_read, 1048576) # Exceptions
limited to 1 MB
OSError: [Errno 4] Interrupted system call

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/atexit.py", line 24, in
_run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.4/threading.py", line 632, in
__exitfunc
    t.join()
  File "/usr/local/lib/python2.4/threading.py", line 539, in
join
    self.__block.wait()
  File "/usr/local/lib/python2.4/threading.py", line 203, in
wait
    waiter.acquire()
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/atexit.py", line 24, in
_run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.4/threading.py", line 632, in
__exitfunc
    t.join()
  File "/usr/local/lib/python2.4/threading.py", line 539, in
join
    self.__block.wait()
  File "/usr/local/lib/python2.4/threading.py", line 203, in
wait
    waiter.acquire()
KeyboardInterrupt
ralf at stronzo:~/tmp$


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1404925&group_id=5470