[ python-Bugs-1307798 ] subprocess.Popen locks on Cygwin

Mon Oct 3 15:11:21 CEST 2005

Bugs item #1307798, was opened at 2005-09-29 12:05
Message generated for change (Comment added) made by jacek_poplawski
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1307798&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Threads
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Jacek Poplawski (jacek_poplawski)
Assigned to: Jason Tishler (jlt63)
Summary: subprocess.Popen locks on Cygwin

Initial Comment:
I have problem running application with
subprocess.Popen, it happens only on Python from Cygwin
or compiled on Cygwin. It works correctly on Linux,
QNX, and Python from Windows installer, it also works
on one version of Python 2.4.0 from Cygwin.

When I run application with shell=True - everything is
OK, but when I run it without it - Popen locks on data
= os.read(errpipe_read, 1048576). 

Using options stdin, stdout, stderr doesn't help.

I can't reproduce it with any simple application,
application I am running is written in C++, multithreaded. 

$ python
Python 2.4.2 (#1, Sep 29 2005, 10:56:12)
[GCC 3.4.4 (cygming special) (gdc 0.12, using dmd
0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for
more information.
>>> import subprocess
>>> subprocess.Popen("./application.exe")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.4/subprocess.py", line
542, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.4/subprocess.py", line
970, in _execute_child
    data = os.read(errpipe_read, 1048576) # Exceptions
limited to 1 MB
KeyboardInterrupt
>>> subprocess.Popen("./application.exe",shell=True)
<subprocess.Popen object at 0x4dd0ec>

----------------------------------------------------------------------

>Comment By: Jacek Poplawski (jacek_poplawski)
Date: 2005-10-03 15:11

Message:
Logged In: YES 
user_id=264913

Problem has been fixed after update of cygwin1.dll, THANKS A
LOT!

PS. could you explain "race-condition"?
PS2. Is it development version of cygwin? Why isn't
available thru setup.exe?
CYGWIN_NT-5.1 widlobrody 1.5.19s(0.138/4/2) 20050930
20:02:20 i686 unknown unknown Cygwin

----------------------------------------------------------------------

Comment By: Peter Åstrand (astrand)
Date: 2005-10-03 14:59

Message:
Logged In: YES 
user_id=344921

My guess is that you have a race-condition in application.exe. 

----------------------------------------------------------------------

Comment By: Jacek Poplawski (jacek_poplawski)
Date: 2005-10-03 14:54

Message:
Logged In: YES 
user_id=264913

Update... we tried to compile new Python version on computer
where 2.4.0 worked correctly, and... it still works. So -
like you said - problem may be related to cygwin setup,
because this one computer has older gcc and older cygwin:

Python 2.4.2 (#1, Oct  3 2005, 14:31:11)
[GCC 3.3.3 (cygwin special)] on cygwin

CYGWIN_NT-5.1 BOLESTY 1.5.13(0.122/4/2) 2005-03-01 11:01
i686 unknown unknown Cygwin

Now we are trying to update gcc on that computer to check
will it fail with never one. However, it would be nice to
have any way to fix that in subprocess.py, because I don't
even know what to report for Cygwin guys.

----------------------------------------------------------------------

Comment By: Jason Tishler (jlt63)
Date: 2005-10-03 14:47

Message:
Logged In: YES 
user_id=86216

> $ uname -a
> CYGWIN_NT-5.1 widlobrody 1.5.18

Please try the latest Cygwin snapshot.

> please notice that in the subprocess.py it locks after
> execvp

Where exactly does it lock up in subprocess.py?

If you can create a simple test case (in C), then the core
Cygwin developers are likely to try to debug the problem.

Since the problem only happens (so far) with application.exe,
I don't know how to debug the problem further. :,(

----------------------------------------------------------------------

Comment By: Jacek Poplawski (jacek_poplawski)
Date: 2005-10-03 14:37

Message:
Logged In: YES 
user_id=264913

$ uname -a
CYGWIN_NT-5.1 widlobrody 1.5.18(0.132/4/2) 2005-07-02 20:30
i686 unknown unknown Cygwin

Our application compiles with cygwin gcc and Microsoft
Visual Studio, problem happens on both versions.

execvp with this application works in Python, problem is
only with subprocess.Popen, please notice that in the
subprocess.py it locks after execvp

in log (after I break application with ctrl+c) I can't find
any cygwin error messages, I found some lines like that:

   52   16939 [main] python2.4 3380 geterrno_from_win_error:
windows error 123 == errno 2
   89   17028 [main] python2.4 3380 symlink_info::check:
GetFileAttributes (c:\tmp\builds\test
BS\cygwin\RELEASE\import subprocess;
subprocess.Popen(".\application.exe").exe) failed
   49   17077 [main] python2.4 3380 geterrno_from_win_error:
windows error 123 == errno 2
   87   17164 [main] python2.4 3380 symlink_info::check:
GetFileAttributes (c:\tmp\builds\test
BS\cygwin\RELEASE\import subprocess;
subprocess.Popen(".\application.exe").exe.lnk) failed

or: 

   49   41683 [main] python2.4 3380 geterrno_from_win_error:
windows error 3 == errno 2
   85   41768 [main] python2.4 3380 symlink_info::check:
GetFileAttributes (C:\cygwin-new\usr\local\bin\lib\python2.4\li
b-dynload.exe) failed

they repeat many times in log

I don't see anything suspicious in the place where
application.exe is forked:

 651 1135006 [main] python2.4 3612 spawnve: spawnve
(./application.exe, ./application.exe, 460090)
(...)
   55 1137090 [main] python2.4 3612 child_info::child_info:
subproc_ready 0x6DC
test_rpc\jp_test\Builds\Demo\LBS\cygwin\RELEASE\application.exe)
   80 1141223 [main] python2.4 3612! spawn_guts: new process
name c:\tmp\builds\test_rpc\jp_test\Builds\Demo\LBS\cygwinRELEASE\application.exe
  227 1141450 [main] python2.4 3612! _pinfo::dup_proc_pipe:
closed wr_proc_pipe 0x7FC for pid 3612(3348)

Do you need full log?

----------------------------------------------------------------------

Comment By: Jason Tishler (jlt63)
Date: 2005-10-03 13:51

Message:
Logged In: YES 
user_id=86216

Jacek,

You can use uname -a to display the Cygwin version.

Is your application.exe a Cygwin application?

AFAICT, this is probably a Cygwin bug.  Please try the
following:

1. Try the latest Cygwin snapshot to see if the problem has
already been fixed. You can download snapshots from:

http://cygwin.com/snapshots/

and replace your cygwin1.dll with this DLL.

2. Try to create a simple test case that does not involve
Python. For example, create a C program to use execvp()
to invoke your program to similuate subprocess.Popen()
with shell=False. If successful, then you've isolated the
problem to Cygwin.

3. Try to strace the problem:

$ strace -o python.log python2.4.exe -c 'import subprocess; 
subprocess.Popen("./application.exe")'

Search through python.log to look for error messages from
Cygwin.

----------------------------------------------------------------------

Comment By: Jacek Poplawski (jacek_poplawski)
Date: 2005-10-03 09:01

Message:
Logged In: YES 
user_id=264913

Only version of Python from Cygwin which doesn't lock:

Python 2.4 (#1, Dec  4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright", "credits" or "license" for more
information.
>>>

Example of version which locks (compiled by hand):
Python 2.4.2 (#1, Sep 29 2005, 10:56:12)
[GCC 3.4.4 (cygming special) (gdc 0.12, using dmd
0.125)] on cygwin

Or Python 2.4.1 from current Cygwin.

Official versions of Python for Windows also don't lock,
because they use different way to implement
subprocess.Popen, but I can't use such Python - it lacks
some features of Cygwin.

PS. I don't know how to report version of Cygwin, it was
installed just by running setup.exe and downloading packages.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2005-10-02 22:22

Message:
Logged In: YES 
user_id=33168

Jacek, are all the failures on cygwin?  Can you report the
version of cygwin on the failing boxes as well as the
version of cygwin on the box that passed the test?

----------------------------------------------------------------------

Comment By: Jacek Poplawski (jacek_poplawski)
Date: 2005-09-30 10:07

Message:
Logged In: YES 
user_id=264913

Yes, I can run test_subprocess.py and all 38 tests are OK
(in 15 seconds).
Just like I wrote I can't reproduce that with any simple
command, so maybe problem is that subprocess.Popen locks
_if_ application does something specific - but what?

PS. But I can reproduce that on few different computers with
different Python installations so it is not just
installation problem.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2005-09-30 07:33

Message:
Logged In: YES 
user_id=33168

Jason, any comments?  Is this cygwin specific or a general
subprocess issue?

----------------------------------------------------------------------

Comment By: Peter Åstrand (astrand)
Date: 2005-09-29 21:32

Message:
Logged In: YES 
user_id=344921

Does the testsuite test_subprocess.py work correctly on this
platform?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1307798&group_id=5470