Cygwin: Problem detecting subprocess termination after _spawn_posix in distutils?
I've been trying to install PIL 1.1.6 for several days now under Cygwin. I have tried the Cygwin, Image and distutils lists with nary a pointer, so I wondered whether python-dev might lead me to an answer (other than "Stop using Cygwin" ;-) Here's the output from a failed setup run with a couple of debug prints inserted which should report how sub-process termination occurred - it hangs after this output. $ python setup.py build running build running build_py running build_ext building '_imaging' extension SPAWN: ['gcc', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O3', '-Wall', '-Wstrict-prototypes', '-DHAVE_LIBZ', '-IlibImaging', '-I/usr/include', '-I/usr/include/python2.5', '-c', 'libImaging/Chops.c', '-o', 'build/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o'] PATH? 1 V: 0 D:0 gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ -IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Chops.c -o build/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o Are we done yet? Waiting on pid 3280 I have to kill a python.exe process that's left running in the background if I CTRL/C the build. Otherwise it will spend all night in a tight loop. The compilation is finished when I kill the process, and the next compile will begin if I repeat the command. I extracted the _spawn_all function from distutils/spawn.py, hacked some exceptions and ran it under command line control. That worked fine with other subtasks, so I wondered whether this was a failure specific to gcc, which would seem kind of unlikely. I ran the same compile using my test function standing alone, seeing success: sholden@bigboy ~/Imaging-1.1.6 $ python ~/Projects/Python/spawntest.py gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ -IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Chops.c -o build/temp.cygwin-1.5.24-i686-2.5/libImaging/Chops.o Are we done yet? Waiting on pid 3244 Got pid, status 3244 0 Got WIFEXITED 0 So it appears unlikely to be gcc-specific, leaving me wondering what exactly is the difference between the build environment and my tests. It would be really nice if test_distutils showed any failures, but it doesn't so any assistance would be welcome. At this point I can't even replicate the failure in a simpler test :-( regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading -------------
It would be really nice if test_distutils showed any failures, but it doesn't so any assistance would be welcome. At this point I can't even replicate the failure in a simpler test :-(
My guess is that it's the environment; if not that, the working directory. Assuming you have already instrumented ccompiler.CCompiler.spawn, I suggest to dump os.environ and print os.getcwd(). Assuming you really meant that you run under Cygwin Python (instead of just using --compiler), you might want to instrument spawn._spawn_posix instead. When you say you extracted _spawn_all from distutils/spawn.py: what version of Python are you talking about? I can't find _spawn_all in the sources of 2.5.x, or 2.6. Regards, Martin
Martin v. Löwis wrote:
It would be really nice if test_distutils showed any failures, but it doesn't so any assistance would be welcome. At this point I can't even replicate the failure in a simpler test :-(
My guess is that it's the environment; if not that, the working directory. Assuming you have already instrumented ccompiler.CCompiler.spawn, I suggest to dump os.environ and print os.getcwd(). Assuming you really meant that you run under Cygwin Python (instead of just using --compiler), you might want to instrument spawn._spawn_posix instead.
When you say you extracted _spawn_all from distutils/spawn.py: what version of Python are you talking about? I can't find _spawn_all in the sources of 2.5.x, or 2.6.
Thanks for taking the time to have a look at this. Sorry, it *was* _spawn_posix I extracted (and have instrumented in the live version) - I have no idea where "_spawn_all" came from. I am indeed running under Cygwin Python. Here-s a diff -u output against the original spawn.py so you can see what I have changed. $ diff -u /lib/python2.5/distutils/{spawn.py.org,spawn.py} --- /lib/python2.5/distutils/spawn.py.org 2007-07-14 09:09:24.114921600 -0400 +++ /lib/python2.5/distutils/spawn.py 2007-07-31 20:53:33.325945600 -0400 @@ -118,7 +118,9 @@ search_path=1, verbose=0, dry_run=0): - + for _k in sorted(os.environ.keys()): + print "%s=%s" % (_k, os.environ[_k]) + print "SPAWN:", cmd, "PATH?", search_path, "V:", verbose, "D:", dry_run log.info(string.join(cmd, ' ')) if dry_run: return @@ -144,20 +146,25 @@ # Loop until the child either exits or is terminated by a signal # (ie. keep waiting if it's merely stopped) while 1: + print "Are we done yet? Waiting on pid", pid try: (pid, status) = os.waitpid(pid, 0) + print "Got pid, status", pid, status except OSError, exc: import errno + print "Got OSError", exc.errno if exc.errno == errno.EINTR: continue raise DistutilsExecError, \ "command '%s' failed: %s" % (cmd[0], exc[-1]) if os.WIFSIGNALED(status): + print "Got WIFSIGNALED", status raise DistutilsExecError, \ "command '%s' terminated by signal %d" % \ (cmd[0], os.WTERMSIG(status)) elif os.WIFEXITED(status): + print "Got WIFEXITED", status exit_status = os.WEXITSTATUS(status) if exit_status == 0: return # hey, it succeeded! @@ -167,9 +174,11 @@ (cmd[0], exit_status) elif os.WIFSTOPPED(status): + print "Got WIFSTOPPED", status continue else: + print "Got unknown exception", status raise DistutilsExecError, \ "unknown error executing '%s': termination status %d" % \ (cmd[0], status) The output now includes the environment: $ python setup.py install running install running build running build_py running build_ext building '_imaging' extension !::=::\ !C:=C:\cygwin\bin ALLUSERSPROFILE=C:\Documents and Settings\All Users APPDATA=C:\Documents and Settings\sholden\Application Data APR_ICONV_PATH=C:\Program Files\Subversion\iconv CDPATH=.:/c/Steve:/c/Steve/Projects:/usr/local CLIENTNAME=Console COMMONPROGRAMFILES=C:\Program Files\Common Files COMPUTERNAME=BIGBOY COMSPEC=C:\WINDOWS\system32\cmd.exe CVSROOT=/usr/local/repository/ CVS_RSH=/bin/ssh FP_NO_HOST_CHECK=NO HOME=/c/Steve HOMEDRIVE=C: HOMEPATH=\Documents and Settings\sholden HOSTNAME=bigboy INFOPATH=/usr/local/info:/usr/share/info:/usr/info: LANG=C LOGONSERVER=\\BIGBOY MAKE_MODE=unix MANPATH=/usr/local/man:/usr/share/man:/usr/man::/usr/ssl/man NUMBER_OF_PROCESSORS=1 OLDPWD=/c/Steve OS=Windows_NT PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/c/WINDOWS/system32:/c/WINDOWS: /c/WINDOWS/System32/Wbem:/c/Program Files/ATI Technologies/ATI Control Panel:/c/ Program Files/Common Files/GTK/2.0/bin:/c/Program Files/Subversion/bin:/c/Python 25:/c/Steve/bin PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH PLAT=cygwin-1.5.24-i686 PRINTER=HP Photosmart C6100 series PROCESSOR_ARCHITECTURE=x86 PROCESSOR_IDENTIFIER=x86 Family 6 Model 13 Stepping 6, GenuineIntel PROCESSOR_LEVEL=6 PROCESSOR_REVISION=0d06 PROGRAMFILES=C:\Program Files PROMPT=$P$G PS1=\[\e]0;\w\a\]\n\[\e[32m\]\u@\h \[\e[33m\]\w\[\e[0m\]\n\$ PWD=/c/Steve/Imaging-1.1.6 PYSVN=svn+ssh://pythondev@svn.python.org/ PYTHONSTARTUP=/c/Steve/.pythonrc SESSIONNAME=Console SHLVL=1 SYSTEMDRIVE=C: SYSTEMROOT=C:\WINDOWS TEMP=/c/DOCUME~1/sholden/LOCALS~1/Temp TERM=cygwin TMP=/c/DOCUME~1/sholden/LOCALS~1/Temp USER=sholden USER1=u35582809@s90820416.onlinehome.us USERDOMAIN=BIGBOY USERNAME=sholden USERPROFILE=C:\Documents and Settings\sholden VISUAL=vi VS80COMNTOOLS=C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\ WINDIR=C:\WINDOWS _=/usr/bin/python SPAWN: ['gcc', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O3', '-Wall', '-Wstri ct-prototypes', '-DHAVE_LIBZ', '-IlibImaging', '-I/usr/include', '-I/usr/include /python2.5', '-c', 'libImaging/Dib.c', '-o', 'build/temp.cygwin-1.5.24-i686-2.5/ libImaging/Dib.o'] PATH? 1 V: 0 D: 0 gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ - IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Dib.c -o build /temp.cygwin-1.5.24-i686-2.5/libImaging/Dib.o Are we done yet? Waiting on pid 416 The only environment variables that don't appear in the shell output from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading -------------
The only environment variables that don't appear in the shell output from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.
At this point, I'd recommend to perform a cygwin update; with Cygwin, these problems often go away with an update. If that doesn't help, you can ask on the Cygwin list also; to analyse this further, ISTM one will need to debug the internals of cygwin. One thing you could try is to add -v to the list of gcc options; you can then see whether gcc is progressing correctly. Regards, Martin
Martin v. Löwis wrote:
The only environment variables that don't appear in the shell output from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.
At this point, I'd recommend to perform a cygwin update; with Cygwin, these problems often go away with an update.
I updated Cygwin and did a rebaseall before posting.
If that doesn't help, you can ask on the Cygwin list also; to analyse this further, ISTM one will need to debug the internals of cygwin.
I posted on Cygwin before asking here.
One thing you could try is to add -v to the list of gcc options; you can then see whether gcc is progressing correctly.
I'll do that, though I have reason to believe the gcc *is* terminating and _spawn_posix isn't detecting the end of the process. At the very least we should get another test out of this dreadfully irritating bug. Thanks again for looking at this. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading -------------
Steve Holden wrote:
Martin v. Löwis wrote:
The only environment variables that don't appear in the shell output from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed. At this point, I'd recommend to perform a cygwin update; with Cygwin, these problems often go away with an update.
I updated Cygwin and did a rebaseall before posting.
If that doesn't help, you can ask on the Cygwin list also; to analyse this further, ISTM one will need to debug the internals of cygwin.
I posted on Cygwin before asking here.
One thing you could try is to add -v to the list of gcc options; you can then see whether gcc is progressing correctly.
I'll do that, though I have reason to believe the gcc *is* terminating and _spawn_posix isn't detecting the end of the process. At the very least we should get another test out of this dreadfully irritating bug.
Thanks again for looking at this.
A further data point (sorry, I've not had a lot of time to look at this). Building Python from the trunk at rev 57076 yielded a system that *would* cleanly build the same release of PIL. This is somewhat annoying because it appears to put the bug in the past where there's less incentive to investigate it. But I may find time to take a look. Hardly worth it if I'm the only one noticing the issue, though. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden --------------- Asciimercial ------------------ Get on the web: Blog, lens and tag the Internet Many services currently offer free registration ----------- Thank You for Reading -------------
participants (2)
-
"Martin v. Löwis" -
Steve Holden