New submission from Steve Barnes:
When running under pythonw, or pyinstaller with the -w flag, modules that use subprocess calls such as popen, run, etc. will crash if the default `stdout=None, stderr=None` behaviour is used rather than PIPE. This is an obscure problem which is very hard to debug yet there is no warning in the documentation on this.
I would like to suggest adding a :warning:`stdout=None, stderr=None` must not be used in any of the calls in this module when running under pythonw due to the lack of sys.stdout & sys.stderr in that case. Please use `stdout=PIPE, stderr=PIPE` instead.
A patch against the default branch would be: diff -r 4243df51fe43 Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst Fri Feb 10 14:19:36 2017 +0100 +++ b/Doc/library/subprocess.rst Thu Mar 16 16:56:24 2017 +0000 @@ -33,6 +33,13 @@ function for all use cases it can handle. For more advanced use cases, the underlying :class:`Popen` interface can be used directly.
+.. warning:: Do not use default parameters on Windows with pythonw. + + As pythonw deletes `sys.stdout` & `sys.stderr` the use of the default + parameters, `stdout=None, stderr=None,`, which defaults to being + `stdout=sys.stdout, stderr=sys.stderr,` may cause unexpected crashes + it is recommended to use `stdout=PIPE, stderr=PIPE,` instead. + The :func:`run` function was added in Python 3.5; if you need to retain compatibility with older versions, see the :ref:`call-function-trio` section.
---------- assignee: docs@python components: Documentation messages: 289722 nosy: Steve Barnes, docs@python priority: normal severity: normal status: open title: Documentation lacks clear warning of subprocess issue with pythonw type: behavior versions: Python 2.7, Python 3.6, Python 3.7
R. David Murray added the comment:
A warning is not appropriate (we reserve those for things that are security related, pretty much). A sentence might be, though. For example, we could change the initial discussion of the run function to say:
This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments. You *must* do this if you run your python program with pythonw on windows, since pythonw closes the standard streams.
Here I'm adding the windows sentence to the existing "This does not capture" sentence.
---------- nosy: +r.david.murray
Eryk Sun added the comment:
An exception (not a crash) is possible if a standard handle is invalid and a Popen call tries to replace one or two of the other standard handles (e.g. stdin is invalid and you pass the argument stdout=PIPE). Note that subprocess uses the standard handles directly, so this problem is unrelated to Python's sys.std* streams. IMO, the presence of an invalid handle in the standard handles should not cause Popen to fail. This is an old problem, and it should have been addressed a long time ago by substituting a handle for \.\NUL.
In Windows 8+ this isn't particular to running pythonw.exe. In older versions such as Windows 7, if you start pythonw.exe from a console application, then the parent's standard handle values are copied to the pythonw.exe child. But these handles are invalid because pythonw.exe doesn't automatically attach to its parent's console. (You have to manually call AttachConsole(-1) or AllocConsole(), but that's complicated because you may need to reset the CRT's standard file descriptors and FILE streams and Pythons sys.std* streams.)
Note that pythonw.exe doesn't delete or close the standard streams. They're normally set to None because, in the default case, the Windows standard handles aren't valid in pythonw.exe, and thus the CRT maps its standard file descriptors (0, 1, and 2) to INVALID_HANDLE_VALUE (i.e. -1 cast as a pointer). For example:
C:\Temp>pyw -2 -c "import msvcrt; open('test.txt', 'w').write(str(msvcrt.get_osfhandle(1)))"
C:\Temp>type test.txt 18446744073709551614
You can override this by starting pythonw.exe with explicit standard handles. For example, the following redirects stderr (2) to stdout (1) and redirects stdout to a pipe:
C:>set "script=import sys; print(sys.executable); print(sys.stdout); print(sys.stderr)"
C:>2>&1 pyw -2 -c "%script%" | more C:\Program Files\Python27\pythonw.exe <open file '<stdout>', mode 'w' at 0x00000000021FB0C0> <open file '<stderr>', mode 'w' at 0x00000000021FB150>
C:>2>&1 pyw -3 -c "%script%" | more C:\Program Files\Python36\pythonw.exe <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'> <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>
---------- components: +Windows nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Eryk Sun email@example.com added the comment:
This problem can be addressed most easily in _get_handles() by combining the case of an invalid standard handle with that of a NULL standard handle, for which a file handle is validated via GetFileType(). However, that leaves a small window for the handle to become invalid before duplicating it. Instead, _get_handles() could deal with a failed _make_inheritable() call by substituting a pipe handle.
---------- versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.6, Python 3.7
Eryk Sun firstname.lastname@example.org added the comment:
I forgot about the much older issue for this problem. I'm closing this issue, because the broken behavior should be fixed, not documented.
---------- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> subprocess failing in GUI applications on Windows