Chris Jerdonek added the comment: Attached are a few test cases showing that Popen *does* consider cwd when searching for the executable (as well as for args[0]), and in particular that you *can* specify the program's path relative to cwd. I also moved the test_cwd test to be adjacent to the other cwd test (the one that tests cwd with the executable argument). I can also prepare the documentation changes for addition to the patch. ---------- keywords: +patch Added file: http://bugs.python.org/file27016/issue-15533-test-cases-1.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Here is a full patch for the default branch (documentation correction and test cases for the documented behavior). If this patch looks acceptable, I can prepare a separate patch for 2.7. ---------- Added file: http://bugs.python.org/file27018/issue-15533-2-default.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: python_dir = os.path.dirname(os.path.realpath(sys.executable)) wrong_cwd = os.path.join(python_dir, 'Doc') Actually, is there a better directory to be using for this? I'd like a directory that is guaranteed to exist that is in the same directory as sys.executable -- so that I can construct a simple relative path from that directory to sys.executable. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Here is a new patch that makes no assumptions about the contents of the directory containing sys.executable. ---------- Added file: http://bugs.python.org/file27026/issue-15533-3-default.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Andrew Svetlov added the comment: Update patch. ---------- Added file: http://bugs.python.org/file27112/issue-15533-4-default.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Andrew Svetlov added the comment: I believe it's trick for Tru64 platform. I've asked to support of this in python-dev. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Andrew, I seem to be getting a test failure for test_executable_with_cwd() with your updated patch (the child process is outputting an absolute path rather than ''). I will update the patch to fix. There are also some stylistic changes I would like to make to the helper method (updated code comment, etc). ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Updating the patch again to tweak the original documentation change. I was concerned that the previous language could be construed to mean that Popen will look in *two* places for the executable (both relative to the current directory and relative to the cwd argument). The change I'm uploading makes this a little more clear. ---------- Added file: http://bugs.python.org/file27116/issue-15533-6-default.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Here is a proposed patch that attempts to minimize the chance of test breakage for Tru64. The patch follows Martin's recommendation on python-dev of being cautious by following existing code. ---------- Added file: http://bugs.python.org/file27123/issue-15533-8-default.patch _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Andrew Svetlov added the comment: Chris, please commit your patch. It's fine for me. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Roundup Robot added the comment: New changeset abfaa4368263 by Chris Jerdonek in branch '3.2': Issue #15533: Clarify docs and add tests for subprocess.Popen()'s cwd argument. http://hg.python.org/cpython/rev/abfaa4368263 New changeset f66ff96f0030 by Chris Jerdonek in branch '3.3': Issue #15533: Merge fix from 3.2. http://hg.python.org/cpython/rev/f66ff96f0030 New changeset 37f4aa15a1c6 by Chris Jerdonek in branch 'default': Issue #15533: Merge fix from 3.3. http://hg.python.org/cpython/rev/37f4aa15a1c6 ---------- nosy: +python-dev _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Chris Jerdonek added the comment: Two of the tests fail on at least some of the Windows bots. I am investigating. ====================================================================== ERROR: test_cwd_with_relative_arg (test.test_subprocess.ProcessTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 222, in test_cwd_with_relative_arg self._assert_cwd(python_dir, rel_python, cwd=python_dir) File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 195, in _assert_cwd **kwargs) File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 745, in __init__ restore_signals, start_new_session) File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 964, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified ====================================================================== ERROR: test_cwd_with_relative_executable (test.test_subprocess.ProcessTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 240, in test_cwd_with_relative_executable cwd=python_dir) File "D:\Buildslave\3.2.moore-windows\build\lib\test\test_subprocess.py", line 195, in _assert_cwd **kwargs) File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 745, in __init__ restore_signals, start_new_session) File "D:\Buildslave\3.2.moore-windows\build\lib\subprocess.py", line 964, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.2/builds/... ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Roundup Robot added the comment: New changeset d8d52b5b4bc2 by Chris Jerdonek in branch '3.2': Issue #15533: Skip test_cwd_with_relative_*() tests on Windows pending resolution of issue. http://hg.python.org/cpython/rev/d8d52b5b4bc2 New changeset 17d709f0b69b by Chris Jerdonek in branch '3.3': Issue #15533: Merge update from 3.2. http://hg.python.org/cpython/rev/17d709f0b69b New changeset d10a7c1ac3a7 by Chris Jerdonek in branch 'default': Issue #15533: Merge update from 3.3. http://hg.python.org/cpython/rev/d10a7c1ac3a7 ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Jan Lachnitt added the comment: Hi, I am using Python 3.2.3 on Windows XP. I encountered a problem with subprocess.call when using the cwd parameter. I used Google to look for a solution and I found this issue in Python tracker. But this issue seems absolutely reversed! The subprocess documentation says this: "In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." But this is NOT true. If I use the path relative to cwd, I get Windows Error 2 (system cannot find the file). If I change the executable's path to be relative to the current directory, where the script is running (i.e. NOT the cwd parameter passed to subprocess.call), it works fine. ---------- nosy: +pepalogik _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Jan Lachnitt added the comment: EDIT: No, it doesn't work fine: the executable reports stack overflow. I thought this had nothing to do with Python, hence I didn't mention it. But If I run the executable without Python, THEN it works fine. But this may be another issue. I'll update to 3.3.0 and then tell you if this has changed. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Jan Lachnitt added the comment: Hi all, I have solved the problem by using absolute path of the executable. The reason why the executable didn't work properly may be that the executable's relative path was inconsistent with current directory. See the following example (I have made an executable which shows its argv and cwd). If it is called normally, then: argv[0] = phsh0.exe cwd = D:\Jenda\AutoLEED\TESTING\default But if it is called by Python's subprocess.call from "D:\Jenda\AutoLEED\TESTING" as I want, then: argv[0] = default\phsh0.exe cwd = D:\Jenda\AutoLEED\TESTING\default The executable may be confused by this inconsistency. So it is not the documentation, but Python itself what should be changed. The executable should be searched in cwd on any platform to avoid the inconsistency. I have not yet updated my Python installation, so my results apply to 3.2.3. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Wolfgang Maier added the comment: Ping. This still isn't fixed several years later, i.e., the documentation still describes the POSIX, but not the Windows behavior. See also issue20927, which reports this as a bug. ---------- nosy: +wolma versions: +Python 3.5, Python 3.6, Python 3.7 -Python 2.7, Python 3.2, Python 3.3 _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Changes by Ned Deily : ---------- nosy: +paul.moore, steve.dower, tim.golden, zach.ware versions: -Python 3.4 _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Jan Lachnitt added the comment: Thank Wolfgang Maier for reminding this issue and providing various details and observations. Having taken a look at my old comments (and at others' comments, too), I feel that the cwd issue deserves a clearer description. Let's use the following simple C program as the callee: #include #include int main(int argc, char* argv[]) { char cwd[FILENAME_MAX+1]; for (int i = 0; i < argc; ++i) printf("argv[%d] = %s\n", i, argv[i]); getcwd(cwd, FILENAME_MAX); printf("cwd = %s\n", cwd); return 0; } As is evident, this program merely prints its arguments and working directory. I have built it using gcc, called it "print_argv+cwd", and placed it in the "subdir" subdirectory of the current directory. Next, let's use the following Python 3 script for testing: import os from subprocess import run # substitute run->call in Python < 3.5 prg_name = 'print_argv+cwd' if os.name == 'nt': prg_name += '.exe' else: prg_name = os.path.join('.',prg_name) dir_name = 'subdir' def execute(path, cwd): print('Executing "{}" in "{}":'.format(path,cwd)) try: run([path], cwd=cwd) # substitute run->call in Python < 3.5 except Exception as err: print(type(err).__qualname__+':', err) print('Script\'s cwd =', os.getcwd()) execute(prg_name, dir_name) execute(os.path.join(dir_name,prg_name), dir_name) execute(os.path.abspath(os.path.join(dir_name,prg_name)), dir_name) Output on Linux with Python 3.5.2: Script's cwd = /home/jenda/Bug reports/Python/subprocess Executing "./print_argv+cwd" in "subdir": argv[0] = ./print_argv+cwd cwd = /home/jenda/Bug reports/Python/subprocess/subdir Executing "subdir/./print_argv+cwd" in "subdir": FileNotFoundError: [Errno 2] No such file or directory: 'subdir/./print_argv+cwd' Executing "/home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd" in "subdir": argv[0] = /home/jenda/Bug reports/Python/subprocess/subdir/print_argv+cwd cwd = /home/jenda/Bug reports/Python/subprocess/subdir Output on Windows with Python 3.5.2: Script's cwd = C:\Users\Jenda\Bug reports\Python\subprocess Executing "print_argv+cwd.exe" in "subdir": FileNotFoundError: [WinError 2] Systém nemůže nalézt uvedený soubor Executing "subdir\print_argv+cwd.exe" in "subdir": argv[0] = subdir\print_argv+cwd.exe cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir Executing "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\print_argv+cwd.exe" in "subdir": argv[0] = C:\Users\Jenda\Bug reports\Python\subprocess\subdir\print_argv+cwd.exe cwd = C:\Users\Jenda\Bug reports\Python\subprocess\subdir Summary: On Linux, subprocess.run (or call or Popen) behaves correctly, in accordance with current documentation. On Windows, both possible relative paths produce incorrect results. With the first one, relative to "subdir", Python fails to find the executable. With the other one, relative to the script's cwd, Python actually executes the program, but argv[0] is inconsistent with cwd. Imagine that the called program wants to resolve its own path: It joins cwd and argv[0] and gets "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe", which is an incorrect (and nonexistent) path. This is why the cwd issue is not just a documentation issue. The only option working correctly on Windows is the last one, using absolute path of the executable. ---------- _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Eryk Sun added the comment: The Unix implementation of subprocess.Popen follows the behavior of os.execvpe, which is an outlier. Other execvpe implementations, such as the one added to glibc in 2009, search PATH in the current environment instead of the passed environment. As such, and given the natural expectations of a Windows programmer, I do not see the current behavior of the Windows implementation as incorrect. It's a documentation bug. On a related note, the Popen documentation for Windows should also mention that defining the environment variable NoDefaultCurrentDirectoryInExePath removes the current directory from the executable search path, in both CreateProcess and cmd.exe (i.e. w/ shell=True). This feature was introduced in Windows Vista, so it applies to Python 3.5+.

Python actually executes the program, but argv[0] is inconsistent with cwd. Imagine that the called program wants to resolve its own path: It joins cwd and argv[0] and gets "C:\Users\Jenda\Bug reports\Python\subprocess\subdir\subdir\print_argv+cwd.exe"

A Windows program would call GetModuleFileName with hModule as NULL, which returns the path of the process executable. There's also the pseudo-environment variable __APPDIR__. Using argv[0] from the command line would be unreliable. For example: >>> _ = run('"spam & eggs" /c echo %__APPDIR__%', ... executable=os.environ['ComSpec']) C:\Windows\system32\ >>> _ = run('"spam & eggs" -m calendar 2017 2', ... executable=sys.executable) February 2017 Mo Tu We Th Fr Sa Su 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ---------- nosy: +eryksun _______________________________________ Python tracker http://bugs.python.org/issue15533 _______________________________________

Damon Atkins added the comment: See also https://bugs.python.org/msg262399 ---------- _______________________________________ Python tracker https://bugs.python.org/issue15533 _______________________________________

Damon Atkins added the comment:

From https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425.aspx

Note Python is using CreateProcess(), consider using CreateProcessW() The Unicode version of this function, CreateProcessW, can modify the contents of this string. Therefore, this parameter cannot be a pointer to read-only memory (such as a const variable or a literal string). If this parameter is a constant string, the function may cause an access violation. ---------- _______________________________________ Python tracker https://bugs.python.org/issue15533 _______________________________________

Gregory P. Smith added the comment: Thanks for pointing me at this issue Ned. It sounds like there is a behavior difference between Windows and POSIX systems related to the current directory and/or which process environment is used by the system call that launches the new process to find the executable. It seems to have existed "forever" in subprocess module API terms, so I don't know if we should reconcile the corner cases when cwd and/or env are supplied to a single cross platform behavior as that could break existing code. Such a behavior change _could_ be made but be 3.8 specific. BUT: If we did that, it becomes a challenge for people writing code that needs to work on multiple Python versions. Popen growing yet another bool flag parameter to choose the new behavior is possible, but quite ugly (and unusable on multi-python version code). I think we should start out by living with the difference - document these platform specific corner case behaviors to minimize surprise. If we want to provide a way for people to have the same behavior on both, we should document a recommended way to do that. I believe that entails telling people get an absolute path to their executable themselves before launching the subprocess as that should work the same no matter the cwd or environment? ---------- _______________________________________ Python tracker https://bugs.python.org/issue15533 _______________________________________

Change by Chris Jerdonek : ---------- assignee: chris.jerdonek -> _______________________________________ Python tracker https://bugs.python.org/issue15533 _______________________________________

Eryk Sun added the comment:

Python is conceptually multi-platform, so its behavior on Linux and Windows should be as much consistent as possible.

It's not expected for the behavior of all Popen() parameters to be the same on all platforms. For example, the behavior and capabilities of shell=True are different in Windows vs POSIX. But I think a basic set of parameters should have been singled out for cross-platform consistency -- at least in the default case. Unfortunately it wasn't designed that way. New behavior that's consistent with POSIX can be implemented, but at this point it would have to be enabled by a parameter.

"In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path."

For POSIX, this should be stated as a "relative path without a slash in it" or a "relative path without a directory in it". An unqualified filename is a relative path that won't be resolved against `cwd`, unless there's a "." entry in PATH. For Windows, the use of CreateProcess() is documented. It could be stated more explicitly that `executable`, `args` / list2cmdline(args), `env`, and `cwd` are passed directly to CreateProcess() as lpApplicationName, lpCommandLine, lpEnvironment, and lpCurrentDirectory. Here are some notes and corrections about the documentation of lpCommandLine, in particular the following paragraph: If the file name does not contain an extension, .exe is appended. Therefore, if the file name extension is .com, this parameter must include the .com extension. If the file name ends in a period (.) with no extension, or if the file name contains a path, .exe is not appended. If the file name does not contain a directory path, the system searches for the executable file in the following sequence [1][2][3]: * %__APPDIR__% * %__CD__% [4] * %SystemRoot%\System32 * %SystemRoot%\System * %SystemRoot% * %PATH% (machine/user extended search sequence) [1] The search sequence is rewritten here succinctly using environment variables in the current process, including the virtual variables __APPDIR__ (application directory) and __CD__ (current directory), which are supported by WinAPI GetEnvironmentVariableW(). [2] A path name is resolved by searching for it if it isn't fully qualified and doesn't explicitly begin with the current directory (".") or its parent (".."). Note that, unlike POSIX, a relative path name is resolved by searching for it even if it contains a directory component (i.e. a slash or backslash). For example, "spam\eggs.exe" is resolved by looking for r"%__APPDIR__%\spam\eggs.exe" and so on. [3] If a path name has to be resolved by searching, and its final component does not contain a "." character, then ".exe" is appended to the name. On the other hand, if a path name does not need to be resolved by searching, because it's fully qualified or the first component is "." or "..", then if the given path name doesn't exist, it also looks for the name with ".exe" appended, even if the final component of the path name contains a "." character. [4] If "NoDefaultCurrentDirectoryInExePath" is defined in the environment and the path name does not contain a directory component (i.e. no slash or backslash), then the current directory is excluded from the search sequence. That %__APPDIR__% always takes precedence means that subprocess.Popen(['python']) runs the Python version of the current process, regardless of PATH, unless shell=True is used. The implementation of lpApplicationName (executable) and lpCurrentDirectory (cwd) means that argv[0] in the child process, as parsed from its command line, does not necessarily resolve to a name in the filesystem. Windows supports GetModuleFileNameW(NULL, ...) to get the path of the application executable. ---------- versions: +Python 3.10, Python 3.9 -Python 3.6, Python 3.7 _______________________________________ Python tracker https://bugs.python.org/issue15533 _______________________________________