[issue20927] Different behaviour on Posix and Windows when using subprocess.Popen(..., cwd=path)
New submission from Jovik: This works on Linux as expected: import subprocess proc = subprocess.Popen(["./app"], stdout=subprocess.PIPE, cwd="workspace") but on Windows I get: FileNotFoundError: [WinError 2] The system cannot find the file specified To successfully execute it on Windows I need to set shell=True first: proc = subprocess.Popen(["app.exe"], stdout=subprocess.PIPE, cwd="workspace", shell=True) which is odd since by default it should use cwd when searching for binary: "If cwd is not None, the function changes the working directory to cwd before executing the child. In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." from http://docs.python.org/3.3/library/subprocess.html ---------- assignee: docs@python components: Documentation messages: 213570 nosy: Jovik, docs@python priority: normal severity: normal status: open title: Different behaviour on Posix and Windows when using subprocess.Popen(..., cwd=path) type: behavior versions: Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
R. David Murray added the comment: Your cwd is relative. What happens if you make it absolute? (What I'm thinking is that the non-shell starting cwd may be different on windows than it is on unix...but I don't know windows very well, so this may be irrelevant...) ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Changes by Eric V. Smith <eric@trueblade.com>: ---------- nosy: +eric.smith _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: I did a test with cwd being set to full path, but the result was the same (still had to use shell=True to execute a command). Let me know if I can provide any more details. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Terry J. Reedy added the comment: I am sure that using / instead of \, which is to say, not using os.sep, is the problem as / is *not* allowed in Windows command names even though the substitution works for paths given as options. In a Windows console,
python # works .\python # works ./python # see /python as an option for the . program, which does not exist.
I suspect that shell=True cause subprocess to execute "shell ./app" (howver 'shell' is spelled on the system), with whatever other options and quotation are needed to make ./app work as an option passed to shell instead of a command to be executed directly. I also suspect that passing ".\\app" might work, which would mean that you should use .%sapp" % os.sep to get a cross-platform string. If so, please close this issue. ---------- nosy: +terry.reedy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: I'm quite aware of the os.sep issues between the systems, but I checked both out of curiosity. Here are latest results: All of the following commands raises the same exception:
proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace\\") proc = subprocess.Popen(".\plink", stdout=subprocess.PIPE, cwd="c:\python33\workspace") proc = subprocess.Popen(".\\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:/python33/workspace")
Traceback (most recent call last): File "C:\Python33\lib\subprocess.py", line 1104, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python33\lib\subprocess.py", line 819, in __init__ restore_signals, start_new_session) File "C:\Python33\lib\subprocess.py", line 1110, in _execute_child raise WindowsError(*e.args) FileNotFoundError: [WinError 2] The system cannot find the file specified But, when I set shell=True, then everything works just fine:
proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:\python33\workspace", shell=True) proc = subprocess.Popen(".\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace", shell=True) proc = subprocess.Popen(".\\plink", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace", shell=True) proc = subprocess.Popen("plink", stdout=subprocess.PIPE, cwd="c:/python33/workspace", shell=True) I can get plink's output afterwards with proc.communicate()
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: In the first example, you switch from "./app" to "app.exe" when using shell=True. What happens to any of your examples if you add ".exe" without shell=True? Popen eventually calls CreateProcess on Windows. From: http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).as..., describing the lpApplicationName parameter: "This parameter must include the file name extension; no default extension is assumed." Running the shell though, you don't need the extension. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: Assuming this is the problem, we should at least document this. It does make cross-platform coding difficult. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Changes by Kathleen Weaver <kathleen@kweaver.org>: ---------- nosy: +kathweaver _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: As requested I did extra tests with extension. Same result as before:
proc = subprocess.Popen("plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") proc = subprocess.Popen(".\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") proc = subprocess.Popen(".\\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") proc = subprocess.Popen(".\plink.exe", stdout=subprocess.PIPE, cwd="c:/python33/workspace") proc = subprocess.Popen("plink.exe", stdout=subprocess.PIPE, cwd="c:/python33/workspace")
Traceback (most recent call last): File "C:\Python33\lib\subprocess.py", line 1104, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python33\lib\subprocess.py", line 819, in __init__ restore_signals, start_new_session) File "C:\Python33\lib\subprocess.py", line 1110, in _execute_child raise WindowsError(*e.args) FileNotFoundError: [WinError 2] The system cannot find the file specified I believe it's a wider issue, since my colleagues insisted on using shell=True on Windows by default (I didn't understand why, until now) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: Where is plink.exe? If it's not in cwd (c:\python33\workspace), note that the documentation for cwd says: "If cwd is not None, the function changes the working directory to cwd before executing the child. In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." Although confusingly, the 2.7 documentation says: "If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd." ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: plink.exe is located in c:\python33\workspace. I'm aware of the docstring difference between Python 2 and 3, thus submitted the bug for Python 3 only. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: I think the 2.7 documentation is correct: the current directory when the call is made, and not cwd, is included in the search path. I'd suggest specifying args as ["c:\\python33\\workspace\\plink.exe"]. I think I may have mislead you earlier on the search path. The Windows call is: CreateProcess(lpApplicationName, lpCommandLine, <other stuff>, lpCurrentDirectory, <other stuff>) The first parameter to Popen ("args") becomes lpCommandLine. The "executable" parameter to Popen, which you're not setting, becomes lpApplicationName. So, you're calling CreateProcess(NULL, "plink.exe, ..., lpCurrentDirectory="c:\\python33\\workspace"). In this case, .exe would be added if missing. But, the search path rules seem to not include the directory pointed to by lpCurrentDirectory (aka cwd). So I think this would work: subprocess.Popen(["c:\\python33\\workspace\\plink.exe"], stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") or subprocess.Popen(["plink"], executable="c:\\python33\\workspace\\plink.exe", stdout=subprocess.PIPE, cwd="c:\\python33\\workspace") Unfortunately, I don't have a copy of 3.x on my Windows machine to test with. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: Why this feature works on Posix systems, but not Windows? If my memory is correct, it didn't work anywhere when I used Python 2.7 (according with docs). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: The underlying APIs are very different. It's debatable how much of a shim we should provide to make all platforms look alike. I think first we should understand what it currently takes to make something work in both environments. Then we can talk about how or if we can make them look more similar. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: Isn't Python's crossplatform functionality a key feature? A quick fix would be to retry the call by adding cwd to arg[0] in case of FileNotFoundError. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Eric V. Smith added the comment: We don't always provide fully cross-platform functionality (see the os module for many examples), but we might be able to do better here. It might be some functionality we can add, it might be a documentation issue. Note, for example, the subprocess.STARTUPINFO and subprocess.Popen creationflags argument. These expose Windows-only functionality. I'm opposed to trying again with cwd added. There's a long history of security problems doing exactly this. It's why '.' is not in the PATH by default on Unix. On my list of things to do is trace through exactly which scenarios work and don't work on Windows. If you really want a more cross-platform solution, Cygwin python might work for you. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Jovik added the comment: I appreciate your suggestion regarding cygwin, but in the new code base we want to avoid this dependency. Thanks for your time on this issue. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> subprocess.Popen(cwd) documentation _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue20927> _______________________________________
participants (6)
-
Eric V. Smith -
Jovik
-
Kathleen Weaver
-
Martin Panter
-
R. David Murray
-
Terry J. Reedy