subprocess insufficiently platform-independent?
Several people at Google seem to have independently discovered that despite all of the platform-independent goodness in subprocess.py, you still need to be platform aware. One of my colleagues summarized it like this: """ Given a straightforward command list like: cmd = ['svn', 'ls', 'http://rietveld.googlecode.com/svn/trunk'] You apparently cannot pass this list to any subprocess function (subprocess.call() or otherwise) with a set of arguments that allow it to "just work" on both Windows and non-Windows systems. If you call: subprocess.call(cmd, shell=False) Then it works on Linux, but fails on Windows because it does not perform the Windows %PATHEXT% search that allows it to find that "svn.exe" is the actual executable to be invoked. If you call: subprocess.call(cmd, shell=True) Then it works on Windows (it finds the "svn.exe" executable), but it fails on Linux because it *only* executes the first argument in the list ("svn") and does not pass the remaining arguments in the list to the "svn" invocation. This forces you to code platform-dependent behavior in how you call subprocess.call() when using a list. For example, you can make the shell= argument depend on the platform you're running on, like: subprocess.call(cmd, shell=(sys.platform=='win32')) Or you could vary cmd[0] in your list based on the platform: if sys.platform == 'win32': svn = 'svn.exe' else: svn = 'svn' cmd = [svn, 'ls', 'http://rietveld.googlecode.com/svn/trunk'] But either way, this seems clearly broken (or at least sub-optimal) for a module that's supposed to abstract platform-specific execution issues from the user. """ -- --Guido van Rossum (home page: http://www.python.org/~guido/)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 25, 2008, at 1:13 PM, Guido van Rossum wrote:
Unless I'm misremembering (I no longer have access to Windows), I believe that if you use ' '.join(cmd) as the first argument, it will work cross-platform. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSLLsU3EjvBPtnXfVAQJapQP+N4HY0I/uczbdQKB1bi6OV0BZgj5JS8an Tz4FEnaD9LDegTnV8fqAx5/blIidZEdPjVkdmW4m4bz8tRNIEdoZyghHUmKycgRj d65FU0e1tL40u0AoKl3ARO6WWkKKhaqn4R17065lh+V1ZNKutu2btiAso6VfWVW5 V7hvo/61ACM= =P0Nb -----END PGP SIGNATURE-----
On Mon, Aug 25, 2008 at 01:30:58PM -0400, Barry Warsaw wrote:
What about arguments that contain spaces? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 25, 2008, at 1:33 PM, Oleg Broytmann wrote:
Oh sure, throw a monkey wrench into it <wink>. Guido's original problem statement is still correct though. It's advertised to take a sequence of arguments, so that should work. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSLLtr3EjvBPtnXfVAQKJpgP/TPmVgsRdRiAE+jvZOCl9unHWU6LtLSx/ uU3gIkQfvPYcyv9oUS0mcTwWsRDCeP42foQxP+MgX2Zx1ItQPi8/QDbW2bS809pF q8igObharc0kSokhTw2zrkNXsEx3S+epPJXaiueY2cs9jC4mCvXnlnt67ZtWEc8r r+eNLlI2/eo= =SfRl -----END PGP SIGNATURE-----
On Mon, Aug 25, 2008 at 10:30 AM, Barry Warsaw <barry@python.org> wrote:
That will mean something different if there are spaces or shell metacharacters in the arguments. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, Aug 25, 2008 at 11:30 AM, Barry Warsaw <barry@python.org> wrote:
FWIW, I've also struggled with similar issues. For example, with no shell= argument, Windows typically opens up an extra window to run the command, while Unix doesn't. My most recent hack around this looked something like:: try: import win32con except ImportError: win32con = None kwargs = dict(...) if win32con is not None: kwargs['creationflags'] = win32con.CREATE_NO_WINDOW subprocess.Popen(cmd, **kwargs) I'd love to see subprocess become more consistent cross-platform. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy
On Mon, 25 Aug 2008 10:13:32 -0700, Guido van Rossum <guido@python.org> wrote:
Launching child processes on Windows is actually pretty hard to do. The idea of an array of arguments is more like a group hallucination than a reality there. If you assume certain things about how the launched process will interpret its command line (something each program gets to decide for itself on Windows), and many programs do actually use the same rules, then you can get something together that mostly works and is mostly general. Twisted's process support does this and presents a cross-platform API (at least as far as arguments are concerned). Perhaps the subprocess module should borrow that implementation? http://twistedmatrix.com/trac/browser/trunk/twisted/python/win32.py#L66 This doesn't handle all possible inputs correctly (you can read about its flaws in some detail at <http://twistedmatrix.com/trac/ticket/1123>) but it does handle *most* inputs. It also isn't appropriate for all programs, but for the programs which don't follow these quoting rules, it's probably a mistake to have a list-based API anyway: they should be invoked using one string giving the entire command line. Jean-Paul
On Aug 25, 2008, at 9:52 PM, Greg Ewing wrote:
+1 to that. It's really nice to be able to *not* invoke a shell, and thus not have to worry about the shell doing nasty things to your process spawning. So, any solution that aids portability to windows without requiring the invocation of a shell seems like a good thing to me. James
Guido quotes a colleague:
I can't reproduce this as described.
The reason this works is that Windows itself (CreateProcess) has support both for implied '.exe' extensions and searching $PATH. Thus, PATHEXT handling isn't required for executables. I *can* reproduce with another extension - eg, 'svn.py' - for this test I had a 'foo.py' on my PATH (but not in the cwd), and .py in PATHEXT.
So I can't see this problem for 'svn.exe' and at face value the behaviour on Windows looks quite correct - you *do* need the shell for this functionality on Windows (in the same way you would to execute a .bat file, for example)
It sounds like in this particular example at least, this behaviour on Linux is the problem? Mark
On Tue, Aug 26, 2008 at 8:08 PM, Mark Hammond <mhammond@skippinet.com.au> wrote:
Which Windows version? This sounds like one of those things that could well vary by Windows version; if it works for you in Vista it may well be broken in XP. It could also vary by other setup parameters besides PATHEXT.
I think so yes. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, Aug 27, 2008 at 9:43 AM, Guido van Rossum <guido@python.org> wrote:
When passing the executable name to CreateProcess via the lpCommandLine parameter, PATH is considered but PATHEXT is ignored. The only extension that's automatically appended is ".exe", and only if no other extension is present. This has been true for as long as I can remember. I've found the documentation for CreateProcess (http://msdn.microsoft.com/en-us/library/ms682425.aspx) to be pretty reliable. And the mention of a ".com" in the docs suggests that the description has been around for a while... -- Curt Hagenlocher curt@hagenlocher.org
Curt Hagenlocher wrote:
And I just described it as pretty vague ;-) Reading it again, I realize that I completely missed the part that says: "If the file name does not contain an extension, .exe is appended. Therefore, if the file name extension is .com, this parameter must include the .com extension. If the file name ends in a period (.) with no extension, or if the file name contains a path, .exe is not appended." </F>
Guido van Rossum wrote:
It works the same way on XP, at least:
According to the MS docs, CreateProcess works the same way on at least 2K, XP and Vista. The documentation is a bit vague (as usual), but it contains an example that implies that CreateProcess always adds ".exe" if not already there, and that you need to use the command interpreter (that is, shell=True) if you want to run something that's not a Windows executable module (e.g. a batch file). </F>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 25, 2008, at 1:13 PM, Guido van Rossum wrote:
Unless I'm misremembering (I no longer have access to Windows), I believe that if you use ' '.join(cmd) as the first argument, it will work cross-platform. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSLLsU3EjvBPtnXfVAQJapQP+N4HY0I/uczbdQKB1bi6OV0BZgj5JS8an Tz4FEnaD9LDegTnV8fqAx5/blIidZEdPjVkdmW4m4bz8tRNIEdoZyghHUmKycgRj d65FU0e1tL40u0AoKl3ARO6WWkKKhaqn4R17065lh+V1ZNKutu2btiAso6VfWVW5 V7hvo/61ACM= =P0Nb -----END PGP SIGNATURE-----
On Mon, Aug 25, 2008 at 01:30:58PM -0400, Barry Warsaw wrote:
What about arguments that contain spaces? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 25, 2008, at 1:33 PM, Oleg Broytmann wrote:
Oh sure, throw a monkey wrench into it <wink>. Guido's original problem statement is still correct though. It's advertised to take a sequence of arguments, so that should work. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSLLtr3EjvBPtnXfVAQKJpgP/TPmVgsRdRiAE+jvZOCl9unHWU6LtLSx/ uU3gIkQfvPYcyv9oUS0mcTwWsRDCeP42foQxP+MgX2Zx1ItQPi8/QDbW2bS809pF q8igObharc0kSokhTw2zrkNXsEx3S+epPJXaiueY2cs9jC4mCvXnlnt67ZtWEc8r r+eNLlI2/eo= =SfRl -----END PGP SIGNATURE-----
On Mon, Aug 25, 2008 at 10:30 AM, Barry Warsaw <barry@python.org> wrote:
That will mean something different if there are spaces or shell metacharacters in the arguments. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, Aug 25, 2008 at 11:30 AM, Barry Warsaw <barry@python.org> wrote:
FWIW, I've also struggled with similar issues. For example, with no shell= argument, Windows typically opens up an extra window to run the command, while Unix doesn't. My most recent hack around this looked something like:: try: import win32con except ImportError: win32con = None kwargs = dict(...) if win32con is not None: kwargs['creationflags'] = win32con.CREATE_NO_WINDOW subprocess.Popen(cmd, **kwargs) I'd love to see subprocess become more consistent cross-platform. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy
On Mon, 25 Aug 2008 10:13:32 -0700, Guido van Rossum <guido@python.org> wrote:
Launching child processes on Windows is actually pretty hard to do. The idea of an array of arguments is more like a group hallucination than a reality there. If you assume certain things about how the launched process will interpret its command line (something each program gets to decide for itself on Windows), and many programs do actually use the same rules, then you can get something together that mostly works and is mostly general. Twisted's process support does this and presents a cross-platform API (at least as far as arguments are concerned). Perhaps the subprocess module should borrow that implementation? http://twistedmatrix.com/trac/browser/trunk/twisted/python/win32.py#L66 This doesn't handle all possible inputs correctly (you can read about its flaws in some detail at <http://twistedmatrix.com/trac/ticket/1123>) but it does handle *most* inputs. It also isn't appropriate for all programs, but for the programs which don't follow these quoting rules, it's probably a mistake to have a list-based API anyway: they should be invoked using one string giving the entire command line. Jean-Paul
On Aug 25, 2008, at 9:52 PM, Greg Ewing wrote:
+1 to that. It's really nice to be able to *not* invoke a shell, and thus not have to worry about the shell doing nasty things to your process spawning. So, any solution that aids portability to windows without requiring the invocation of a shell seems like a good thing to me. James
Guido quotes a colleague:
I can't reproduce this as described.
The reason this works is that Windows itself (CreateProcess) has support both for implied '.exe' extensions and searching $PATH. Thus, PATHEXT handling isn't required for executables. I *can* reproduce with another extension - eg, 'svn.py' - for this test I had a 'foo.py' on my PATH (but not in the cwd), and .py in PATHEXT.
So I can't see this problem for 'svn.exe' and at face value the behaviour on Windows looks quite correct - you *do* need the shell for this functionality on Windows (in the same way you would to execute a .bat file, for example)
It sounds like in this particular example at least, this behaviour on Linux is the problem? Mark
On Tue, Aug 26, 2008 at 8:08 PM, Mark Hammond <mhammond@skippinet.com.au> wrote:
Which Windows version? This sounds like one of those things that could well vary by Windows version; if it works for you in Vista it may well be broken in XP. It could also vary by other setup parameters besides PATHEXT.
I think so yes. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, Aug 27, 2008 at 9:43 AM, Guido van Rossum <guido@python.org> wrote:
When passing the executable name to CreateProcess via the lpCommandLine parameter, PATH is considered but PATHEXT is ignored. The only extension that's automatically appended is ".exe", and only if no other extension is present. This has been true for as long as I can remember. I've found the documentation for CreateProcess (http://msdn.microsoft.com/en-us/library/ms682425.aspx) to be pretty reliable. And the mention of a ".com" in the docs suggests that the description has been around for a while... -- Curt Hagenlocher curt@hagenlocher.org
Curt Hagenlocher wrote:
And I just described it as pretty vague ;-) Reading it again, I realize that I completely missed the part that says: "If the file name does not contain an extension, .exe is appended. Therefore, if the file name extension is .com, this parameter must include the .com extension. If the file name ends in a period (.) with no extension, or if the file name contains a path, .exe is not appended." </F>
Guido van Rossum wrote:
It works the same way on XP, at least:
According to the MS docs, CreateProcess works the same way on at least 2K, XP and Vista. The documentation is a bit vague (as usual), but it contains an example that implies that CreateProcess always adds ".exe" if not already there, and that you need to use the command interpreter (that is, shell=True) if you want to run something that's not a Windows executable module (e.g. a batch file). </F>
participants (11)
-
Barry Warsaw -
Curt Hagenlocher -
Fredrik Lundh -
Greg Ewing -
Guido van Rossum -
James Y Knight -
Jean-Paul Calderone -
Mark Hammond -
Oleg Broytmann -
Robert Brewer -
Steven Bethard