Someone changed test_popen to "quote" the path to python: cmd = '"%s" -c "import sys;print sys.argv" %s' % (sys.executable, cmdline) ^ ^ The double-quote characters above the carets are new. This causes test_popen to fail on Win2K, but not on Win98. The relevant difference appears to be the default shell (cmd.exe on the former, command.com on the latter). Simplifed example, on Win2K:
p = os.popen('python -c "print 666"') p.read() '666\n' p.close()
Worked fine, but doesn't if python is quoted:
p = os.popen('"python" -c "print 666"') p.read() '' p.close() 1
The same kind of behavior can be observed directly from a DOS-box prompt: C:\Code\python\PCbuild>cmd /c python -c "print 666" 666 C:\Code\python\PCbuild> Worked fine, but quoting the program name flops: C:\Code\python\PCbuild>cmd /c "python" -c "print 666" 'python" -c "print' is not recognized as an internal or external command, operable program or batch file. C:\Code\python\PCbuild> So it looks like it stripped off the first and last double-quote characters, leaving two senseless double-quote characters "in the middle".
Tim Peters wrote:
Someone changed test_popen to "quote" the path to python:
cmd = '"%s" -c "import sys;print sys.argv" %s' % (sys.executable, cmdline) ^ ^
The double-quote characters above the carets are new.
Having to quote arguments to popen and system is a pet peave of mine. 99% of the time I don't not want or need the shell. Is it possible to write versions of system() and popen() that do not use the shell on Windows? I know it's possible on Unix systems. It would be really nice if both popen() and system() could take a sequence for the command and arguments in addition to a string. Neil
[Neil Schemenauer]
Having to quote arguments to popen and system is a pet peave of mine. 99% of the time I don't not want or need the shell. Is it possible to write versions of system() and popen() that do not use the shell on Windows? I know it's possible on Unix systems. It would be really nice if both popen() and system() could take a sequence for the command and arguments in addition to a string.
Those would be quite different functions, then, unless you proposed to have Python interpret native shell metacharacters on its own too (e.g., set up pipes, do the indicated file redirections, interpolate envars, and fake whatever other shell gimmicks people may use). The spawn family of functions take a list of arguments and are sometimes more convenient. IIRC, though, on Windows the MS spawn implementation pastes them back into a cmdline, and then you get some *really* bizarre quoting problems. I always thought Tcl's "exec" cmd was worthy of stealing. That defines a sh-like syntax for specifying OS commands, but arranges to interpret them the same way on all platforms (so, e.g, "2>&1" redirects stderr to stdout even on Win95; last I looked, there were thousands of lines in the Tcl implementation devoted to making this command work).
Those would be quite different functions, then, unless you proposed to have Python interpret native shell metacharacters on its own too (e.g., set up pipes, do the indicated file redirections, interpolate envars, and fake whatever other shell gimmicks people may use).
What we need is a function which does all those things, but uses some way of specifying them *other* than shell metacharacters. E.g. os.plumb(("sed", "-e", "s/dead/resting/", "parrots"), ("grep", "norwegian"), output = myfile)) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[attribution lost]
Those would be quite different functions, then, unless you proposed to have Python interpret native shell metacharacters on its own too (e.g., set up pipes, do the indicated file redirections, interpolate envars, and fake whatever other shell gimmicks people may use).
On Mon, Mar 10, 2003 at 11:52:39AM +1300, Greg Ewing wrote:
What we need is a function which does all those things, but uses some way of specifying them *other* than shell metacharacters. E.g.
os.plumb(("sed", "-e", "s/dead/resting/", "parrots"), ("grep", "norwegian"), output = myfile))
+1 on the concept. +1 on something that can be transformed to use tcl's "exec" so that it'll begin working on several common arches immediately. Jeff
[Greg Ewing]
What we need is a function which does all those things, but uses some way of specifying them *other* than shell metacharacters. E.g.
os.plumb(("sed", "-e", "s/dead/resting/", "parrots"), ("grep", "norwegian"), output = myfile))
[Jeff Epler]
+1 on the concept. +1 on something that can be transformed to use tcl's "exec" so that it'll begin working on several common arches immediately.
They're really the same thing -- Tcl's exec would be a simple transformation of a cross-platform sh-like syntax into Greg's hypothesized functions. The pain in Tcl's exec implementation was in providing the functionality across platforms, not in parsing the sh-like syntax. Then again, Tcl was trying to run all the way back to Windows 3.1, and Python already gave up on that.
Tim Peters <tim.one@comcast.net> writes:
[Greg Ewing]
What we need is a function which does all those things, but uses some way of specifying them *other* than shell metacharacters. E.g.
os.plumb(("sed", "-e", "s/dead/resting/", "parrots"), ("grep", "norwegian"), output = myfile))
[Jeff Epler]
+1 on the concept. +1 on something that can be transformed to use tcl's "exec" so that it'll begin working on several common arches immediately.
They're really the same thing -- Tcl's exec would be a simple transformation of a cross-platform sh-like syntax into Greg's hypothesized functions.
I think Jeff was suggesting that we implement it like this: def plumb(cmd): import Tkinter return Tkinter.call('exec ' + cmd) or whatever. Cheers, M. -- at any rate, I'm satisfied that not only do they know which end of the pointy thing to hold, but where to poke it for maximum effect. -- Eric The Read, asr, on google.com
I think Jeff was suggesting that we implement it like this:
def plumb(cmd): import Tkinter return Tkinter.call('exec ' + cmd)
But then you'd be going through the bottleneck of a string syntax with all its attendant quoting problems. The whole point of my suggestion was to avoid that! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
On Mon, Mar 10, 2003 at 11:52:39AM +1300, Greg Ewing wrote:
Those would be quite different functions, then, unless you proposed to have Python interpret native shell metacharacters on its own too (e.g., set up pipes, do the indicated file redirections, interpolate envars, and fake whatever other shell gimmicks people may use).
What we need is a function which does all those things, but uses some way of specifying them *other* than shell metacharacters. E.g.
os.plumb(("sed", "-e", "s/dead/resting/", "parrots"), ("grep", "norwegian"), output = myfile))
How about this: cmd.sed('-e', 's/dead/resting', 'parrots') / cmd.grep('norwegian') >> myfile or this: def mygrep(pattern): def tran(upstream): for s in upstream: if re.search(pattern, s): yield s return transformation(tran) open('parrots') / (lambda s:s.replace('dead','resting')) / mygrep('norwegian')) >> open('myfile', 'w') This is not some hypothetical syntax - I have a module that actually does this. It can mix python functions, generators and external commands in the same flow, use any iterable object as source, use a file, list or other data consumer as destination and a few more goodies. It's not finished but it mostly works. I don't have much time to work on it, though. oh-dear-what-have-I-done-now-I'll-have-to-finish-it-ly yours, Oren
Tim Peters <tim.one@comcast.net> writes:
Someone changed test_popen to "quote" the path to python:
cmd = '"%s" -c "import sys;print sys.argv" %s' % (sys.executable, cmdline) ^ ^
The double-quote characters above the carets are new.
This causes test_popen to fail on Win2K, but not on Win98. The relevant difference appears to be the default shell (cmd.exe on the former, command.com on the latter).
In distutils we had a similar problem. I don't remember the details at the moment exactly, but I think enclosing sys.executable in double quotes *only* when it contains spaces should do the trick. Thomas
[Thomas Heller]
... In distutils we had a similar problem. I don't remember the details at the moment exactly, but I think enclosing sys.executable in double quotes *only* when it contains spaces should do the trick.
That's what I checked in, but doubt it works in general. The cmdline test_popen passes to cmd.exe would have 4 double-quote characters then, and the docs I quoted clearly say it falls into the second case then (so it would strip the first and last quotes, leaving the second and third, which don't make sense anymore). The trick would work if the executable path were the only quoted thing on the cmdline.
From: Thomas Heller
Tim Peters <tim.one@comcast.net> writes:
Someone changed test_popen to "quote" the path to python:
cmd = '"%s" -c "import sys;print sys.argv" %s' % (sys.executable, cmdline) ^ ^
The double-quote characters above the carets are new.
This causes test_popen to fail on Win2K, but not on Win98. The relevant difference appears to be the default shell (cmd.exe on the former, command.com on the latter).
In distutils we had a similar problem. I don't remember the details at the moment exactly, but I think enclosing sys.executable in double quotes *only* when it contains spaces should do the trick.
My example isn't for popen, but this sounds familiar. There are a few places where I had to do things like this for some Win98 folks that installed 'Python22' into 'C:\Program Files\' instead of at 'C:\' if ' ' in sys.executable: python = '"' + sys.executable + '"' else: python = sys.executable os.spawnv(os.P_NOWAIT, python, [python, filename]) there have also been some quote issues with the arguments like filename and I'm still not sure all the cases on various versions of Windows, Mac OS X, and Linux work correctly all the time. David Ascher is on vacation, otherwise he could tell us all about the process.py module and how it relates to these issues :) ka
On vrijdag, maa 7, 2003, at 22:48 Europe/Amsterdam, Thomas Heller wrote:
In distutils we had a similar problem. I don't remember the details at the moment exactly, but I think enclosing sys.executable in double quotes *only* when it contains spaces should do the trick.
But only spaces may not be good enough. What I think we really want is a function that makes any string safe for popen/exec/shell script (or raises an exception if it can't be done?). As this function will have to be platform-specific it seems os.path would be a suitable place for it. Or would this give a false sense of security to people who write cgi scripts or something and then suddenly get hit by an IFS hack or similar trick? -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
When I tackled this problem for a program of mine, I ended up making sure that I always used the "short filename" form for the program to be executed. This way, there were no spaces in the filename and no need to quote them. However, the function I used to do this comes from win32<something>, so test_popen can't use it. Nor can Python fix this up for all users of os.popen() Jeff
-----Original Message----- From: python-dev-admin@python.org [mailto:python-dev-admin@python.org] On Behalf Of Jeff Epler Sent: Saturday, March 08, 2003 9:39 AM To: Tim Peters Cc: PythonDev Subject: Re: [Python-Dev] test_popen broken on Win2K
When I tackled this problem for a program of mine, I ended up making sure that I always used the "short filename" form for the program to be executed. This way, there were no spaces in the filename and no need to quote them.
However, the function I used to do this comes from win32<something>, so test_popen can't use it. Nor can Python fix this up for all users of os.popen()
Jeff
Note that there's a policy/reghack that disables short filenames. This allegedly improves file IO by up to 25 % and is commonly recommended as a performance enhancement for domains that only have W2K + servers and clients. It's also part of Microsoft's Server lockdown documentation. I believe the official stance by Microsoft itself is that short filenames are a legacy feature.
participants (10)
-
Greg Ewing -
Jack Jansen -
Jeff Epler -
Kevin Altis -
logistix -
Michael Hudson -
Neil Schemenauer -
Oren Tirosh -
Thomas Heller -
Tim Peters