[Twisted-Python] Question on using processes
![](https://secure.gravatar.com/avatar/b0667007a32c9222194ccf52abd3aa43.jpg?s=120&d=mm&r=g)
I'm using twisted.internet.utils.getProcessOutput() to run commands from a reactor. I've pretty much got this working okay (with a few false starts related to how args are put together... you can't send in a big long string; you literally have to make a list of argument words... what a PITA). Now, I've got a command that needs to pipe something through another command, something like this: /bin/cat foo.txt | /usr/bin/tail If I set up my "process" like this: CMD = "/bin/cat" ARGS = [ "foo.txt", "|", "/usr/bin/tail" ] d = utils.getProcessOutput( CMD, ARGS ) It doesn't work. Now, this is a stupidly-trivial example, but it seems to have trouble with thinking that the pipe symbol is a filename. My less trivial example merely hangs up the console and does nothing. I figured that this had something to do with the pipe (obviously) so I tried going deeper and using reactor.spawnProcess... This is a little over my head, but I tried using an instance of twisted.internet.protocol.ProcessProtocol as the protocol, and turning on usePTY. Still, no dice. So, my next theory is that it has something to do with the way that the process is literally launched. This is extremely low level, and has resisted my attempts to figure it out even with a debugger and stepping through. My question: what's the best way to go about this? Options: - write my own Process protocol somehow? - figure out what child descriptors may be required to fix this? - simply make a bash script that wraps up the pipe inside of a black box and make the reactor call that? Thanks!
![](https://secure.gravatar.com/avatar/152986af8e990c9c8b61115f298b9cb2.jpg?s=120&d=mm&r=g)
On 6 Aug 2004, at 06:53, Matt Feifarek wrote:
I'm using twisted.internet.utils.getProcessOutput() to run commands from a reactor.
I've pretty much got this working okay (with a few false starts related to how args are put together... you can't send in a big long string; you literally have to make a list of argument words... what a PITA).
This is due to Python's (and C's) Unix heritage. The kernel call to spawn a new process takes the full path to the executable, and an array of null-terminated strings which are the arguments. This can be quite tedious to set up in C, but shouldn't be too painful under Python. Contrast this behaviour with MS-DOS and Win32, whose 'new process' API takes a single string of command-line parameters, and passes that single string directly to the new process. When you type a command with some parameters at the command line, how does the program wind up with a sequence of command-line parameters? Under Unix, the shell (usually /bin/sh or compatible) reads through the command-line and does all its special parsing - adding environment variables, constructing pipes, breaking up the string into space-delimited chunks, doing backslash-escaping and so forth, it searches the PATH environment variable for the location of the named program, and winds up with a full executable path and an array of string parameters which it hands off to the kernel, and your process is started. Because most shells have a well-designed meta-character set, you can spawn pretty much any program you like with pretty much any command-line you like (except for embedded nulls, of course) grouped in any way you like. Under DOS/Win32, the shell (usually CMD.EXE) takes the command-line you give, searches $PATH for the executable, and hands the executable's path and the command-line string off to the kernel. The program that's started is responsible for parsing the given string into a list of parameters, and of course they all do it slightly differently. Some don't let you use filenames with spaces, some let you use filenames with spaces but don't let you use filenames with quotes, and so forth. For example, under Unix, I can use 'python -c' to make instant Python scripts: python -c 'print "Hello, \"World\""' Under Win32, I don't believe I can do the above step because CMD.EXE always treats double-quotes as delimiting a filename with spaces. I can't make a single commandline parameter that includes embedded quotes.
Now, I've got a command that needs to pipe something through another command, something like this:
/bin/cat foo.txt | /usr/bin/tail
The reason for this is that piping is set up and executed by the shell, not the kernel. What you really need (to solve both these problems) is a way to get the shell to do all the dirty work for you - and luckily, such a feature exists. For the above example, instead of executing /bin/cat and handing it the other parameters try executing /bin/sh and giving it the parameters ['-c', '/bin/cat foo.txt | /usr/bin/tail']
![](https://secure.gravatar.com/avatar/9a7b191de8b6ec7fe0d6c66d0a5512fa.jpg?s=120&d=mm&r=g)
Thank you for your very informative reply. I didn't realize that the reactor was getting THAT low-level. Cool. On Fri, 6 Aug 2004 08:12:13 +1000, Tim Allen <screwtape@froup.com> wrote:
The reason for this is that piping is set up and executed by the shell, not the kernel. What you really need (to solve both these problems) is a way to get the shell to do all the dirty work for you - and luckily, such a feature exists. For the above example, instead of executing /bin/cat and handing it the other parameters try executing /bin/sh and giving it the parameters ['-c', '/bin/cat foo.txt | /usr/bin/tail']
![](https://secure.gravatar.com/avatar/333c456fe87d4b313037ffc1a0d479f2.jpg?s=120&d=mm&r=g)
Matt Feifarek wrote:
I've pretty much got this working okay (with a few false starts related to how args are put together... you can't send in a big long string; you literally have to make a list of argument words... what a PITA).
not that this is what your asking, but I just thought you might like to know: args = ['a', 'long', 'list', 'of', 'annoying', 'PITA', 'args'] is less of a pita as: args = 'a long list of annoying PITA args'.split() -tjs -- (o_ Timothy Stebbing, Pythonista, Nunatak Systems //\ 03 6226 6259, tim.stebbing@nunatak.com.au V_/_ ><>--------------------------------------<><
![](https://secure.gravatar.com/avatar/c565aef253600553567b390d70c5876a.jpg?s=120&d=mm&r=g)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Or shlex.split, if you want it to be parsed more like a command line really would be parsed. |>> shlex.split('a huge argument "list of things"') ['a', 'huge', 'argument', 'list of things'] Tim Stebbing wrote: | Matt Feifarek wrote: | |> I've pretty much got this working okay (with a few false starts |> related to how args are put together... you can't send in a big long |> string; you literally have to make a list of argument words... what a |> PITA). | | | not that this is what your asking, but I just thought you might like to | know: | | args = ['a', 'long', 'list', 'of', 'annoying', 'PITA', 'args'] | | is less of a pita as: | | args = 'a long list of annoying PITA args'.split() | | -tjs | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3-nr1 (Windows XP) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBGUDL3A5SrXAiHQcRAnI5AJ9ziX2LdUK22hrIq482lghaJaua6gCginq8 lFBOHYSMj5tGVZrVmLmQdII= =0rxp -----END PGP SIGNATURE-----
participants (5)
-
Cory Dodt
-
Matt Feifarek
-
Matt Feifarek
-
Tim Allen
-
Tim Stebbing