[Tutor] subprocess.call list vs. str argument

[eryksun]

On Wed, Feb 26, 2014 at 3:50 AM, Albert-Jan Roskam <fomcl at yahoo.com> wrote:
> On Tue, Feb 25, 2014 at 4:54 PM, Dave Angel <davea at davea.name> wrote:
>> CreateProcess has its own design bobbles as well. For
>> example,  if you forget to put quotes around the program
>> name, it will happily try to add ".exe" to *multiple*
>> places in the hopes that one of them will work.
>> Adding a file c:\program.exe to a system will blow up
>> lots of programs that were working by mistake for years.
>
> Yesterday evening (it was *late* so forgive me if I wrong) I
> realized that part of my confusion was also caused by the fact
> that I ran my code in Idle. If I called subprocess.call with a
> list argument, it returned code 0 (success) but the generated
> sphinx code was not the same as when I ran the program from
> the terminal. I concluded that this is some weird interaction
> between Idle (which may also run in a subprocess??) and my own
> program. I also had no problems when I ran (in the terminal):
> python -c "import subprocess;
> subprocess.call(['sphinx-apidoc'...(etc)])"

Run IDLE from the terminal to have sphinx-apidoc inherit the terminal
for standard I/O. Then you can see the TTY output. I assume you're
using a POSIX system. I tacked on the bit about Windows CreateProcess
just so you'd be aware of the differences.

I'd flip the quoting around in a POSIX shell: python -c 'import
subprocess; subprocess.call(["sphinx-apidoc", "..."])'.

The shell expands $ and `cmd` in a double-quoted string:

    $ msg=spam
    $ echo "$msg"
    spam
    $ echo "`uname -o`"
    GNU/Linux

But not in a single-quoted string:

    $ echo '$msg'
    $msg
    $ echo '`uname -o`'
    `uname -o`

> I was surprised that the list elements of the argument for
> subprocess.call *have to be* separate tokens, e.g. (earlier in
> one of Danny's replies): a token (list element) '-f -F' won't
> work, it has to be two separate elements/tokens: '-f', '-F'.
> Apparently, subprocess.call is more than just a convenience
> function that " ".joins the list and escapes the resulting
> string.

subprocess.call doesn't join the arguments:

    def call(*popenargs, **kwargs):
        return Popen(*popenargs, **kwargs).wait()

On a POSIX system, Popen._execute_child doesn't join the arguments,
either. It sets executable=args[0] and calls os.execvp(executable,
args) in the forked child process. If executable is a relative path,
os.execvp uses a loop over the paths in the PATH environment variable
to create an absolute path.

In CPython, os.execvp calls posix.execv(path, args), which is written
in C to call the system function execv(const char *path, char *const
argv[]). This replaces the current process image with the executable
file located at the absolute `path`.

To set up the system call, posix.execv transforms the tuple/list of
args into an array of char * pointers of length len(args) + 1. The
final item of the array is the NULL terminator. Any unicode strings
are encoded with the file-system encoding (probably UTF-8).

If the new process image isn't successfully executed (replacing Python
in the process), then OSError is raised. This is in the forked child,
so Popen._execute_child has to pickle the exception and write it back
to the parent process via os.write(errpipe_write,
pickle.dumps(exc_value)). It reads the `data` from the other end of
the pipe in the parent, and if it's non-empty it'll `raise
pickle.loads(data)`. That's the source of the OSError from calling
subprocess.call(cmd), where cmd was the entire command line as a
string.

As to what a program does if you mash two options into a single item
of its argv array, such as '-f -F', assuming it isn't trying to be
clever and expects the command-line to be parsed by a standard POSIX
shell, then '-f -F' won't match any of its known options.