[Python-Dev] subprocess.Popen and win32

[David Aguilar]

Hi,

I just joined python-dev because I found the need to add some code to
paper over python3's subprocess API, and I'm wondering whether I'm
missing something.

On python2 and python3, the (only?) way to get utf-8 arguments to
subprocess was to ensure that all unicode strings are encoded into
bytes before subprocess sees them. This has worked for a long time
(currently compatible across python2 and 3).

On python3, this still works for normal platforms, but on windows we
can't pass a list of byte strings. We have to pass a list of unicode
strings.

This means that the application code ends up needing to do this:
https://github.com/git-cola/git-cola/commit/1109aeb4354c49931d9b0435d2b7cfdc2d5d6966

basically,

def start_command(cmd):
    if sys.platform == 'win32':
        # Python on windows always goes through list2cmdline() internally inside
        # of subprocess.py so we must provide unicode strings here otherwise
        # Python3 breaks when bytes are provided.
        cmd = [decode(c) for c in cmd]
    else:
        cmd = [encode(c) for c in cmd]
     return subprocess.Popen(cmd)

That seems broken to me, so I wonder if this is a bug in the way
python3 is handling Popen with list-of-bytestring on win32?

I'm not a windows user, but I was able to install python3 under wine
and the same traceback happens without the paper bag fix. This is what
the traceback looks like; it dies in list2cmdline (which I am not
calling directly, Popen does it under the covers):

    File "E:\Program Files
(E)\git-cola\share\git-cola\lib\cola\core.py", line 109, in
start_command
    universal_newlines=universal_newlines)
    File "C:\Python32\lib\subprocess.py", line 744, in __init__
    restore_signals, start_new_session)
    File "C:\Python32\lib\subprocess.py", line 936, in _execute_child
    args = list2cmdline(args)
    File "C:\Python32\lib\subprocess.py", line 564, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
    TypeError: Type str doesn't support the buffer API

This is an issue for folks that use python to write cross-platform
code. The unix code paths expect list-of-bytes, but win32 only expects
list-of-unicode, which pushes the burden onto the application
programmer.

It's my opinion that the win32 code path on python3 is the odd man
out. If it allowed list-of-bytes like python2/win32 and python2+3/unix
then this wouldn't be an issue.

Is this an actual problem, or is it something that should be handled
by application-level code as I've done?

Thanks,
-- 
David