[Distutils] build system abstraction PEP, take #2

[Robert Collins]

Updated - tl;dr:
- specify encoding of stdout streams that matter
- change build command template to a list of strings so we don't need
to attempt to address shell escaping for paths with spaces(or other
such things)
- be clear about which variables apply in command templating, and
which are environment variables [we dont' want an implicit dependency
on shell, since this is for Windows too.

I've written a proof of concept of a  setuptools shim (looks like
setup.py, calls the build system from pypa.json) - right now it only
handles 'develop', so its missing wheel support, and for pip < 6 also
install support.

It is however sufficient to let us reason about things.
https://pypi.python.org/pypi/setuptools_shim

The thing I'm least happy about is that implementing install support
will require recursively calling back into pip, that or reimplementing
the installation of wheels logic from within pip - because
sufficiently old pip's won't call wheel at all. And even modern pips
can be told *not to call wheel*.

-Rob

diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
index 762cd88..a6e4712 100644
--- a/build-system-abstraction.rst
+++ b/build-system-abstraction.rst
@@ -84,18 +84,19 @@ bootstrap_requires
     Optional list of dependency specifications [#dependencyspec] that must be
     installed before running the build tool. For instance, if using flit, then
     the requirements might be::
-
+
         bootstrap_requires: ["flit"]

 build_command
-    A mandatory Python format string [#strformat]_ describing the command to
-    run. For instance, if using flit then the build command might be::
+    A mandatory key, this is a list of Python format strings [#strformat]_
+    describing the command to run. For instance, if using flit then the build
+    command might be::

-        build_command: "flit"
+        build_command: ["flit"]

     If using a command which is a runnable module fred::

-        {PYTHON} -m fred
+        build_command: ["{PYTHON}", "-m", "fred"]

 Process interface
 -----------------
@@ -114,14 +115,23 @@ redirected and no communication with the user is possible.

 As usual with processes, a non-zero exit status indicates an error.

-Available variables
--------------------
+Available format variables
+--------------------------

 PYTHON
     The Python interpreter in use. This is important to enable calling things
     which are just Python entry points.

-        ${PYTHON} -m foo
+        {PYTHON} -m foo
+
+Available environment variables
+-------------------------------
+
+These variables are set by the caller of the build system and will always be
+available.
+
+PYTHON
+    As for format variables.

 PYTHONPATH
     Used to control sys.path per the normal Python mechanisms.
@@ -133,9 +143,9 @@ There are a number of separate subcommands that
build systems must support.
 The examples below use a build_command of ``flit`` for illustrative purposes.

 build_requires
-    Query build requirements. Build requirements are returned as a JSON
-    document with one key ``build_requires`` consisting of a list of
-    dependency specifications [#dependencyspec]_. Additional keys must be
+    Query build requirements. Build requirements are returned as a UTF-8
+    encoded JSON document with one key ``build_requires`` consisting of a list
+    of dependency specifications [#dependencyspec]_. Additional keys must be
     ignored. The build_requires command is the only command run without
     setting up a build environment.

@@ -145,9 +155,9 @@ build_requires

 metadata
     Query project metadata.  The metadata and only the metadata should
-    be output on stdout. pip would run metadata just once to determine what
-    other packages need to be downloaded and installed. The metadata is output
-    as a wheel METADATA file per PEP-427 [#pep427]_.
+    be output on stdout in UTF-8 encoding. pip would run metadata just once to
+    determine what other packages need to be downloaded and installed. The
+    metadata is output as a wheel METADATA file per PEP-427 [#pep427]_.

     Note that the metadata generated by the metadata command, and the metadata
     present in a generated wheel must be identical.
@@ -169,7 +179,7 @@ wheel -d OUTPUT_DIR
 develop [--prefix PREFIX] [--root ROOT]
     Command to do an in-place 'development' installation of the project.
     Stdout and stderr have no semantic meaning.
-
+
     Not all build systems will be able to perform develop installs. If a build
     system cannot do develop installs, then it should error when run. Note
     that doing so will cause use operations like ``pip install -e foo`` to