[Distutils] Update to my skeletal PEP for a new build system interface

[Paul Moore]

On 9 November 2015 at 05:20, Nathaniel Smith <njs at pobox.com> wrote:
> A *source tree* is something like a VCS checkout. We need a standard
> interface for installing from this format, to support usages like
> ``pip install some-directory/``.

I still find these two definitions unhelpful, sorry.

We don't *need* an interface to install from a source tree. It's
entirely feasible to have a standard interface to build a sdist from a
source tree and go source tree -> sdist -> wheel -> install. That
doesn't cater for editable installs, nor does it cater for reusing
things like object files from previous builds, so there may be
*benefits* to having a richer interface than this, but it's wrong to
say it's needed.

I suspect you're reluctant to require a "source tree -> sdist"
interface, because the author of flit isn't comfortable with having
such a thing. That's OK - if you want to note that a benefit of going
direct to install (or wheel) is that tools that don't allow you to
create a sdist are supported, then let's make that explicit. Expect
plenty of pushback on the idea of tools that don't supply sdists
though...

> A *source distribution* is a static snapshot representing a particular
> release of some source code, like ``lxml-3.4.4.zip``. Source
> distributions serve many purposes: they form an archival record of
> releases, they provide a stupid-simple de facto standard for tools
> that want to ingest and process large corpora of code, possibly
> written in many languages (e.g. code search), they act as the input to
> downstream packaging systems like Debian/Fedora/Conda/..., and so
> forth. In the Python ecosystem they additionally have a particularly
> important role to play, because packaging tools like ``pip`` are able
> to use source distributions to fulfill binary dependencies, e.g. if
> there is a distribution ``foo.whl`` which declares a dependency on
> ``bar``, then we need to support the case where ``pip install bar`` or
> ``pip install foo`` automatically locates the sdist for ``bar``,
> downloads it, builds it, and installs the resulting package.
>
> Source distributions are also known as *sdists* for short.

One key feature of the current sdists that you are either overlooking
or ignoring is that they can, and do, contain *built* files. The best
example is projects using Cython. The sdist contains generated C
files, so that users building wheels from the sdist don't need cython
installed.

Certainly your definition of a sdist is general enough that it doesn't
preclude such things. But on the other hand, it doesn't offer any
suggestion that this is an important feature of a sdist (and it is - I
say that as someone who has needed to build wheels from a sdist and
doesn't have Cython installed). From your definition, people will
infer that zipping up a development directory makes a sdist, and so
that's what they'll do. Because after all, making Cython a build
requirement and generating the C at build time is *also* an option,
it's just not as friendly to the average user.

Paul