[Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 6 July 2017 at 15:54, Donald Stufft <donald at stufft.io> wrote:
> The fundamental problem here is that sdists *are* a key part of the build
> pipeline and are always going to be unless pip stops supporting sdists all
> together. I think it is a complete non-starter to suggest removing
> installation from sdist support from pip (particularly since it would
> immediately lose support for every platform but Windows, MacOS and many
> common Linux’s (but not all of them!).

I wonder how true this is. Certainly the route "acquire sdist ->
unpack -> build wheel -> install" is a fundamental route, as is
"acquire wheel -> install". But as Nick pointed out, the awkward cases
are all in the *other* area, which is "get a random source tree -> ???
-> install". That's where all the debate about isolated builds,
incremental compiles, etc, occur. We've been focusing on the sdist as
a means of copying trees, and maybe that makes it feel like sdists are
more fundamental than maybe they need to be. The fundamental operation
is really "copy this arbitrary source tree safely".

I'm not sure I have a solution here, but as a starting point maybe we
need to conceptually separate source trees into "publishing trees"
(ones that the backend is capable of building a sdist from) and "build
trees" (ones that the backend only supports building wheels from).
Whether unpacking a sdist gives a publishing tree or a build tree is
backend-defined (setuptools says yes, flit says no) but frontends need
to deal with the distinction cleanly.

Isolated and incremental build questions are answered differently
depending on whether you have a publishing or a build tree. And some
of those questions prompt the need for copying trees (or at least
creating equivalent build trees from whatever you have). For a
publishing tree, "make sdist and unpack" works, but that's not
possible for a build tree.

There's also the fact that tox uses sdists to populate its
environments. But bluntly, that's tox's problem, not distutils-sig's.
How tox handles flit-based projects is a different question, that we
don't really have the relevant experts present here to answer. The
same is true of any *other* potential consumers of PEP 517 backends
such as hypothetical "unified sdist builders". I'm inclined to say
that we shouldn't even try to consider these, but should limit PEP 517
to the pip (or equivalent) <-> backend interface. Future PEPs can
expand the interface as needed.

I don't know if any of this helps. If not, that's fine (it at least
helped me to clarify my thinking about source trees and sdists). But
I'm posting it in case it prompts any new insights.

Paul