[Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 17 July 2017 at 18:53, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 17 July 2017 at 18:29, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> On Mon, Jul 17, 2017 at 7:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> The minimal specification for in-place builds is "Whatever you would
>>> do to build a wheel file from an unpacked sdist".
>>
>> Eh no, in-place has nothing to do with building a wheel. Several people have
>> already pointed this out, you're mixing unrelated concepts and that's likely
>> due to you using a definition for in-place/out-of-place that's nonstandard.
>
> I'm using in-place specifically to mean any given PEP 517 backend's
> equivalent of an unqualified "./setup.py build_wheel".

Slight correction, since I left out `--dist-dir`, which is relevant to
both in-place and out-of-tree builds:

    ./setup.py build_wheel --dist-dir <wheel_directory>

> I'm using out-of-tree to mean (as a baseline) what Daniel suggested:
> any given backend's equivalent of "./setup.py build -b
> <build_directory> build_wheel" (e.g. variant directories in Scons).

And for out-of-tree builds:

    ./setup.py build -b <build_directory> build_wheel --dist-dir
<wheel_directory>

And to elaborate a bit further on my perspective in the latest round
of API discussions:

As far as I can tell, Nathaniel is suggesting either that:

1. We shouldn't allow frontends to explicitly request out-of-tree
builds (which I disagree with, since out-of-tree builds are useful for
a whole host of reasons, hence why most build systems natively support
them, even setuptools); or
2. Even with out-of-tree builds, we should allow frontends to
*separately* indicate "I tried to build the sdist first, but that
failed, so I'm resorting to building the wheel directly"

And I can completely see the point of folks saying "But you're
conflating two different questions" when I declare the latter notion
redundant given out-of-tree build support, since that's exactly what
I'm doing:

1. Daniel raised the idea of a build directory parameter to request
out-of-tree builds as a replacement for prepare_input_for_build_wheel
2. Thomas liked that idea and added it to the PEP
3. I/we realised that if the pip devs were willing to rely on
out-of-tree builds as a fallback when build_sdist failed, then we
could *avoid* adding a Python-specific sdist-consistency concept
directly to the backend API definition (beyond some aspirational text
in the definition of build_wheel) until we had clear evidence that we
actually needed it
4. I personally switched from requesting any further API changes to
assisting with clarifications aimed at addressing the concerns of
folks trying to follow what just happened and get us to a point where
the PEP could be accepted

The end result of all of that is that my position now is this:

- Out-of-tree builds are an inherently useful thing for the backend
API to model, independently of how pip and other frontends
specifically end up using them. Now that they've been added, I'll be
highly reluctant to accept any version of PEP 517 that attempts to
take them out again.
- I *think* we'll find that out-of-tree builds using a pip-managed
intermediate directory are sufficient in practice to meet pip's needs
when it comes to handling cases where build_sdist fails
- *If* I turn out to be wrong about that, *then* we can look at adding
back the "prepare_input_for_build_wheel" hook (or something along
those lines) as a fallback specifically for the case where a frontend
would really prefer to be building via sdist, but sdist creation isn't
possible for some reason, informed by specific knowledge of cases
where falling back to out-of-tree builds proved to be inadequate

By doing things this way, we're not risking lumbering ourselves
permanently with a Python-specific build flag or optional hook that
may turn out to only be relevant to the way setuptools/distutils
currently work. Instead, in the ideal case where we don't need to
revise the backend API, we'll *only* have the concept of
in-place/out-of-tree builds, which is a general one, not specific to
any particular build system or any particular use case.

And that now seems to be the core point of contention: my
understanding is that Nathaniel would prefer to add the ill-defined
and Python-specific notion of sdist-consistency to the backend API
*now*, and *postpone* adding the general concept of out-of-tree
builds. I don't think that's a good idea, since our only well-defined
notion of sdist consistency involves actually building the sdist
first, and then building the wheel from that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia