[Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 14 July 2017 at 20:59, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Jul 14, 2017 at 3:32 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> it appears to be non-negotiable that there is some
>> way of building without affecting the source directory, so whatever the
>> interface is, we need some way to do this.
>
> But this is confusing the means with the ends. Obviously no cares
> about that *per se*; they want it because it accomplishes something
> they actually care about.
>
> Maybe because they don't trust build tools to do incremental builds.
> Maybe because they want to exercise the sdist path to reduce chances
> for error. Maybe because pip has always done it that way. What is that
> thing? What are the advantages of this design, as compared to the
> fallback of doing unconditional copytree (like pip does now and may
> well continue doing for years to come regardless of what we say here),
> or the slightly fancier fallback that my draft supports of attempting
> to build an sdist and if that fails doing a copytree instead? Is this
> the simplest way to accomplish the real goal? All the goals that I can
> think of seem to allow for solutions that have fewer complications and
> unknowns...
>
> ...and if pip's goal is to go via sdist whenever possible while always
> being careful never to modify the source tree, then why did we end up
> with a design where sdist generation is the one case that *is*
> encouraged to modify the source tree? This doesn't make any sense.

You're confusing two different aspects of the design here:

- the pip developer's desire for publisher flows to closely match end
user flows to reduce the risk of novice publishers shipping broken
archives
- the general build tool convention that out-of-tree builds shouldn't
modify the source tree

pip's design requirement is met by pip defaulting to doing build_sdist
and then doing build_wheel based on that sdist, so that's the simplest
possible design we could approve and ship.

The question then is what the fallback should be when build_sdist
fails, but only a wheel archive or installed package is needed.

Candidates:

- fail anyway (unecessarily poor UX if the wheel build would have worked)
- build in-place (reasonable option)
- build out-of-tree (reasonable option)
- shutil.copytree then build in-place (status quo for legacy sdists,
but problematic due to VCS directories & other large sets of data)

The design that PEP 517 has settled on is to say that since both
in-place and out-of-tree builds are reasonable things for a frontend
to request, the *API* will allow frontends to request either an
in-place build ("build_directory is None" or
"os.path.samefile(build_directory, os.getcwd())") or an out-of-tree
build ("os.path.samefile(build_directory, os.getcwd())").

We're not inventing the concept of an out-of-tree build, we're just
choosing to make it a requirement for PEP 517 backends to support
them. This is low risk, since by pushing out-of-tree build support to
the backend, we also make it straightforward for the *backend* to tell
publishers how to define a filtered shutli.copytree operation if the
backend doesn't natively support out-of-tree builds. That way we don't
need to standardise on a universal protocol for defining which files
are needed for the wheel building process - that can be a backend
dependent operation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia