[Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 15 July 2017 at 07:31, Daniel Holth <dholth at gmail.com> wrote:
> I proposed the build directory parameter because the copytree hook made no
> sense to me. It is not a perfect substitute but perhaps a configurable build
> directory is nice on its own without having to satisfy all older arguments
> in favor of copytree. I think true in-place builds are the oddball (for
> example 2to3 or any build where sources have the same name as outputs needs
> a build directory to put the translated .py files, otherwise it would
> overwrite the source). What people think of as in-place builds in distutils
> are usually just builds into the default build directory.
>
> IIRC the main reason we are worried sdist generation would fail is when you
> are trying to do new development from an unpacked sdist. Suggest not
> worrying that this use case is a perfect experience. It's good enough for
> quick patching, but if you need to do a new source release then you probably
> have time to meet the requirements of your build backend.

This reminds me of something I should have mentioned on list, but made
the mistake of assuming would be equally clear to everyone else: the
last time that Thomas elaborated on his concerns about VCS metadata
potentially being unavailable, I realised there's a *very* common
build use case where we can safely assume that metadata will be
missing.

Specifically: source builds in systems like RPM, deb, and conda. Those
start with a release tarball, *not* a VCS checkout, and while they'll
likely start with the sdist rather than a raw tarball for most Python
projects, that *won't* be the case for projects that include generated
artifacts like Cython output files in their sdists.

So to handle that kind of scenario under PEP 517, we're going to
*need* wheel builds to work even when sdist builds would fail due to a
lack of VCS metadata.

That doesn't specifically say anything about in-tree vs out-of-tree
build support (aside from the fact that Debian's packaging policies
call for out-of-tree builds), but it does mean that having the backend
API *only* support the "source tree -> sdist -> wheel" build path is
genuinely insufficient for our full array of use cases.

However, backends that *don't* have any special requirements for
creating sdists will still be free to implement their out-of-tree
build support that way, which is what I'm proposing we do for the
example backend in the PEP:
https://github.com/python/peps/pull/310/files

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia