[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Paul Moore]

On 31 May 2017 at 18:03, Donald Stufft <donald at stufft.io> wrote:
> No you’re correct, it currently just invokes ``setup.py sdist bdist_wheel``.
> The hook is needed so that Travis can have a singular tool to invoke (likely
> twine?) instead of needing to determine if it needs to invoke flit or
> setuptools or mytotallyradbuildthing. The thing I’m trying to express (and
> doing poorly it seems :( ) is that generating a sdist is an important thing
> to have be possible, and it needs to be done in a way that it can be invoked
> generically.

I don't think that's either unclear or in dispute. The question here
is whether "produce a sdist" is in scope for this particular PEP.

The problem is that you're proposing using a "build a sdist" hook as
the means for pip to do its "copy the source to an isolated build
directory" step. Currently, doing that fails because tools like
setuptools_scm work differently when run from a VCS checkout instead
of a sdist. The long term plan for pip is something we've always
described in terms of going via a sdist, but there's lots of details
we haven't thrashed out yet. I don't think we're at a point where we
can insist that in a post-PEP 517 world, we can switch straight to
building via a sdist. However, I agree with you that we want PEP 517
to assist us with moving in that direction, and we *definitely* don't
want to get into a situation like we're in now, where a PEP 517
compliant backend can leave us with no option better than "copy the
whole source tree".

That's why I'm focusing at the moment on asking that PEP 517 require
backends to specify the minimal set of files needed to do the build.
With that we can create isolated build directories efficiently, and
those files are going to be an essential component of building a
sdist. They aren't *sufficient* to build a sdist by themselves, but
specifying in a PEP how we do build a sdist is a perfectly reasonable
next step, and can be done independently of PEP 517 (and indeed of pip
- there's no "pip sdist" command, so pip doesn't even have a stake in
that debate).

One other point that I think we're glossing over here. The only reason
that I know of why current setuptools builds can't be isolated via a
"build sdist" step, is because of helpers like setuptools_scm. I don't
know much about such tools, and I don't know where they'd fit in a
landscape of PEP 517 backends. Would setuptools_scm be a backend
itself, or would it be a plugin for other backends (would backends
want to *have* a plugin system for things like this)? The only
concrete backends I know of are flit and
something-that-will-do-what-setuptools-does-involving-compilers.
Neither of those need a "tell me what files constitute a build" hook -
they'd be fine with pip (or a library like packaging) implementing a
"copy everything except VCS directories, tox directories, etc"
heuristic[1]. So the cases where we *need* a hook are those where an
as-yet undefined "use VCS information to generate wheel metadata"
backends (or backend plugin) is involved - and nobody knows what shape
that will take, so we're guessing. My feeling is thatsuch a tool
should be a backend that wraps another backend, and modifies the
generate metadata hook. On that basis, it's perfectly capable of also
modifying the "specify the build files" hook. But that's nothing more
than a guess at this point.

Thoughts?

Paul

[1] They can do a better job than that, certainly, which is why the
hook is a good thing, but it's not a showstopper. The key point is
that they need *less* than the obvious heuristic would give them, not
more.