[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Paul Moore]

On 31 May 2017 at 15:05, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Wed, May 31, 2017, at 03:00 PM, Thomas Kluyver wrote:
>> I would be fine with specifying a hook to copy only the files needed to
>> build the wheel, but if we do that, let's not call it 'sdist' or
>> anything that suggests that.

I agree - I've been trying to push away from using the term "sdist"
myself, as there's too much history with it, most of which isn't (IMO)
relevant in this context.

> Also, if this is all we're after, I'd like to push again for making it
> optional - for flit, the resulting wheel will always be the same whether
> you copy the files somewhere else first or just build from the original
> source tree. It's less to implement and more performant if flit can just
> build the wheel directly, and skip the copying step.

Hmm. The proposed API (whether it's "create a sdist" or "tell me what
files to copy") is intended so that a frontend can use the following
process:

1. Create a temp directory
2. Call a hook to get the "source"
3. Put the source in the temp directory (unpack, copy, ask the hook to
put the files in here directly, whatever)
4. Call the "build wheel" hook on the temp directory

Pip basically does this currently, and it's probably out of scope for
this PEP to look at changing that. The current discussion is really
only about whether we need a hook at step 2, or whether "copy the
whole source tree" is sufficient. Clearly, in one sense, it is
sufficient, because that's what pip has to do at the moment, but we
have outstanding bug reports flagging that our copy step is sometimes
severely inefficient, and without the proposed hook, we have no way of
improving things. The benefit of a formal hook is that it gives the
backend control of the process, but that only helps if the tools that
cause us issues currently, like setuptools_scm, work with the backend,
specifically to control what gets copied so that it's sufficient.

Backends like flit probably have no need for this flexibility, but
should probably follow some form of standard rule like "exclude VCS
and tox control directories". It may make sense to implement that rule
in the frontend (on the usual "one frontend, multiple backends" basis)
but I'm inclined to think that choice should be opt-in, not make it
the default if a backend doesn't implement the hook. I'm prepared to
be convinced otherwise, though.

By the way, in theory with this workflow, there's some pretty bizarre
misbehaviours that backends could implement. Consider a backend that
does "build sdist" by preprocessing the current directory, and writing
a sdist that uses a completely different backend (e.g., a
"setuptools_scm" backend that writes sdists that freeze the version
data and use flit to build). Then step (4) needs to re-read the
backend before running the "build wheel" hook. That's actually a
not-unreasonable way to handle tools like setuptools_scm, so we should
probably take a view on whether it's allowable. Specifically, there's
a choice the spec needs to make between

1. Frontends MAY determine the backend once and assume that it won't
change while processing a project
   1a. Hooks MUST NOT modify the pyproject.toml metadata specifying
the backend details, and MUST copy that file unchanged if asked to
copy it

vs

2. Frontends MUST allow for the possibility that pyproject.toml could
change after a hook is called

Ever get the feeling that you shouldn't have looked too closely at
something? :-)

Paul