[Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 6 July 2017 at 18:57, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> Thank-you all for the discussion and the attempts to accommodate flit,
> but I'll bow out now. It's become clear that the way flit approaches
> packaging is fundamentally incompatible with the priorities other people
> have for the ecosystem.

While I can completely understand how the current debate over whether
or not the prepare_input_for_build_wheel hook is necessary or not
would make you feel that way, I hope I can convince you that we're
really just quibbling over a genuinely trivial arcane technical detail
that I'd never let get in the way of flit being a full-fledged
participant in the Python packaging ecosystem.

So I'll be completely clear: if *you* want that hook to be part of the
API, that's sufficient reason for me to approve keeping it in the
specification. However, I'll also make one last attempt at explaining
why we suspect it may actually be redundant, especially if some
additional configuration settings were added to pip.

(I'll also note that I have an entirely selfish reason for ensuring
that you're happy to continue active development on flit, which is
that I've found it to be a delight to use for my own pure Python
projects)

> Namely, I see sdists as archival artifacts to be
> made approximately once per release, but the general trend is to make
> them a key part of the build pipeline.

Sort of.

>From my point of view, the single most critical build path is:

1. Publisher uploads sdist to PyPI
2. Consumers and redistributors do their own from source builds
targeting the binary format of their choice

That's the path that enables the "publish my stuff once, let
redistributors deal with getting that source code into arbitrary
binary formats" dream:
http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html

However, it's closely followed by the beginner-friendly:

1. Publisher uploads universal wheels or wheels for popular platforms to PyPI
2. Consumers use the appropriate wheel rather than doing their own builds

Your relative priorities for those two paths are presumably the other
way around, and that's fine.

The only impact the current debate over whether or not to include a
dedicated out-of-tree input preparation hook has on either of those
paths relates to how much implicit testing the sdist building hook
gets when developers do their own local builds (as if you're not using
something like tox for your local testing, it's otherwise fairly easy
to inadvertently publish sdists that don't actually include all the
files they need to successfully build a wheel file).

That is, the current point of contention is specifically about how we
want tools to behave when we're starting with a source directory that:

1. Doesn't include VCS metadata (e.g. it's been exported as a tarball
rather than cloned)
2. The build frontend doesn't want to use as the basis for an in-place build
3. The build frontend doesn't want to blindly copy into a separate
build directory

So just by way of those preconditions, we're already well outside the
most common package installation workflows.

Now, I'm personally entirely OK with backend developers saying that
dealing with that scenario is entirely a frontend problem, since the
problem is in large part *created* by the frontend's refusal to either
do an in-place build, or just blindly copy the entire directory. It's
a reasonable stance to take, and frontends and their users are in the
best position to know how they want to handle it for their particular
use case.

That perspective is embodied in the hypothetical proposal to add a
"--build-strategy" option to pip that would allow folks building
wheels to choose between:

- creating and unpacking an sdist and building a wheel from that
- copying the directory tree and building a wheel from that
- building a wheel directly from the original directory

(Perhaps with a variant that tries to create and unpack the sdist
first, and only if that fails falls back to copying the entire tree)

>From a backend point of view, developers would then only need to worry
about two cases:

- given a directory, either create an sdist, or fail, and report why
it didn't work
- given a directory, either create a wheel, or fail, and report why it
didn't work

The rest of the UX becomes a frontend developer concern.

However, if you don't find that perspective compelling, and would
strongly prefer to include a way to let frontends skip building an
sdist just to do an out-of-tree wheel build, then I *don't* think
that's a barrier to accepting the PEP. It would just be up to
frontends how they wanted to prioritise that hook:

- always use it in preference to build_sdist if present
- use it as a fallback if build_sdist fails
- never use it (for frontends that specifically want the sdist)

The benefit I see to that approach is that it means that backend
developers can clearly communicate the difference between what's
required to make a source archive that's good enough to later build
the wheel archive, and what's required to actually capture all the
files that the publisher wants to publish.

> Making a guerilla tool with no concern for integration was fun. It
> became frustrating as people began to use it and expected it to play
> well with other tools, so I jumped on PEP 517 as a way to bring it into
> the fold. That didn't work out, and a tool that doesn't play well with
> pip can only be an attractive nuisance at best, even if it technically
> complies with the relevant specs.
>
> Flit is therefore deprecated, and I recommend anyone using it migrate
> back to setup.py packaging.

I stated this above, but it bears repeating: as the one currently
offering to do the work for PEP 517, you *do* have the right to say
that your willingness to make that contribution is contingent on the
"prepare_input_for_build_wheel" hook being included in the design.

While I see merit in the arguments raised against it, I'm personally
OK with it, and ultimately, that's part of why we have the
BDFL-Delegate system: so we don't lose potentially major contributions
over details that ultimately don't matter all that much. The idea is
for folks to end up mad at *me* for any such decisions that they don't
like, rather than at the folks offering to volunteer their time to
help improve the ecosystem for everyone (in this case, you).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia