[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Paul Moore]

On 1 June 2017 at 21:45, Brett Cannon <brett at python.org> wrote:
> And so I think in a very wordy way, I just said we need to stop saying "pip
> needs a standardized way to produce an sdist" and instead start saying
> "twine needs a way to produce an sdist". And that leads to the question
> about whether PEP 517 must cover sdist production for twine because we want
> to have that solved before we have the consumption side in pip in place. Or
> put another way, are we okay with pip consuming things that twine simply
> can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't
> be held up, while a "no" means we need a solution for twine.

The question is more, are we okay with pip consuming things
(pyproject.toml source trees) for which we don't define any means of
uploading to a package index (which is basically what a sdist is - a
way of publishing sources on PyPI). My answer to that is "no".

pip doesn't need to be able to provide a user interface that builds a
sdist. It *could* provide one, as an enhancement, but it's not
*necessary*. Whether the canonical "build a sdist" command should be
pip or (say) twine is a side issue here (although the ability to
*have* a canonical command, and not rely on each backend having its
own incompatible way, *is* important IMO - but that's not the point
here). However, there are a number of places where pip indirectly
needs the ability to build a sdist (or do something closely
equivalent).

pip needs a way to deal with "pip install foo". There are 2 scenarios here:

1. The index (PyPI) contains an appropriate wheel. Then pip just
installs it (using the wheel library). PEP 517 isn't involved, no
build, easy.
2. There is no appropriate wheel. There are 2 subcases here:
   2a. There is no sdist. We're stuck. See below.
   2b. There is a sdist (whatever that means). As you say, pip needs
to be able to consume that sdist. PEP 517 covers this as it stands.

pip also needs a way to deal with "pip install <local directory>. In
this case, pip (under its current model) copies that directory to a
working area. In that area, it runs the build command to create a
wheel, and proceeds from there. In principle, there's little change in
a PEP 517 world. But again, see below.

There are other things here, but they are largely marginal, or similar
to the above cases (the big remaining case is editable installs, but
that's a whole other question).

Notes:

- The case 2a in the "pip install foo" example is of concern. At the
moment, building sdists is trivial (python setup.py sdist) and there's
effectively no barrier to publishing sdists. In a PEP 517 world,
backends may or may not provide a "build a sdist" command (flit
doesn't, and for a long time I believe didn't propose to do so -
partially because what constituted a sdist at the time was
inappropriate for them, but they were happy to be wheel-only). That
means that users of that backend are basically unable to upload
source, and the likelihood of scenario 2a goes *way* up. Making sure
that PEP 517 mandates that backends at least *allow* tools like twine
to build a sdist from a source tree significantly alleviates this
risk.
- Copying a local directory to a temporary area is demonstrably a
serious performance issue for pip. We have a number of bugs raised
over this. The problem is that we have to blindly copy *everything*,
including arbitrarily large amounts of irrelevant data. We wanted to
switch to "build a sdist from the local directory, proceed from the
"we have a sdist" case. But some uses of setup.py break with that
approach. One of the hopes for PEP 517 is that we avoid that problem,
because we make it the backend's problem (by asking the backend to
tell us how to know what to copy, in one form or another). If there's
no support in the PEP for this, we end up having to accept that going
forward we'll *still* have this problem (unless we add an extra
mandatory backend hook in a new PEP, but backends can still claim
support for only PEP 517 and not the new PEP). This is admittedly an
enhancement to pip, not essential functionality, but the point of this
whole process is to enable the packaging ecosystem (including pip!) to
move forward, rather than being paralyzed by the constraints of the
old setup.py system. So leaving the same constraints in the new system
isn't helpful.
- The local directory -> sdist -> wheel -> install model has its
issues, but it's a much cleaner model for pip (if we can get to it
without breaking use cases) - for all the same reasons that switching
from direct installs to installs via wheels was a huge win for us. To
implement that model internally, pip would need a means of building a
sdist.

I hope this helps. If nothing else, it makes your comments look less
wordy by comparison :-)

Paul