[Distutils] Provisionally accepting PEP 517's declarative build system interface

Brett Cannon brett at python.org
Thu Jun 1 16:45:53 EDT 2017

On Wed, 31 May 2017 at 14:14 Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Wed, May 31, 2017, at 09:16 PM, Donald Stufft wrote:
> How you build the release-quality sdist isn’t really of concern of PEP 517
> any more than building a release quality wheel is, it’s up to the build
> tool to implement that as it makes sense for them.
> But if we have a hook for building something called an sdist, we need to
> define what an sdist is. The definition you're referring to is a functional
> definition sufficient for a tool that only wants to install it, but it
> doesn't cover what an sdist means or how it fits into workflows.
> I could see this as an argument that the PEP should have *both* a
> build_sdist and a prepare_build_files hook, if you don’t think that the
> build_sdist hook is suitable on it’s own.
> I would prefer that to the the current status of one hook that tries to
> cover both use cases.
> I'd still rather specify prepare_build_files in this PEP, and leave the
> sdist hook to a later PEP. I don't see a great need for a standard
> interface to building an sdist: the developer doing that by calling their
> build tool directly seems adequate for the majority of use cases.

So it sounds like the list_build_files() part of the API is still useful
for isolated builds versus in-place builds, correct?

As for Thomas' idea that "calling a build tool directly seems adequate" is
somewhat interesting and not something I thought about. Let's look at
Donald's list of current ways to get from something to installation (which
I know we want to scale back):

1) VCS Checkout -> Installed
2) VCS Checkout -> Sdist -> Installed
3) VCS Checkout -> Wheel -> Installed
4) VCS Checkout -> Sdist -> Wheel -> Installed
5) VCS Checkout -> Editable Install
6) VCS Checkout -> Sdist -> Editable Install

OK, so what Thomas is suggesting is this isn't all necessarily directly
under pip's purview. So if you take that list and break out those steps
into "back-end responsibility" and "front-end responsibility" then you end
up with:

Back-end (e.g. flit):
1) VCS Checkout -> wheel (driven by pip)
2) VCS Checkout -> sdist
2) sdist -> wheel (driven by pip)

Front-end (e.g. pip):
1) wheel -> installed

You can then generate the non-editable install steps above by combining
these roles. And I think the point Thomas is trying to make is if you look
at back-end#2 you can simply leave pip out of it in a post-PEP 517 world if
you view an sdist as a trimmed down VCS checkout that pip just needs to
know how to unpack.

But thinking from a more fundamental perspective, why does pip even need to
be involved with any part of PEP 517? If pip is meant to be viewed as a
package manager then isn't its key focus on fetching and installing
appropriate things?

Well, it's because pip may have to build a wheel from a post-517 sdist that
it downloaded from PyPI when a project doesn't provide e.g. a Windows
wheel. That's why pip needs to be PEP 517-aware at all. Otherwise pip could
just work in a wheel-only world and not even care about building wheels to
begin with.

But where does building sdists come into play? You need to be able to make
them, and you need to be able to get them up to PyPI. But beyond pip
needing to know how to unpack an sdist to make the wheel it wants to work
with, pip doesn't actually *need* to produce an sdist.

But I think *twine* is the tool that needs a way to specify how to produce
an sdist. If we want to view twine as the tool to upload artifacts to PyPI
then we need twine to know how to produce sdists and wheels in a PEP 517
world, not pip.

Maybe I'm missing something fundamental here about pip and all of this is
wrong, but from where I'm sitting it seems the key thing pip needs from PEP
517 is how to go from a bunch of source to a wheel so that it can install
that wheel. Now pip needs to know how to *consume* an sdist that it gets
from PyPI that has a pyproject.toml, but it technically doesn't need to
know how to *produce* one if in the end it really just wants a wheel.
Twine, OTOH, does need a way to produce an sdist as well as a wheel so it
can upload those to PyPI.

And so I think in a very wordy way, I just said we need to stop saying "pip
needs a standardized way to produce an sdist" and instead start saying
"twine needs a way to produce an sdist". And that leads to the question
about whether PEP 517 must cover sdist *production* for twine because we
want to have that solved before we have the *consumption* side in pip in
place. Or put another way, are we okay with pip consuming things that twine
simply can't make from a pyproject.toml file (yet)? A "yes" means PEP 517
shouldn't be held up, while a "no" means we need a solution for twine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170601/94294685/attachment-0001.html>

More information about the Distutils-SIG mailing list