On Wed, 31 May 2017 at 14:14 Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Wed, May 31, 2017, at 09:16 PM, Donald Stufft wrote:
How you build the release-quality sdist isn’t really of concern of PEP 517 any more than building a release quality wheel is, it’s up to the build tool to implement that as it makes sense for them.

But if we have a hook for building something called an sdist, we need to define what an sdist is. The definition you're referring to is a functional definition sufficient for a tool that only wants to install it, but it doesn't cover what an sdist means or how it fits into workflows.

I could see this as an argument that the PEP should have *both* a build_sdist and a prepare_build_files hook, if you don’t think that the build_sdist hook is suitable on it’s own.

I would prefer that to the the current status of one hook that tries to cover both use cases.

I'd still rather specify prepare_build_files in this PEP, and leave the sdist hook to a later PEP. I don't see a great need for a standard interface to building an sdist: the developer doing that by calling their build tool directly seems adequate for the majority of use cases.

So it sounds like the list_build_files() part of the API is still useful for isolated builds versus in-place builds, correct?

As for Thomas' idea that "calling a build tool directly seems adequate" is somewhat interesting and not something I thought about. Let's look at Donald's list of current ways to get from something to installation (which I know we want to scale back):

1) VCS Checkout -> Installed
2) VCS Checkout -> Sdist -> Installed
3) VCS Checkout -> Wheel -> Installed
4) VCS Checkout -> Sdist -> Wheel -> Installed
5) VCS Checkout -> Editable Install
6) VCS Checkout -> Sdist -> Editable Install

OK, so what Thomas is suggesting is this isn't all necessarily directly under pip's purview. So if you take that list and break out those steps into "back-end responsibility" and "front-end responsibility" then you end up with:

Back-end (e.g. flit):
1) VCS Checkout -> wheel (driven by pip)
2) VCS Checkout -> sdist
2) sdist -> wheel (driven by pip)

Front-end (e.g. pip):
1) wheel -> installed

You can then generate the non-editable install steps above by combining these roles. And I think the point Thomas is trying to make is if you look at back-end#2 you can simply leave pip out of it in a post-PEP 517 world if you view an sdist as a trimmed down VCS checkout that pip just needs to know how to unpack.

But thinking from a more fundamental perspective, why does pip even need to be involved with any part of PEP 517? If pip is meant to be viewed as a package manager then isn't its key focus on fetching and installing appropriate things?

Well, it's because pip may have to build a wheel from a post-517 sdist that it downloaded from PyPI when a project doesn't provide e.g. a Windows wheel. That's why pip needs to be PEP 517-aware at all. Otherwise pip could just work in a wheel-only world and not even care about building wheels to begin with.

But where does building sdists come into play? You need to be able to make them, and you need to be able to get them up to PyPI. But beyond pip needing to know how to unpack an sdist to make the wheel it wants to work with, pip doesn't actually need to produce an sdist.

But I think twine is the tool that needs a way to specify how to produce an sdist. If we want to view twine as the tool to upload artifacts to PyPI then we need twine to know how to produce sdists and wheels in a PEP 517 world, not pip.

Maybe I'm missing something fundamental here about pip and all of this is wrong, but from where I'm sitting it seems the key thing pip needs from PEP 517 is how to go from a bunch of source to a wheel so that it can install that wheel. Now pip needs to know how to consume an sdist that it gets from PyPI that has a pyproject.toml, but it technically doesn't need to know how to produce one if in the end it really just wants a wheel. Twine, OTOH, does need a way to produce an sdist as well as a wheel so it can upload those to PyPI.

And so I think in a very wordy way, I just said we need to stop saying "pip needs a standardized way to produce an sdist" and instead start saying "twine needs a way to produce an sdist". And that leads to the question about whether PEP 517 must cover sdist production for twine because we want to have that solved before we have the consumption side in pip in place. Or put another way, are we okay with pip consuming things that twine simply can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't be held up, while a "no" means we need a solution for twine.