On May 30, 2017, at 7:37 AM, Donald Stufft <donald@stufft.io> wrote:



Post a to-be-written sdist-source-tree-export PEP:

- pip (et al) can use setup.py to build from an unfiltered source tree
- pip (et al) can use setup.py to build from an sdist
- pip (et al) can use the pyproject.toml build-backend setting to
build from an unfiltered source tree
- pip (et al) can use the pyproject.toml build-backend setting to
build from an sdist
- pip (et al) can use a new pyproject.toml setting defined in that PEP
("source-filter" perhaps?) to export a filtered version of a source
tree, otherwise they fall back on copying the entire unfiltered tree
(similar to the way build-backend falls back to setuptools & wheel if
not otherwise specified)

That approach would decouple the export backends from the build
backends, so we might even end up with a common VCS-aware source
exporter that projects could nominate (e.g. by adding this
functionality to twine), rather than every build backend having to
define its own source export logic.

Note that I'm also fine with pip as a project saying that it will only
ship support for the build-backend interface once the source filtering
interface is also defined and implemented.

I'm just saying that I don't see a close enough link between "here is
how to build this component from source" and "here is how to export a
filtered source tree for this component from an unfiltered VCS
checkout" for it to make sense to define them as part of the same
backend API. The only connection I'm aware of is that it makes sense
for projects to ensure that their source filtering when creating an
sdist isn't leaving out any files needed by their build process, but
that's no different from making sure that your build process produces
a wheel file that actually works when installed.


I don’t think there is any value in decoupling the generation of what goes into an sdist from the tool that builds them. If we did that, I suspect that 100% of the time the exact same tool is going to be used to handle both anyways (or people just won’t bother to put in the extra effort to produce sdists). I think trying to split them up serves only to make the entire toolchain harder and more complicated for people who aren’t stepped in packaging lore to figure out what goes where and what does what. The fact that we have different mechanisms just to control what goes into a sdist (MANIFEST.in) vs what gets installed (package_data) already confuses people, further splitting these two steps apart is only going to make that worse.

Keeping the two together also completely sidesteps the problems around “well what if only the sdist tool is defined but the build tool isn’t?” Or “what if only the build tool is defined but the sdist tool isn’t?”.

The only value I can even think of, is that some of the code is going to be re-usable, but we already have a perfectly serviceable way of allowing code re-use: publish a library and have end tools consume it. 



I think my other thing here is that I don’t even think as written you *can* separate the two concepts. The reason being they’re both going to need the same metadata (like name, version, etc). If you want to have two separate tools handling then, that the build wheel step should accept *only* an sdist (unpacked or otherwise) as the “input” to this build tool and it should not accept a VCS input at all. The wheel build tool can then read the metadata like name, version, etc from whatever metadata exists in the hypothetical sdist format. Otherwise you’ll need two different to both implement the same scheme for determining version (since you’d be able to run both options from a VCS checkout).

Now I’m perfectly fine mandating that builds go VCS -> sdist -> wheel -> install [1], but that’s not what the current PEP does and making it do that would be significantly more effort since you’d have to spend more time hammering out what metadata is “sdist metadata” and what metadata is “Wheel metadata” and I really don’t think that particular rabbit hole is an effective use of our time.

[1] Indeed I think this being the only path (besides of course editable) is a overall good thing and I’d support it, but even if we did I still would think that separate tools for this is a waste of time which can also be added after the fact with a simple shim package if people do end up finding it very useful.

—
Donald Stufft