On Tue, 30 May 2017 at 11:40 Donald Stufft <donald@stufft.io> wrote:

On May 30, 2017, at 2:17 PM, Brett Cannon <brett@python.org> wrote:

Just to make sure I'm following this correctly, Donald is asking for:
  • A way for pip to ask back-ends what files should be in an sdist from a source checkout or to make an actual sdist
    • Because sdists are a thing and so we should support them properly
    • To make it so building wheels interact with just the files from an sdist instead of skipping that and going straight from a source checkout
  • Have this be a part of PEP 517 or at least a requirement for back-ends to support so it doesn't get left out
Am I missing anything?

More or less. If I had my druthers we’d just add another mandatory method to the API, something like:

    def build_sdist_tree(sdist_directory, config_settings) -> None:

All this does is assemble the would be sdist tree into sdist_directory, handling any sdist creation time steps WITHOUT actually tar+gz’ing up this tree. My tests show that this is basically as fast as the copytree option in the simple case (which makes sense, that’s basically all it is) and is way faster than actually assembling the entire sdist tarball and all.

So the back-ends then need to provide a way for users to specify what files to go into the sdist (e.g. the MANIFEST.in case). Obviously a fallback is to just copy everything or just what is used to build the wheel (overly board or overly selective, respectively), but I doubt tools will want to do either in general and so they will need to come up with some solution to specify extra files like READMEs and stuff.

To handle creation, we can either have ``twine sdist`` or something like that which just calls the above API and then runs ``shutil.make_archive()`` on the final directory.

Do we want to say that sdist creation is the front-end's job? If so that is different from PEP 517 where the back-end packages zip up the wheel versus simply providing all the data and files that make up a wheel as seems to be suggested in the sdist case here.

When building a wheel, pip can then just inside of this directory, call into the wheel metadata/wheel build API and ultimately install that wheel. It may even make sense for the build_wheel API to produce an “unpacked wheel” as well and again let something like ``twine wheel`` handle running ``shutil.make_archive()`` on it. Similar to the sdist case, this would further reduce the install time by avoiding the need to zip and then immediately unzip inside of pip.

If we're going to go down a route where back-ends simply provide the files and the front-ends do the packaging then it makes sense to do that for wheels as well in PEP 517, separating the back-ends to create the components of the final artifact while the front-ends handle the bundling up for all of it into its final form. This makes the back-ends purely a compiler-and-metadata thing and the front-ends a manage-final-files thing.


One unrelated thing I just noticed, I don’t think the PEP states how pip is supposed to communicate to this API what it’s actually building. Is it just assumed to be ``os.curdir()``? Ideally I think we’d add another parameter to all of these functions that is the directory containing the “input” to each of these.

Donald Stufft