On 1 June 2017 at 21:45, Brett Cannon
And so I think in a very wordy way, I just said we need to stop saying "pip needs a standardized way to produce an sdist" and instead start saying "twine needs a way to produce an sdist". And that leads to the question about whether PEP 517 must cover sdist production for twine because we want to have that solved before we have the consumption side in pip in place. Or put another way, are we okay with pip consuming things that twine simply can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't be held up, while a "no" means we need a solution for twine.
The question is more, are we okay with pip consuming things (pyproject.toml source trees) for which we don't define any means of uploading to a package index (which is basically what a sdist is - a way of publishing sources on PyPI). My answer to that is "no". pip doesn't need to be able to provide a user interface that builds a sdist. It *could* provide one, as an enhancement, but it's not *necessary*. Whether the canonical "build a sdist" command should be pip or (say) twine is a side issue here (although the ability to *have* a canonical command, and not rely on each backend having its own incompatible way, *is* important IMO - but that's not the point here). However, there are a number of places where pip indirectly needs the ability to build a sdist (or do something closely equivalent). pip needs a way to deal with "pip install foo". There are 2 scenarios here: 1. The index (PyPI) contains an appropriate wheel. Then pip just installs it (using the wheel library). PEP 517 isn't involved, no build, easy. 2. There is no appropriate wheel. There are 2 subcases here: 2a. There is no sdist. We're stuck. See below. 2b. There is a sdist (whatever that means). As you say, pip needs to be able to consume that sdist. PEP 517 covers this as it stands. pip also needs a way to deal with "pip install <local directory>. In this case, pip (under its current model) copies that directory to a working area. In that area, it runs the build command to create a wheel, and proceeds from there. In principle, there's little change in a PEP 517 world. But again, see below. There are other things here, but they are largely marginal, or similar to the above cases (the big remaining case is editable installs, but that's a whole other question). Notes: - The case 2a in the "pip install foo" example is of concern. At the moment, building sdists is trivial (python setup.py sdist) and there's effectively no barrier to publishing sdists. In a PEP 517 world, backends may or may not provide a "build a sdist" command (flit doesn't, and for a long time I believe didn't propose to do so - partially because what constituted a sdist at the time was inappropriate for them, but they were happy to be wheel-only). That means that users of that backend are basically unable to upload source, and the likelihood of scenario 2a goes *way* up. Making sure that PEP 517 mandates that backends at least *allow* tools like twine to build a sdist from a source tree significantly alleviates this risk. - Copying a local directory to a temporary area is demonstrably a serious performance issue for pip. We have a number of bugs raised over this. The problem is that we have to blindly copy *everything*, including arbitrarily large amounts of irrelevant data. We wanted to switch to "build a sdist from the local directory, proceed from the "we have a sdist" case. But some uses of setup.py break with that approach. One of the hopes for PEP 517 is that we avoid that problem, because we make it the backend's problem (by asking the backend to tell us how to know what to copy, in one form or another). If there's no support in the PEP for this, we end up having to accept that going forward we'll *still* have this problem (unless we add an extra mandatory backend hook in a new PEP, but backends can still claim support for only PEP 517 and not the new PEP). This is admittedly an enhancement to pip, not essential functionality, but the point of this whole process is to enable the packaging ecosystem (including pip!) to move forward, rather than being paralyzed by the constraints of the old setup.py system. So leaving the same constraints in the new system isn't helpful. - The local directory -> sdist -> wheel -> install model has its issues, but it's a much cleaner model for pip (if we can get to it without breaking use cases) - for all the same reasons that switching from direct installs to installs via wheels was a huge win for us. To implement that model internally, pip would need a means of building a sdist. I hope this helps. If nothing else, it makes your comments look less wordy by comparison :-) Paul