Re: [Distutils] Provisionally accepting PEP 517's declarative build system interface

On May 31, 2017, at 2:01 PM, Paul Moore p.f.moore@gmail.com wrote:

On 31 May 2017 at 18:03, Donald Stufft donald@stufft.io wrote:

No you’re correct, it currently just invokes setup.py sdist bdist_wheel. The hook is needed so that Travis can have a singular tool to invoke (likely twine?) instead of needing to determine if it needs to invoke flit or setuptools or mytotallyradbuildthing. The thing I’m trying to express (and doing poorly it seems :( ) is that generating a sdist is an important thing to have be possible, and it needs to be done in a way that it can be invoked generically.

I don't think that's either unclear or in dispute. The question here is whether "produce a sdist" is in scope for this particular PEP.

The problem is that you're proposing using a "build a sdist" hook as the means for pip to do its "copy the source to an isolated build directory" step. Currently, doing that fails because tools like setuptools_scm work differently when run from a VCS checkout instead of a sdist. The long term plan for pip is something we've always described in terms of going via a sdist, but there's lots of details we haven't thrashed out yet. I don't think we're at a point where we can insist that in a post-PEP 517 world, we can switch straight to building via a sdist. However, I agree with you that we want PEP 517 to assist us with moving in that direction, and we definitely don't want to get into a situation like we're in now, where a PEP 517 compliant backend can leave us with no option better than "copy the whole source tree”.

I don’t think we can start telling projects if they using a PEP 517 they can delete their setup.py and live in the brave new world (assuming all of their tools have been updated to PEP 517) when doing so is removing a “standard” interface for producing a sdist. Either a replacement for setup.py should support the things we want to keep from setup.py and explicitly unsupport things we don’t want, or I don’t think that thing is actually a replacement for setup.py and I don’t think we should support it.

Taking pip completely off the table a second, let’s take a look at tox. Tox’s default mode of operation is to produce a sdist. Now let’s say I’m writing a project that I want to use PEP 517 and get rid of setup.py, except now tox is broken with no path forward because PEP 517 doesn’t define how to produce a sdist.

The same story is true for TravisCI’s PyPI deployment pipeline, as soon as any project starts depending on PEP 517, we completely break that feature for them without a path for them to fix it (besides writing a PEP of course).

The same is true for Gem Fury’s private PyPI repositories where you can git push fury and have them build a sdist automatically for you.

This same pattern is over and over and over again, projects depend on the ability to produce a sdist for any Python project. PEP 517 says that people can delete their setup.py but doesn’t provide the mechanism for producing a sdist, thus breaking parts of the ecosystem. Simply changing the PEP to say “ok you can’t delete your setup.py yet” isn’t acceptable yet either, because then you have two competing build systems both who think they should be in charge, which makes the entire process more confusing for the end users than just baking the concept of sdist generation into PEP 517.

Now, independently of that, pip needs a way to take an arbitrary directory that might contain a git clone with a bunch of extraneous files in it, or it might also just be a sdist that was already unpacked. For a variety of reasons we want to copy this directory into a temporary location, but doing a blind copy of everything can trigger a bad path where a simple pip install . can take a long time (up to minutes long have been reported in the wild) trying to copy the entire directory, including files that we don’t even need or want. We need some mechanism for copying these files over, and it just so happens that the exact same process needs to occur when computing what files going into a sdist, and since I believe that for completely unrelated reasons, computing a sdist must be a part of any attempt to replace setup.py, reusing that simplifies the process of creating a PEP 517 backend (since having to only implement build_sdist is simpler than having to implement build_sdist AND copy_files_for_build).

In addition to all of the above, we currently have like 7 different “paths” installation can go through on the process of going from a VCS checkout/developer copy to a installed distribution, we have:

1) VCS Checkout -> Installed 2) VCS Checkout -> Sdist -> Installed 3) VCS Checkout -> Wheel -> Installed 4) VCS Checkout -> Sdist -> Wheel -> Installed 5) VCS Checkout -> Editable Install 6) VCS Checkout -> Sdist -> Editable Install

Unless you’re careful to have your packaging done exactly correct, each of those 6 can end up having different (and often times surprising behavior) that regular end users who are new to packaging (or hell, even old hands) hit with some regularity. One of my long term goals is try and reduce the number of those paths down, which will make it more likely that people are not surprised by edge cases in how their own uses are calling pip install and will ultimately provide a more enjoyable experience using pip. We obviously cannot reduce the number of supported methods down to 1, but we can reduce them down to:

A) VCS Checkout -> Sdist -> Wheel -> Installed B) VCS Checkout -> Editable Install

Implementing build_sdist in PEP 517 and using that to handle copying the files from what could either be a VCS checkout OR an unpacked sdist, means that we eliminate (1) and (3) from the first list. Ensuring that we only ever install a PEP 517 style project by always using build_wheel after having used build_sdist then eliminates (2). We’re not dealing with editable installs here (and it kind of pains me we aren’t, but they’re a much bigger topic so I think it’s unavoidable) but preventing an editable install of an sdist would eliminate (6) from above, leaving us with just two paths (and the second path requiring an explicit flag to opt into, rather than being implicit by nature of what you’re passing into pip and what other libraries you have installed).

In addition to all of the above, any part of building a sdist that is more complicated than “copy some files”, these build backends are already going to have to support by nature of the fact we’re expecting them to generate wheel metadata. The wheel metadata has to include the version number, so if someone wants to dynamically compute the version number from git, a PEP 517 backend must already handle that or it simply won’t work.

Finally, we’re should/are assuming that these build projects are going to be capable of producing sdists. Thus they already have to implement 99% of build_sdist anyways, and the only additional effort on their part is just the glue code that wires up their internal mechanism for producing sdists to the API that allows a standard mechanism for calling those mechanisms. Hopefully it is not controversial that a build tool must be capable of producing a sdist, since otherwise we’re throwing away support for any non Windows/macOS/Linux platform. Implementing a custom “copy these files” is more effort than exposing the mechanism that they should already have.

So yes, one of the things I want to do with this hook is copy the source files to an isolated directory, but that’s not the only thing I want to do with that hook, and when I see single solution that can solve multiple problems, I always vastly prefer that over a single solution that only solves a single problem.

—

Donald Stufft

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig