[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Daniel Holth]

In my experience tools think an archive is an sdist if it is named
properly, contains PKG-INFO with a minimum number of fields

Metadata-Version: 1.1Name: bobVersion: 1.0


and setup.py. setuptools sdists also contain .egg-info but it is unnecessary.


On Wed, May 31, 2017 at 2:41 PM Donald Stufft <donald at stufft.io> wrote:

> On May 31, 2017, at 2:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> On 31 May 2017 at 18:03, Donald Stufft <donald at stufft.io> wrote:
>
> No you’re correct, it currently just invokes ``setup.py sdist
> bdist_wheel``.
> The hook is needed so that Travis can have a singular tool to invoke
> (likely
> twine?) instead of needing to determine if it needs to invoke flit or
> setuptools or mytotallyradbuildthing. The thing I’m trying to express (and
> doing poorly it seems :( ) is that generating a sdist is an important thing
> to have be possible, and it needs to be done in a way that it can be
> invoked
> generically.
>
>
> I don't think that's either unclear or in dispute. The question here
> is whether "produce a sdist" is in scope for this particular PEP.
>
> The problem is that you're proposing using a "build a sdist" hook as
> the means for pip to do its "copy the source to an isolated build
> directory" step. Currently, doing that fails because tools like
> setuptools_scm work differently when run from a VCS checkout instead
> of a sdist. The long term plan for pip is something we've always
> described in terms of going via a sdist, but there's lots of details
> we haven't thrashed out yet. I don't think we're at a point where we
> can insist that in a post-PEP 517 world, we can switch straight to
> building via a sdist. However, I agree with you that we want PEP 517
> to assist us with moving in that direction, and we *definitely* don't
> want to get into a situation like we're in now, where a PEP 517
> compliant backend can leave us with no option better than "copy the
> whole source tree”.
>
>
> I don’t think we can start telling projects if they using a PEP 517 they
> can delete their ``setup.py`` and live in the brave new world (assuming all
> of their tools have been updated to PEP 517) when doing so is removing a
> “standard” interface for producing a sdist. Either a replacement for
> setup.py should support the things we want to keep from setup.py and
> explicitly unsupport things we don’t want, or I don’t think that thing is
> actually a replacement for setup.py and I don’t think we should support it.
>
> Taking pip completely off the table a second, let’s take a look at tox.
> Tox’s default mode of operation is to produce a sdist. Now let’s say I’m
> writing a project that I want to use PEP 517 and get rid of setup.py,
> except now tox is broken with no path forward because PEP 517 doesn’t
> define how to produce a sdist.
>
> The same story is true for TravisCI’s PyPI deployment pipeline, as soon as
> any project starts depending on PEP 517, we completely break that feature
> for them without a path for them to fix it (besides writing a PEP of
> course).
>
> The same is true for Gem Fury’s private PyPI repositories where you can
> ``git push fury`` and have them build a sdist automatically for you.
>
> This same pattern is over and over and over again, projects depend on the
> ability to produce a sdist for any Python project. PEP 517 says that people
> can delete their setup.py but doesn’t provide the mechanism for producing a
> sdist, thus breaking parts of the ecosystem. Simply changing the PEP to say
> “ok you can’t delete your setup.py yet” isn’t acceptable yet either,
> because then you have two competing build systems both who think they
> should be in charge, which makes the entire process *more* confusing for
> the end users than just baking the concept of sdist generation into PEP 517.
>
> Now, independently of that, pip needs a way to take an arbitrary directory
> that might contain a git clone with a bunch of extraneous files in it, or
> it might also just be a sdist that was already unpacked. For a variety of
> reasons we want to copy this directory into a temporary location, but doing
> a blind copy of everything can trigger a bad path where a simple ``pip
> install .`` can take a long time (up to minutes long have been reported in
> the wild) trying to copy the entire directory, including files that we
> don’t even need or want. We need some mechanism for copying these files
> over, and it just so happens that the exact same process needs to occur
> when computing what files going into a sdist, and since I believe that for
> completely unrelated reasons, computing a sdist *must* be a part of any
> attempt to replace setup.py, reusing that simplifies the process of
> creating a PEP 517 backend (since having to only implement build_sdist is
> simpler than having to implement build_sdist AND copy_files_for_build).
>
> In addition to all of the above, we currently have like 7 different
> “paths” installation can go through on the process of going from a VCS
> checkout/developer copy to a installed distribution, we have:
>
> 1) VCS Checkout -> Installed
> 2) VCS Checkout -> Sdist -> Installed
> 3) VCS Checkout -> Wheel -> Installed
> 4) VCS Checkout -> Sdist -> Wheel -> Installed
> 5) VCS Checkout -> Editable Install
> 6) VCS Checkout -> Sdist -> Editable Install
>
> Unless you’re careful to have your packaging done exactly correct, each of
> those 6 can end up having different (and often times surprising behavior)
> that regular end users who are new to packaging (or hell, even old hands)
> hit with some regularity. One of my long term goals is try and reduce the
> number of those paths down, which will make it more likely that people are
> not surprised by edge cases in how their own uses are calling ``pip
> install`` and will ultimately provide a more enjoyable experience using
> pip. We obviously cannot reduce the number of supported methods down to 1,
> but we can reduce them down to:
>
> A) VCS Checkout -> Sdist -> Wheel -> Installed
> B) VCS Checkout -> Editable Install
>
> Implementing build_sdist in PEP 517 and using that to handle copying the
> files from what could either be a VCS checkout OR an unpacked sdist, means
> that we eliminate (1) and (3) from the first list. Ensuring that we only
> ever install a PEP 517 style project by always using build_wheel after
> having used build_sdist then eliminates (2). We’re not dealing with
> editable installs here (and it kind of pains me we aren’t, but they’re a
> much bigger topic so I think it’s unavoidable) but preventing an editable
> install of an sdist would eliminate (6) from above, leaving us with just
> two paths (and the second path requiring an explicit flag to opt into,
> rather than being implicit by nature of what you’re passing into pip and
> what other libraries you have installed).
>
> In addition to all of the above, any part of building a sdist that is more
> complicated than “copy some files”, these build backends are already going
> to have to support by nature of the fact we’re expecting them to generate
> wheel metadata. The wheel metadata has to include the version number, so if
> someone wants to dynamically compute the version number from git, a PEP 517
> backend must already handle that or it simply won’t work.
>
> Finally, we’re should/are assuming that these build projects are going to
> be capable of producing sdists. Thus they already have to implement 99% of
> build_sdist anyways, and the only additional effort on their part is just
> the glue code that wires up their internal mechanism for producing sdists
> to the API that allows a standard mechanism for calling those mechanisms.
> Hopefully it is not controversial that a build tool *must* be capable of
> producing a sdist, since otherwise we’re throwing away support for any non
> Windows/macOS/Linux platform. Implementing a custom “copy these files” is
> *more* effort than exposing the mechanism that they should already have.
>
> So yes, one of the things I want to do with this hook is copy the source
> files to an isolated directory, but that’s not the *only* thing I want to
> do with that hook, and when I see single solution that can solve multiple
> problems, I always vastly prefer that over a single solution that only
> solves a single problem.
>
>
> —
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/dc2999a3/attachment.html>