[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Nick Coghlan]

On 29 May 2017 at 15:37, Donald Stufft <donald at stufft.io> wrote:
>
> On May 28, 2017, at 11:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Hi folks,
>
> I am hereby provisionally accepting PEP 517 as our CLI independent
> interface for build backends, allowing interested parties to move
> forward with implementing it as part of pip and other tools.
>
> As with other provisionally accepted PyPA interoperability
> specifications, it won't be considered completely settled until it has
> been implemented and released in pip and we have at least one
> iteration of end user feedback.
>
> Thanks to Nathaniel & Thomas for their work on the PEP, and to
> everyone that participated in the related discussions!
>
>
>
> Bleh, I had it on my stack to respond to PEP 517, but life has been super
> hectic so I hadn’t gotten around to it.
>
> Here are a few thoughts FWIW:
>
> 1. Using the {package-name}-{package-version}.dist-info in the
> get_wheel_metadata() metadata is a mistake I think. In pip currently we have
> a bug we have not yet been able to track down because there is nothing
> systematically preventing both foobar-1.0.dist-info and foobar-2.0.distinfo
> from existing side by side in a build directory (or inside a wheel for that
> matter). Thus I think this naming scheme is a nuisance and we shouldn’t
> propagate it any further. I would just use something like DIST-INFO/ which
> will completely side step this issue. The only reason I can think of to use
> the current scheme is to make it easier to shutil.copytree it into the
> wheel, but handling that case is trivial.

I think this is an ambiguity in the current spec, since the name of
the metadata directory is wholly controlled by the front-end, but I
now see that "This directory MUST be a valid .dist-info directory as
defined in the wheel specification, except that it need not contain
RECORD or signatures." can be taken as suggesting it is required to
follow the directory naming scheme and hence backends can parse it
accordingly to find out the name of the wheel being built.

That's not the way I actually read it when accepting the PEP, so I
think it should be clarified to say:

* the *name* of the directory is arbitrary, and entirely up to the
build frontend. Build backends MUST NOT attempt to parse it for
information.
* the *contents* of the directory, as produced by the build backend,
MUST be as defined in the wheel specification (only omitting RECORD
and signatures)

If we want to later introduce an RPM style "multiple wheels from a
single sdist" model, that should be its own PEP, with its own
additions to pyproject.toml to define the available alternate wheel
files (so that tools like pyp2rpm and conda skeleton can read it and
adjust their output accordingly) and to the build backend API (to
request building an alternate wheel rather than the default one that's
named after the sdist).

> 2. As I mentioned in previous discussions on this, I think that this
> interface *needs* some mechanism to ask it to build a sdist. Ideally this
> would either fail or recreate the sdist when being run from something that
> is already an sdist. In pip when we’re doing ``pip install .`` we currently
> copy tree the entire thing into a temporary directory, and we run the build
> out of there. For a variety of reasons we’re going to keep build isolation,
> but the current mechanism is painful because it also grabs things like .tox,
> .git, etc. At one point we tried to simply filter those out but it broke
> some packages expectations. The planned (which is on my long list of things
> to do…) mechanism to fix this is to create a sdist from a ``.`` and then
> install from that (building an intermediate wheel as well). This also solves
> another class of bugs that people run into where ``pip install .`` and
> ``python setup.py sdist && pip install dist/*`` give different results. As
> written, this PEP prevents that from happening (and thus, when I implement
> it, I’ll only be able to implement it for old style sdists, and will need to
> tell people to continue to use old style if they don’t want pip to grab
> random directories from ``.``).

As with multiplexed wheel creation, I think "give me a filtered source
tree to use as an out-of-tree build root" can be a separate follow-up
PEP (I'll even propose a name for the API:
"export_sdist_source_tree").

My rationale for seeing it that way, is that while PEP 517 requires
that the current working directory correspond to the root of the
source tree, it *doesn't* require that the source tree be the
unfiltered contents of a VCS checkout. It can't, since the source tree
might have come from an sdist, and MANIFEST.in and friends already
allow that to be a filtered subset of the full VCS contents.

And when it comes to *publishing* an sdist, folks are free to call
their build backend directly for that, rather than necessarily going
through pip to do it.

That said, I *do* think it makes sense for pip to offer an option to
use out-of-tree builds when running `pip install` (even if it isn't
the default), and that means extending the build backend API to cover
creating an exported sdist tree directly from a VCS checkout or other
local directory without necessarily making the sdist first.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia