[Distutils] abstract build system approaches redux

[Robert Collins]

This is my attempt to consolidate the three different proposals we
have at the moment into a single set of design choices.

We have three draft designs - from Nathaniel (517), Donald (unnumberd)
and mine (516).

For purposes of comparison, I've skipped all rationale here, and only
listed the highlights that differ across the proposals. If you haven't
read all three, you should before continuing.

My goal is to let us pick a final shape - and then we can shave the
rough edges off of that shape to get a final PEP (or PEPs)

517:
 - Python interface to the system
 - build requires, optional wheel metadata (to directory on disk),
build wheel. install editable commands

516:
 - CLI interface to the system
 - build requires, mandatory wheel metadata, build wheel, install
editable commands

Donald's:
 - Python interface to the system
 - defines new 'source wheel' concept
 - source_metadata, source_wheel, binary_metadata, binary_wheel commands
 - 'develop' // editable not addressed currently
 - outputs directories on disk, rather than zipped up things


Ok, now here's my view on the differences...

Python vs CLI - I don't care enough to argue. I think a CLI is better
in this space, as running commands is the lingua franca of CLI's;
previously Donald has advocated for that as well, but his proposal is
now a Python API. *shrug*. I want the BDFL-Delegate to choose.

516 and 517 are otherwise identical in all meaningful ways - there's
plenty of stuff to bikeshed on (e.g. should wheel metadata be optional
or mandatory), but I don't think it is crucial: we can always change
this in future by issuing a new version. Its not free, but if we can't
agree, we can at least find out later. Donalds and 516 both have the
ability to get metadata out explicitly as a mandatory thing, so I
think we should make it mandatory and weaken it in future if e.g. flit
eventually come back saying its a burden. [I don't believe it will
be].

File formats - BDFL-Delegate can pick. I so don't care :).

New source format. I'm quite strongly against this at the moment:
Building from VCS is now a fairly major thing in the community, and we
can't mandate new URLs for all sources. We're going to thus be forcing
a change to all tooling (e.g. Debian and Fedora and pip and others)
that know how to build things from an arbitrary VCS url. A new source
extension in pypi doesn't reduce the impact of that at all, and the
necessary code to handle that will also handle sdists's on pypi that
have the new config file and may be missing a setup.py. If there is a
new source extension in pypi, we should expect that folk using e.g.
flit will *not* upload old style sdists to pypi - they don't at the
moment, and the objection to doing so will *remain*. -> We don't save
any grief or friction for the adoption of the new thing, nor enable
new build systems any more easily. A shim to the new shiny would
enable current sdists for anything adopting the new thing in exactly
the same manner as it would if we reuse the extension. I can see an
argument for a new iteration of the sdist format for better metadata,
and to be able to tell what metadata is trustable etc, but I don't see
that as related to the problem of enabling third party build systems -
and since all the easy cases will be solved by binary wheels I think
we're in rapidly diminishing territory here - if something can't
upload a binary wheel, its quite likely to have complex dependencies -
and we haven't solved the numpy ABI problem yet, so I'd like to
actually focus on that well and solve it - and once solved see if we
can retrofit it into the existing sdist format, or if a new one is
actually needed.

Develop is essential to tackle IMO. We can either fully PEP define it
now - and if so we should stop talking about this and start talking
about that. If we don't have bandwidth to do that yet (and AFAICT we
don't, based on IRC discussions and the headaches of interactions with
the three different namespace package possibilities, virtualenvs and
user installs....).

Outputting the wheel as an unzipped directly on disk would be a nice
optimisation for big trees, I think, so perhaps we should apply that
to 516/517 to reduce the set of things we're discussing.

-Rob

516: https://github.com/pypa/interoperability-peps/blob/master/pep-0516-build-system-abstraction.rst
517: https://github.com/pypa/interoperability-peps/blob/master/pep-0517-build-system-abstraction.rst
Donalds: https://github.com/pypa/interoperability-peps/compare/master...dstufft:build-pipeline




-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud