[Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 19:47, Nathaniel Smith <njs at pobox.com> wrote:
> The reason I'm being so intense about this is that AFAICT these are all true:
>
> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.
>
> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.
>
> Conclusion: If we want to ever reach a state where builds are
> reliable, we need to tie build isolation to the new static metadata.

OK, I think I see where we're talking past each other here.

Yes, being able to do isolated builds is important, but we don't need
to invent a Python specific solution to build isolation, as build
isolation can already be handled by running a build in a fresh VM, or
in a container, and continuous integration systems already let people
do exactly that.

This means that even if the original publisher of a package doesn't
regularly run a "Can I reliably rebuild this from source on a clean
system?" check, plenty of consumers of their software will, and those
folks will complain if the build dependencies are wrong. (And folks
consuming pre-built binaries won't care in the first place).

Longer term, as an example of what increasing automation makes
possible, folks in Fedora are exploring what would be involved in
doing automatic mass rebuilds of PyPI as RPM packages [1], and I
assume they'll eventually get to a point where the problems in the
automation pipeline are ironed out, so they'll instead be looking at
problems like expressing external deps in the upstream metadata [2],
as well as finding errors in the dependency definitions of individual
packages.

The only thing that absolutely *has* to be handled centrally by
distutils-sig is ensuring that build requirements can be expressed
accurately enough to allow for fully automated builds on a clean
system. Everything else (including quality assurance on build
dependencies) is more amenable to distributed effort.

Cheers,
Nick.

[1] http://miroslav.suchy.cz/blog/archives/2016/04/21/wip_rebuilding_all_pypi_modules_as_rpm_packages/index.html
[2] https://github.com/pypa/interoperability-peps/pull/30

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia