[Distutils] moving things forward (was: wheel including files it shouldn't)

[Daniel Holth]

>From my point of view mandatory build isolation will make building thinks
slow and bad, besides being totally unrelated to the idea of a working
bootstrap requirements feature.

On Thu, May 5, 2016 at 9:37 PM Robert Collins <robertc at robertcollins.net>
wrote:

> On 5 May 2016 at 21:47, Nathaniel Smith <njs at pobox.com> wrote:
> > On Wed, May 4, 2016 at 11:57 PM, Robert Collins
> ...
> >>> I don't think I've ever seen a package that had accurate
> >>> setup_requires (outside the trivial case of packages where
> >>> setup_requires=[] is accurate). Scientific packages in particular
> >>> universally have undeclared setup requirements.
> >>
> >> Are those requirements pip installable?
> >
> > Either they are, or they will be soon.
>
> Thats good. It occurs to me that scientific builds may be univerally
> broken because folk want to avoid easy-install and the high cost of
> double builds of things. E.g. adding bootstrap_requires will let folk
> fix things?
>
> But the main question is : why are these packages staying inaccurate?
> Even in the absence of a systematic solution I'd expect bug reports
> and pull requests to converge on good dependencies.
>
> ...
> >> So the 10x thing is defining how the thing doing the isolation (e.g.
> >> pip) should handle things that can't be installed but are already
> >> available on the system.
> >>
> >> And that has to tunnel all the way out to the user, because its
> >> context specific, its not an attribute of the dependencies per se
> >> (since new releases can add or remove this situation), nor of the
> >> consuming thing (same reason).
> >
> > # User experience today on i386
> > $ pip install foo
> > <... error: missing pyqt5 ...>
> > $ apt install python-pyqt5
> > $ pip install foo
> >
> > # User experience with build isolation on i386
> > $ pip install foo
> > <... error: missing pyqt5 ...>
> > $ apt install python-pyqt5
> > $ pip install --no-isolated-environment foo
>
> So thats one way that isolation can be controlled yes.
>
> > It'd even be straightforward for pip to notice that the requirement
> > that it failed to satisfy is already satisfied by the ambient
> > environment, and suggest --no-isolated-environment as a solution.
> >
> >> Ultimately, its not even an interopability question: pip could do
> >> isolated builds now, if it chose, and it has no ramifications as far
> >> as PEPs etc are concerned.
> >
> > That's not true. In fact, it seems dangerously naive :-/
> >
> > If pip just went ahead and flipped a switch to start doing isolated
> > builds now, then everything would burst into flame and there would be
> > a howling mob in the bug tracker. Sure, there's no PEP saying we
> > *can't* do that, but in practice it's utterly impossible.
>
> We've done similarly omg it could be bad changes before - e.g.
> introducing wheel caching. A couple of iterations later and we're in
> good shape. Paul seems to think we could introduce it gracefully - opt
> in, then default with fallback, then solely opt-out.
>
> > If we roll out this feature without build isolation, then next year
> > we'll still be in the exact same situation we are today -- we'll have
> > the theoretical capability of enabling build isolation, but everything
> > would break, so in practice, we won't be able to.
> >
> > The reason I'm being so intense about this is that AFAICT these are all
> true:
> >
> > Premise 1: Without build isolation enabled by default, then in
> > practice everyone will putter along putting up with broken builds all
> > the time. It's *incredibly* easy to forget to declare a build
> > dependency, it's the kind of mistake that every new user makes, and
> > experienced users too.
>
> I know lots of projects that already build in [mostly] clean
> environments all the time - e.g. anything using tox[*], most things
> that use Travis, Appveyor, Jenkins etc. Yes, if its not there by
> default it requires effort to opt-in, and there will be a tail of some
> sort. I don't disagree with the effect of the premise, but I do
> disagree about the intensity.
>
> [*]: yes, to runs setup.py sdist outside the environment, so
> setup_requires doesn't get well exercised. IIRC tox is adding a
> feature to build in a venv so they will be exercised.
>
> > Premise 2: We can either enable build isolation together with the new
> > static bootstrap requirements, or we can never enable build isolation
> > at all, ever.
>
> I don't agree with this one at all. We can enable it now if we want:
> yes, its a behavioural change, and we'd need to phase it in, but the
> logistics are pretty direct.
>
> > Conclusion: If we want to ever reach a state where builds are
> > reliable, we need to tie build isolation to the new static metadata.
> >
> > If you have some clever plan for how we could practically transition
> > to build isolation without having them opt-in via a new feature, then
> > that would be an interesting counter-argument; or an alternative plan
> > for how to reach a point where build requirements are accurate without
> > being enforced; or ...?
>
> As Paul said, add it off by default, phase it in over a couple of releases.
> 0: opt-in
> 1: opt-out but when builds fail disable and retry (and warn)
> 2: opt out
>
> >> pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
> >> is going to be super common, Power8 another one, let alone less common
> >> but still extant and used architectures like PPC, itanium, or new ones
> >> like x86_32 [If I remember the abbreviation right - no, its not i386].
> >
> > (it's x32)
>
> Thanks :)
>
> > manylinux is helpful here, but it's not necessary -- build isolation
> > just requires that the dependencies be pip installable, could be from
> > source or whatever. In practice the wheel cache will kick in and
> > handle most of the work.
>
> *only* for things that honour setup.py - which the vast majority of
> SWIG and SWIG style extension modules do not.
>
> >> Solve that underlying problem - great, then isolation becomes an
> >> optimisation question for things without manylinux wheels. But if we
> >> don't solve it then isolation becomes a 'Can build X at all' question,
> >> which is qualitatively different.
> >
> > More like 'can build X at all (without adding one command line
> > option)'. And even this is only if you're in some kind of environment
> > that X upstream doesn't support -- no developer is going to make a
> > release of X with build isolation turned on unless build isolation
> > works on the platforms they care about.
>
> There's some thinking here I don't follow: upstreams don't opt
> into-or-outof build isolation AFAICT, *other* than by your proposal to
> couple it to declaring bootstrap requirements. The idea that upstream
> should care about whether their thing is built in an isolated
> environment or not doesn't make any sense to me.
> Upstream *should* make sure their thing can build in an isolated
> fashion - and then not care anymore.
>
> -Rob
> --
> Robert Collins <rbtcollins at hpe.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160506/6fc7f2c5/attachment-0001.html>