From my point of view mandatory build isolation will make building thinks slow and bad, besides being totally unrelated to the idea of a working bootstrap requirements feature.
On Thu, May 5, 2016 at 9:37 PM Robert Collins <robertc@robertcollins.net> wrote:
On 5 May 2016 at 21:47, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, May 4, 2016 at 11:57 PM, Robert Collins ...
I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.
Are those requirements pip installable?
Either they are, or they will be soon.
Thats good. It occurs to me that scientific builds may be univerally broken because folk want to avoid easy-install and the high cost of double builds of things. E.g. adding bootstrap_requires will let folk fix things?
But the main question is : why are these packages staying inaccurate? Even in the absence of a systematic solution I'd expect bug reports and pull requests to converge on good dependencies.
...
So the 10x thing is defining how the thing doing the isolation (e.g. pip) should handle things that can't be installed but are already available on the system.
And that has to tunnel all the way out to the user, because its context specific, its not an attribute of the dependencies per se (since new releases can add or remove this situation), nor of the consuming thing (same reason).
# User experience today on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install foo
# User experience with build isolation on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install --no-isolated-environment foo
So thats one way that isolation can be controlled yes.
It'd even be straightforward for pip to notice that the requirement that it failed to satisfy is already satisfied by the ambient environment, and suggest --no-isolated-environment as a solution.
Ultimately, its not even an interopability question: pip could do isolated builds now, if it chose, and it has no ramifications as far as PEPs etc are concerned.
That's not true. In fact, it seems dangerously naive :-/
If pip just went ahead and flipped a switch to start doing isolated builds now, then everything would burst into flame and there would be a howling mob in the bug tracker. Sure, there's no PEP saying we *can't* do that, but in practice it's utterly impossible.
We've done similarly omg it could be bad changes before - e.g. introducing wheel caching. A couple of iterations later and we're in good shape. Paul seems to think we could introduce it gracefully - opt in, then default with fallback, then solely opt-out.
If we roll out this feature without build isolation, then next year we'll still be in the exact same situation we are today -- we'll have the theoretical capability of enabling build isolation, but everything would break, so in practice, we won't be able to.
The reason I'm being so intense about this is that AFAICT these are all true:
Premise 1: Without build isolation enabled by default, then in practice everyone will putter along putting up with broken builds all the time. It's *incredibly* easy to forget to declare a build dependency, it's the kind of mistake that every new user makes, and experienced users too.
I know lots of projects that already build in [mostly] clean environments all the time - e.g. anything using tox[*], most things that use Travis, Appveyor, Jenkins etc. Yes, if its not there by default it requires effort to opt-in, and there will be a tail of some sort. I don't disagree with the effect of the premise, but I do disagree about the intensity.
[*]: yes, to runs setup.py sdist outside the environment, so setup_requires doesn't get well exercised. IIRC tox is adding a feature to build in a venv so they will be exercised.
Premise 2: We can either enable build isolation together with the new static bootstrap requirements, or we can never enable build isolation at all, ever.
I don't agree with this one at all. We can enable it now if we want: yes, its a behavioural change, and we'd need to phase it in, but the logistics are pretty direct.
Conclusion: If we want to ever reach a state where builds are reliable, we need to tie build isolation to the new static metadata.
If you have some clever plan for how we could practically transition to build isolation without having them opt-in via a new feature, then that would be an interesting counter-argument; or an alternative plan for how to reach a point where build requirements are accurate without being enforced; or ...?
As Paul said, add it off by default, phase it in over a couple of releases. 0: opt-in 1: opt-out but when builds fail disable and retry (and warn) 2: opt out
pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64 is going to be super common, Power8 another one, let alone less common but still extant and used architectures like PPC, itanium, or new ones like x86_32 [If I remember the abbreviation right - no, its not i386].
(it's x32)
Thanks :)
manylinux is helpful here, but it's not necessary -- build isolation just requires that the dependencies be pip installable, could be from source or whatever. In practice the wheel cache will kick in and handle most of the work.
*only* for things that honour setup.py - which the vast majority of SWIG and SWIG style extension modules do not.
Solve that underlying problem - great, then isolation becomes an optimisation question for things without manylinux wheels. But if we don't solve it then isolation becomes a 'Can build X at all' question, which is qualitatively different.
More like 'can build X at all (without adding one command line option)'. And even this is only if you're in some kind of environment that X upstream doesn't support -- no developer is going to make a release of X with build isolation turned on unless build isolation works on the platforms they care about.
There's some thinking here I don't follow: upstreams don't opt into-or-outof build isolation AFAICT, *other* than by your proposal to couple it to declaring bootstrap requirements. The idea that upstream should care about whether their thing is built in an isolated environment or not doesn't make any sense to me. Upstream *should* make sure their thing can build in an isolated fashion - and then not care anymore.
-Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig