On 5 May 2016 at 21:47, Nathaniel Smith <njs@pobox.com> wrote:
> On Wed, May 4, 2016 at 11:57 PM, Robert Collins
...
>>> I don't think I've ever seen a package that had accurate
>>> setup_requires (outside the trivial case of packages where
>>> setup_requires=[] is accurate). Scientific packages in particular
>>> universally have undeclared setup requirements.
>>
>> Are those requirements pip installable?
>
> Either they are, or they will be soon.

Thats good. It occurs to me that scientific builds may be univerally
broken because folk want to avoid easy-install and the high cost of
double builds of things. E.g. adding bootstrap_requires will let folk
fix things?

But the main question is : why are these packages staying inaccurate?
Even in the absence of a systematic solution I'd expect bug reports
and pull requests to converge on good dependencies.

...
>> So the 10x thing is defining how the thing doing the isolation (e.g.
>> pip) should handle things that can't be installed but are already
>> available on the system.
>>
>> And that has to tunnel all the way out to the user, because its
>> context specific, its not an attribute of the dependencies per se
>> (since new releases can add or remove this situation), nor of the
>> consuming thing (same reason).
>
> # User experience today on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install foo
>
> # User experience with build isolation on i386
> $ pip install foo
> <... error: missing pyqt5 ...>
> $ apt install python-pyqt5
> $ pip install --no-isolated-environment foo

So thats one way that isolation can be controlled yes.

> It'd even be straightforward for pip to notice that the requirement
> that it failed to satisfy is already satisfied by the ambient
> environment, and suggest --no-isolated-environment as a solution.
>
>> Ultimately, its not even an interopability question: pip could do
>> isolated builds now, if it chose, and it has no ramifications as far
>> as PEPs etc are concerned.
>
> That's not true. In fact, it seems dangerously naive :-/
>
> If pip just went ahead and flipped a switch to start doing isolated
> builds now, then everything would burst into flame and there would be
> a howling mob in the bug tracker. Sure, there's no PEP saying we
> *can't* do that, but in practice it's utterly impossible.

We've done similarly omg it could be bad changes before - e.g.
introducing wheel caching. A couple of iterations later and we're in
good shape. Paul seems to think we could introduce it gracefully - opt
in, then default with fallback, then solely opt-out.

> If we roll out this feature without build isolation, then next year
> we'll still be in the exact same situation we are today -- we'll have
> the theoretical capability of enabling build isolation, but everything
> would break, so in practice, we won't be able to.
>
> The reason I'm being so intense about this is that AFAICT these are all true:
>
> Premise 1: Without build isolation enabled by default, then in
> practice everyone will putter along putting up with broken builds all
> the time. It's *incredibly* easy to forget to declare a build
> dependency, it's the kind of mistake that every new user makes, and
> experienced users too.

I know lots of projects that already build in [mostly] clean
environments all the time - e.g. anything using tox[*], most things
that use Travis, Appveyor, Jenkins etc. Yes, if its not there by
default it requires effort to opt-in, and there will be a tail of some
sort. I don't disagree with the effect of the premise, but I do
disagree about the intensity.

[*]: yes, to runs setup.py sdist outside the environment, so
setup_requires doesn't get well exercised. IIRC tox is adding a
feature to build in a venv so they will be exercised.

> Premise 2: We can either enable build isolation together with the new
> static bootstrap requirements, or we can never enable build isolation
> at all, ever.

I don't agree with this one at all. We can enable it now if we want:
yes, its a behavioural change, and we'd need to phase it in, but the
logistics are pretty direct.

> Conclusion: If we want to ever reach a state where builds are
> reliable, we need to tie build isolation to the new static metadata.
>
> If you have some clever plan for how we could practically transition
> to build isolation without having them opt-in via a new feature, then
> that would be an interesting counter-argument; or an alternative plan
> for how to reach a point where build requirements are accurate without
> being enforced; or ...?

As Paul said, add it off by default, phase it in over a couple of releases.
0: opt-in
1: opt-out but when builds fail disable and retry (and warn)
2: opt out

>> pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
>> is going to be super common, Power8 another one, let alone less common
>> but still extant and used architectures like PPC, itanium, or new ones
>> like x86_32 [If I remember the abbreviation right - no, its not i386].
>
> (it's x32)

Thanks :)

> manylinux is helpful here, but it's not necessary -- build isolation
> just requires that the dependencies be pip installable, could be from
> source or whatever. In practice the wheel cache will kick in and
> handle most of the work.

*only* for things that honour setup.py - which the vast majority of
SWIG and SWIG style extension modules do not.

>> Solve that underlying problem - great, then isolation becomes an
>> optimisation question for things without manylinux wheels. But if we
>> don't solve it then isolation becomes a 'Can build X at all' question,
>> which is qualitatively different.
>
> More like 'can build X at all (without adding one command line
> option)'. And even this is only if you're in some kind of environment
> that X upstream doesn't support -- no developer is going to make a
> release of X with build isolation turned on unless build isolation
> works on the platforms they care about.

There's some thinking here I don't follow: upstreams don't opt
into-or-outof build isolation AFAICT, *other* than by your proposal to
couple it to declaring bootstrap requirements. The idea that upstream
should care about whether their thing is built in an isolated
environment or not doesn't make any sense to me.
Upstream *should* make sure their thing can build in an isolated
fashion - and then not care anymore.

-Rob
--
Robert Collins <rbtcollins@hpe.com>
Distinguished Technologist
HP Converged Cloud
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig