[Distutils] Working toward Linux wheel support

[Nate Coraor]

Hi all,

I think Nathaniel raised a lot of important points below and I do see the
case for a "base environment" meta tag. The implementation of sniffing out
those environments on a wide array of systems may be complicated, but
perhaps we can, er, borrow from conda here. I do think the glibc tag is
useful as well, although it may be unnecessary if there's a way to deal
with the glibc version in a base environment.

However, I don't think I'm qualified to make a decision on what direction
to go, and I'd like to work on updating PEP 425 for improved platform tags.
So, I'm hoping to kickstart the discussion again and see if we can get a
consensus on what to do.

One proposal - if PEP 425 were updated to indicate that the platform tag
can be more than simply `distutils.util.get_platform()`, and some language
as to its intent, without specifying exactly what it must be, we could
separate out the exact details into the packaging documentation as Nick has
suggested.

--nate

On Wed, Sep 9, 2015 at 7:49 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Sep 9, 2015 at 8:06 AM, Nate Coraor <nate at bx.psu.edu> wrote:
> > On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft <donald at stufft.io> wrote:
> >> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu)
> wrote:
> >> >> >>>
> >> >> >>> I'll create PRs for this against wheel and pip shortly. I can
> also
> >> >> >>> work
> >> >> >>> on a PEP for the platform tag - I don't think it's going to need
> to
> >> >> >>> be a
> >> >> >>> big one. Are there any preferences as to whether this should be a
> >> >> >>> new PEP
> >> >> >>> or an update to 425?
> >> >> >>>
> >> >
> >> > Coming back to this, I'm wondering if we should include the libc
> >> > implementation/version in a less generic, but still generic linux
> wheel.
> >> > Right
> >> > now if you staticly link I think the only platform ABIs you need to
> >> > worry about
> >> > are libc and Python itself. Python itself is handled already but libc
> is
> >> > not.
> >> > The only thing I've seen so far is "build on an old enough version of
> >> > glibc
> >> > that it handles anything sane", but not all versions of Linux even use
> >> > glibc at
> >> > all.
> >>
> >> This feels kinda half-baked to me?
> >>
> >> "linux" is a useful tag because it has a clear meaning: "there exists
> >> a linux system somewhere that can run this, but no guarantees about
> >> which one, good luck". When building a wheel it's easy to tell whether
> >> this tag can be correctly applied.
> >
> >
> > I'm not sure how it'd be possible to tell. The same meaning for a generic
> > tag would be true of any wheel built, regardless of whether the wheel has
> > dependencies in addition to libc.
>
> Sure... my point is just that "linux" is unambiguous and fills a
> niche: it unambiguously says "you're on your own", and sometimes
> that's the best we can hope to say.
>
> >> Distro-specific tags are useful because they also have a fairly clear
> >> meaning: "here's a specific class of systems that can run this, so
> >> long as you install enough packages to fulfill the external
> >> dependencies". Again, when building a wheel it's pretty easy to tell
> >> whether this tag can be correctly applied. (Of course someone could
> >> screw this up, e.g. by building on a system is technically distro X
> >> but has some incompatible hand-compiled libraries installed, but 99%
> >> of the time we can guess correctly.)
> >>
> >> If we define a LSB-style base system and give it a tag, like I don't
> >> know, the "Python base environment", call it "linux_pybe1_core" or
> >> something, that describes what libraries are and aren't available and
> >> their ABIs, and provide docs/tooling to help people explicitly create
> >> such wheels and check whether they're compatible with their system,
> >> then this is also useful -- we have proof that this is sufficient to
> >> actually distribute arbitrary software usefully, given that multiple
> >> distributors have converged on this strategy. (I've been talking to
> >> some people off-list about maybe actually putting together a proposal
> >> like this...)
> >>
> >> To me "linux_glibc_2.18" falls between the cracks though. If this
> >> starts being what you get by default when you build a wheel, then
> >> people will use it for wheels that are *not* statically linked, and
> >> what that tag will mean is "there exists some system that can run
> >> this, and that has glibc 2.18 on it, and also some other unspecified
> >> stuff, good luck". Which is pretty useless -- we might as well just
> >> stick with "linux" in this case. OTOH if it's something that builders
> >> have to opt into, then we could document that it's only to be used for
> >> wheels that are statically linked except for glibc, and make it mean
> >> "*any* system which has glibc 2.18 or later on it can run this". Which
> >> would be useful in some cases.
> >
> >
> > This is a tooling issue. If wheel (the package) inspects the built .so
> files
> > and finds they are not dynamically linked to anything not included with
> > glibc, it can apply the glibc tag. Otherwise, it'd apply the distro tag.
> > There's no possibility for human error here, unless they explicitly
> override
> > the platform tag.
> >
> >>
> >> But at this point it's basically a
> >> version of the "defined base environment" approach, and once you've
> >> gone that far you might as well take advantage of the various
> >> distributors' experience about what should actually be in that
> >> environment -- glibc isn't enough.
> >
> > While I agree that glibc isn't always enough, defining a base environment
> > that may not be met by the "standard" install of popular distributions
> makes
> > unprivileged wheel installation much more difficult.
>
> Yeah, which is why my suggestion is that we steal the "base
> environment" definition from the folks like Continuum and Enthought
> who have already done the work of determining what is in the
> "standard" install of popular distributions, and have spent years
> actually distributing packages to unprivileged users :-).
>
> > It's also not going to
> > work out of the box on older distributions that wouldn't provide whatever
> > standardized mechanism is defined for a list of "base environments
> currently
> > provided by this system" (unless pip does the work itself at runtime to
> > determine whether a base environment is met).
>
> Right -- which is basically what pip will have to do to figure out the
> current glibc version too, right?
>
> Trying to guess whether the installed versions of several libraries
> are really ABI compatible with what we expect is harder than trying to
> guess whether the installed version of glibc alone is really ABI
> compatible with what we expect, but in both cases it's basically a
> heuristic (some distros could have local patches to their glibc that
> break ABI, who knows) and in both cases it's basically safe to just
> assume it will work (because if we stick to libraries that other
> distributors are already depending on then we have years of experience
> that it pretty much always works).
>
> > Maybe an important question:
> > how many popular packages with C Extensions have dependencies in
> addition to
> > glibc?
>
> Certainly enough that the major distributors of binary packages on
> Linux, like Continuum and Enthought, have decided that they need to
> require more than glibc :-).
>
> libstdc++ is an example of one particularly common external dependency.
>
> To be clear: if you're talking specifically about the model where we
> validate that the extensions are statically linked before we enable
> the glibc tag, then I don't think it will do any harm to have it as an
> option. It just seems redundant with the more general solution.
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150921/1a50d891/attachment.html>