[Distutils] Working toward Linux wheel support

[Donald Stufft]

On September 8, 2015 at 1:29:53 PM, Nate Coraor (nate at bx.psu.edu) wrote:
> On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote:
> 
> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote:
> > > >>>
> > > >>> I'll create PRs for this against wheel and pip shortly. I can also
> > work
> > > >>> on a PEP for the platform tag - I don't think it's going to need to
> > be a
> > > >>> big one. Are there any preferences as to whether this should be a
> > new PEP
> > > >>> or an update to 425?
> > > >>>
> >
> > Coming back to this, I'm wondering if we should include the libc
> > implementation/version in a less generic, but still generic linux wheel.
> > Right
> > now if you staticly link I think the only platform ABIs you need to worry
> > about
> > are libc and Python itself. Python itself is handled already but libc is
> > not.
> 
> The only thing I've seen so far is "build on an old enough version of glibc
> > that it handles anything sane", but not all versions of Linux even use
> > glibc at
> > all.
> 
> 
> This proposal makes a lot of sense to me. pip will need an update to do the
> backwards compatibility, and it may be a little ugly to do this all on the
> platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be
> installed on systems that identify as linux_x86_64_ubuntu_14_04, but
> linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as
> linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag
> prefixes or patterns should be considered backward compatible, and which
> should not. Granted, new libcs do not pop up overnight, so it's not exactly
> a nightmare scenario.
> 
> Wheel should be updated to generate the "libc-generic" wheels by default
> when nothing other than libc is dynamically linked. It'll need libc
> vendor/version detection.
> 
> Alternatively, the platform tag could be split in two, in which case you
> have a "generic" portion (which would probably be what it currently is,
> distutils.util.get_platform()) and a "specific" portion (the distro or
> libc), possibly prefixed with something to avoid having to maintain a list
> of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs.
> 'c_glibc_2_19')?
> 
> I don't think there is a strong case to include the libc version in the
> specific portion when a distro version will also be specified, because the
> distro is supposed to define the ABI (at least in the case of distros with
> stable ABIs), and that includes the libc compatibility. So for psycopg2
> wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for
> SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19).
> 
> It's then up to PyPI project owners to build on whatever platforms they
> wish to support.
> 

I think it's reasonable to not include the libc when the wheel is distro
specific. I think the barrier to entry on adding new tags is far lower than
adding a whole new type of tag. Right now, I think our longest tag is for OSX
which is something like macosx_10_10_x86_64 at 19 chars, I don't think it's
much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or
linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special c or
d prefix, we can just treat it as ==, and special case glibc as >= like we're
currently special casing the macosx wheels to be >=.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA