[Distutils] Working toward Linux wheel support

[Nate Coraor]

I've started down this road of Linux platform detection, here's the work so
far:

    https://bitbucket.org/natefoo/wheel/src/tip/wheel/platform/linux.py

I'm collecting distribution details here:

    https://gist.github.com/natefoo/814c5bf936922dad97ff

One thing to note, although it's not used, I'm attempting to label a
particular ABI as stable or unstable, so for example, Debian testing is
unstable, whereas full releases are stable. Arch and Gentoo are always
unstable, Ubuntu is always stable, etc. Hopefully this would be useful in
making a decision about what wheels to allow into PyPI.

--nate

On Mon, Aug 24, 2015 at 2:17 PM, Nate Coraor <nate at bx.psu.edu> wrote:

> On Mon, Aug 24, 2015 at 1:51 PM, Wes Turner <wes.turner at gmail.com> wrote:
>
>>
>>
>> On Mon, Aug 24, 2015 at 10:03 AM, Nate Coraor <nate at bx.psu.edu> wrote:
>>
>>> On Fri, Aug 21, 2015 at 2:51 AM, Nick Coghlan <ncoghlan at gmail.com>
>>> wrote:
>>>
>>>> On 21 August 2015 at 05:58, Robert Collins <robertc at robertcollins.net>
>>>> wrote:
>>>> > On 21 August 2015 at 07:25, Donald Stufft <donald at stufft.io> wrote:
>>>> >>
>>>> >> On August 20, 2015 at 3:23:09 PM, Daniel Holth (dholth at gmail.com)
>>>> wrote:
>>>> >>> If you need that for some reason just put the longer information in
>>>> the
>>>> >>> metadata, inside the WHEEL file for example. Surely "does it work
>>>> on my
>>>> >>> system" dominates, as opposed to "I have a wheel with this mnemonic
>>>> tag,
>>>> >>> now let me install debian 5 so I can get it to run".
>>>> >>>
>>>> >>>
>>>> >>
>>>> >> It’s less about “now let me install Debian 5” and more like tooling
>>>> that doesn’t run *on* the platform but which needs to make decisions based
>>>> on what platform a wheel is built for.
>>>> >
>>>> > Cramming that into the file name is a mistake IMO.
>>>> >
>>>> > Make it declarative data, make it indexable, and index it. We can do
>>>> > that locally as much as via the REST API.
>>>> >
>>>> > That btw is why the draft for referencing external dependencies
>>>> > specifies file names (because file names give an ABI in the context of
>>>> > a platform) - but we do need to identify the platform, and
>>>> > platform.distribution should be good enough for that (or perhaps we
>>>> > start depending on lsb-release for detection
>>>>
>>>> LSB has too much stuff in it, so most distros aren't LSB compliant out
>>>> of the box - you have to install extra packages.
>>>>
>>>> /etc/os-release is a better option:
>>>> http://www.freedesktop.org/software/systemd/man/os-release.html
>>>
>>>
>>> As per this discussion, and because I've discovered that the entire
>>> platform module is deprecated in 3.5 (and other amusements, like a
>>> Ubuntu-modified version of platform that ships on Ubuntu - platform as
>>> shipped with CPython detects Ubuntu as debian), I'm switching to
>>> os-release, but even that is unreliable - the file does not exist in
>>> CentOS/RHEL 6, for example. On Debian testing/sid installs, VERSION and
>>> VERSION_ID are unset (which is not wrong - there is no release of testing,
>>> but it does make identifying the platform more complicated since even the
>>> codename is not provided other than at the end of PRETTY_NAME). Regardless
>>> of whether a hash or a human-identifiable string is used to identify the
>>> platform, there still needs to be a way to reliably detect it.
>>>
>>> Unless someone tells me not to, I'm going to default to using os-release
>>> and then fall back to other methods in the event that os-release isn't
>>> available, and this will be in some sort of library alongside pep425tags in
>>> wheel/pip.
>>>
>>> FWIW, os-release's `ID_LIKE` gives us some ability to make assumptions
>>> without explicit need for a binary-compatibility.cfg (although not blindly
>>> - for example, CentOS sets this to "rhel fedora", but of course RHEL/CentOS
>>> and Fedora versions are not congruent).
>>>
>>
>> IIUC, then the value of os-release
>> will be used to generalize
>> the compatible versions of *.so deps
>> of a given distribution at a point in time?
>>
>> This works for distros that don't change [libc] much during a release,
>> but for rolling release models (e.g. arch, gentoo),
>> IDK how this simplification will work.
>> (This is a graph with nodes and edges (with attributes), and rules).
>>
>
> Arch, Gentoo, and other rolling release distributions don't have a stable
> ABI, so by definition I don't think we can support redistributable wheels
> on them. I'm adding platform detection support for them regardless, but I
> don't think there's any way to allow wheels built for these platforms in
> PyPI.
>
>
>> * Keying/namespacing is a simplification which may work.
>>   * *conda preprocessing selectors* (and ~LSB-Python-Conda)
>>      ~'prune' large parts of the graph
>>
>> * Someone mentioned LSB[-Python-Base] (again as a simplification)
>>   * [[package, [version<=>verstr]]]
>>
>> Salt
>> * __salt__['grains']['os']  = "Fedora" || "Ubuntu"
>> * __salt__['grains']['os_family'] = "RedHat" || "Debian"
>> * __salt__['grains']['osrelease'] = "22" || "14.04"
>> * __salt__['grains']['oscodename'] = "Twenty Two" || "trusty"
>> * Docs: http://docs.saltstack.com/en/latest/topics/targeting/grains.html
>> * Docs:
>> http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.grains.html#salt.modules.grains.get
>> * Src:
>> https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1018
>> ("def os_data()")
>>
>> $ sudo salt-call --local grains.item os_family os osrelease oscodename
>> local:
>>     ----------
>>     os:
>>         Fedora
>>     os_family:
>>         RedHat
>>     oscodename:
>>         Twenty Two
>>     osrelease:
>>         22
>>
>>
>>
>>> --nate
>>>
>>>
>>>>
>>>>
>>>> My original concern with using that was that it *over*specifies the
>>>> distro (e.g. not only do CentOS and RHEL releases show up as different
>>>> platforms, but so do X.Y releases within a series), but the
>>>> binary-compatibility.txt idea resolves that issue, since a derived
>>>> distro can explicitly identify itself as binary compatible with its
>>>> upstream and be able to use the corresponding wheel files.
>>>>
>>>> Regards,
>>>> Nick.
>>>>
>>>> --
>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>> _______________________________________________
>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>
>>>
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150825/1f75a6d7/attachment.html>