[Distutils] Working toward Linux wheel support

Fri Aug 14 04:24:32 CEST 2015

On Aug 13, 2015 9:14 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
>
> On Thu, Aug 13, 2015 at 6:31 PM, Robert Collins
> <robertc at robertcollins.net> wrote:
> > On 14 August 2015 at 13:25, Nathaniel Smith <njs at pobox.com> wrote:
> >> On Thu, Aug 13, 2015 at 7:07 AM, Nate Coraor <nate at bx.psu.edu> wrote:
> >>> On Wed, Aug 12, 2015 at 9:05 PM, Nathaniel Smith <njs at pobox.com>
wrote:
> >>>>
> >>>> From my reading of what the Enthought and Continuum folks were saying
> >>>> about how they are successfully distributing binaries across
different
> >>>> distributions, it sounds like the additional piece that would take
this from
> >>>> a interesting experiment to basically-immediately-usable would be to
teach
> >>>> pip that if no binary-compatibility.cfg is provided, then it should
assume
> >>>> by default that the compatible systems whose wheels should be
installed are:
> >>>> (1) the current system's exact tag,
> >>>
> >>> This should already be the case - the default tag will no longer be
> >>> -linux_x86_64, it'd be linux_x86_64_distro_version.
> >>>
> >>>>
> >>>> (2) the special hard-coded tag "centos5". (That's what everyone
actually
> >>>> uses in practice, right?)
> >>>
> >>> The idea here is that we should attempt to install centos5 wheels if
more
> >>> specific wheels for the platform aren't available?
> >>
> >> Yes.
> >>
> >> Or more generally, we should pick some common baseline build
> >> environment such that we're pretty sure wheels built there can run on
> >> 99% of end-user systems and give this environment a name. (Doesn't
> >> have to be "centos5", though IIUC CentOS 5 is what people are using
> >> for this baseline build environment right now.) That way when distros
> >> catch up and start providing binary-compatibility.cfg files, we can
> >> give tell them that this is an environment that they should try to
> >> support because it's what everyone is using, and to kick start that
> >> process we should assume it as a default until the distros do catch
> >> up. This has two benefits: it means that these wheels would actually
> >> become useful in some reasonable amount of time, and as a bonus, it
> >> would provide a clear incentive for those rare distros that *aren't*
> >> compatible to document that by starting to provide a
> >> binary-compatibility.cfg.
> >
> > Sounds like a reinvention of LSB, which is still a thing I think, but
> > really didn't take the vendor world by storm.
>
> Yeah, I've been carefully not mentioning LSB because LSB is a disaster
> :-). But, I think this is different.
>
> IIUC, the problem with LSB is that it's trying to make it possible for
> big enterprise software vendors to stop saying "This RDBMS is
> certified to work on RHEL 6" and start saying "This RDBMS is certified
> to work on any distribution that meets the LSB criteria".

That's great. Is there a Dockerfile invocation for:
- running the tests
- building a binary in a mapped path
- posting build state and artifacts to a central server

> But in
> practice this creates more risk and work for the vendor, while not
> actually solving any real problem -- if a customer is spending $$$$ on
> some enterprise database then they might as well throw in an extra $$
> for a RHEL license, so the customers don't care, so the vendor doesn't
> either. And the folks building free software like Postgres don't care
> either because the distros do the support for them. So the LSB
> continues to limp along through the ISO process because just enough
> managers have been convinced that it *ought* to be useful that they
> continue to throw some money at it, and hey, it's probably useful to
> some people sometimes, just not very many people very often.
>
> We, on the other hand, are trying to solve a real problem that our
> users feel keenly (lots of people want to be able to distribute some
> little binary python extension in a way that just works for a wide
> range of users), and the proposed mechanism for solving this problem
> is not "let's form an ISO committee and hire contractors to write a
> Grand Unified Test Suite", it's codifying an existing working solution
> in the form of a wiki page or PEP or something.
>
> Of course if you have an alternative proposal than I'm all ears :-).

Required_caps = [('blas1', None), ('blas', '>= 1'), ('np17', None)]

Re-post [TODO: upgrade mailman]
"""
... BLAS may not be the best example, but should we expect such linked
interfaces to change over time? (And e.g. be versioned dependencies with
shim packages that have check functions)?

... How is an ABI constraint different from a package dependency?

iiuc, ABI tags are thus combinatorial with package/wheel dependency strings?

Conda/pycosat solve this with "preprocessing selectors" :
http://conda.pydata.org/docs/building/meta-yaml.html#preprocessing-selectors
:

```
linux True if the platform is Linux
linux32 True if the platform is Linux and the Python architecture is 32-bit
linux64 True if the platform is Linux and the Python architecture is 64-bit
armv6 True if the platform is Linux and the Python architecture is armv6l
osx True if the platform is OS X
unix True if the platform is Unix (OS X or Linux)
win True if the platform is Windows
win32 True if the platform is Windows and the Python architecture is 32-bit
win64 True if the platform is Windows and the Python architecture is 64-bit
py The Python version as a two digit string (like '27'). See also the
CONDA_PY environment variable below.
py3k True if the Python major version is 3
py2k True if the Python major version is 2
py26 True if the Python version is 2.6
py27 True if the Python version is 2.7
py33 True if the Python version is 3.3
py34 True if the Python version is 3.4
np The NumPy version as a two digit string (like '17'). See also the
CONDA_NPY environment variable below.
Because the selector is any valid Python expression, complicated logic is
possible.
```

>
> -n
>
> P.S.: since probably not everyone on the mailing list has been
> following Linux inside baseball for decades, some context...:
>   https://en.wikipedia.org/wiki/Linux_Standard_Base
>   http://www.linuxfoundation.org/collaborate/workgroups/lsb/download
>   https://lwn.net/Articles/152580/
>   http://udrepper.livejournal.com/8511.html
> (Last two links are from 2005, I can't really say how accurate they
> still are in details but they do describe some of the structural
> reasons why the LSB has not been massively popular)
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150813/61065654/attachment-0001.html>