[Distutils] draft PEP: manylinux1

[Nathaniel Smith]

On Thu, Jan 21, 2016 at 12:18 PM, Matthias Klose <doko at ubuntu.com> wrote:
> On 21.01.2016 04:55, Nathaniel Smith wrote:
>
> the choice of compiler is questionable.  Just a pick into a release series.
> Not even the last released version on this release series.  Is this a good
> choice? Maybe for x86_64 and i386, but not for anything else.

There's no mandatory compiler. The rule is basically: "if your wheel
works when given access to CentOS 5's versions of the following
packages: ..., then your wheel is manylinux1 compatible". Any method
for achieving that is fair game :-). The RH devtoolset fork of gcc 4.8
happens to be the only C++11-supporting compiler that we know of that
can produce binaries that accomplish that, but if you have another
compiler that works better for you then go for it.

>> The permitted external shared libraries are: ::
>>
>>      libpanelw.so.5
>>      libncursesw.so.5
>
>
> this doesn't include libtinfo, dependency of libncursesw.
>
>>      libgcc_s.so.1
>>      libstdc++.so.6
>
>
> if you insist on a version built from before GCC 5, you are in trouble with
> the libstdc++ ABI. So maybe link that one statically.

The libstdc++ ABI is forward compatible across the GCC 4/GCC 5
transition; the exception is if you want to pass C++ stdlib objects
directly between code compiled with GCC 4 and code compiled with GCC
5. This is extremely rare, since Python packages almost never provide
a C++ ABI to other Python packages. (I don't know of any examples of
this.) If it does come up then those package authors will have to
figure out how they want to handle it, yes, possibly via picking a
more modern libstdc++ and agreeing to all use it together.

>>      libgobject-2.0.so.0
>>      libgthread-2.0.so.0
>>      libglib-2.0.so.0
>
>
> while glib2.0 is somehow frozen, what will you do if these change the
> soname?
>
> libgfortran is missing from this list while later down you mention gfortran.
> This will include libquadmath too.

libgfortran is missing intentionally, because it turns out from
experience that lots of end-user systems don't have it installed out
of the box. You'll notice that the numpy wheels I posted earlier
include libgfortran inside them.

(libquadmath is optional though -- you can build gfortran with
--disable-libquadmath-support and get a toolchain that generates
binaries that don't depend on libquadmath.)

> Any reason to not list libz?
>
>> Compilation and Tooling
>> =======================
>
>
> so how are people supposed to build these wheels?  will you provide a
> development distribution, or do you "trust" people building such wheels?

We currently provide a docker image (docker pull
quay.io/manylinux/manylinux) and a tool that checks wheels for
compliance and can in many cases automatically vendor any needed
shared libraries (pip3 install auditwheel). In the end of course we
can't force people to use these tools -- there's nothing that will
actually stop someone from renaming a Windows wheel to say
"manylinux1" in the platform tag and uploading it somewhere :-) -- but
our experience with OS X wheels is that if you provide good docs and
tools then people will generally get it right.

Package authors generally don't set out to build broken packages that
frustrate users ;-)

>> Platform Detection for Installers
>> =================================
>>
>> Because the ``manylinux1`` profile is already known to work for the many
>> thousands of users of popular commercial Python distributions, we suggest
>> that
>> installation tools like ``pip`` should error on the side of assuming that
>> a
>> system *is* compatible, unless there is specific reason to think
>> otherwise.
>>
>> We know of three main sources of potential incompatibility that are likely
>> to
>> arise in practice:
>>
>> * A linux distribution that is too old (e.g. RHEL 4)
>> * A linux distribution that does not use glibc (e.g. Alpine Linux, which
>> is
>>    based on musl libc, or Android)
>> * Eventually, in the future, there may exist distributions that break
>>    compatibility with this profile
>
>
> add: "A Linux distribution that is built with clang", e.g. Mageia (libc++
> instead of libstdc++).

Interesting point. It looks like from some quick googling that even
Mageia does provide a libstdc++6 package, though, presumably because
of these kinds of compatibility issues?

>> Security Implications
>> =====================
>>
>> One of the advantages of dependencies on centralized libraries in Linux is
>> that bugfixes and security updates can be deployed system-wide, and
>> applications which depend on on these libraries will automatically feel
>> the
>> effects of these patches when the underlying libraries are updated. This
>> can
>> be particularly important for security updates in packages communication
>> across the network or cryptography.
>>
>> ``manylinux1`` wheels distributed through PyPI that bundle
>> security-critical
>> libraries like OpenSSL will thus assume responsibility for prompt updates
>> in
>> response disclosed vulnerabilities and patches. This closely parallels the
>> security implications of the distribution of binary wheels on Windows
>> that,
>> because the platform lacks a system package manager, generally bundle
>> their
>> dependencies. In particular, because its lacks a stable ABI, OpenSSL
>> cannot be
>> included in the ``manylinux1`` profile.
>
>
> so you rely on the python build to provide this and access OpenSSL through
> the standard library?

That is one strategy that's available to packages who need access to
OpenSSL, yes.

> From my point of view this draft is too much influenced by Anaconda and
> their needs.  It doesn't talk at all about interaction with other system
> libraries, or interaction with extensions provided by distributions.

It's true that it doesn't do those things, just like there's no PEP
talking about how pip on Windows should interact with nuget or how pip
on OS X should interact with homebrew. This is a first step, the
perfect is the enemy of the good, etc.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org