Robert McGibbon: Thanks for writing up this PEP :-) Some comments below... On 21.01.2016 04:55, Nathaniel Smith wrote:
The ``manylinux1`` policy =========================
For these reasons, to achieve broad portability, Python wheels
* should depend only on an extremely limited set of external shared libraries; and * should depend only on ``old`` symbol versions in those external shared libraries.
The ``manylinux1`` policy thus encompasses a standard for what the permitted external shared libraries a wheel may depend on, and the maximum depended-upon symbol versions therein.
The permitted external shared libraries are: ::
libpanelw.so.5 libncursesw.so.5 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0
The list is good start, but limiting the possible external references to only these libraries will make it impossible to have manylinux1 wheels which link against other, similarly old, but very common libraries, or alternatively rather new ones, which are then optionally included via subpackage instead of being mandatory. At eGenix we have been tackling this problem for years with our extensions and the approach that's been the most successful was to simply use Linux build systems which are at least 5 years old. In our case, that's openSUSE 11.3. I think a better approach is to use the above list to test for used library *versions* and then apply the tag based on the findings. If a package includes binaries which link to e.g. later libc.so versions, it would be rejected. If it includes other libraries not listed in the above listing, that's fine, as long as these libraries also comply to the version limitation. What I'm getting at here is that incompatibilities are not caused by libraries being absent on the system (the package simply won't load, but that's not due to the the package being incompatible to the platform, only due to the system lacking a few packages), but instead by having the packages use more recent versions of these system libraries.
Compilation and Tooling =======================
To support the compilation of wheels meeting the ``manylinux1`` standard, we provide initial drafts of two tools.
The first is a Docker image based on CentOS 5.11, which is recommended as an easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_. Compiling on a more recently-released linux distribution will generally introduce dependencies on too-new versioned symbols. The image comes with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and pip.
The second tool is a command line executable called ``auditwheel`` [5]_. First, it inspects all of the ELF files inside a wheel to check for dependencies on versioned symbols or external shared libraries, and verifies conformance with the ``manylinux1`` policy. This includes the ability to add the new platform tag to conforming wheels.
In addition, ``auditwheel`` has the ability to automatically modify wheels that depend on external shared libraries by copying those shared libraries from the system into the wheel itself, and modifying the appropriate RPATH entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system.
This approach has a few problems: * Libraries typically depend on a lot more context than just the code that is provided in the libraries file, e.g. config files, external resources, other libraries which are loaded on demand, etc. * By including the libraries in the wheel you are distributing the binary, which can lead to licensing problems, esp. with GPLed or LGPLed code.
Neither of these tools are necessary to build wheels which conform with the ``manylinux1`` policy. Similar results can usually be achieved by statically linking external dependencies and/or using certain inline assembly constructs to instruct the linker to prefer older symbol versions, however these tricks can be quite esoteric.
Static linking only helps in very few cases, where the context needed for the external library to work is minimal.
Platform Detection for Installers =================================
Because the ``manylinux1`` profile is already known to work for the many thousands of users of popular commercial Python distributions, we suggest that installation tools like ``pip`` should error on the side of assuming that a system *is* compatible, unless there is specific reason to think otherwise.
We know of three main sources of potential incompatibility that are likely to arise in practice:
* A linux distribution that is too old (e.g. RHEL 4) * A linux distribution that does not use glibc (e.g. Alpine Linux, which is based on musl libc, or Android) * Eventually, in the future, there may exist distributions that break compatibility with this profile
To handle the first two cases, we propose the following simple and reliable check: :: def have_glibc_version(major, minimum_minor): [...]
It would be better to use platform.libc_ver() for this.
``manylinux1`` wheels distributed through PyPI that bundle security-critical libraries like OpenSSL will thus assume responsibility for prompt updates in response disclosed vulnerabilities and patches. This closely parallels the security implications of the distribution of binary wheels on Windows that, because the platform lacks a system package manager, generally bundle their dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be included in the ``manylinux1`` profile.
The OpenSSL ABI has been quite stable in recent years (unlike in the days of 0.9.7 and earlier). Since many libraries do link against OpenSSL (basically everything that uses network connections nowadays), using the fixed scheme outlined in the PEP would severely limited the usefulness. By using the version based approach, we'd not run into this problem and gain a lot more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 21 2016)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/