[Distutils] draft PEP: manylinux1
rmcgibbo at gmail.com
Thu Jan 21 04:27:09 EST 2016
After I posted the PEP link on social media, a friend of mine, Kyle
Beauchamp, asked: "I wonder if there are speed and correctness implications
to always reverting to the lowest common denominator of glibc."
For speed, I believe there is some consequence, in that the maximum gcc
version you can realistically use with glibc <= 2.5 is gcc 4.8.2, which is
presumably somewhat (although not much) slower than the latest gcc release.
As to correctness, it seems like a reasonable concern, and I don't know one
way or the other. I thought maybe someone on the list would know.
There are also potential compatibility issues in wheels that use certain
instruction set extensions (SSE 4, AVX, etc) that might not be available on
certain platforms. I think policies on this issue are better left up to
individual packages than set at a PEP-wide level, but it also may be worth
On Thu, Jan 21, 2016 at 1:03 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> Robert McGibbon: Thanks for writing up this PEP :-)
> Some comments below...
> On 21.01.2016 04:55, Nathaniel Smith wrote:
> > The ``manylinux1`` policy
> > =========================
> > For these reasons, to achieve broad portability, Python wheels
> > * should depend only on an extremely limited set of external shared
> > libraries; and
> > * should depend only on ``old`` symbol versions in those external shared
> > libraries.
> > The ``manylinux1`` policy thus encompasses a standard for what the
> > permitted external shared libraries a wheel may depend on, and the
> > depended-upon symbol versions therein.
> > The permitted external shared libraries are: ::
> > libpanelw.so.5
> > libncursesw.so.5
> > libgcc_s.so.1
> > libstdc++.so.6
> > libm.so.6
> > libdl.so.2
> > librt.so.1
> > libcrypt.so.1
> > libc.so.6
> > libnsl.so.1
> > libutil.so.1
> > libpthread.so.0
> > libX11.so.6
> > libXext.so.6
> > libXrender.so.1
> > libICE.so.6
> > libSM.so.6
> > libGL.so.1
> > libgobject-2.0.so.0
> > libgthread-2.0.so.0
> > libglib-2.0.so.0
> The list is good start, but limiting the possible external
> references to only these libraries will make it impossible
> to have manylinux1 wheels which link against other, similarly
> old, but very common libraries, or alternatively rather
> new ones, which are then optionally included via subpackage
> instead of being mandatory.
> At eGenix we have been tackling this problem for years with
> our extensions and the approach that's been the most
> successful was to simply use Linux build systems which
> are at least 5 years old. In our case, that's openSUSE 11.3.
> I think a better approach is to use the above list to
> test for used library *versions* and then apply the tag
> based on the findings.
> If a package includes binaries which link to e.g.
> later libc.so versions, it would be rejected. If it
> includes other libraries not listed in the above listing,
> that's fine, as long as these libraries also comply to
> the version limitation.
> What I'm getting at here is that incompatibilities are
> not caused by libraries being absent on the system
> (the package simply won't load, but that's not due to the
> the package being incompatible to the platform, only due
> to the system lacking a few packages), but instead by
> having the packages use more recent versions of these
> system libraries.
> > Compilation and Tooling
> > =======================
> > To support the compilation of wheels meeting the ``manylinux1``
> standard, we
> > provide initial drafts of two tools.
> > The first is a Docker image based on CentOS 5.11, which is recommended
> as an
> > easy to use self-contained build box for compiling ``manylinux1`` wheels
> > Compiling on a more recently-released linux distribution will generally
> > introduce dependencies on too-new versioned symbols. The image comes
> with a
> > full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2)
> > well as the latest releases of Python and pip.
> > The second tool is a command line executable called ``auditwheel`` _.
> > it inspects all of the ELF files inside a wheel to check for
> dependencies on
> > versioned symbols or external shared libraries, and verifies conformance
> > the ``manylinux1`` policy. This includes the ability to add the new
> > tag to conforming wheels.
> > In addition, ``auditwheel`` has the ability to automatically modify
> wheels that
> > depend on external shared libraries by copying those shared libraries
> > the system into the wheel itself, and modifying the appropriate RPATH
> > such that these libraries will be picked up at runtime. This
> accomplishes a
> > similar result as if the libraries had been statically linked without
> > changes to the build system.
> This approach has a few problems:
> * Libraries typically depend on a lot more context than just
> the code that is provided in the libraries file, e.g. config
> files, external resources, other libraries which are loaded
> on demand, etc.
> * By including the libraries in the wheel you are distributing
> the binary, which can lead to licensing problems, esp. with
> GPLed or LGPLed code.
> > Neither of these tools are necessary to build wheels which conform with
> > ``manylinux1`` policy. Similar results can usually be achieved by
> > linking external dependencies and/or using certain inline assembly
> > to instruct the linker to prefer older symbol versions, however these
> > can be quite esoteric.
> Static linking only helps in very few cases, where the context
> needed for the external library to work is minimal.
> > Platform Detection for Installers
> > =================================
> > Because the ``manylinux1`` profile is already known to work for the many
> > thousands of users of popular commercial Python distributions, we
> suggest that
> > installation tools like ``pip`` should error on the side of assuming
> that a
> > system *is* compatible, unless there is specific reason to think
> > We know of three main sources of potential incompatibility that are
> likely to
> > arise in practice:
> > * A linux distribution that is too old (e.g. RHEL 4)
> > * A linux distribution that does not use glibc (e.g. Alpine Linux, which
> > based on musl libc, or Android)
> > * Eventually, in the future, there may exist distributions that break
> > compatibility with this profile
> > To handle the first two cases, we propose the following simple and
> > check: ::
> > def have_glibc_version(major, minimum_minor):
> > [...]
> It would be better to use platform.libc_ver() for this.
> > ``manylinux1`` wheels distributed through PyPI that bundle
> > libraries like OpenSSL will thus assume responsibility for prompt
> updates in
> > response disclosed vulnerabilities and patches. This closely parallels
> > security implications of the distribution of binary wheels on Windows
> > because the platform lacks a system package manager, generally bundle
> > dependencies. In particular, because its lacks a stable ABI, OpenSSL
> cannot be
> > included in the ``manylinux1`` profile.
> The OpenSSL ABI has been quite stable in recent years (unlike in the
> days of 0.9.7 and earlier).
> Since many libraries do link against OpenSSL (basically everything
> that uses network connections nowadays), using the fixed scheme
> outlined in the PEP would severely limited the usefulness.
> By using the version based approach, we'd not run into this
> problem and gain a lot more.
> Marc-Andre Lemburg
> Professional Python Services directly from the Experts (#1, Jan 21 2016)
> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
> >>> Python Database Interfaces ... http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
> ::: We implement business ideas - efficiently in both time and costs :::
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Distutils-SIG