[Python-checkins] peps: Apply Nathaniel's changes to PEP 513
chris.angelico
python-checkins at python.org
Wed Feb 10 00:55:12 EST 2016
https://hg.python.org/peps/rev/161a5afedcdc
changeset: 6221:161a5afedcdc
user: Chris Angelico <rosuav at gmail.com>
date: Wed Feb 10 16:54:46 2016 +1100
summary:
Apply Nathaniel's changes to PEP 513
files:
pep-0513.txt | 118 +++++++++++++++++++++++++++++++-------
1 files changed, 94 insertions(+), 24 deletions(-)
diff --git a/pep-0513.txt b/pep-0513.txt
--- a/pep-0513.txt
+++ b/pep-0513.txt
@@ -126,8 +126,7 @@
To be eligible for the ``manylinux1`` platform tag, a Python wheel must
therefore both (a) contain binary executables and compiled code that links
-*only* to libraries (other than the appropriate ``libpython`` library, which is
-always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs
+*only* to libraries with SONAMEs
included in the following list: ::
libpanelw.so.5
@@ -153,11 +152,7 @@
libglib-2.0.so.0
and, (b) work on a stock CentOS 5.11 [6]_ system that contains the system
-package manager's provided versions of these libraries. In addition,
-for wheels targeting CPython 3.2 and earlier (including all 2.x
-versions), there is an extra requirement that (c) the wheel be
-built against a version of CPython compiled with 4-byte unicode
-support (i.e. one where ``sys.maxunicode > 0xFFFF``).
+package manager's provided versions of these libraries.
Because CentOS 5 is only available for x86_64 and i686 architectures,
these are the only architectures currently supported by the ``manylinux1``
@@ -216,6 +211,81 @@
problematic or add libraries that have turned out to be safe.
+libpythonX.Y.so.1
+-----------------
+
+Note that ``libpythonX.Y.so.1`` is *not* on the list of libraries that
+a ``manylinux1`` extension is allowed to link to. Explicitly linking
+to ``libpythonX.Y.so.1`` is unnecessary in almost all cases: the way
+ELF linking works, extension modules that are loaded into the
+interpreter automatically get access to all of the interpreter's
+symbols, regardless of whether or not the extension itself is
+explicitly linked against libpython. Furthermore, explicit linking to
+libpython creates problems in the common configuration where Python is
+not built with ``--enable-shared``. In particular, on Debian and
+Ubuntu systems, ``apt install pythonX.Y`` does not even install
+``libpythonX.Y.so.1``, meaning that any wheel that *did* depend on
+``libpythonX.Y.so.1`` could fail to import.
+
+There is one situation where extensions that are linked in this way
+can fail to work: if a host program (e.g., ``apache2``) uses
+``dlopen()`` to load a module (e.g., ``mod_wsgi``) that embeds the
+CPython interpreter, and the host program does *not* pass the
+``RTLD_GLOBAL`` flag to ``dlopen()``, then the embedded CPython will
+be unable to load any extension modules that do not themselves link
+explicitly to ``libpythonX.Y.so.1``. Fortunately, ``apache2`` *does*
+set the ``RTLD_GLOBAL`` flag, as do all the other programs that
+embed-CPython-via-a-dlopened-plugin that we could locate, so this does
+not seem to be a serious problem in practice. The incompatibility with
+Debian/Ubuntu is more of an issue than the theoretical incompatibility
+with a rather obscure corner case.
+
+This is a rather complex and subtle issue that extends beyond
+the scope of ``manylinux1``; for more discussion see: [9]_, [10]_,
+[11]_.
+
+UCS-2 vs UCS-4 builds
+---------------------
+
+All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can be
+built in two ABI-incompatible modes: builds using the
+``--enable-unicode=ucs2`` configure flag store Unicode data in UCS-2
+(or really UTF-16) format, while builds using the
+``--enable-unicode=ucs4`` configure flag store Unicode data in
+UCS-4. (CPython 3.3 and greater use a different storage method that
+always supports UCS-4.) If we want to make sure ``ucs2`` wheels don't
+get installed into ``ucs4`` CPythons and vice-versa, then something
+must be done.
+
+An earlier version of this PEP included a requirement that
+``manylinux1`` wheels targeting these older CPython versions should
+always use the ``ucs4`` ABI. But then, in between the PEP's initial
+acceptance and its implementation, ``pip`` and ``wheel`` gained
+first-class support for tracking and checking this aspect of ABI
+compatibility for the relevant CPython versions, which is a better
+solution. So we now allow the ``manylinux1`` platform tags to be used
+in combination with any ABI tag. However, to maintain compatibility it
+is crucial to ensure that all ``manylinux1`` wheels include a
+non-trivial abi tag. For example, a wheel built against a ``ucs4``
+CPython might have a name like::
+
+ PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl
+ ^^^^^^ Good!
+
+While a wheel built against the ``ucs2`` ABI might have a name like::
+
+ PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl
+ ^^^^^ Okay!
+
+But you should never have a wheel with a name like::
+
+ PKG-VERSION-cp27-none-manylinux1_x86_64.whl
+ ^^^^ BAD! Don't do this!
+
+We note for information that the ``ucs4`` ABI appears to be much more
+widespread among Linux CPython distributors.
+
+
Compilation of Compliant Wheels
===============================
@@ -237,7 +307,7 @@
The first tool is a Docker image based on CentOS 5.11, which is recommended as
an easy to use self-contained build box for compiling ``manylinux1`` wheels
-[9]_. Compiling on a more recently-released linux distribution will generally
+[12]_. Compiling on a more recently-released linux distribution will generally
introduce dependencies on too-new versioned symbols. The image comes with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
well as the latest releases of Python and ``pip``.
@@ -245,7 +315,7 @@
Auditwheel
----------
-The second tool is a command line executable called ``auditwheel`` [10]_ that
+The second tool is a command line executable called ``auditwheel`` [13]_ that
may aid in package maintainers in dealing with third-party external
dependencies.
@@ -285,7 +355,7 @@
dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1``
policy encourages bundling external dependencies, a practice
which runs counter to the package management policies of many linux
-distributions' system package managers [11]_, [12]_. The primary purpose of
+distributions' system package managers [14]_, [15]_. The primary purpose of
this is cross-distro compatibility. Furthermore, ``manylinux1`` wheels on PyPI
occupy a different niche than the Python packages available through the
system package manager.
@@ -352,8 +422,6 @@
We know of four main sources of potential incompatibility that are
likely to arise in practice:
-* "Narrow" unicode builds of Python 3.2 and earlier (including all
- versions of Python 2)
* Eventually, in the future, there may exist distributions that break
compatibility with this profile (e.g., if one of the libraries in
the profile changes its ABI in a backwards-incompatible way)
@@ -361,8 +429,7 @@
* A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is
based on musl ``libc``, or Android)
-Checking for unicode configuration compatibility is straightforward,
-but the other cases are more subtle. We propose a two-pronged
+To address these we propose a two-pronged
approach. To handle potential future incompatibilities, we standardize
a mechanism for a Python distributor to signal that a particular
Python install definitely is or is not compatible with ``manylinux1``:
@@ -389,11 +456,6 @@
if get_platform() not in ["linux-x86_64", "linux-i686"]:
return False
- # "wide" Unicode mode is mandatory (always true on CPython 3.3+)
- import sys
- if sys.maxunicode <= 0xFFFF:
- return False
-
# Check for presence of _manylinux module
try:
import _manylinux
@@ -522,13 +584,21 @@
(https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U)
.. [8] distutils-sig discussion
(https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html)
-.. [9] manylinux1 docker image
- (https://quay.io/repository/manylinux/manylinux)
-.. [10] auditwheel tool
+.. [9] distutils-sig discussion
+ (https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html)
+.. [10] github issue discussion
+ (https://github.com/pypa/manylinux/issues/30)
+.. [11] python bug tracker discussion
+ (https://bugs.python.org/issue21536)
+.. [12] manylinux1 docker images
+ (Source: https://github.com/pypa/manylinux;
+ x86-64: https://quay.io/repository/pypa/manylinux1_x86_64;
+ x86-32: https://quay.io/repository/pypa/manylinux1_i686)
+.. [13] auditwheel tool
(https://pypi.python.org/pypi/auditwheel)
-.. [11] Fedora Bundled Software Policy
+.. [14] Fedora Bundled Software Policy
(https://fedoraproject.org/wiki/Bundled_Software_policy)
-.. [12] Debian Policy Manual -- 4.13: Convenience copies of code
+.. [15] Debian Policy Manual -- 4.13: Convenience copies of code
(https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list