[Python-checkins] r82909 - peps/trunk/pep-3149.txt
barry.warsaw
python-checkins at python.org
Thu Jul 15 01:56:40 CEST 2010
Author: barry.warsaw
Date: Thu Jul 15 01:56:40 2010
New Revision: 82909
Log:
Added PEP 3149
Added:
peps/trunk/pep-3149.txt
Added: peps/trunk/pep-3149.txt
==============================================================================
--- (empty file)
+++ peps/trunk/pep-3149.txt Thu Jul 15 01:56:40 2010
@@ -0,0 +1,267 @@
+PEP: 3149
+Title: ABI version tagged .so files
+Version: $Revision: 81577 $
+Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $
+Author: Barry Warsaw <barry at python.org>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 2010-07-09
+Python-Version: 3.2
+Post-History: 2010-07-14
+Resolution: TBD
+
+
+Abstract
+========
+
+PEP 3147 [1]_ described an extension to Python's import machinery that
+improved the sharing of Python source code, by allowing more than one
+byte compilation file (.pyc) to be co-located with each source file.
+
+This PEP defines an adjunct feature which allows the co-location of
+extension module files (.so) in a similar manner. This optional,
+build-time feature will enable downstream distributions of Python to
+more easily provide more than one Python major version at a time.
+
+
+Background
+==========
+
+PEP 3147 defined the file system layout for a pure-Python package,
+where multiple versions of Python are available on the system. For
+example, where the `alpha` package containing source modules `one.py`
+and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte
+compilation file system layout would be::
+
+ alpha/
+ __pycache__/
+ __init__.cpython-32.pyc
+ __init__.cpython-33.pyc
+ one.cpython-32.pyc
+ one.cpython-33.pyc
+ two.cpython-32.pyc
+ two.cpython-33.pyc
+ __init__.py
+ one.py
+ two.py
+
+For packages with extension modules, a similar differentiation is
+needed for the module's .so files. Extension modules compiled for
+different Python major versions are incompatible with each other due
+to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it
+will minimize, but not eliminate extension module incompatibilities
+between Python major versions. Thus a mechanism for discriminating
+extension module file names is proposed.
+
+
+Rationale
+=========
+
+Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more
+than one Python version at the same time to their users. For example,
+Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
+with Python 2.6 being the default.
+
+In order to share as much as possible between the available Python
+versions, these distributions install third party (i.e. non-standard
+library) packages into `/usr/share/pyshared` and symlink to them from
+`/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a
+pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from
+byte compilation by the various installed Pythons will name collide
+with each other. For Python versions >= 3.2, all pure-Python packages
+can be shared, because the `.pyc` files will no longer cause file
+system naming conflicts. Eliminating these symlinks makes for a
+simpler, more robust Python distribution.
+
+A similar situation arises with shared library extensions. Because
+extension modules are typically named `foo.so` for a `foo` extension
+module, these would also name collide if `foo` was provided for more
+than one Python version. There are several approaches that could be
+taken to avoid this, which will be explored below, but this PEP
+proposes a fairly simple compile-time option to allow extension
+modules to live in the same file system directory and avoid any name
+collisions.
+
+
+Proposal
+========
+
+A new configure option is added for building Python, called
+`--with-so-abi-tag`. This takes as an argument a unique, but
+arbitrary string, e.g.::
+
+ ./configure --with-so-abi-tag=cpython-32
+
+This string is passed into the `Makefile` and affects two aspects of
+the Python build. First, it is compiled into `Python/dynload_shlib.c`
+where it defines some additional `.so` file names to search for when
+importing extension modules. Second, it modifies the `Makefile`'s
+`$SO` variable, which in turn controls the `distutils` module's default
+filename when compiling extension modules.
+
+When `--with-so-abi-tag` is not given to `configure` nothing changes
+in the way the Python executable is built, or acts. Thus, this
+configure switch is completely optional and has no effect if not used.
+
+What this allows is for distributions that want to distinguish among
+extension modules built for different versions of Python, but shared
+in the same file system path, to arrange for `.so` names that are
+unique and non-colliding.
+
+For example, let's say Python 3.2 was built with::
+
+ ./configure --with-so-abi-tag=cpython-32
+
+and Python 3.3 was built with::
+
+ ./configure --with-so-abi-tag=cpython-33
+
+For an arbitrary package `foo`, you would see these files when the
+distribution package was installed::
+
+ /usr/share/pyshared/foo.cpython-32.so
+ /usr/share/pyshared/foo.cpython-33.so
+
+
+Proven approach
+===============
+
+The approach described here is already proven, in a sense, on Debian
+and Ubuntu system where different extensions are used for debug builds
+of Python and extension modules. Debug builds on Windows also already
+use a different file extension for dynamic libraries.
+
+
+PEP 384
+=======
+
+PEP 384 defines a stable ABI for extension modules. Universal
+adoption of PEP 384 would eliminate the need for this PEP because all
+extension modules would be compatible with any Python version. In
+practice of course, it will be impossible to achieve universal
+adoption. Older extensions may not be ported to PEP 384, or an
+extension may require Python APIs outside of PEP 384 definition.
+Therefore there will always be a (hopefully diminishing, but never
+zero) need for ABI version tagged shared libraries.
+
+Further, it is anticipated that the stable ABI will evolve over time,
+meaning that existing PEP 384 compatible extension modules may be
+incompatible with future versions of Python. While a complete
+specification is reserved for PEP 384, here is a discussion of the
+relevant issues.
+
+PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
+passed as the API version if the extension was complied with
+``Py_LIMITED_API``. This should be formalized into an official macro
+called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If
+and when the ABI changes in an incompatible way, this version number
+would be bumped. To facilitate sharing, Python would be extended to
+search for extension modules with the ``PYTHON_ABI_VERSION`` number in
+its name. The prefix ``abi`` is reserved for Python's use.
+
+Thus for example, an initial implementation of PEP 384, compiled with
+`--with-so-abi-tag=cpython-xy` would search for the following file
+names when extension module `foo` is imported (in this order)::
+
+ foo.abi3.so
+ foo.cpython-xy.so
+ foo.so
+
+The distutils [7]_ ``build_ext`` command would also have to be
+extended to compile to shared library files with the ``abi3`` tag,
+when the module author indicates that their extension supports that
+version of the ABI. This could be done in a backward compatible way
+by adding a keyword argument to the ``Extension`` class, such as::
+
+ Extension('foo', ['foo.c'], abi=3)
+
+
+Alternatives
+============
+
+In the initial python-dev thread [8]_ where this idea was first
+introduced, several alternatives were suggested. For completeness
+they are listed here, along with the reasons for not adopting them.
+
+
+Independent directories or symlinks
+-----------------------------------
+
+Debian and Ubuntu could simply add a version-specific directory to
+``sys.path`` that would contain just the extension modules for that
+version of Python. Or the symlink trick eliminated in PEP 3147 could
+be retained for just shared libraries. This approach is rejected
+because it propagates the essential complexity that PEP 3147 tries to
+avoid, and adds yet another directory to search for all modules, even
+when the number of extension modules is much fewer than the total
+number of Python packages. It also makes for more robust management
+when all of a package's module files live in the same directory,
+because it allows systems such as `dpkg` to detect file conflicts
+between distribution packages.
+
+
+Don't share packages with extension modules
+-------------------------------------------
+
+It has been suggested that Python packages with extension modules not
+be shared among all supported Python versions on a distribution. Even
+with adoption of PEP 3149, extension modules will have to be compiled
+for every supported Python version, so perhaps sharing of such
+packages isn't useful anyway. Not sharing packages with extensions
+though is infeasible for several reasons.
+
+If a pure-Python package is shared in one version, should it suddenly
+be not-shared if the next release adds an extension module for speed?
+Also, even though all extension shared libraries will be compiled and
+distributed once for every supported Python, there's a big difference
+between duplicating the `.so` files and duplicating all `.py` files.
+The extra space increases the download time for such packages, and
+more immediately, increases the space pressures on already constrained
+distribution CD-ROMs.
+
+
+Reference implementation
+========================
+
+Work on this code is tracked in a Bazaar branch on Launchpad [5]_
+until it's ready for merge into Python 3.2. The work-in-progress diff
+can also be viewed [6]_ and is updated automatically as new changes
+are uploaded.
+
+
+References
+==========
+
+.. [1] PEP 3147
+
+.. [2] PEP 384
+
+.. [3] Ubuntu: <http://www.ubuntu.com>
+
+.. [4] Debian: <http://www.debian.org>
+
+.. [5] https://code.edge.launchpad.net/~barry/python/sovers
+
+.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
+
+.. [7] http://docs.python.org/py3k/distutils/index.html
+
+.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
+
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+
+..
+ Local Variables:
+ mode: indented-text
+ indent-tabs-mode: nil
+ sentence-end-double-space: t
+ fill-column: 70
+ coding: utf-8
+ End:
More information about the Python-checkins
mailing list