[Python-Dev] versioned .so files for Python 3.2
Barry Warsaw
barry at python.org
Thu Jul 15 01:59:55 CEST 2010
On Jun 25, 2010, at 08:35 AM, Nick Coghlan wrote:
>I like the idea, but I think summarising the rest of this discussion
>in its own (relatively short) PEP would be good (there are a few
>things that are tricky - exact versioning scheme, PEP 384 forward
>compatibility, impact on distutils, articulating the benefits for
>distro packaging, etc).
The first draft of PEP 3149 is ready for review.
http://www.python.org/dev/peps/pep-3149/
Plain text attached here for your convenience. Comments, suggestions as
always are welcome. Thanks to everyone who participated in the original
discussion.
-Barry
PEP: 3149
Title: ABI version tagged .so files
Version: $Revision: 81577 $
Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $
Author: Barry Warsaw <barry at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2010-07-09
Python-Version: 3.2
Post-History: 2010-07-14
Resolution: TBD
Abstract
========
PEP 3147 [1]_ described an extension to Python's import machinery that
improved the sharing of Python source code, by allowing more than one
byte compilation file (.pyc) to be co-located with each source file.
This PEP defines an adjunct feature which allows the co-location of
extension module files (.so) in a similar manner. This optional,
build-time feature will enable downstream distributions of Python to
more easily provide more than one Python major version at a time.
Background
==========
PEP 3147 defined the file system layout for a pure-Python package,
where multiple versions of Python are available on the system. For
example, where the `alpha` package containing source modules `one.py`
and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte
compilation file system layout would be::
alpha/
__pycache__/
__init__.cpython-32.pyc
__init__.cpython-33.pyc
one.cpython-32.pyc
one.cpython-33.pyc
two.cpython-32.pyc
two.cpython-33.pyc
__init__.py
one.py
two.py
For packages with extension modules, a similar differentiation is
needed for the module's .so files. Extension modules compiled for
different Python major versions are incompatible with each other due
to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it
will minimize, but not eliminate extension module incompatibilities
between Python major versions. Thus a mechanism for discriminating
extension module file names is proposed.
Rationale
=========
Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more
than one Python version at the same time to their users. For example,
Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
with Python 2.6 being the default.
In order to share as much as possible between the available Python
versions, these distributions install third party (i.e. non-standard
library) packages into `/usr/share/pyshared` and symlink to them from
`/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a
pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from
byte compilation by the various installed Pythons will name collide
with each other. For Python versions >= 3.2, all pure-Python packages
can be shared, because the `.pyc` files will no longer cause file
system naming conflicts. Eliminating these symlinks makes for a
simpler, more robust Python distribution.
A similar situation arises with shared library extensions. Because
extension modules are typically named `foo.so` for a `foo` extension
module, these would also name collide if `foo` was provided for more
than one Python version. There are several approaches that could be
taken to avoid this, which will be explored below, but this PEP
proposes a fairly simple compile-time option to allow extension
modules to live in the same file system directory and avoid any name
collisions.
Proposal
========
A new configure option is added for building Python, called
`--with-so-abi-tag`. This takes as an argument a unique, but
arbitrary string, e.g.::
./configure --with-so-abi-tag=cpython-32
This string is passed into the `Makefile` and affects two aspects of
the Python build. First, it is compiled into `Python/dynload_shlib.c`
where it defines some additional `.so` file names to search for when
importing extension modules. Second, it modifies the `Makefile`'s
`$SO` variable, which in turn controls the `distutils` module's default
filename when compiling extension modules.
When `--with-so-abi-tag` is not given to `configure` nothing changes
in the way the Python executable is built, or acts. Thus, this
configure switch is completely optional and has no effect if not used.
What this allows is for distributions that want to distinguish among
extension modules built for different versions of Python, but shared
in the same file system path, to arrange for `.so` names that are
unique and non-colliding.
For example, let's say Python 3.2 was built with::
./configure --with-so-abi-tag=cpython-32
and Python 3.3 was built with::
./configure --with-so-abi-tag=cpython-33
For an arbitrary package `foo`, you would see these files when the
distribution package was installed::
/usr/share/pyshared/foo.cpython-32.so
/usr/share/pyshared/foo.cpython-33.so
Proven approach
===============
The approach described here is already proven, in a sense, on Debian
and Ubuntu system where different extensions are used for debug builds
of Python and extension modules. Debug builds on Windows also already
use a different file extension for dynamic libraries.
PEP 384
=======
PEP 384 defines a stable ABI for extension modules. Universal
adoption of PEP 384 would eliminate the need for this PEP because all
extension modules would be compatible with any Python version. In
practice of course, it will be impossible to achieve universal
adoption. Older extensions may not be ported to PEP 384, or an
extension may require Python APIs outside of PEP 384 definition.
Therefore there will always be a (hopefully diminishing, but never
zero) need for ABI version tagged shared libraries.
Further, it is anticipated that the stable ABI will evolve over time,
meaning that existing PEP 384 compatible extension modules may be
incompatible with future versions of Python. While a complete
specification is reserved for PEP 384, here is a discussion of the
relevant issues.
PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
passed as the API version if the extension was complied with
``Py_LIMITED_API``. This should be formalized into an official macro
called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If
and when the ABI changes in an incompatible way, this version number
would be bumped. To facilitate sharing, Python would be extended to
search for extension modules with the ``PYTHON_ABI_VERSION`` number in
its name. The prefix ``abi`` is reserved for Python's use.
Thus for example, an initial implementation of PEP 384, compiled with
`--with-so-abi-tag=cpython-xy` would search for the following file
names when extension module `foo` is imported (in this order)::
foo.abi3.so
foo.cpython-xy.so
foo.so
The distutils [7]_ ``build_ext`` command would also have to be
extended to compile to shared library files with the ``abi3`` tag,
when the module author indicates that their extension supports that
version of the ABI. This could be done in a backward compatible way
by adding a keyword argument to the ``Extension`` class, such as::
Extension('foo', ['foo.c'], abi=3)
Alternatives
============
In the initial python-dev thread [8]_ where this idea was first
introduced, several alternatives were suggested. For completeness
they are listed here, along with the reasons for not adopting them.
Independent directories or symlinks
-----------------------------------
Debian and Ubuntu could simply add a version-specific directory to
``sys.path`` that would contain just the extension modules for that
version of Python. Or the symlink trick eliminated in PEP 3147 could
be retained for just shared libraries. This approach is rejected
because it propagates the essential complexity that PEP 3147 tries to
avoid, and adds yet another directory to search for all modules, even
when the number of extension modules is much fewer than the total
number of Python packages. It also makes for more robust management
when all of a package's module files live in the same directory,
because it allows systems such as `dpkg` to detect file conflicts
between distribution packages.
Don't share packages with extension modules
-------------------------------------------
It has been suggested that Python packages with extension modules not
be shared among all supported Python versions on a distribution. Even
with adoption of PEP 3149, extension modules will have to be compiled
for every supported Python version, so perhaps sharing of such
packages isn't useful anyway. Not sharing packages with extensions
though is infeasible for several reasons.
If a pure-Python package is shared in one version, should it suddenly
be not-shared if the next release adds an extension module for speed?
Also, even though all extension shared libraries will be compiled and
distributed once for every supported Python, there's a big difference
between duplicating the `.so` files and duplicating all `.py` files.
The extra space increases the download time for such packages, and
more immediately, increases the space pressures on already constrained
distribution CD-ROMs.
Reference implementation
========================
Work on this code is tracked in a Bazaar branch on Launchpad [5]_
until it's ready for merge into Python 3.2. The work-in-progress diff
can also be viewed [6]_ and is updated automatically as new changes
are uploaded.
References
==========
.. [1] PEP 3147
.. [2] PEP 384
.. [3] Ubuntu: <http://www.ubuntu.com>
.. [4] Debian: <http://www.debian.org>
.. [5] https://code.edge.launchpad.net/~barry/python/sovers
.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
.. [7] http://docs.python.org/py3k/distutils/index.html
.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100714/7365b435/attachment.pgp>
More information about the Python-Dev
mailing list