[Python-ideas] bdist naming scheme (compatibility tags) PEP

Daniel Holth dholth at gmail.com
Wed Aug 8 13:14:39 CEST 2012

Today pypy and CPython's "setup.py bdist" generate the same filename
but incompatible bdists. This makes it difficult to share both bdists
in the same folder or index. Instead, they should generate different
bdist filenames because one won't work with the other implementation.
This PEP specifies a tagging system that includes enough information
to decide whether a particular bdist is expected to work on a
particular Python.

Also at https://bitbucket.org/dholth/python-peps/raw/98cd36228c2e/pep-CTAG.txt

Thanks for your feedback,

Daniel Holth
-------------- next part --------------
Title: Compatibility Tags for Built Distributions 
Version: $Revision$
Last-Modified: 07-Aug-2012
Author: Daniel Holth <dholth at fastmail.fm>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Jul-2012


This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible.  A set of three
tags indicate which Python implementation and language version, ABI,
and platform a built distribution requires.  The tags are terse because
they will be included in filenames.


Today "python setup.py bdist" generates the same filename on PyPy
and CPython, but an incompatible archive, making it inconvenient to
share built distributions in the same folder or index.  Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.

Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the Python
implementation, language version, ABI, and platform as a set of tags.

By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.


The tag format is {python tag}-{abi tag}-{platform tag}

python tag
    ?py27?, ?cp33?
abi tag
    ?cp33dmu?, ?none?
platform tag
    ?linux_x86_64?, ?any?

For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any platform.


Python Tag

The Python tag indicates both the implementation and the language version
required by a distribution.  Major implementations have abbreviated
codes, initially:

* py: Generic Python (does not require implementation-specific features)
* cp: CPython
* ip: IronPython
* pp: PyPy
* jy: Jython

Other Python implementations should use `sys.implementation.name`.

The language version is `py_version_nodot`, or just the major version
`2` or `3` for many pure-Python distributions.  CPython gets away with
no dot, but if one is needed the underscore `_` is used instead.

A single-source Python 2/3 compatible distribution can use the compound
tag `py2.py3`.  See `Compressed Tag Sets`, below.


The ABI tag indicates which Python ABI is required by any included
extension modules.  For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
the CPython 3.3 ABI with debugging.

As a special case, the CPython stable ABI starts with `py`; `py32`
is that ABI with only the operations available from Python 3.2 onward.

Implementations with a very unstable ABI may use the first 6 bytes (as
8 base64-encoded characters) of the SHA-256 hash of ther source code
revision and compiler flags, etc, but will probably not have a great need
to distribute binary distributions. Each implementation's community may
decide how to best use the ABI tag.

Platform Tag

The platform tag is simply `distutils.util.get_platform()` with all
hyphens `-` and periods `.` replaced with underscore `_`.


The tags are used by installers to decide which built distribution
(if any) to download from a list of potential built distributions.
Installers will have a list of (python, abi, plat) that the current
Python installation can run sorted by order of preference.  Each built
distribution recieves a score based on its tag's position in the list,
and the most-preferred distribution is the one that is installed.
If no built distribution matches the list of supported tag tuples then
the installer falls back to installing from the source distribution.
Tags are only compared for equality; they are never greater or less than
another tag, and a tag that 'startswith' another tag is not a subset of
the shorter tag.

For example, an installer running under CPython 3.3 on an imaginary MMIX
system might prefer, in order:

0. (cp33, cp33, mmix) # built for this specific version of Python
1. (cp33, py32, mmix) # using the stable ABI as defined by Python 3.2
2. (cp33, none, mmix) # using no ABI, but still depending on the specific
platform (e.g. through ctypes or os.system)
3. (cp33, none, any) # pure-Python distribution for the current Python
4. (py33, none, any) # pure-Python distribution for the current (generic)
5. (py32, none, any) # pure-Python distributions for older versions of
6. (py31, none, any) # ""
6. (py30, none, any) # ""
7. (py3, none, any) # ""

A distribution that requires CPython 3.3 or CPython 2.7 and has an
optional extension module could distribute built distributions tagged
`cp33-cp3-mmix`, `cp33-none-any`, and `cp27-none-any`.  (Our imaginary
program is using 2to3, so the built distribution is not compatible across
major releases.)  `cp33-cp3-mmix` gets a score of 1, `cp33-none-any`
gets a score of 3, and `cp27-none-any` is not in the list at all.  Since
`cp33-cp3-mmix` has the best score, that built distribution is installed.

A user could instruct their installer to fall back to building from an
sdist more or less often by configuring this list of tags.

Compressed Tag Sets

To allow for compact filenames of bdists that work with more than
one compatibility tag triple, each tag in a filename can instead be a
'.'-separated, sorted, set of tags.  For example, pip, a pure-Python
package that is written to run under Python 2 and 3 with the same source
code, could distribute a bdist with the tag `py2.py3-none-any`.
The full list of simple tags is::

    for x in pytag.split('.'):
        for y in abitag.split('.'):
            for z in archtag.split('.'):
                yield '-'.join((x, y, z))

A bdist format that implements this scheme should include the expanded
tags in bdist-specific metadata.  This compression scheme can generate
large numbers of unsupported tags and "impossible" tags that are supported
by no Python implementation e.g. "cp33-cp31u-win64", so use it sparingly.


Can I have a tag `py32+` to indicate a minimum Python minor release?
    No.  Inspect the Trove classifiers to determine this level of
    cross-release compatibility.  Similar to the announcements "beaglevote
    versions 3.2 and above no longer supports Python 1.52", you will
    have to manually keep track of the maximum (PEP-386) release that
    still supports your version of Python.

Why isn't there a `.` in the Python version number?
    CPython has lasted 20+ years without a 3-digit major release. This
    should continue for some time.  Other implementations may use _ as
    a delimeter, since both - and . delimit the surrounding filename.

Who will maintain the registry of abbreviated implementations?
    New two-letter abbreviations can be requested on the python-dev
    mailing list.  As a rule of thumb, abbreviations are reserved for
    the current 4 most prominent implementations. 

Does the compatibility tag go into METADATA or PKG-INFO?
    No.  The compatibility tag is part of the built distribution's
    metadata.  METADATA / PKG-INFO should be valid for an entire
    distribution, not a single build of that distribution.

Why didn't you mention my favorite Python implementation?
    The abbreviated tags facilitate sharing compiled Python code in a
    public index.  Your Python implementation can use this specification
    too, but with longer tags.
    Recall that all "pure Python" built distributions just use 'py'.


.. [1] Egg Filename-Embedded Metadata

.. [2] Creating Built Distributions

.. [3] PEP 3147 -- PYC Repository Directories


The author thanks Paul Moore, Nick Coughlan, Mark Abramowitz, and
Mr. Michele Lacchia for their valuable advice and help with this effort.


This document has been placed in the public domain.

   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8

More information about the Python-ideas mailing list