[Python-Dev] Submitting PEP 425 (compatibility tags)

[Nick Coghlan]

On Sat, Feb 16, 2013 at 2:06 PM, Daniel Holth <dholth at gmail.com> wrote:
> This is the improved compatibility tags PEP 425, specifying how part of the
> Wheel PEP 427 filenames work. Last time we discussed whether replacing .
> with _ was ugly but I concluded it was harmless.
>
> Submitted for your consideration,
>
> PEP: 425
> Title: Compatibility Tags for Built Distributions
> Version: $Revision$
> Last-Modified: 07-Aug-2012
> Author: Daniel Holth <dholth at fastmail.fm>
> BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 27-Jul-2012
> Python-Version: 3.4
> Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013
>
>
> Abstract
> ========
>
> This PEP specifies a tagging system to indicate with which versions of
> Python a built or binary distribution is compatible.  A set of three
> tags indicate which Python implementation and language version, ABI,
> and platform a built distribution requires.  The tags are terse because
> they will be included in filenames.
>
>
> PEP Editor's Note
> =================
>
> While the naming scheme described in this PEP will not be supported directly
> in the standard library until Python 3.4 at the earliest, draft
> implementations may be made available in third party projects.
>
>
> Rationale
> =========
>
> Today "python setup.py bdist" generates the same filename on PyPy
> and CPython, but an incompatible archive, making it inconvenient to
> share built distributions in the same folder or index.  Instead, built
> distributions should have a file naming convention that includes enough
> information to decide whether or not a particular archive is compatible
> with a particular implementation.
>
> Previous efforts come from a time where CPython was the only important
> implementation and the ABI was the same as the Python language release.
> This specification improves upon the older schemes by including the Python
> implementation, language version, ABI, and platform as a set of tags.
>
> By comparing the tags it supports with the tags listed by the
> distribution, an installer can make an educated decision about whether
> to download a particular built distribution without having to read its
> full metadata.
>
> Overview
> ========
>
> The tag format is {python tag}-{abi tag}-{platform tag}
>
> python tag
>     ‘py27’, ‘cp33’
> abi tag
>     ‘cp32dmu’, ‘none’
> platform tag
>     ‘linux_x86_64’, ‘any’
>
> For example, the tag py27-none-any indicates compatible with Python 2.7
> (any Python 2.7 implementation) with no abi requirement, on any platform.
>
> Use
> ===
>
> The `wheel` built package format includes these tags in its filenames,
> of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
> tag}-{platform tag}.whl``.  Other package formats may have their own
> conventions.
>
> Details
> =======
>
> Python Tag
> ----------
>
> The Python tag indicates the implementation and version required by
> a distribution.  Major implementations have abbreviated codes, initially:
>
> * py: Generic Python (does not require implementation-specific features)
> * cp: CPython
> * ip: IronPython
> * pp: PyPy
> * jy: Jython
>
> Other Python implementations should use `sys.implementation.name`.
>
> The version is `py_version_nodot`.  CPython gets away with no dot,
> but if one is needed the underscore `_` is used instead.  PyPy should
> probably use its own versions here `pp18`, `pp19`.
>
> The version can be just the major version `2` or `3` `py2`, `py3` for
> many pure-Python distributions.
>
> Importantly, major-version-only tags like `py2` and `py3` are not
> shorthand for `py20` and `py30`.  Instead, these tags mean the packager
> intentionally released a cross-version-compatible distribution.
>
> A single-source Python 2/3 compatible distribution can use the compound
> tag `py2.py3`.  See `Compressed Tag Sets`, below.
>
> ABI Tag
> -------
>
> The ABI tag indicates which Python ABI is required by any included
> extension modules.  For implementation-specific ABIs, the implementation
> is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
> the CPython 3.3 ABI with debugging.
>
> The CPython stable ABI is `abi3` as in the shared library suffix.
>
> Implementations with a very unstable ABI may use the first 6 bytes (as
> 8 base64-encoded characters) of the SHA-256 hash of ther source code
> revision and compiler flags, etc, but will probably not have a great need
> to distribute binary distributions. Each implementation's community may
> decide how to best use the ABI tag.
>
> Platform Tag
> ------------
>
> The platform tag is simply `distutils.util.get_platform()` with all
> hyphens `-` and periods `.` replaced with underscore `_`.
>
> * win32
> * linux_i386
> * linux_x86_64
>
> Use
> ===
>
> The tags are used by installers to decide which built distribution
> (if any) to download from a list of potential built distributions.
> The installer maintains a list of (pyver, abi, arch) tuples that it
> will support.  If the built distribution's tag is `in` the list, then
> it can be installed.

I suggest adjusting the specific compatibility tags in the example,
and making it clear that is also a *recommended* default order.

================
It is recommended that installers, by default, try to give users
access to the most feature complete built distribution available, by
favouring those versions more specific to the installation
environment, and then those marked as explicitly cross-version
compatible, before falling back to pure Python versions published for
older Python releases. It is also recommended that installers allow
users to configure the list of acceptable compatibility tags,
including their relative precedence.

For example, an installer running under CPython 3.3 on a linux_x86_64
system might support::

  cp33-cp33m-linux_x86_64
  cp33-abi3-linux_x86_64
  cp3-abi3-linux_x86_64  <--- Added
  cp33-none-linux_x86_64*
  cp3-none-linux_x86_64*  <--- Added
  py33-none-linux_x86_64*  <--- Added
  py3-none-linux_x86_64*  <--- Added
  cp33-none-any
  cp3-none-any
  py33-none-any
  py3-none-any
  py32-none-any
  py31-none-any
  py30-none-any

*Built distributions may be platform specific for reasons other than C
extensions, such as using "sudo" to invoke a subprocess with elevated
privileges.
================

> Can I have a tag `py32+` to indicate a minimum Python minor release?
>     No.  Inspect the Trove classifiers to determine this level of
>     cross-release compatibility.  Similar to the announcements "beaglevote
>     versions 3.2 and above no longer supports Python 1.52", you will
>     have to manually keep track of the maximum (PEP-386) release that
>     still supports your version of Python.
>

The answer to this question needs to be updated given that installers
may fall back to trying earlier pure Python (or stable ABI) versions.
(That is, installers are free to offer builds like py32-none-any and
cp32-abi3-linux_x86_64 as possible solutions for a Python 3.3 install
if there's nothing more recent)

Additionally, I suggest including FAQ entries covering your answers to
a couple of other questions that were raised in the last PEP 425
thread (I went back and looked at that thread to check the major
points had been addressed for this version and a couple were missing):

=================
Why normalise hyphens and other non-alphanumeric characters to underscores?
    To avoid conflicting with the "." and "-" characters that separate
components of the filename, and for better compatibility with the
widest range of filesystem limitations for filenames (including being
usable in URL paths without quoting).

Why not use special character <X> rather than "." or "-"?
    Either because that character is inconvenient or potentially
confusing in some contexts (for example, "+" must be quoted in URLs,
"~" is used to denote the user's home directory in POSIX), or because
the advantages weren't sufficiently compelling to justify changing the
existing reference implementation for the wheel format defined in PEP
427 (for example, using "," rather than "." to separate components in
a compressed tag)
=================

The compatibility tag spec itself looks fine to me, so with the above
clarifications, I will accept the PEP.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia