Round 6 - PEP 440 - Version Identification and Dependency Specification Version
I’m happy to announce the 6th, and hopefully final, draft of PEP 440. The updates to this PEP take into account feedback from several large projects and users of various systems (such as Linux package tools) where this would be expected to interface with. It also takes into account experience gained from attempting to implement this PEP fully as a proof of concept within pip. Significant updates to the PEP include: * Switching the Epoch identifier from : to ! as : is not valid in a directory name on Windows. * Local version identifiers use + as a seperator in order to reduce ambigiuty with existing versions on PyPI. * Allow alpha numerics for local versions. * Define a sorting algorithm for local versions. * Moved the source label to PEP 426 * Normalization rules for parsing more versions following along the idea of Postel's Law. * Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also should be used for Metadata 1.0 and 1.1. * Declare how invalid versions should be handled. * Add the escape hatch "Arbitrary equality" operator === to allow depending on versions which cannot be parsed by PEP 440. * Make specifier syntax match what setuptools uses (foo==1.0 instead of foo (==1.0)) * Remove the default specifier. * Use @ for direct references instead of "from". * Create a reference implementation and Link To it. * Lots more minor changes. The outcome of these changes is that we were able to raise compatability with all the versions registered with PyPI up to 98.12% and we sort 99.88% of projects registered with PyPI the same as pkg_resources does when filtering invalid versions from the list of versions. The fallout is that 498 projects, or 1.06%, are no longer installable without using the ``===`` operator and 190 projects, or 0.4%, of projects have a different result for what the "latest" version is. Of the 498 projects a number of them are nonsensical versions like ``.`` or the repr of lazy objects and of the 190 projects projects it's about evenly split between projects where pkg_resources supported something we didn't and where PEP 440 just simply does a better job at parsing and sorting versions. You can see the pip proof of concept and a large discussion about normalization at https://github.com/pypa/pip/pull/1894 and the reference implementation can be found at https://github.com/pypa/packaging/pull/1. The Proof of Concept does not allow using the new specifiers inside of an install_requires in a source distribution because setuptools itself does not support it, but it does support them on the command line and in requirements.txt files. The online view of the PEP can be found at https://www.python.org/dev/peps/pep-0440/ and the changes since the last posting can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and http://hg.python.org/peps/rev/257822378672. Without further ado, the PEP itself: Abstract ======== This PEP describes a scheme for identifying versions of Python software distributions, and declaring dependencies on particular versions. This document addresses several limitations of the previous attempt at a standardized approach to versioning, as described in PEP 345 and PEP 386. Definitions =========== The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. The following terms are to be interpreted as described in PEP 426: * "Distributions" * "Releases" * "Build tools" * "Index servers" * "Publication tools" * "Installation tools" * "Automated tools" * "Projects" Version scheme ============== Distributions are identified by a public version identifier which supports all defined version comparison operations The version scheme is used both to describe the distribution version provided by a particular distribution archive, as well as to place constraints on the version of dependencies needed in order to build or run the software. Public version identifiers -------------------------- Public version identifiers MUST comply with the following scheme:: [N!]N(.N)*[{a|b|c}N][.postN][.devN] Public version identifiers MUST NOT include leading or trailing whitespace. Public version identifiers MUST be unique within a given distribution. Installation tools SHOULD ignore any public versions which do not comply with this scheme. Installation tools MAY warn the user when non-compliant or ambiguous versions are detected. Public version identifiers are separated into up to five segments: * Epoch segment: ``N!`` * Release segment: ``N(.N)*`` * Pre-release segment: ``{a|b|c}N`` * Post-release segment: ``.postN`` * Development release segment: ``.devN`` Any given release will be a "final release", "pre-release", "post-release" or "developmental release" as defined in the following sections. All numeric components MUST be non-negative integers. All numeric components MUST be interpreted and ordered according to their numeric value, not as text strings. All numeric components MAY be zero. Except as described below for the release segment, a numeric component of zero has no special significance aside from always being the lowest possible value in the version ordering. .. note:: Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects. Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections. Local version identifiers ------------------------- Local version identifiers MUST comply with the following scheme:: <public version identifier>[+<local version label>] They consist of a normal public version identifier (as defined in the previous section), along with an arbitrary "local version label", separated from the public version identifier by a plus. Local version labels have no specific semantics assigned, but some syntactic restrictions are imposed. Local version identifiers are used to denote fully API (and, if applicable, ABI) compatible patched versions of upstream projects. For example, these may be created by application developers and system integrators by applying specific backported bug fixes when upgrading to a new upstream release would be too disruptive to the application or other integrated system (such as a Linux distribution). The inclusion of the local version label makes it possible to differentiate upstream releases from potentially altered rebuilds by downstream integrators. The use of a local version identifier does not affect the kind of a release but, when applied to a source distribution, does indicate that it may not contain the exact same code as the corresponding upstream release. To ensure local version identifiers can be readily incorporated as part of filenames and URLs, and to avoid formatting inconsistencies in hexadecimal hash representations, local version labels MUST be limited to the following set of permitted characters: * ASCII letters (``[a-zA-Z]``) * ASCII digits (``[0-9]``) * periods (``.``) Local version labels MUST start and end with an ASCII letter or digit. Comparison and ordering of local versions considers each segment of the local version (divided by a ``.``) separately. If a segment consists entirely of ASCII digits then that section should be considered an integer for comparison purposes and if a segment contains any ASCII letters than that segment is compared lexicographically with case insensitivity. When comparing a numeric and lexicographic segment, the numeric section always compares as greater than the lexicographic segment. Additionally a local version with a great number of segments will always compare as greater than a local version with fewer segments, as long as the shorter local version's segments match the beginning of the longer local version's segments exactly. Local version identifiers may be used in most locations where a public version identifier is expected, with the exception of any version specifiers that explicitly rely on being able to unambiguously order candidate versions. Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions. Source distributions using a local version identifier SHOULD provide the ``python.integrator`` extension metadata (as defined in :pep:`459`). Final releases -------------- A version identifier that consists solely of a release segment and optionally an epoch identifier is termed a "final release". The release segment consists of one or more non-negative integer values, separated by dots:: N[.N]+ Final releases within a project MUST be numbered in a consistently increasing fashion, otherwise automated tools will not be able to upgrade them correctly. Comparison and ordering of release segments considers the numeric value of each component of the release segment in turn. When comparing release segments with different numbers of components, the shorter segment is padded out with additional zeros as necessary. While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). For example:: 0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 ... A release series is any set of final release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series. .. note:: ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as the release segment comparison rules implicit expand the two component form to ``X.Y.0`` when comparing it to any release segment that includes three components. Date based release segments are also permitted. An example of a date based release scheme using the year and month of the release:: 2012.04 2012.07 2012.10 2013.01 2013.06 ... Pre-releases ------------ Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a final release. If used as part of a project's development cycle, these pre-releases are indicated by including a pre-release segment in the version identifier:: X.YaN # Alpha release X.YbN # Beta release X.YcN # Candidate release X.Y # Final release A version identifier that consists solely of a release segment and a pre-release segment is termed a "pre-release". The pre-release segment consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase. Installation tools MAY accept both ``c`` and ``rc`` releases for a common release segment in order to handle some existing legacy distributions. Installation tools SHOULD interpret ``rc`` versions as being equivalent to ``c`` versions (that is, ``rc1`` indicates the same version as ``c1``). Build tools, publication tools and index servers SHOULD disallow the creation of both ``c`` and ``rc`` releases for a common release segment. Post-releases ------------- Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes). If used as part of a project's development cycle, these post-releases are indicated by including a post-release segment in the version identifier:: X.Y.postN # Post-release A version identifier that includes a post-release segment without a developmental release segment is termed a "post-release". The post-release segment consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release. .. note:: The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release. Post-releases are also permitted for pre-releases:: X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate .. note:: Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component. Developmental releases ---------------------- Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases directly from source control which do not conflict with later project releases. If used as part of a project's development cycle, these developmental releases are indicated by including a developmental release segment in the version identifier:: X.Y.devN # Developmental release A version identifier that includes a developmental release segment is termed a "developmental release". The developmental release segment consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases with the same release segment), and following any previous release (including any post-releases). Developmental releases are also permitted for pre-releases and post-releases:: X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release .. note:: Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component. Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes. Version epochs -------------- If included in a version identifier, the epoch appears before all other components, separated from the release segment by an exclamation mark:: E!X.Y # Version identifier with epoch If no explicit epoch is given, the implicit epoch is ``0``. Most version identifiers will not include an epoch, as an explicit epoch is only needed if a project *changes* the way it handles version numbering in a way that means the normal version ordering rules will give the wrong answer. For example, if a project is using date based versions like ``2014.04`` and would like to switch to semantic versions like ``1.0``, then the new releases would be identified as *older* than the date based releases when using the normal sorting scheme:: 1.0 1.1 2.0 2013.10 2014.04 However, by specifying an explicit epoch, the sort order can be changed appropriately, as all versions from a later epoch are sorted after versions from an earlier epoch:: 2013.10 2014.04 1!1.0 1!1.1 1!2.0 Normalization ------------- In order to maintain better compatibility with existing versions there are a number of "alternative" syntaxes that MUST be taken into account when parsing versions. These syntaxes MUST be considered when parsing a version, however they should be "normalized" to the standard syntax defined above. Case sensitivity ~~~~~~~~~~~~~~~~ All ascii letters should be interpreted case insensitively within a version and the normal form is lowercase. This allows versions such as ``1.1RC1`` which would be normalized to ``1.1c1``. Integer Normalization ~~~~~~~~~~~~~~~~~~~~~ All integers are interpreted via the ``int()`` built in and normalize to the string form of the output. This means that an integer version of ``00`` would normalize to ``0`` while ``09000`` would normalize to ``9000``. This does not hold true for integers inside of an alphanumeric segment of a local version such as ``1.0+foo0100`` which is already in its normalized form. Pre-release separators ~~~~~~~~~~~~~~~~~~~~~~ Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the release segment and the pre-release segment. The normal form for this is without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1`` which would be normalized to ``1.1a1``. It should also allow a seperator to be used between the pre-release signifier and the numeral. This allows versions such as ``1.0a.1`` which would be normalized to ``1.0a1``. Pre-release spelling ~~~~~~~~~~~~~~~~~~~~ Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``, ``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively. This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional spelling should be considered equivalent to their normal forms. Implicit pre-release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Pre releases allow omitting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2a`` which is normalized to ``1.2a0``. Post release separators ~~~~~~~~~~~~~~~~~~~~~~~ Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to ``1.2.post2``. Like the pre-release seperator this also allows an optional separator between the post release signifier and the numeral. This allows versions like ``1.2.post-2`` which would normalize to ``1.2.post2``. Post release spelling ~~~~~~~~~~~~~~~~~~~~~ Post-releases allow the additional spellings of ``rev`` and ``r``. This allows versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the pre-releases the additional spellings should be considered equivalent to their normal forms. Implicit post release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Post releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``. Implicit post releases ~~~~~~~~~~~~~~~~~~~~~~ Post releases allow omitting the ``post`` signifier all together. When using this form the separator MUST be ``-`` and no other form is allowed. This allows versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular normalization MUST NOT be used in conjunction with the implicit post release number rule. In other words ``1.0-`` is *not* a valid version and it does *not* normalize to ``1.0.post0``. Development release separators ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Development releases allow a ``.``, ``-``, or a ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which normalize to ``1.2.dev2``. Implicit development release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Development releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.dev`` which is normalized to ``1.2.dev0``. Local version segments ~~~~~~~~~~~~~~~~~~~~~~ With a local version, in addition to the use of ``.`` as a separator of segments, the use of ``-`` and ``_`` is also acceptable. The normal form is using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be normalized to ``1.0+ubuntu.1``. Preceding v character ~~~~~~~~~~~~~~~~~~~~~ In order to support the common version notation of ``v1.0`` versions may be preceded by a single literal ``v`` character. This character MUST be ignored for all purposes and should be omitted from all normalized forms of the version. The same version with and without the ``v`` is considered equivalent. Leading and Trailing Whitespace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Leading and trailing whitespace must be silently ignored and removed from all normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``, ``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly, such as a version like ``1.0\n`` which normalizes to ``1.0``. Examples of compliant version schemes ------------------------------------- The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently. The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools. Simple "major.minor" versioning:: 0.1 0.2 0.3 1.0 1.1 ... Simple "major.minor.micro" versioning:: 1.1.0 1.1.1 1.1.2 1.2.0 ... "major.minor" versioning with alpha, beta and candidate pre-releases:: 0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ... "major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections:: 0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ... Date based releases, using an incrementing serial within each year, skipping zero:: 2012.1 2012.2 2012.3 ... 2012.15 2013.1 2013.2 ... Summary of permitted suffixes and relative ordering --------------------------------------------------- .. note:: This section is intended primarily for authors of tools that automatically process distribution metadata, rather than developers of Python distributions deciding on a versioning scheme. The epoch segment of version identifiers MUST be sorted according to the numeric value of the given epoch. If no epoch segment is present, the implicit numeric value is ``0``. The release segment of version identifiers MUST be sorted in the same order as Python's tuple sorting when the release segment is parsed as follows:: tuple(map(int, release_segment.split("."))) All release segments involved in the comparison MUST be converted to a consistent length by padding shorter segments with zeros as needed. Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and MUST be ordered as shown:: .devN, aN, bN, cN/rcN, <no suffix>, .postN Note that `rc` is considered to be semantically equivalent to `c` and must be sorted as if it were `c`. Tools MAY reject the case of having the same ``N`` for both a ``rc`` and a ``c`` in the same release segment as ambiguous and remain in compliance with the PEP. Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be ordered as shown:: .devN, <no suffix>, .postN Within a post-release (``1.0.post1``), the following suffixes are permitted and MUST be ordered as shown:: .devN, <no suffix> Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a pre-release, post-release or development release segment with a shared prefix, ordering MUST be by the value of the numeric component. The following example covers many of the possible combinations:: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0+abc.5 1.0+abc.7 1.0+5 1.0.post456.dev34 1.0.post456 1.1.dev1 Version ordering across different metadata versions --------------------------------------------------- Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. However metadata v1.2 (PEP 345) does specify a scheme which is defined in PEP 386. Due to the nature of the simple installer API it is not possible for an installer to be aware of which metadata version a particular distribution was using. Additionally installers required the ability to create a reasonably prioritized list that includes all, or as many as possible, versions of a project to determine which versions it should install. These requirements necessitate a standardization across one parsing mechanism to be used for all versions of a project. Due to the above, this PEP MUST be used for all versions of metadata and supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions which cannot be parsed by the rules in this PEP, but MAY fall back to implementation defined version parsing and ordering schemes if no versions complying with this PEP are available. Distribution users may wish to explicitly remove non-compliant versions from any private package indexes they control. Compatibility with other version schemes ---------------------------------------- Some projects may choose to use a version scheme which requires translation in order to comply with the public version scheme defined in this PEP. In such cases, the project specific version can be stored in the metadata while the translated public version is published in the version field. This allows automated distribution tools to provide consistently correct ordering of published releases, while still allowing developers to use the internal versioning scheme they prefer for their projects. Semantic versioning ~~~~~~~~~~~~~~~~~~~ `Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on. The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged. Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public version field. One possible mechanism to translate such semantic versioning based source labels to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order. Specific build information may also be included in local version labels. .. _Semantic versioning: http://semver.org/ DVCS based version labels ~~~~~~~~~~~~~~~~~~~~~~~~~ Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public version field. As with semantic versioning, the public ``.devN`` suffix may be used to uniquely identify such releases for publication, while the original DVCS based label can be stored in the project metadata. Identifying hash information may also be included in local version labels. Olson database versioning ~~~~~~~~~~~~~~~~~~~~~~~~~ The ``pytz`` project inherits its versioning scheme from the corresponding Olson timezone database versioning scheme: the year followed by a lowercase character indicating the version of the database within that year. This can be translated to a compliant public version identifier as ``<year>.<serial>``, where the serial starts at zero or one (for the '<year>a' release) and is incremented with each subsequent database update within the year. As with other translated version identifiers, the corresponding Olson database version could be recorded in the project metadata. Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. For example:: ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0 The comparison operator determines the kind of version clause: * ``~=``: `Compatible release`_ clause * ``==``: `Version matching`_ clause * ``!=``: `Version exclusion`_ clause * ``<=``, ``>=``: `Inclusive ordered comparison`_ clause * ``<``, ``>``: `Exclusive ordered comparison`_ clause * ``===``: `Arbitrary equality`_ clause. The comma (",") is equivalent to a logical **and** operator: a candidate version must match all given version clauses in order to match the specifier as a whole. Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas. When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard `Version scheme`_. Whether or not pre-releases are considered as candidate versions SHOULD be handled as described in `Handling of pre-releases`_. Except where specifically noted below, local version identifiers MUST NOT be permitted in version specifiers, and local version labels MUST be ignored entirely when checking if candidate versions match a given version specifier. Compatible release ------------------ A compatible release clause consists of either a version identifier without any comparison operator or else the compatible release operator ``~=`` and a version identifier. It matches any candidate version that is expected to be compatible with the specified version. The specified version identifier must be in the standard format described in `Version scheme`_. Local version identifiers are NOT permitted in this version specifier. For a given release identifier ``V.N``, the compatible release clause is approximately equivalent to the pair of comparison clauses:: >= V.N, == V.* This operator MUST NOT be used with a single segment version number such as ``~=1``. For example, the following groups of version clauses are equivalent:: 2.2 ~= 2.2 >= 2.2, == 2.* 1.4.5 ~= 1.4.5 >= 1.4.5, == 1.4.* If a pre-release, post-release or developmental release is named in a compatible release clause as ``V.N.suffix``, then the suffix is ignored when determining the required prefix match:: 2.2.post3 ~= 2.2.post3 >= 2.2.post3, == 2.* 1.4.5a4 ~= 1.4.5a4 >= 1.4.5a4, == 1.4.* The padding rules for release segment comparisons means that the assumed degree of forward compatibility in a compatible release clause can be controlled by appending additional zeros to the version specifier:: 2.2.0 ~= 2.2.0 >= 2.2.0, == 2.2.* 1.4.5.0 ~= 1.4.5.0 >= 1.4.5.0, == 1.4.5.* Version matching ---------------- A version matching clause includes the version matching operator ``==`` and a version identifier. The specified version identifier must be in the standard format described in `Version scheme`_, but a trailing ``.*`` is permitted on public version identifiers as described below. By default, the version matching operator is based on a strict equality comparison: the specified version must be exactly the same as the requested version. The *only* substitution performed is the zero padding of the release segment to ensure the release segments are compared with the same length. Whether or not strict version matching is appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when strict version matches are used inappropriately. Prefix matching may be requested instead of strict comparison, by appending a trailing ``.*`` to the version identifier in the version matching clause. This means that additional trailing segments will be ignored when determining whether or not a version identifier matches the clause. If the specified version includes only a release segment, than trailing components (or the lack thereof) in the release segment are also ignored. For example, given the version ``1.1.post1``, the following clauses would match or not as shown:: == 1.1 # Not equal, so 1.1.post1 does not match clause == 1.1.post1 # Equal, so 1.1.post1 matches clause == 1.1.* # Same prefix, so 1.1.post1 matches clause For purposes of prefix matching, the pre-release segment is considered to have an implied preceding ``.``, so given the version ``1.1a1``, the following clauses would match or not as shown:: == 1.1 # Not equal, so 1.1a1 does not match clause == 1.1a1 # Equal, so 1.1a1 matches clause == 1.1.* # Same prefix, so 1.1a1 matches clause An exact match is also considered a prefix match (this interpreation is implied by the usual zero padding rules for the release segment of version identifiers). Given the version ``1.1``, the following clauses would match or not as shown:: == 1.1 # Equal, so 1.1 matches clause == 1.1.0 # Zero padding expands 1.1 to 1.1.0, so it matches clause == 1.1.dev1 # Not equal (dev-release), so 1.1 does not match clause == 1.1a1 # Not equal (pre-release), so 1.1 does not match clause == 1.1.post1 # Not equal (post-release), so 1.1 does not match clause == 1.1.* # Same prefix, so 1.1 matches clause It is invalid to have a prefix match containing a development or local release such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development release segment is always the final segment in the public version, and the local version is ignored for comparison purposes, so using either in a prefix match wouldn't make any sense. The use of ``==`` (without at least the wildcard suffix) when defining dependencies for published distributions is strongly discouraged as it greatly complicates the deployment of security fixes. The strict version comparison operator is intended primarily for use when defining dependencies for repeatable *deployments of applications* while using a shared distribution index. If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions. If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison. Version exclusion ----------------- A version exclusion clause includes the version exclusion operator ``!=`` and a version identifier. The allowed version identifiers and comparison semantics are the same as those of the `Version matching`_ operator, except that the sense of any match is inverted. For example, given the version ``1.1.post1``, the following clauses would match or not as shown:: != 1.1 # Not equal, so 1.1.post1 matches clause != 1.1.post1 # Equal, so 1.1.post1 does not match clause != 1.1.* # Same prefix, so 1.1.post1 does not match clause Inclusive ordered comparison ---------------------------- An inclusive ordered comparison clause includes a comparison operator and a version identifier, and will match any version where the comparison is correct based on the relative position of the candidate version and the specified version given the consistent ordering defined by the standard `Version scheme`_. The inclusive ordered comparison operators are ``<=`` and ``>=``. As with version matching, the release segment is zero padded as necessary to ensure the release segments are compared with the same length. Local version identifiers are NOT permitted in this version specifier. Exclusive ordered comparison ---------------------------- Exclusive ordered comparisons are similar to inclusive ordered comparisons, except that the comparison operators are ``<`` and ``>`` and the clause MUST be effectively interpreted as implying the prefix based version exclusion clause ``!= V.*``. The exclusive ordered comparison ``> V`` MUST NOT match a post-release or maintenance release of the given version. Maintenance releases can be permitted by using the clause ``> V.0``, while both post releases and maintenance releases can be permitted by using the inclusive ordered comparison ``>= V.post1``. The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of the given version, even if acceptance of pre-releases is enabled as described in the section below. Local version identifiers are NOT permitted in this version specifier. Arbitrary equality ------------------ Arbitrary equality comparisons are simple string equality operations which do not take into account any of the semantic information such as zero padding or local versions. This operator also does not support prefix matching as the ``==`` operator does. The primary use case for arbitrary equality is to allow for specifying a version which cannot otherwise be represented by this PEP. This operator is special and acts as an escape hatch to allow someone using a tool which implements this PEP to still install a legacy version which is otherwise incompatible with this PEP. An example would be ``===foobar`` which would match a version of ``foobar``. This operator may also be used to explicitly require an unpatched version of a project such as ``===1.0`` which would not match for a version ``1.0+downstream1``. Use of this operator is heavily discouraged and tooling MAY display a warning when it is used. Handling of pre-releases ------------------------ Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release. By default, dependency resolution tools SHOULD: * accept already installed pre-releases for all version specifiers * accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier * exclude all other pre-releases from consideration Dependency resolution tools MAY issue a warning if a pre-release is needed to satisfy a version specifier. Dependency resolution tools SHOULD also allow users to request the following alternative behaviours: * accepting pre-releases for all version specifiers * excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier) Dependency resolution tools MAY also allow the above behaviour to be controlled on a per-distribution basis. Post-releases and final releases receive no special treatment in version specifiers - they are always included unless explicitly excluded. Examples -------- * ``3.1``: version 3.1 or later, but not version 4.0 or later. * ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later. * ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later. * ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases, post releases, developmental releases and any 3.1.x maintenance releases. * ``== 3.1.*``: any version that starts with 3.1. Equivalent to the ``3.1.0`` compatible release clause. * ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and not version 3.2.0 or later. Direct references ================= Some automated tools may permit the use of a direct reference as an alternative to a normal version specifier. A direct reference consists of the specifier ``@`` and an explicit URL. Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately. Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers. Depending on the use case, some appropriate targets for a direct URL reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or a wheel binary archive. The exact URLs and targets supported will be tool dependent. For example, a local source archive may be referenced directly:: pip @ file:///localbuilds/pip-1.3.1.zip Alternatively, a prebuilt archive may also be referenced:: pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl All direct references that do not refer to a local file URL SHOULD specify a secure transport mechanism (such as ``https``) AND include an expected hash value in the URL for verification purposes. If a direct reference is specified without any hash information, with hash information that the tool doesn't understand, or with a selected hash algorithm that the tool considers too weak to trust, automated tools SHOULD at least emit a warning and MAY refuse to rely on the URL. If such a direct reference also uses an insecure transport, automated tools SHOULD NOT rely on the URL. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used for source archive hashes. At time of writing, that list consists of ``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and ``'sha512'``. For source archive and wheel references, an expected hash value may be specified by including a ``<hash-algorithm>=<expected-hash>`` entry as part of the URL fragment. For version control references, the ``VCS+protocol`` scheme SHOULD be used to identify both the version control system and the secure transport, and a version control system with hash based commit identifiers SHOULD be used. Automated tools MAY omit warnings about missing hashes for version control systems that do not provide hash based commit identifiers. To handle version control systems that do not support including commit or tag references directly in the URL, that information may be appended to the end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>`` notation. .. note:: This isn't *quite* the same as the existing VCS reference notation supported by pip. Firstly, the distribution name is moved in front rather than embedded as part of the URL. Secondly, the commit hash is included even when retrieving based on a tag, in order to meet the requirement above that *every* link should include a hash to make things harder to forge (creating a malicious repo with a particular tag is easy, creating one with a specific *hash*, less so). Remote URL examples:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a... pip @ git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d pip @ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee6... Updating the versioning specification ===================================== The versioning specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Actually changing the version comparison semantics still requires a new versioning scheme and metadata version defined in new PEPs. Summary of differences from \PEP 386 ==================================== * Moved the description of version specifiers into the versioning PEP * Added the "direct reference" concept as a standard notation for direct references to resources (rather than each tool needing to invent its own) * Added the "local version identifier" and "local version label" concepts to allow system integrators to indicate patched builds in a way that is supported by the upstream tools, as well as to allow the incorporation of build tags into the versioning of binary distributions. * Added the "compatible release" clause * Added the trailing wildcard syntax for prefix based version matching and exclusion * Changed the top level sort position of the ``.devN`` suffix * Allowed single value version numbers * Explicit exclusion of leading or trailing whitespace * Explicit support for date based versions * Explicit normalisation rules to improve compatibility with existing version metadata on PyPI where it doesn't introduce ambiguity * Implicitly exclude pre-releases unless they're already present or needed to satisfy a dependency * Treat post releases the same way as unqualified releases * Discuss ordering and dependencies across metadata versions The rationale for major changes is given in the following sections. Changing the version scheme --------------------------- One key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their final releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers, but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions. The exclusion of leading and trailing whitespace was made explicit after a couple of projects with version identifiers differing only in a trailing ``\n`` character were found on PyPI. Various other normalisation rules were also added as described in the separate section on version normalisation below. `Appendix A` shows detailed results of an analysis of PyPI distribution version information, as collected on 8th August, 2014. This analysis compares the behavior of the explicitly ordered version scheme defined in this PEP with the de facto standard defined by the behavior of setuptools. These metrics are useful, as the intent of this PEP is to follow existing setuptools behavior as closely as is feasible, while still throwing exceptions for unorderable versions (rather than trying to guess an appropriate order as setuptools does). A more opinionated description of the versioning scheme ------------------------------------------------------- As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes). For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice. The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions. Describing version specifiers alongside the versioning scheme ------------------------------------------------------------- The main reason to even have a standardised version scheme in the first place is to make it easier to do reliable automated dependency analysis. It makes more sense to describe the primary use case for version identifiers alongside their definition. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as even marking the package as hidden wasn't enough to keep automated tools from downloading it, and also made it harder for users to obtain the test release manually through the main PyPI web interface. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while still allowing pre-release versions to be retrieved automatically when that's the only way to satisfy a dependency. The "some forward compatibility assumed" version constraint is derived from the Ruby community's "pessimistic version constraint" operator [2]_ to allow projects to take a cautious approach to forward compatibility promises, while still easily setting a minimum required version for their dependencies. The spelling of the compatible release clause (``~=``) is inspired by the Ruby (``~>``) and PHP (``~``) equivalents. Further improvements are also planned to the handling of parallel installation of multiple versions of the same library, but these will depend on updates to the installation database definition along with improved tools for dynamic path manipulation. The trailing wildcard syntax to request prefix based version matching was added to make it possible to sensibly define both compatible release clauses and the desired pre- and post-release handling semantics for ``<`` and ``>`` ordered comparison clauses. Support for date based version identifiers ------------------------------------------ Excluding date based versions caused significant problems in migrating ``pytz`` to the new metadata standards. It also caused concerns for the OpenStack developers, as they use a date based versioning scheme and would like to be able to migrate to the new metadata standards without changing it. Adding version epochs --------------------- Version epochs are added for the same reason they are part of other versioning schemes, such as those of the Fedora and Debian Linux distributions: to allow projects to gracefully change their approach to numbering releases, without having a new release appear to have a lower version number than previous releases and without having to change the name of the project. In particular, supporting version epochs allows a project that was previously using date based versioning to switch to semantic versioning by specifying a new version epoch. The ``!`` character was chosen to delimit an epoch version rather than the ``:`` character, which is commonly used in other systems, due to the fact that ``:`` is not a valid character in a Windows directory name. Adding direct references ------------------------ Direct references are added as an "escape clause" to handle messy real world situations that don't map neatly to the standard distribution model. This includes dependencies on unpublished software for internal use, as well as handling the more complex compatibility issues that may arise when wrapping third party libraries as C extensions (this is of especial concern to the scientific community). Index servers are deliberately given a lot of freedom to disallow direct references, since they're intended primarily as a tool for integrators rather than publishers. PyPI in particular is currently going through the process of *eliminating* dependencies on external references, as unreliable external services have the effect of slowing down installation operations, as well as reducing PyPI's own apparent reliability. Adding arbitrary equality ------------------------- Arbitrary equality is added as an "escape clause" to handle the case where someone needs to install a project which uses a non compliant version. Although this PEP is able to attain ~97% compatibility with the versions that are already on PyPI there are still ~3% of versions which cannot be parsed. This operator gives a simple and effective way to still depend on them without having to "guess" at the semantics of what they mean (which would be required if anything other than strict string based equality was supported). Adding local version identifiers -------------------------------- It's a fact of life that downstream integrators often need to backport upstream bug fixes to older versions. It's one of the services that gets Linux distro vendors paid, and application developers may also apply patches they need to bundled dependencies. Historically, this practice has been invisible to cross-platform language specific distribution tools - the reported "version" in the upstream metadata is the same as for the unmodified code. This inaccuracy can then cause problems when attempting to work with a mixture of integrator provided code and unmodified upstream code, or even just attempting to identify exactly which version of the software is installed. The introduction of local version identifiers and "local version labels" into the versioning scheme, with the corresponding ``python.integrator`` metadata extension allows this kind of activity to be represented accurately, which should improve interoperability between the upstream tools and various integrated platforms. The exact scheme chosen is largely modeled on the existing behavior of ``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``, with the main distinction being that where ``pkg_resources`` currently always takes the suffix into account when comparing versions for exact matches, the PEP requires that the local version label of the candidate version be ignored when no local version label is present in the version specifier clause. Furthermore, the PEP does not attempt to impose any structure on the local version labels (aside from limiting the set of permitted characters and defining their ordering). This change is designed to ensure that an integrator provided version like ``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version specifier like ``pip>=1.5``. The plus is chosen primarily for readability of local version identifiers. It was chosen instead of the hyphen to prevent ``pkg_resources.parse_version`` from parsing it as a prerelease, which is important for enabling a successful migration to the new, more structured, versioning scheme. The plus was chosen instead of a tilde because of the significance of the tilde in Debian's version ordering algorithm. Providing explicit version normalization rules ---------------------------------------------- Historically, the de facto standard for parsing versions in Python has been the ``pkg_resources.parse_version`` command from the setuptools project. This does not attempt to reject *any* version and instead tries to make something meaningful, with varying levels of success, out of whatever it is given. It has a few simple rules but otherwise it more or less relies largely on string comparison. The normalization rules provided in this PEP exist primarily to either increase the compatability with ``pkg_resources.parse_version``, particularly in documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something more reasonable with versions that already exist on PyPI. All possible normalization rules were weighed against whether or not they were *likely* to cause any ambiguity (e.g. while someone might devise a scheme where ``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone actually doing that, much less on any scale that is noticeable, is fairly low). They were also weighed against how ``pkg_resources.parse_version`` treated a particular version string, especially with regards to how it was sorted. Finally each rule was weighed against the kinds of additional versions it allowed, how "ugly" those versions looked, how hard there were to parse (both mentally and mechanically) and how much additional compatibility it would bring. The breadth of possible normalizations were kept to things that could easily be implemented as part of the parsing of the version and not pre-parsing transformations applied to the versions. This was done to limit the side effects of each transformation as simple search and replace style transforms increase the likelihood of ambiguous or "junk" versions. For an extended discussion on the various types of normalizations that were considered, please see the proof of concept for PEP 440 within pip [4]_. References ========== The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] Reference Implementation of PEP 440 Versions and Specifiers https://github.com/pypa/packaging/pull/1 .. [2] Version compatibility analysis script: https://github.com/pypa/packaging/blob/master/tasks/check.py .. [3] Pessimistic version constraint http://docs.rubygems.org/read/chapter/16 .. [4] Proof of Concept: PEP 440 within pip https://github.com/pypa/pip/pull/1894 Appendix A ========== Metadata v2.0 guidelines versus setuptools:: $ invoke check.pep440 Total Version Compatibility: 245806/250521 (98.12%) Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%) Total Sorting Compatibility (Filtered): 47057/47114 (99.88%) Projects with No Compatible Versions: 498/47114 (1.06%) Projects with Differing Latest Version: 688/47114 (1.46%) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 8 August 2014 22:53, Donald Stufft <donald@stufft.io> wrote:
Direct references =================
Some automated tools may permit the use of a direct reference as an alternative to a normal version specifier. A direct reference consists of the specifier ``@`` and an explicit URL.
Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately.
Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.
Depending on the use case, some appropriate targets for a direct URL reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or a wheel binary archive. The exact URLs and targets supported will be tool dependent.
For example, a local source archive may be referenced directly::
pip @ file:///localbuilds/pip-1.3.1.zip
Alternatively, a prebuilt archive may also be referenced::
pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl
One very minor question. Is URL format required here? I guess so. On Windows, file URLs are confusing and annoying to type - use of / rather than \ makes tab-completion useless, it's difficult to remember how many slashes go at the start and where the drive letter goes, and do UNC paths need 157 or 158 slashes at the start (:-)), and the documentation is inconsistent and hard to find. Allowing a pathname here would be convenient for users, but it's 100% not important enough to need a new version of the spec. A clarification (presumably, that a URL is required) with a pointer to a document that explains the relevant filename->URL translation algorithm, would be good, though. Paul
On Aug 9, 2014, at 4:26 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 8 August 2014 22:53, Donald Stufft <donald@stufft.io> wrote:
Direct references =================
Some automated tools may permit the use of a direct reference as an alternative to a normal version specifier. A direct reference consists of the specifier ``@`` and an explicit URL.
Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately.
Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.
Depending on the use case, some appropriate targets for a direct URL reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or a wheel binary archive. The exact URLs and targets supported will be tool dependent.
For example, a local source archive may be referenced directly::
pip @ file:///localbuilds/pip-1.3.1.zip
Alternatively, a prebuilt archive may also be referenced::
pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl
One very minor question. Is URL format required here? I guess so. On Windows, file URLs are confusing and annoying to type - use of / rather than \ makes tab-completion useless, it's difficult to remember how many slashes go at the start and where the drive letter goes, and do UNC paths need 157 or 158 slashes at the start (:-)), and the documentation is inconsistent and hard to find.
Allowing a pathname here would be convenient for users, but it's 100% not important enough to need a new version of the spec. A clarification (presumably, that a URL is required) with a pointer to a document that explains the relevant filename->URL translation algorithm, would be good, though.
Paul
To be clear, the direct reference is mostly for use in the install_requires. On the CLI pip can still just take a path to a file or whatever. This feature is intended to replace dependency_links in a way that people can use them for private packages but that they won't be allowed on PyPI or the like. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 9 August 2014 17:02, Donald Stufft <donald@stufft.io> wrote:
To be clear, the direct reference is mostly for use in the install_requires. On the CLI pip can still just take a path to a file or whatever. This feature is intended to replace dependency_links in a way that people can use them for private packages but that they won't be allowed on PyPI or the like.
Understood. I'd still like an explicit pointer to the canonical URL details tools should (or must?) use. See http://en.wikipedia.org/wiki/File_URI_scheme (specifically the "Windows" sections) for why I'm bothered - specifically the quote "Here are *some* examples which *may* be accepted by some applications" (emphasis mine). I'd be more than happy if the spec was just "tools will use XXX from the Python standard library. But note (the first 4 are the supposedly acceptable URLs from that Wikipedia article):
py Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
from urllib.request import url2pathname url2pathname("file://localhost/c|/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi url2pathname("file:///c|/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|///c|/WINDOWS/clock.avi url2pathname("file://localhost/c:/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi url2pathname("file:///c:/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|///c|/WINDOWS/clock.avi url2pathname("file:////WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file:///WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file://WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file:/WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi'
I have no idea where Python got that E drive from - my current drive is C:. At this point in time, I have no idea how to correctly write a file URL for the file C:\Windows\clock.avi on my PC. While it's very definitely a corner case in the PEP, and it's clearly not the PEP's responsibility to explain the file URL spec, I do think it's relevant to the usability of the PEP. Paul
On Aug 9, 2014, at 1:41 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 9 August 2014 17:02, Donald Stufft <donald@stufft.io> wrote:
To be clear, the direct reference is mostly for use in the install_requires. On the CLI pip can still just take a path to a file or whatever. This feature is intended to replace dependency_links in a way that people can use them for private packages but that they won't be allowed on PyPI or the like.
Understood. I'd still like an explicit pointer to the canonical URL details tools should (or must?) use. See http://en.wikipedia.org/wiki/File_URI_scheme (specifically the "Windows" sections) for why I'm bothered - specifically the quote "Here are *some* examples which *may* be accepted by some applications" (emphasis mine).
I'd be more than happy if the spec was just "tools will use XXX from the Python standard library. But note (the first 4 are the supposedly acceptable URLs from that Wikipedia article):
py Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
from urllib.request import url2pathname url2pathname("file://localhost/c|/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi url2pathname("file:///c|/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|///c|/WINDOWS/clock.avi url2pathname("file://localhost/c:/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi url2pathname("file:///c:/WINDOWS/clock.avi") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname raise OSError(error) OSError: Bad URL: file|///c|/WINDOWS/clock.avi url2pathname("file:////WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file:///WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file://WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi' url2pathname("file:/WINDOWS/clock.avi") 'E:\\WINDOWS\\clock.avi'
I have no idea where Python got that E drive from - my current drive is C:.
At this point in time, I have no idea how to correctly write a file URL for the file C:\Windows\clock.avi on my PC. While it's very definitely a corner case in the PEP, and it's clearly not the PEP's responsibility to explain the file URL spec, I do think it's relevant to the usability of the PEP.
Paul
Does: file:///c:/WINDOWS/clock.avi work? That’s pointed out in the wikipage you linked as to what IE has done since IE4 and seems to be the most reasonable candidate. It also uses file://remotehost/sharename/dir/file.txt for UNC paths apparently. I misunderstood the problem though, I thought you were worried about needing to type out the file:// and the slash order on the CLI, not referencing files at all. We can absolutely add more details to the PEP. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 9 August 2014 18:50, Donald Stufft <donald@stufft.io> wrote:
Does: file:///c:/WINDOWS/clock.avi work?
Never mind, it's my mistake. Reading the docs more closely, "Convert the path component path from a percent-encoded URL to the local syntax for a path. This does not accept a complete URL." (Note that second sentence...)
url2pathname("///c:/WINDOWS/clock.avi") 'C:\\WINDOWS\\clock.avi'
Looks like the PEP would be covered by referring to pathname2url/url2pathname (with a note that you strip the "file:" prefix for idiots like me ;-)) and giving a couple of examples. FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how you'd refer to \\myserver\share\WINDOWS\clock.avi. See what I mean about the slashes? I can't rationalise the 4 slashes here resulting in two backslashes, when the 3 slashes above disappear entirely... Paul.
Paul Moore wrote:
FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how you'd refer to \\myserver\share\WINDOWS\clock.avi.
Where did you get that from? According to this it's wrong: http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx It should be file://myserver/share/WINDOWS/clock.avi i.e. the hosthame part of the path goes into the hostname part of the URI, which makes sense. If the path uses a drive letter instead, there is no hostname, so that part of the URI is empty, resulting in three consecutive slashes: file:///c:/WINDOWS/clock.avi -- Greg
On 10 August 2014 01:03, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Paul Moore wrote:
FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how you'd refer to \\myserver\share\WINDOWS\clock.avi.
Where did you get that from? According to this it's wrong:
http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx
It should be
file://myserver/share/WINDOWS/clock.avi
from urllib.request import pathname2url pathname2url(r'\\myserver\share\WINDOWS\clock.avi') '////myserver/share/WINDOWS/clock.avi'
Bug in urllib? Paul
Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions.
I'm thinking this should just say "PyPI" and not "Public" broadly. The point is for local versions not to confused with the one authoritative upstream version sequence, not that it couldn't be "public". I can imagine locally versioned distributions needing to be distributed or available "publicly" (e.g. for a specific platform or system). Considering the recommendation to use the "python.integrator" extension, which is generally about "downstream" modification and redistribution, it seems inconsistent to say that this redistribution couldn't be public. On Fri, Aug 8, 2014 at 2:53 PM, Donald Stufft <donald@stufft.io> wrote:
I’m happy to announce the 6th, and hopefully final, draft of PEP 440. The updates to this PEP take into account feedback from several large projects and users of various systems (such as Linux package tools) where this would be expected to interface with. It also takes into account experience gained from attempting to implement this PEP fully as a proof of concept within pip.
Significant updates to the PEP include:
* Switching the Epoch identifier from : to ! as : is not valid in a directory name on Windows. * Local version identifiers use + as a seperator in order to reduce ambigiuty with existing versions on PyPI. * Allow alpha numerics for local versions. * Define a sorting algorithm for local versions. * Moved the source label to PEP 426 * Normalization rules for parsing more versions following along the idea of Postel's Law. * Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also should be used for Metadata 1.0 and 1.1. * Declare how invalid versions should be handled. * Add the escape hatch "Arbitrary equality" operator === to allow depending on versions which cannot be parsed by PEP 440. * Make specifier syntax match what setuptools uses (foo==1.0 instead of foo (==1.0)) * Remove the default specifier. * Use @ for direct references instead of "from". * Create a reference implementation and Link To it. * Lots more minor changes.
The outcome of these changes is that we were able to raise compatability with all the versions registered with PyPI up to 98.12% and we sort 99.88% of projects registered with PyPI the same as pkg_resources does when filtering invalid versions from the list of versions. The fallout is that 498 projects, or 1.06%, are no longer installable without using the ``===`` operator and 190 projects, or 0.4%, of projects have a different result for what the "latest" version is.
Of the 498 projects a number of them are nonsensical versions like ``.`` or the repr of lazy objects and of the 190 projects projects it's about evenly split between projects where pkg_resources supported something we didn't and where PEP 440 just simply does a better job at parsing and sorting versions.
You can see the pip proof of concept and a large discussion about normalization at https://github.com/pypa/pip/pull/1894 and the reference implementation can be found at https://github.com/pypa/packaging/pull/1. The Proof of Concept does not allow using the new specifiers inside of an install_requires in a source distribution because setuptools itself does not support it, but it does support them on the command line and in requirements.txt files.
The online view of the PEP can be found at https://www.python.org/dev/peps/pep-0440/ and the changes since the last posting can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and http://hg.python.org/peps/rev/257822378672.
Without further ado, the PEP itself:
Abstract ========
This PEP describes a scheme for identifying versions of Python software distributions, and declaring dependencies on particular versions.
This document addresses several limitations of the previous attempt at a standardized approach to versioning, as described in PEP 345 and PEP 386.
Definitions ===========
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
The following terms are to be interpreted as described in PEP 426:
* "Distributions" * "Releases" * "Build tools" * "Index servers" * "Publication tools" * "Installation tools" * "Automated tools" * "Projects"
Version scheme ==============
Distributions are identified by a public version identifier which supports all defined version comparison operations
The version scheme is used both to describe the distribution version provided by a particular distribution archive, as well as to place constraints on the version of dependencies needed in order to build or run the software.
Public version identifiers --------------------------
Public version identifiers MUST comply with the following scheme::
[N!]N(.N)*[{a|b|c}N][.postN][.devN]
Public version identifiers MUST NOT include leading or trailing whitespace.
Public version identifiers MUST be unique within a given distribution.
Installation tools SHOULD ignore any public versions which do not comply with this scheme. Installation tools MAY warn the user when non-compliant or ambiguous versions are detected.
Public version identifiers are separated into up to five segments:
* Epoch segment: ``N!`` * Release segment: ``N(.N)*`` * Pre-release segment: ``{a|b|c}N`` * Post-release segment: ``.postN`` * Development release segment: ``.devN``
Any given release will be a "final release", "pre-release", "post-release" or "developmental release" as defined in the following sections.
All numeric components MUST be non-negative integers.
All numeric components MUST be interpreted and ordered according to their numeric value, not as text strings.
All numeric components MAY be zero. Except as described below for the release segment, a numeric component of zero has no special significance aside from always being the lowest possible value in the version ordering.
.. note::
Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects.
Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections.
Local version identifiers -------------------------
Local version identifiers MUST comply with the following scheme::
<public version identifier>[+<local version label>]
They consist of a normal public version identifier (as defined in the previous section), along with an arbitrary "local version label", separated from the public version identifier by a plus. Local version labels have no specific semantics assigned, but some syntactic restrictions are imposed.
Local version identifiers are used to denote fully API (and, if applicable, ABI) compatible patched versions of upstream projects. For example, these may be created by application developers and system integrators by applying specific backported bug fixes when upgrading to a new upstream release would be too disruptive to the application or other integrated system (such as a Linux distribution).
The inclusion of the local version label makes it possible to differentiate upstream releases from potentially altered rebuilds by downstream integrators. The use of a local version identifier does not affect the kind of a release but, when applied to a source distribution, does indicate that it may not contain the exact same code as the corresponding upstream release.
To ensure local version identifiers can be readily incorporated as part of filenames and URLs, and to avoid formatting inconsistencies in hexadecimal hash representations, local version labels MUST be limited to the following set of permitted characters:
* ASCII letters (``[a-zA-Z]``) * ASCII digits (``[0-9]``) * periods (``.``)
Local version labels MUST start and end with an ASCII letter or digit.
Comparison and ordering of local versions considers each segment of the local version (divided by a ``.``) separately. If a segment consists entirely of ASCII digits then that section should be considered an integer for comparison purposes and if a segment contains any ASCII letters than that segment is compared lexicographically with case insensitivity. When comparing a numeric and lexicographic segment, the numeric section always compares as greater than the lexicographic segment. Additionally a local version with a great number of segments will always compare as greater than a local version with fewer segments, as long as the shorter local version's segments match the beginning of the longer local version's segments exactly.
Local version identifiers may be used in most locations where a public version identifier is expected, with the exception of any version specifiers that explicitly rely on being able to unambiguously order candidate versions.
Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions.
Source distributions using a local version identifier SHOULD provide the ``python.integrator`` extension metadata (as defined in :pep:`459`).
Final releases --------------
A version identifier that consists solely of a release segment and optionally an epoch identifier is termed a "final release".
The release segment consists of one or more non-negative integer values, separated by dots::
N[.N]+
Final releases within a project MUST be numbered in a consistently increasing fashion, otherwise automated tools will not be able to upgrade them correctly.
Comparison and ordering of release segments considers the numeric value of each component of the release segment in turn. When comparing release segments with different numbers of components, the shorter segment is padded out with additional zeros as necessary.
While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro").
For example::
0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 ...
A release series is any set of final release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series.
.. note::
``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as the release segment comparison rules implicit expand the two component form to ``X.Y.0`` when comparing it to any release segment that includes three components.
Date based release segments are also permitted. An example of a date based release scheme using the year and month of the release::
2012.04 2012.07 2012.10 2013.01 2013.06 ...
Pre-releases ------------
Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a final release.
If used as part of a project's development cycle, these pre-releases are indicated by including a pre-release segment in the version identifier::
X.YaN # Alpha release X.YbN # Beta release X.YcN # Candidate release X.Y # Final release
A version identifier that consists solely of a release segment and a pre-release segment is termed a "pre-release".
The pre-release segment consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase.
Installation tools MAY accept both ``c`` and ``rc`` releases for a common release segment in order to handle some existing legacy distributions.
Installation tools SHOULD interpret ``rc`` versions as being equivalent to ``c`` versions (that is, ``rc1`` indicates the same version as ``c1``).
Build tools, publication tools and index servers SHOULD disallow the creation of both ``c`` and ``rc`` releases for a common release segment.
Post-releases -------------
Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes).
If used as part of a project's development cycle, these post-releases are indicated by including a post-release segment in the version identifier::
X.Y.postN # Post-release
A version identifier that includes a post-release segment without a developmental release segment is termed a "post-release".
The post-release segment consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release.
.. note::
The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release.
Post-releases are also permitted for pre-releases::
X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate
.. note::
Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component.
Developmental releases ----------------------
Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases directly from source control which do not conflict with later project releases.
If used as part of a project's development cycle, these developmental releases are indicated by including a developmental release segment in the version identifier::
X.Y.devN # Developmental release
A version identifier that includes a developmental release segment is termed a "developmental release".
The developmental release segment consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases with the same release segment), and following any previous release (including any post-releases).
Developmental releases are also permitted for pre-releases and post-releases::
X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release
.. note::
Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component.
Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes.
Version epochs --------------
If included in a version identifier, the epoch appears before all other components, separated from the release segment by an exclamation mark::
E!X.Y # Version identifier with epoch
If no explicit epoch is given, the implicit epoch is ``0``.
Most version identifiers will not include an epoch, as an explicit epoch is only needed if a project *changes* the way it handles version numbering in a way that means the normal version ordering rules will give the wrong answer. For example, if a project is using date based versions like ``2014.04`` and would like to switch to semantic versions like ``1.0``, then the new releases would be identified as *older* than the date based releases when using the normal sorting scheme::
1.0 1.1 2.0 2013.10 2014.04
However, by specifying an explicit epoch, the sort order can be changed appropriately, as all versions from a later epoch are sorted after versions from an earlier epoch::
2013.10 2014.04 1!1.0 1!1.1 1!2.0
Normalization -------------
In order to maintain better compatibility with existing versions there are a number of "alternative" syntaxes that MUST be taken into account when parsing versions. These syntaxes MUST be considered when parsing a version, however they should be "normalized" to the standard syntax defined above.
Case sensitivity ~~~~~~~~~~~~~~~~
All ascii letters should be interpreted case insensitively within a version and the normal form is lowercase. This allows versions such as ``1.1RC1`` which would be normalized to ``1.1c1``.
Integer Normalization ~~~~~~~~~~~~~~~~~~~~~
All integers are interpreted via the ``int()`` built in and normalize to the string form of the output. This means that an integer version of ``00`` would normalize to ``0`` while ``09000`` would normalize to ``9000``. This does not hold true for integers inside of an alphanumeric segment of a local version such as ``1.0+foo0100`` which is already in its normalized form.
Pre-release separators ~~~~~~~~~~~~~~~~~~~~~~
Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the release segment and the pre-release segment. The normal form for this is without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1`` which would be normalized to ``1.1a1``. It should also allow a seperator to be used between the pre-release signifier and the numeral. This allows versions such as ``1.0a.1`` which would be normalized to ``1.0a1``.
Pre-release spelling ~~~~~~~~~~~~~~~~~~~~
Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``, ``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively. This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional spelling should be considered equivalent to their normal forms.
Implicit pre-release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pre releases allow omitting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2a`` which is normalized to ``1.2a0``.
Post release separators ~~~~~~~~~~~~~~~~~~~~~~~
Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to ``1.2.post2``. Like the pre-release seperator this also allows an optional separator between the post release signifier and the numeral. This allows versions like ``1.2.post-2`` which would normalize to ``1.2.post2``.
Post release spelling ~~~~~~~~~~~~~~~~~~~~~
Post-releases allow the additional spellings of ``rev`` and ``r``. This allows versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the pre-releases the additional spellings should be considered equivalent to their normal forms.
Implicit post release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Post releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``.
Implicit post releases ~~~~~~~~~~~~~~~~~~~~~~
Post releases allow omitting the ``post`` signifier all together. When using this form the separator MUST be ``-`` and no other form is allowed. This allows versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular normalization MUST NOT be used in conjunction with the implicit post release number rule. In other words ``1.0-`` is *not* a valid version and it does *not* normalize to ``1.0.post0``.
Development release separators ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Development releases allow a ``.``, ``-``, or a ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which normalize to ``1.2.dev2``.
Implicit development release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Development releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.dev`` which is normalized to ``1.2.dev0``.
Local version segments ~~~~~~~~~~~~~~~~~~~~~~
With a local version, in addition to the use of ``.`` as a separator of segments, the use of ``-`` and ``_`` is also acceptable. The normal form is using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be normalized to ``1.0+ubuntu.1``.
Preceding v character ~~~~~~~~~~~~~~~~~~~~~
In order to support the common version notation of ``v1.0`` versions may be preceded by a single literal ``v`` character. This character MUST be ignored for all purposes and should be omitted from all normalized forms of the version. The same version with and without the ``v`` is considered equivalent.
Leading and Trailing Whitespace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Leading and trailing whitespace must be silently ignored and removed from all normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``, ``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly, such as a version like ``1.0\n`` which normalizes to ``1.0``.
Examples of compliant version schemes -------------------------------------
The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently.
The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools.
Simple "major.minor" versioning::
0.1 0.2 0.3 1.0 1.1 ...
Simple "major.minor.micro" versioning::
1.1.0 1.1.1 1.1.2 1.2.0 ...
"major.minor" versioning with alpha, beta and candidate pre-releases::
0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ...
"major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections::
0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ...
Date based releases, using an incrementing serial within each year, skipping zero::
2012.1 2012.2 2012.3 ... 2012.15 2013.1 2013.2 ...
Summary of permitted suffixes and relative ordering ---------------------------------------------------
.. note::
This section is intended primarily for authors of tools that automatically process distribution metadata, rather than developers of Python distributions deciding on a versioning scheme.
The epoch segment of version identifiers MUST be sorted according to the numeric value of the given epoch. If no epoch segment is present, the implicit numeric value is ``0``.
The release segment of version identifiers MUST be sorted in the same order as Python's tuple sorting when the release segment is parsed as follows::
tuple(map(int, release_segment.split(".")))
All release segments involved in the comparison MUST be converted to a consistent length by padding shorter segments with zeros as needed.
Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and MUST be ordered as shown::
.devN, aN, bN, cN/rcN, <no suffix>, .postN
Note that `rc` is considered to be semantically equivalent to `c` and must be sorted as if it were `c`. Tools MAY reject the case of having the same ``N`` for both a ``rc`` and a ``c`` in the same release segment as ambiguous and remain in compliance with the PEP.
Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be ordered as shown::
.devN, <no suffix>, .postN
Within a post-release (``1.0.post1``), the following suffixes are permitted and MUST be ordered as shown::
.devN, <no suffix>
Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``).
Within a pre-release, post-release or development release segment with a shared prefix, ordering MUST be by the value of the numeric component.
The following example covers many of the possible combinations::
1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0+abc.5 1.0+abc.7 1.0+5 1.0.post456.dev34 1.0.post456 1.1.dev1
Version ordering across different metadata versions ---------------------------------------------------
Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. However metadata v1.2 (PEP 345) does specify a scheme which is defined in PEP 386.
Due to the nature of the simple installer API it is not possible for an installer to be aware of which metadata version a particular distribution was using. Additionally installers required the ability to create a reasonably prioritized list that includes all, or as many as possible, versions of a project to determine which versions it should install. These requirements necessitate a standardization across one parsing mechanism to be used for all versions of a project.
Due to the above, this PEP MUST be used for all versions of metadata and supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions which cannot be parsed by the rules in this PEP, but MAY fall back to implementation defined version parsing and ordering schemes if no versions complying with this PEP are available.
Distribution users may wish to explicitly remove non-compliant versions from any private package indexes they control.
Compatibility with other version schemes ----------------------------------------
Some projects may choose to use a version scheme which requires translation in order to comply with the public version scheme defined in this PEP. In such cases, the project specific version can be stored in the metadata while the translated public version is published in the version field.
This allows automated distribution tools to provide consistently correct ordering of published releases, while still allowing developers to use the internal versioning scheme they prefer for their projects.
Semantic versioning ~~~~~~~~~~~~~~~~~~~
`Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on.
The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged.
Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public version field.
One possible mechanism to translate such semantic versioning based source labels to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order.
Specific build information may also be included in local version labels.
.. _Semantic versioning: http://semver.org/
DVCS based version labels ~~~~~~~~~~~~~~~~~~~~~~~~~
Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public version field.
As with semantic versioning, the public ``.devN`` suffix may be used to uniquely identify such releases for publication, while the original DVCS based label can be stored in the project metadata.
Identifying hash information may also be included in local version labels.
Olson database versioning ~~~~~~~~~~~~~~~~~~~~~~~~~
The ``pytz`` project inherits its versioning scheme from the corresponding Olson timezone database versioning scheme: the year followed by a lowercase character indicating the version of the database within that year.
This can be translated to a compliant public version identifier as ``<year>.<serial>``, where the serial starts at zero or one (for the '<year>a' release) and is incremented with each subsequent database update within the year.
As with other translated version identifiers, the corresponding Olson database version could be recorded in the project metadata.
Version specifiers ==================
A version specifier consists of a series of version clauses, separated by commas. For example::
~= 0.9, >= 1.0, != 1.3.4.*, < 2.0
The comparison operator determines the kind of version clause:
* ``~=``: `Compatible release`_ clause * ``==``: `Version matching`_ clause * ``!=``: `Version exclusion`_ clause * ``<=``, ``>=``: `Inclusive ordered comparison`_ clause * ``<``, ``>``: `Exclusive ordered comparison`_ clause * ``===``: `Arbitrary equality`_ clause.
The comma (",") is equivalent to a logical **and** operator: a candidate version must match all given version clauses in order to match the specifier as a whole.
Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas.
When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard `Version scheme`_. Whether or not pre-releases are considered as candidate versions SHOULD be handled as described in `Handling of pre-releases`_.
Except where specifically noted below, local version identifiers MUST NOT be permitted in version specifiers, and local version labels MUST be ignored entirely when checking if candidate versions match a given version specifier.
Compatible release ------------------
A compatible release clause consists of either a version identifier without any comparison operator or else the compatible release operator ``~=`` and a version identifier. It matches any candidate version that is expected to be compatible with the specified version.
The specified version identifier must be in the standard format described in `Version scheme`_. Local version identifiers are NOT permitted in this version specifier.
For a given release identifier ``V.N``, the compatible release clause is approximately equivalent to the pair of comparison clauses::
>= V.N, == V.*
This operator MUST NOT be used with a single segment version number such as ``~=1``.
For example, the following groups of version clauses are equivalent::
2.2 ~= 2.2 >= 2.2, == 2.*
1.4.5 ~= 1.4.5 >= 1.4.5, == 1.4.*
If a pre-release, post-release or developmental release is named in a compatible release clause as ``V.N.suffix``, then the suffix is ignored when determining the required prefix match::
2.2.post3 ~= 2.2.post3 >= 2.2.post3, == 2.*
1.4.5a4 ~= 1.4.5a4 >= 1.4.5a4, == 1.4.*
The padding rules for release segment comparisons means that the assumed degree of forward compatibility in a compatible release clause can be controlled by appending additional zeros to the version specifier::
2.2.0 ~= 2.2.0 >= 2.2.0, == 2.2.*
1.4.5.0 ~= 1.4.5.0 >= 1.4.5.0, == 1.4.5.*
Version matching ----------------
A version matching clause includes the version matching operator ``==`` and a version identifier.
The specified version identifier must be in the standard format described in `Version scheme`_, but a trailing ``.*`` is permitted on public version identifiers as described below.
By default, the version matching operator is based on a strict equality comparison: the specified version must be exactly the same as the requested version. The *only* substitution performed is the zero padding of the release segment to ensure the release segments are compared with the same length.
Whether or not strict version matching is appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when strict version matches are used inappropriately.
Prefix matching may be requested instead of strict comparison, by appending a trailing ``.*`` to the version identifier in the version matching clause. This means that additional trailing segments will be ignored when determining whether or not a version identifier matches the clause. If the specified version includes only a release segment, than trailing components (or the lack thereof) in the release segment are also ignored.
For example, given the version ``1.1.post1``, the following clauses would match or not as shown::
== 1.1 # Not equal, so 1.1.post1 does not match clause == 1.1.post1 # Equal, so 1.1.post1 matches clause == 1.1.* # Same prefix, so 1.1.post1 matches clause
For purposes of prefix matching, the pre-release segment is considered to have an implied preceding ``.``, so given the version ``1.1a1``, the following clauses would match or not as shown::
== 1.1 # Not equal, so 1.1a1 does not match clause == 1.1a1 # Equal, so 1.1a1 matches clause == 1.1.* # Same prefix, so 1.1a1 matches clause
An exact match is also considered a prefix match (this interpreation is implied by the usual zero padding rules for the release segment of version identifiers). Given the version ``1.1``, the following clauses would match or not as shown::
== 1.1 # Equal, so 1.1 matches clause == 1.1.0 # Zero padding expands 1.1 to 1.1.0, so it matches clause == 1.1.dev1 # Not equal (dev-release), so 1.1 does not match clause == 1.1a1 # Not equal (pre-release), so 1.1 does not match clause == 1.1.post1 # Not equal (post-release), so 1.1 does not match clause == 1.1.* # Same prefix, so 1.1 matches clause
It is invalid to have a prefix match containing a development or local release such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development release segment is always the final segment in the public version, and the local version is ignored for comparison purposes, so using either in a prefix match wouldn't make any sense.
The use of ``==`` (without at least the wildcard suffix) when defining dependencies for published distributions is strongly discouraged as it greatly complicates the deployment of security fixes. The strict version comparison operator is intended primarily for use when defining dependencies for repeatable *deployments of applications* while using a shared distribution index.
If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions.
If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison.
Version exclusion -----------------
A version exclusion clause includes the version exclusion operator ``!=`` and a version identifier.
The allowed version identifiers and comparison semantics are the same as those of the `Version matching`_ operator, except that the sense of any match is inverted.
For example, given the version ``1.1.post1``, the following clauses would match or not as shown::
!= 1.1 # Not equal, so 1.1.post1 matches clause != 1.1.post1 # Equal, so 1.1.post1 does not match clause != 1.1.* # Same prefix, so 1.1.post1 does not match clause
Inclusive ordered comparison ----------------------------
An inclusive ordered comparison clause includes a comparison operator and a version identifier, and will match any version where the comparison is correct based on the relative position of the candidate version and the specified version given the consistent ordering defined by the standard `Version scheme`_.
The inclusive ordered comparison operators are ``<=`` and ``>=``.
As with version matching, the release segment is zero padded as necessary to ensure the release segments are compared with the same length.
Local version identifiers are NOT permitted in this version specifier.
Exclusive ordered comparison ----------------------------
Exclusive ordered comparisons are similar to inclusive ordered comparisons, except that the comparison operators are ``<`` and ``>`` and the clause MUST be effectively interpreted as implying the prefix based version exclusion clause ``!= V.*``.
The exclusive ordered comparison ``> V`` MUST NOT match a post-release or maintenance release of the given version. Maintenance releases can be permitted by using the clause ``> V.0``, while both post releases and maintenance releases can be permitted by using the inclusive ordered comparison ``>= V.post1``.
The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of the given version, even if acceptance of pre-releases is enabled as described in the section below.
Local version identifiers are NOT permitted in this version specifier.
Arbitrary equality ------------------
Arbitrary equality comparisons are simple string equality operations which do not take into account any of the semantic information such as zero padding or local versions. This operator also does not support prefix matching as the ``==`` operator does.
The primary use case for arbitrary equality is to allow for specifying a version which cannot otherwise be represented by this PEP. This operator is special and acts as an escape hatch to allow someone using a tool which implements this PEP to still install a legacy version which is otherwise incompatible with this PEP.
An example would be ``===foobar`` which would match a version of ``foobar``.
This operator may also be used to explicitly require an unpatched version of a project such as ``===1.0`` which would not match for a version ``1.0+downstream1``.
Use of this operator is heavily discouraged and tooling MAY display a warning when it is used.
Handling of pre-releases ------------------------
Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.
By default, dependency resolution tools SHOULD:
* accept already installed pre-releases for all version specifiers * accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier * exclude all other pre-releases from consideration
Dependency resolution tools MAY issue a warning if a pre-release is needed to satisfy a version specifier.
Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:
* accepting pre-releases for all version specifiers * excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)
Dependency resolution tools MAY also allow the above behaviour to be controlled on a per-distribution basis.
Post-releases and final releases receive no special treatment in version specifiers - they are always included unless explicitly excluded.
Examples --------
* ``3.1``: version 3.1 or later, but not version 4.0 or later. * ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later. * ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later. * ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases, post releases, developmental releases and any 3.1.x maintenance releases. * ``== 3.1.*``: any version that starts with 3.1. Equivalent to the ``3.1.0`` compatible release clause. * ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and not version 3.2.0 or later.
Direct references =================
Some automated tools may permit the use of a direct reference as an alternative to a normal version specifier. A direct reference consists of the specifier ``@`` and an explicit URL.
Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately.
Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.
Depending on the use case, some appropriate targets for a direct URL reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or a wheel binary archive. The exact URLs and targets supported will be tool dependent.
For example, a local source archive may be referenced directly::
pip @ file:///localbuilds/pip-1.3.1.zip
Alternatively, a prebuilt archive may also be referenced::
pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl
All direct references that do not refer to a local file URL SHOULD specify a secure transport mechanism (such as ``https``) AND include an expected hash value in the URL for verification purposes. If a direct reference is specified without any hash information, with hash information that the tool doesn't understand, or with a selected hash algorithm that the tool considers too weak to trust, automated tools SHOULD at least emit a warning and MAY refuse to rely on the URL. If such a direct reference also uses an insecure transport, automated tools SHOULD NOT rely on the URL.
It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used for source archive hashes. At time of writing, that list consists of ``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and ``'sha512'``.
For source archive and wheel references, an expected hash value may be specified by including a ``<hash-algorithm>=<expected-hash>`` entry as part of the URL fragment.
For version control references, the ``VCS+protocol`` scheme SHOULD be used to identify both the version control system and the secure transport, and a version control system with hash based commit identifiers SHOULD be used. Automated tools MAY omit warnings about missing hashes for version control systems that do not provide hash based commit identifiers.
To handle version control systems that do not support including commit or tag references directly in the URL, that information may be appended to the end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>`` notation.
.. note::
This isn't *quite* the same as the existing VCS reference notation supported by pip. Firstly, the distribution name is moved in front rather than embedded as part of the URL. Secondly, the commit hash is included even when retrieving based on a tag, in order to meet the requirement above that *every* link should include a hash to make things harder to forge (creating a malicious repo with a particular tag is easy, creating one with a specific *hash*, less so).
Remote URL examples::
pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a... pip @ git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d pip @ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee6...
Updating the versioning specification =====================================
The versioning specification may be updated with clarifications without requiring a new PEP or a change to the metadata version.
Actually changing the version comparison semantics still requires a new versioning scheme and metadata version defined in new PEPs.
Summary of differences from \PEP 386 ====================================
* Moved the description of version specifiers into the versioning PEP
* Added the "direct reference" concept as a standard notation for direct references to resources (rather than each tool needing to invent its own)
* Added the "local version identifier" and "local version label" concepts to allow system integrators to indicate patched builds in a way that is supported by the upstream tools, as well as to allow the incorporation of build tags into the versioning of binary distributions.
* Added the "compatible release" clause
* Added the trailing wildcard syntax for prefix based version matching and exclusion
* Changed the top level sort position of the ``.devN`` suffix
* Allowed single value version numbers
* Explicit exclusion of leading or trailing whitespace
* Explicit support for date based versions
* Explicit normalisation rules to improve compatibility with existing version metadata on PyPI where it doesn't introduce ambiguity
* Implicitly exclude pre-releases unless they're already present or needed to satisfy a dependency
* Treat post releases the same way as unqualified releases
* Discuss ordering and dependencies across metadata versions
The rationale for major changes is given in the following sections.
Changing the version scheme ---------------------------
One key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their final releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates.
The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools).
Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard.
Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers, but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions.
The exclusion of leading and trailing whitespace was made explicit after a couple of projects with version identifiers differing only in a trailing ``\n`` character were found on PyPI.
Various other normalisation rules were also added as described in the separate section on version normalisation below.
`Appendix A` shows detailed results of an analysis of PyPI distribution version information, as collected on 8th August, 2014. This analysis compares the behavior of the explicitly ordered version scheme defined in this PEP with the de facto standard defined by the behavior of setuptools. These metrics are useful, as the intent of this PEP is to follow existing setuptools behavior as closely as is feasible, while still throwing exceptions for unorderable versions (rather than trying to guess an appropriate order as setuptools does).
A more opinionated description of the versioning scheme -------------------------------------------------------
As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes).
For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice.
The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions.
Describing version specifiers alongside the versioning scheme -------------------------------------------------------------
The main reason to even have a standardised version scheme in the first place is to make it easier to do reliable automated dependency analysis. It makes more sense to describe the primary use case for version identifiers alongside their definition.
Changing the interpretation of version specifiers -------------------------------------------------
The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as even marking the package as hidden wasn't enough to keep automated tools from downloading it, and also made it harder for users to obtain the test release manually through the main PyPI web interface.
The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason.
The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while still allowing pre-release versions to be retrieved automatically when that's the only way to satisfy a dependency.
The "some forward compatibility assumed" version constraint is derived from the Ruby community's "pessimistic version constraint" operator [2]_ to allow projects to take a cautious approach to forward compatibility promises, while still easily setting a minimum required version for their dependencies. The spelling of the compatible release clause (``~=``) is inspired by the Ruby (``~>``) and PHP (``~``) equivalents.
Further improvements are also planned to the handling of parallel installation of multiple versions of the same library, but these will depend on updates to the installation database definition along with improved tools for dynamic path manipulation.
The trailing wildcard syntax to request prefix based version matching was added to make it possible to sensibly define both compatible release clauses and the desired pre- and post-release handling semantics for ``<`` and ``>`` ordered comparison clauses.
Support for date based version identifiers ------------------------------------------
Excluding date based versions caused significant problems in migrating ``pytz`` to the new metadata standards. It also caused concerns for the OpenStack developers, as they use a date based versioning scheme and would like to be able to migrate to the new metadata standards without changing it.
Adding version epochs ---------------------
Version epochs are added for the same reason they are part of other versioning schemes, such as those of the Fedora and Debian Linux distributions: to allow projects to gracefully change their approach to numbering releases, without having a new release appear to have a lower version number than previous releases and without having to change the name of the project.
In particular, supporting version epochs allows a project that was previously using date based versioning to switch to semantic versioning by specifying a new version epoch.
The ``!`` character was chosen to delimit an epoch version rather than the ``:`` character, which is commonly used in other systems, due to the fact that ``:`` is not a valid character in a Windows directory name.
Adding direct references ------------------------
Direct references are added as an "escape clause" to handle messy real world situations that don't map neatly to the standard distribution model. This includes dependencies on unpublished software for internal use, as well as handling the more complex compatibility issues that may arise when wrapping third party libraries as C extensions (this is of especial concern to the scientific community).
Index servers are deliberately given a lot of freedom to disallow direct references, since they're intended primarily as a tool for integrators rather than publishers. PyPI in particular is currently going through the process of *eliminating* dependencies on external references, as unreliable external services have the effect of slowing down installation operations, as well as reducing PyPI's own apparent reliability.
Adding arbitrary equality -------------------------
Arbitrary equality is added as an "escape clause" to handle the case where someone needs to install a project which uses a non compliant version. Although this PEP is able to attain ~97% compatibility with the versions that are already on PyPI there are still ~3% of versions which cannot be parsed. This operator gives a simple and effective way to still depend on them without having to "guess" at the semantics of what they mean (which would be required if anything other than strict string based equality was supported).
Adding local version identifiers --------------------------------
It's a fact of life that downstream integrators often need to backport upstream bug fixes to older versions. It's one of the services that gets Linux distro vendors paid, and application developers may also apply patches they need to bundled dependencies.
Historically, this practice has been invisible to cross-platform language specific distribution tools - the reported "version" in the upstream metadata is the same as for the unmodified code. This inaccuracy can then cause problems when attempting to work with a mixture of integrator provided code and unmodified upstream code, or even just attempting to identify exactly which version of the software is installed.
The introduction of local version identifiers and "local version labels" into the versioning scheme, with the corresponding ``python.integrator`` metadata extension allows this kind of activity to be represented accurately, which should improve interoperability between the upstream tools and various integrated platforms.
The exact scheme chosen is largely modeled on the existing behavior of ``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``, with the main distinction being that where ``pkg_resources`` currently always takes the suffix into account when comparing versions for exact matches, the PEP requires that the local version label of the candidate version be ignored when no local version label is present in the version specifier clause. Furthermore, the PEP does not attempt to impose any structure on the local version labels (aside from limiting the set of permitted characters and defining their ordering).
This change is designed to ensure that an integrator provided version like ``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version specifier like ``pip>=1.5``.
The plus is chosen primarily for readability of local version identifiers. It was chosen instead of the hyphen to prevent ``pkg_resources.parse_version`` from parsing it as a prerelease, which is important for enabling a successful migration to the new, more structured, versioning scheme. The plus was chosen instead of a tilde because of the significance of the tilde in Debian's version ordering algorithm.
Providing explicit version normalization rules ----------------------------------------------
Historically, the de facto standard for parsing versions in Python has been the ``pkg_resources.parse_version`` command from the setuptools project. This does not attempt to reject *any* version and instead tries to make something meaningful, with varying levels of success, out of whatever it is given. It has a few simple rules but otherwise it more or less relies largely on string comparison.
The normalization rules provided in this PEP exist primarily to either increase the compatability with ``pkg_resources.parse_version``, particularly in documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something more reasonable with versions that already exist on PyPI.
All possible normalization rules were weighed against whether or not they were *likely* to cause any ambiguity (e.g. while someone might devise a scheme where ``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone actually doing that, much less on any scale that is noticeable, is fairly low). They were also weighed against how ``pkg_resources.parse_version`` treated a particular version string, especially with regards to how it was sorted. Finally each rule was weighed against the kinds of additional versions it allowed, how "ugly" those versions looked, how hard there were to parse (both mentally and mechanically) and how much additional compatibility it would bring.
The breadth of possible normalizations were kept to things that could easily be implemented as part of the parsing of the version and not pre-parsing transformations applied to the versions. This was done to limit the side effects of each transformation as simple search and replace style transforms increase the likelihood of ambiguous or "junk" versions.
For an extended discussion on the various types of normalizations that were considered, please see the proof of concept for PEP 440 within pip [4]_.
References ==========
The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386.
.. [1] Reference Implementation of PEP 440 Versions and Specifiers https://github.com/pypa/packaging/pull/1
.. [2] Version compatibility analysis script: https://github.com/pypa/packaging/blob/master/tasks/check.py
.. [3] Pessimistic version constraint http://docs.rubygems.org/read/chapter/16
.. [4] Proof of Concept: PEP 440 within pip https://github.com/pypa/pip/pull/1894
Appendix A ==========
Metadata v2.0 guidelines versus setuptools::
$ invoke check.pep440 Total Version Compatibility: 245806/250521 (98.12%) Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%) Total Sorting Compatibility (Filtered): 47057/47114 (99.88%) Projects with No Compatible Versions: 498/47114 (1.06%) Projects with Differing Latest Version: 688/47114 (1.46%)
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Aug 11, 2014, at 11:11 AM, Marcus Smith <qwcode@gmail.com> wrote:
Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions.
I'm thinking this should just say "PyPI" and not "Public" broadly. The point is for local versions not to confused with the one authoritative upstream version sequence, not that it couldn't be "public". I can imagine locally versioned distributions needing to be distributed or available "publicly" (e.g. for a specific platform or system). Considering the recommendation to use the "python.integrator" extension, which is generally about "downstream" modification and redistribution, it seems inconsistent to say that this redistribution couldn't be public.
We actually have a definition for Public Index Server, It’s ""Public index servers" are index servers which allow distribution uploads from untrusted third parties. The Python Package Index [3] is a public index server.”. This is defined in PEP 426. But thinking about it, that doesn't particularly match it either, because something like binstar allows this, but should allow local versions. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 12 Aug 2014 01:23, "Donald Stufft" <donald@stufft.io> wrote:
On Aug 11, 2014, at 11:11 AM, Marcus Smith <qwcode@gmail.com> wrote:
Public index servers SHOULD NOT allow the use of local version
identifiers for uploaded distributions.
I'm thinking this should just say "PyPI" and not "Public" broadly. The point is for local versions not to confused with the one
authoritative upstream version sequence, not that it couldn't be "public".
I can imagine locally versioned distributions needing to be distributed or available "publicly" (e.g. for a specific platform or system). Considering the recommendation to use the "python.integrator" extension, which is generally about "downstream" modification and redistribution, it seems inconsistent to say that this redistribution couldn't be public.
We actually have a definition for Public Index Server, It’s ""Public index servers" are index servers which allow distribution uploads from untrusted third parties. The Python Package Index [3] is a public index server.”. This is defined in PEP 426.
But thinking about it, that doesn't particularly match it either, because something like binstar allows this, but should allow local versions.
In this particular case, I think Marcus is right - the restriction is specific to PyPI as the keeper of the authoritative shared namespace, rather than applying to public index servers in general. Cheers, Nick.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Thanks Donald for the extensive work on this. It all looks generally good. One thing that stuck out as slightly surprising – the use of ‘_’ as a separator. I imagine most people consider the underscore to be yet another alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest that the PEP give some examples. As is, I didn’t see any examples that showed ‘_’ as a separator. Related, I would suggest a consistent scheme for local version tags. Why not have local version tags have the same syntax as primary version numbers? That is, allow the same character set, pre and post versions, etc, just separated by a +. At the very least, I would expect local versions to allow underscores. These are all just nitpicks, though. A lot of work has gone into this spec, and it’s good to see the community coalescing on an implementable standard. It’s surely acceptable in the current form. From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Donald Stufft Sent: Friday, 08 August, 2014 17:53 To: distutils-sig@python.org Subject: [Distutils] Round 6 - PEP 440 - Version Identification and Dependency Specification Version I’m happy to announce the 6th, and hopefully final, draft of PEP 440. The updates to this PEP take into account feedback from several large projects and users of various systems (such as Linux package tools) where this would be expected to interface with. It also takes into account experience gained from attempting to implement this PEP fully as a proof of concept within pip. Significant updates to the PEP include: * Switching the Epoch identifier from : to ! as : is not valid in a directory name on Windows. * Local version identifiers use + as a seperator in order to reduce ambigiuty with existing versions on PyPI. * Allow alpha numerics for local versions. * Define a sorting algorithm for local versions. * Moved the source label to PEP 426 * Normalization rules for parsing more versions following along the idea of Postel's Law. * Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also should be used for Metadata 1.0 and 1.1. * Declare how invalid versions should be handled. * Add the escape hatch "Arbitrary equality" operator === to allow depending on versions which cannot be parsed by PEP 440. * Make specifier syntax match what setuptools uses (foo==1.0 instead of foo (==1.0)) * Remove the default specifier. * Use @ for direct references instead of "from". * Create a reference implementation and Link To it. * Lots more minor changes. The outcome of these changes is that we were able to raise compatability with all the versions registered with PyPI up to 98.12% and we sort 99.88% of projects registered with PyPI the same as pkg_resources does when filtering invalid versions from the list of versions. The fallout is that 498 projects, or 1.06%, are no longer installable without using the ``===`` operator and 190 projects, or 0.4%, of projects have a different result for what the "latest" version is. Of the 498 projects a number of them are nonsensical versions like ``.`` or the repr of lazy objects and of the 190 projects projects it's about evenly split between projects where pkg_resources supported something we didn't and where PEP 440 just simply does a better job at parsing and sorting versions. You can see the pip proof of concept and a large discussion about normalization at https://github.com/pypa/pip/pull/1894 and the reference implementation can be found at https://github.com/pypa/packaging/pull/1. The Proof of Concept does not allow using the new specifiers inside of an install_requires in a source distribution because setuptools itself does not support it, but it does support them on the command line and in requirements.txt files. The online view of the PEP can be found at https://www.python.org/dev/peps/pep-0440/ and the changes since the last posting can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and http://hg.python.org/peps/rev/257822378672. Without further ado, the PEP itself: Abstract ======== This PEP describes a scheme for identifying versions of Python software distributions, and declaring dependencies on particular versions. This document addresses several limitations of the previous attempt at a standardized approach to versioning, as described in PEP 345 and PEP 386. Definitions =========== The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. The following terms are to be interpreted as described in PEP 426: * "Distributions" * "Releases" * "Build tools" * "Index servers" * "Publication tools" * "Installation tools" * "Automated tools" * "Projects" Version scheme ============== Distributions are identified by a public version identifier which supports all defined version comparison operations The version scheme is used both to describe the distribution version provided by a particular distribution archive, as well as to place constraints on the version of dependencies needed in order to build or run the software. Public version identifiers -------------------------- Public version identifiers MUST comply with the following scheme:: [N!]N(.N)*[{a|b|c}N][.postN][.devN] Public version identifiers MUST NOT include leading or trailing whitespace. Public version identifiers MUST be unique within a given distribution. Installation tools SHOULD ignore any public versions which do not comply with this scheme. Installation tools MAY warn the user when non-compliant or ambiguous versions are detected. Public version identifiers are separated into up to five segments: * Epoch segment: ``N!`` * Release segment: ``N(.N)*`` * Pre-release segment: ``{a|b|c}N`` * Post-release segment: ``.postN`` * Development release segment: ``.devN`` Any given release will be a "final release", "pre-release", "post-release" or "developmental release" as defined in the following sections. All numeric components MUST be non-negative integers. All numeric components MUST be interpreted and ordered according to their numeric value, not as text strings. All numeric components MAY be zero. Except as described below for the release segment, a numeric component of zero has no special significance aside from always being the lowest possible value in the version ordering. .. note:: Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects. Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections. Local version identifiers ------------------------- Local version identifiers MUST comply with the following scheme:: <public version identifier>[+<local version label>] They consist of a normal public version identifier (as defined in the previous section), along with an arbitrary "local version label", separated from the public version identifier by a plus. Local version labels have no specific semantics assigned, but some syntactic restrictions are imposed. Local version identifiers are used to denote fully API (and, if applicable, ABI) compatible patched versions of upstream projects. For example, these may be created by application developers and system integrators by applying specific backported bug fixes when upgrading to a new upstream release would be too disruptive to the application or other integrated system (such as a Linux distribution). The inclusion of the local version label makes it possible to differentiate upstream releases from potentially altered rebuilds by downstream integrators. The use of a local version identifier does not affect the kind of a release but, when applied to a source distribution, does indicate that it may not contain the exact same code as the corresponding upstream release. To ensure local version identifiers can be readily incorporated as part of filenames and URLs, and to avoid formatting inconsistencies in hexadecimal hash representations, local version labels MUST be limited to the following set of permitted characters: * ASCII letters (``[a-zA-Z]``) * ASCII digits (``[0-9]``) * periods (``.``) Local version labels MUST start and end with an ASCII letter or digit. Comparison and ordering of local versions considers each segment of the local version (divided by a ``.``) separately. If a segment consists entirely of ASCII digits then that section should be considered an integer for comparison purposes and if a segment contains any ASCII letters than that segment is compared lexicographically with case insensitivity. When comparing a numeric and lexicographic segment, the numeric section always compares as greater than the lexicographic segment. Additionally a local version with a great number of segments will always compare as greater than a local version with fewer segments, as long as the shorter local version's segments match the beginning of the longer local version's segments exactly. Local version identifiers may be used in most locations where a public version identifier is expected, with the exception of any version specifiers that explicitly rely on being able to unambiguously order candidate versions. Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions. Source distributions using a local version identifier SHOULD provide the ``python.integrator`` extension metadata (as defined in :pep:`459`). Final releases -------------- A version identifier that consists solely of a release segment and optionally an epoch identifier is termed a "final release". The release segment consists of one or more non-negative integer values, separated by dots:: N[.N]+ Final releases within a project MUST be numbered in a consistently increasing fashion, otherwise automated tools will not be able to upgrade them correctly. Comparison and ordering of release segments considers the numeric value of each component of the release segment in turn. When comparing release segments with different numbers of components, the shorter segment is padded out with additional zeros as necessary. While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). For example:: 0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 ... A release series is any set of final release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series. .. note:: ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as the release segment comparison rules implicit expand the two component form to ``X.Y.0`` when comparing it to any release segment that includes three components. Date based release segments are also permitted. An example of a date based release scheme using the year and month of the release:: 2012.04 2012.07 2012.10 2013.01 2013.06 ... Pre-releases ------------ Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a final release. If used as part of a project's development cycle, these pre-releases are indicated by including a pre-release segment in the version identifier:: X.YaN # Alpha release X.YbN # Beta release X.YcN # Candidate release X.Y # Final release A version identifier that consists solely of a release segment and a pre-release segment is termed a "pre-release". The pre-release segment consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase. Installation tools MAY accept both ``c`` and ``rc`` releases for a common release segment in order to handle some existing legacy distributions. Installation tools SHOULD interpret ``rc`` versions as being equivalent to ``c`` versions (that is, ``rc1`` indicates the same version as ``c1``). Build tools, publication tools and index servers SHOULD disallow the creation of both ``c`` and ``rc`` releases for a common release segment. Post-releases ------------- Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes). If used as part of a project's development cycle, these post-releases are indicated by including a post-release segment in the version identifier:: X.Y.postN # Post-release A version identifier that includes a post-release segment without a developmental release segment is termed a "post-release". The post-release segment consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release. .. note:: The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release. Post-releases are also permitted for pre-releases:: X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate .. note:: Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component. Developmental releases ---------------------- Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases directly from source control which do not conflict with later project releases. If used as part of a project's development cycle, these developmental releases are indicated by including a developmental release segment in the version identifier:: X.Y.devN # Developmental release A version identifier that includes a developmental release segment is termed a "developmental release". The developmental release segment consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases with the same release segment), and following any previous release (including any post-releases). Developmental releases are also permitted for pre-releases and post-releases:: X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release .. note:: Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component. Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes. Version epochs -------------- If included in a version identifier, the epoch appears before all other components, separated from the release segment by an exclamation mark:: E!X.Y # Version identifier with epoch If no explicit epoch is given, the implicit epoch is ``0``. Most version identifiers will not include an epoch, as an explicit epoch is only needed if a project *changes* the way it handles version numbering in a way that means the normal version ordering rules will give the wrong answer. For example, if a project is using date based versions like ``2014.04`` and would like to switch to semantic versions like ``1.0``, then the new releases would be identified as *older* than the date based releases when using the normal sorting scheme:: 1.0 1.1 2.0 2013.10 2014.04 However, by specifying an explicit epoch, the sort order can be changed appropriately, as all versions from a later epoch are sorted after versions from an earlier epoch:: 2013.10 2014.04 1!1.0 1!1.1 1!2.0 Normalization ------------- In order to maintain better compatibility with existing versions there are a number of "alternative" syntaxes that MUST be taken into account when parsing versions. These syntaxes MUST be considered when parsing a version, however they should be "normalized" to the standard syntax defined above. Case sensitivity ~~~~~~~~~~~~~~~~ All ascii letters should be interpreted case insensitively within a version and the normal form is lowercase. This allows versions such as ``1.1RC1`` which would be normalized to ``1.1c1``. Integer Normalization ~~~~~~~~~~~~~~~~~~~~~ All integers are interpreted via the ``int()`` built in and normalize to the string form of the output. This means that an integer version of ``00`` would normalize to ``0`` while ``09000`` would normalize to ``9000``. This does not hold true for integers inside of an alphanumeric segment of a local version such as ``1.0+foo0100`` which is already in its normalized form. Pre-release separators ~~~~~~~~~~~~~~~~~~~~~~ Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the release segment and the pre-release segment. The normal form for this is without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1`` which would be normalized to ``1.1a1``. It should also allow a seperator to be used between the pre-release signifier and the numeral. This allows versions such as ``1.0a.1`` which would be normalized to ``1.0a1``. Pre-release spelling ~~~~~~~~~~~~~~~~~~~~ Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``, ``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively. This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional spelling should be considered equivalent to their normal forms. Implicit pre-release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Pre releases allow omitting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2a`` which is normalized to ``1.2a0``. Post release separators ~~~~~~~~~~~~~~~~~~~~~~~ Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to ``1.2.post2``. Like the pre-release seperator this also allows an optional separator between the post release signifier and the numeral. This allows versions like ``1.2.post-2`` which would normalize to ``1.2.post2``. Post release spelling ~~~~~~~~~~~~~~~~~~~~~ Post-releases allow the additional spellings of ``rev`` and ``r``. This allows versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the pre-releases the additional spellings should be considered equivalent to their normal forms. Implicit post release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Post releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``. Implicit post releases ~~~~~~~~~~~~~~~~~~~~~~ Post releases allow omitting the ``post`` signifier all together. When using this form the separator MUST be ``-`` and no other form is allowed. This allows versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular normalization MUST NOT be used in conjunction with the implicit post release number rule. In other words ``1.0-`` is *not* a valid version and it does *not* normalize to ``1.0.post0``. Development release separators ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Development releases allow a ``.``, ``-``, or a ``_`` separator as well as omitting the separator all together. The normal form of this is with the ``.`` separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which normalize to ``1.2.dev2``. Implicit development release number ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Development releases allow omiting the numeral in which case it is implicitly assumed to be ``0``. The normal form for this is to include the ``0`` explicitly. This allows versions such as ``1.2.dev`` which is normalized to ``1.2.dev0``. Local version segments ~~~~~~~~~~~~~~~~~~~~~~ With a local version, in addition to the use of ``.`` as a separator of segments, the use of ``-`` and ``_`` is also acceptable. The normal form is using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be normalized to ``1.0+ubuntu.1``. Preceding v character ~~~~~~~~~~~~~~~~~~~~~ In order to support the common version notation of ``v1.0`` versions may be preceded by a single literal ``v`` character. This character MUST be ignored for all purposes and should be omitted from all normalized forms of the version. The same version with and without the ``v`` is considered equivalent. Leading and Trailing Whitespace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Leading and trailing whitespace must be silently ignored and removed from all normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``, ``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly, such as a version like ``1.0\n`` which normalizes to ``1.0``. Examples of compliant version schemes ------------------------------------- The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently. The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools. Simple "major.minor" versioning:: 0.1 0.2 0.3 1.0 1.1 ... Simple "major.minor.micro" versioning:: 1.1.0 1.1.1 1.1.2 1.2.0 ... "major.minor" versioning with alpha, beta and candidate pre-releases:: 0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ... "major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections:: 0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ... Date based releases, using an incrementing serial within each year, skipping zero:: 2012.1 2012.2 2012.3 ... 2012.15 2013.1 2013.2 ... Summary of permitted suffixes and relative ordering --------------------------------------------------- .. note:: This section is intended primarily for authors of tools that automatically process distribution metadata, rather than developers of Python distributions deciding on a versioning scheme. The epoch segment of version identifiers MUST be sorted according to the numeric value of the given epoch. If no epoch segment is present, the implicit numeric value is ``0``. The release segment of version identifiers MUST be sorted in the same order as Python's tuple sorting when the release segment is parsed as follows:: tuple(map(int, release_segment.split("."))) All release segments involved in the comparison MUST be converted to a consistent length by padding shorter segments with zeros as needed. Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and MUST be ordered as shown:: .devN, aN, bN, cN/rcN, <no suffix>, .postN Note that `rc` is considered to be semantically equivalent to `c` and must be sorted as if it were `c`. Tools MAY reject the case of having the same ``N`` for both a ``rc`` and a ``c`` in the same release segment as ambiguous and remain in compliance with the PEP. Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be ordered as shown:: .devN, <no suffix>, .postN Within a post-release (``1.0.post1``), the following suffixes are permitted and MUST be ordered as shown:: .devN, <no suffix> Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a pre-release, post-release or development release segment with a shared prefix, ordering MUST be by the value of the numeric component. The following example covers many of the possible combinations:: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0+abc.5 1.0+abc.7 1.0+5 1.0.post456.dev34 1.0.post456 1.1.dev1 Version ordering across different metadata versions --------------------------------------------------- Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. However metadata v1.2 (PEP 345) does specify a scheme which is defined in PEP 386. Due to the nature of the simple installer API it is not possible for an installer to be aware of which metadata version a particular distribution was using. Additionally installers required the ability to create a reasonably prioritized list that includes all, or as many as possible, versions of a project to determine which versions it should install. These requirements necessitate a standardization across one parsing mechanism to be used for all versions of a project. Due to the above, this PEP MUST be used for all versions of metadata and supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions which cannot be parsed by the rules in this PEP, but MAY fall back to implementation defined version parsing and ordering schemes if no versions complying with this PEP are available. Distribution users may wish to explicitly remove non-compliant versions from any private package indexes they control. Compatibility with other version schemes ---------------------------------------- Some projects may choose to use a version scheme which requires translation in order to comply with the public version scheme defined in this PEP. In such cases, the project specific version can be stored in the metadata while the translated public version is published in the version field. This allows automated distribution tools to provide consistently correct ordering of published releases, while still allowing developers to use the internal versioning scheme they prefer for their projects. Semantic versioning ~~~~~~~~~~~~~~~~~~~ `Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on. The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged. Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public version field. One possible mechanism to translate such semantic versioning based source labels to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order. Specific build information may also be included in local version labels. .. _Semantic versioning: http://semver.org/ DVCS based version labels ~~~~~~~~~~~~~~~~~~~~~~~~~ Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public version field. As with semantic versioning, the public ``.devN`` suffix may be used to uniquely identify such releases for publication, while the original DVCS based label can be stored in the project metadata. Identifying hash information may also be included in local version labels. Olson database versioning ~~~~~~~~~~~~~~~~~~~~~~~~~ The ``pytz`` project inherits its versioning scheme from the corresponding Olson timezone database versioning scheme: the year followed by a lowercase character indicating the version of the database within that year. This can be translated to a compliant public version identifier as ``<year>.<serial>``, where the serial starts at zero or one (for the '<year>a' release) and is incremented with each subsequent database update within the year. As with other translated version identifiers, the corresponding Olson database version could be recorded in the project metadata. Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. For example:: ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0 The comparison operator determines the kind of version clause: * ``~=``: `Compatible release`_ clause * ``==``: `Version matching`_ clause * ``!=``: `Version exclusion`_ clause * ``<=``, ``>=``: `Inclusive ordered comparison`_ clause * ``<``, ``>``: `Exclusive ordered comparison`_ clause * ``===``: `Arbitrary equality`_ clause. The comma (",") is equivalent to a logical **and** operator: a candidate version must match all given version clauses in order to match the specifier as a whole. Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas. When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard `Version scheme`_. Whether or not pre-releases are considered as candidate versions SHOULD be handled as described in `Handling of pre-releases`_. Except where specifically noted below, local version identifiers MUST NOT be permitted in version specifiers, and local version labels MUST be ignored entirely when checking if candidate versions match a given version specifier. Compatible release ------------------ A compatible release clause consists of either a version identifier without any comparison operator or else the compatible release operator ``~=`` and a version identifier. It matches any candidate version that is expected to be compatible with the specified version. The specified version identifier must be in the standard format described in `Version scheme`_. Local version identifiers are NOT permitted in this version specifier. For a given release identifier ``V.N``, the compatible release clause is approximately equivalent to the pair of comparison clauses:: >= V.N, == V.* This operator MUST NOT be used with a single segment version number such as ``~=1``. For example, the following groups of version clauses are equivalent:: 2.2 ~= 2.2 >= 2.2, == 2.* 1.4.5 ~= 1.4.5 >= 1.4.5, == 1.4.* If a pre-release, post-release or developmental release is named in a compatible release clause as ``V.N.suffix``, then the suffix is ignored when determining the required prefix match:: 2.2.post3 ~= 2.2.post3 >= 2.2.post3, == 2.* 1.4.5a4 ~= 1.4.5a4 >= 1.4.5a4, == 1.4.* The padding rules for release segment comparisons means that the assumed degree of forward compatibility in a compatible release clause can be controlled by appending additional zeros to the version specifier:: 2.2.0 ~= 2.2.0 >= 2.2.0, == 2.2.* 1.4.5.0 ~= 1.4.5.0 >= 1.4.5.0, == 1.4.5.* Version matching ---------------- A version matching clause includes the version matching operator ``==`` and a version identifier. The specified version identifier must be in the standard format described in `Version scheme`_, but a trailing ``.*`` is permitted on public version identifiers as described below. By default, the version matching operator is based on a strict equality comparison: the specified version must be exactly the same as the requested version. The *only* substitution performed is the zero padding of the release segment to ensure the release segments are compared with the same length. Whether or not strict version matching is appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when strict version matches are used inappropriately. Prefix matching may be requested instead of strict comparison, by appending a trailing ``.*`` to the version identifier in the version matching clause. This means that additional trailing segments will be ignored when determining whether or not a version identifier matches the clause. If the specified version includes only a release segment, than trailing components (or the lack thereof) in the release segment are also ignored. For example, given the version ``1.1.post1``, the following clauses would match or not as shown:: == 1.1 # Not equal, so 1.1.post1 does not match clause == 1.1.post1 # Equal, so 1.1.post1 matches clause == 1.1.* # Same prefix, so 1.1.post1 matches clause For purposes of prefix matching, the pre-release segment is considered to have an implied preceding ``.``, so given the version ``1.1a1``, the following clauses would match or not as shown:: == 1.1 # Not equal, so 1.1a1 does not match clause == 1.1a1 # Equal, so 1.1a1 matches clause == 1.1.* # Same prefix, so 1.1a1 matches clause An exact match is also considered a prefix match (this interpreation is implied by the usual zero padding rules for the release segment of version identifiers). Given the version ``1.1``, the following clauses would match or not as shown:: == 1.1 # Equal, so 1.1 matches clause == 1.1.0 # Zero padding expands 1.1 to 1.1.0, so it matches clause == 1.1.dev1 # Not equal (dev-release), so 1.1 does not match clause == 1.1a1 # Not equal (pre-release), so 1.1 does not match clause == 1.1.post1 # Not equal (post-release), so 1.1 does not match clause == 1.1.* # Same prefix, so 1.1 matches clause It is invalid to have a prefix match containing a development or local release such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development release segment is always the final segment in the public version, and the local version is ignored for comparison purposes, so using either in a prefix match wouldn't make any sense. The use of ``==`` (without at least the wildcard suffix) when defining dependencies for published distributions is strongly discouraged as it greatly complicates the deployment of security fixes. The strict version comparison operator is intended primarily for use when defining dependencies for repeatable *deployments of applications* while using a shared distribution index. If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions. If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison. Version exclusion ----------------- A version exclusion clause includes the version exclusion operator ``!=`` and a version identifier. The allowed version identifiers and comparison semantics are the same as those of the `Version matching`_ operator, except that the sense of any match is inverted. For example, given the version ``1.1.post1``, the following clauses would match or not as shown:: != 1.1 # Not equal, so 1.1.post1 matches clause != 1.1.post1 # Equal, so 1.1.post1 does not match clause != 1.1.* # Same prefix, so 1.1.post1 does not match clause Inclusive ordered comparison ---------------------------- An inclusive ordered comparison clause includes a comparison operator and a version identifier, and will match any version where the comparison is correct based on the relative position of the candidate version and the specified version given the consistent ordering defined by the standard `Version scheme`_. The inclusive ordered comparison operators are ``<=`` and ``>=``. As with version matching, the release segment is zero padded as necessary to ensure the release segments are compared with the same length. Local version identifiers are NOT permitted in this version specifier. Exclusive ordered comparison ---------------------------- Exclusive ordered comparisons are similar to inclusive ordered comparisons, except that the comparison operators are ``<`` and ``>`` and the clause MUST be effectively interpreted as implying the prefix based version exclusion clause ``!= V.*``. The exclusive ordered comparison ``> V`` MUST NOT match a post-release or maintenance release of the given version. Maintenance releases can be permitted by using the clause ``> V.0``, while both post releases and maintenance releases can be permitted by using the inclusive ordered comparison ``>= V.post1``. The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of the given version, even if acceptance of pre-releases is enabled as described in the section below. Local version identifiers are NOT permitted in this version specifier. Arbitrary equality ------------------ Arbitrary equality comparisons are simple string equality operations which do not take into account any of the semantic information such as zero padding or local versions. This operator also does not support prefix matching as the ``==`` operator does. The primary use case for arbitrary equality is to allow for specifying a version which cannot otherwise be represented by this PEP. This operator is special and acts as an escape hatch to allow someone using a tool which implements this PEP to still install a legacy version which is otherwise incompatible with this PEP. An example would be ``===foobar`` which would match a version of ``foobar``. This operator may also be used to explicitly require an unpatched version of a project such as ``===1.0`` which would not match for a version ``1.0+downstream1``. Use of this operator is heavily discouraged and tooling MAY display a warning when it is used. Handling of pre-releases ------------------------ Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release. By default, dependency resolution tools SHOULD: * accept already installed pre-releases for all version specifiers * accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier * exclude all other pre-releases from consideration Dependency resolution tools MAY issue a warning if a pre-release is needed to satisfy a version specifier. Dependency resolution tools SHOULD also allow users to request the following alternative behaviours: * accepting pre-releases for all version specifiers * excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier) Dependency resolution tools MAY also allow the above behaviour to be controlled on a per-distribution basis. Post-releases and final releases receive no special treatment in version specifiers - they are always included unless explicitly excluded. Examples -------- * ``3.1``: version 3.1 or later, but not version 4.0 or later. * ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later. * ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later. * ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases, post releases, developmental releases and any 3.1.x maintenance releases. * ``== 3.1.*``: any version that starts with 3.1. Equivalent to the ``3.1.0`` compatible release clause. * ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and not version 3.2.0 or later. Direct references ================= Some automated tools may permit the use of a direct reference as an alternative to a normal version specifier. A direct reference consists of the specifier ``@`` and an explicit URL. Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately. Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers. Depending on the use case, some appropriate targets for a direct URL reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or a wheel binary archive. The exact URLs and targets supported will be tool dependent. For example, a local source archive may be referenced directly:: pip @ file:///localbuilds/pip-1.3.1.zip<file:///\\localbuilds\pip-1.3.1.zip> Alternatively, a prebuilt archive may also be referenced:: pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl<file:///\\localbuilds\pip-1.3.1-py33-none-any.whl> All direct references that do not refer to a local file URL SHOULD specify a secure transport mechanism (such as ``https``) AND include an expected hash value in the URL for verification purposes. If a direct reference is specified without any hash information, with hash information that the tool doesn't understand, or with a selected hash algorithm that the tool considers too weak to trust, automated tools SHOULD at least emit a warning and MAY refuse to rely on the URL. If such a direct reference also uses an insecure transport, automated tools SHOULD NOT rely on the URL. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used for source archive hashes. At time of writing, that list consists of ``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and ``'sha512'``. For source archive and wheel references, an expected hash value may be specified by including a ``<hash-algorithm>=<expected-hash>`` entry as part of the URL fragment. For version control references, the ``VCS+protocol`` scheme SHOULD be used to identify both the version control system and the secure transport, and a version control system with hash based commit identifiers SHOULD be used. Automated tools MAY omit warnings about missing hashes for version control systems that do not provide hash based commit identifiers. To handle version control systems that do not support including commit or tag references directly in the URL, that information may be appended to the end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>`` notation. .. note:: This isn't *quite* the same as the existing VCS reference notation supported by pip. Firstly, the distribution name is moved in front rather than embedded as part of the URL. Secondly, the commit hash is included even when retrieving based on a tag, in order to meet the requirement above that *every* link should include a hash to make things harder to forge (creating a malicious repo with a particular tag is easy, creating one with a specific *hash*, less so). Remote URL examples:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a... pip @ git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d pip @ git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee6... Updating the versioning specification ===================================== The versioning specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Actually changing the version comparison semantics still requires a new versioning scheme and metadata version defined in new PEPs. Summary of differences from \PEP 386 ==================================== * Moved the description of version specifiers into the versioning PEP * Added the "direct reference" concept as a standard notation for direct references to resources (rather than each tool needing to invent its own) * Added the "local version identifier" and "local version label" concepts to allow system integrators to indicate patched builds in a way that is supported by the upstream tools, as well as to allow the incorporation of build tags into the versioning of binary distributions. * Added the "compatible release" clause * Added the trailing wildcard syntax for prefix based version matching and exclusion * Changed the top level sort position of the ``.devN`` suffix * Allowed single value version numbers * Explicit exclusion of leading or trailing whitespace * Explicit support for date based versions * Explicit normalisation rules to improve compatibility with existing version metadata on PyPI where it doesn't introduce ambiguity * Implicitly exclude pre-releases unless they're already present or needed to satisfy a dependency * Treat post releases the same way as unqualified releases * Discuss ordering and dependencies across metadata versions The rationale for major changes is given in the following sections. Changing the version scheme --------------------------- One key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their final releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers, but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions. The exclusion of leading and trailing whitespace was made explicit after a couple of projects with version identifiers differing only in a trailing ``\n`` character were found on PyPI. Various other normalisation rules were also added as described in the separate section on version normalisation below. `Appendix A` shows detailed results of an analysis of PyPI distribution version information, as collected on 8th August, 2014. This analysis compares the behavior of the explicitly ordered version scheme defined in this PEP with the de facto standard defined by the behavior of setuptools. These metrics are useful, as the intent of this PEP is to follow existing setuptools behavior as closely as is feasible, while still throwing exceptions for unorderable versions (rather than trying to guess an appropriate order as setuptools does). A more opinionated description of the versioning scheme ------------------------------------------------------- As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes). For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice. The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions. Describing version specifiers alongside the versioning scheme ------------------------------------------------------------- The main reason to even have a standardised version scheme in the first place is to make it easier to do reliable automated dependency analysis. It makes more sense to describe the primary use case for version identifiers alongside their definition. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as even marking the package as hidden wasn't enough to keep automated tools from downloading it, and also made it harder for users to obtain the test release manually through the main PyPI web interface. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while still allowing pre-release versions to be retrieved automatically when that's the only way to satisfy a dependency. The "some forward compatibility assumed" version constraint is derived from the Ruby community's "pessimistic version constraint" operator [2]_ to allow projects to take a cautious approach to forward compatibility promises, while still easily setting a minimum required version for their dependencies. The spelling of the compatible release clause (``~=``) is inspired by the Ruby (``~>``) and PHP (``~``) equivalents. Further improvements are also planned to the handling of parallel installation of multiple versions of the same library, but these will depend on updates to the installation database definition along with improved tools for dynamic path manipulation. The trailing wildcard syntax to request prefix based version matching was added to make it possible to sensibly define both compatible release clauses and the desired pre- and post-release handling semantics for ``<`` and ``>`` ordered comparison clauses. Support for date based version identifiers ------------------------------------------ Excluding date based versions caused significant problems in migrating ``pytz`` to the new metadata standards. It also caused concerns for the OpenStack developers, as they use a date based versioning scheme and would like to be able to migrate to the new metadata standards without changing it. Adding version epochs --------------------- Version epochs are added for the same reason they are part of other versioning schemes, such as those of the Fedora and Debian Linux distributions: to allow projects to gracefully change their approach to numbering releases, without having a new release appear to have a lower version number than previous releases and without having to change the name of the project. In particular, supporting version epochs allows a project that was previously using date based versioning to switch to semantic versioning by specifying a new version epoch. The ``!`` character was chosen to delimit an epoch version rather than the ``:`` character, which is commonly used in other systems, due to the fact that ``:`` is not a valid character in a Windows directory name. Adding direct references ------------------------ Direct references are added as an "escape clause" to handle messy real world situations that don't map neatly to the standard distribution model. This includes dependencies on unpublished software for internal use, as well as handling the more complex compatibility issues that may arise when wrapping third party libraries as C extensions (this is of especial concern to the scientific community). Index servers are deliberately given a lot of freedom to disallow direct references, since they're intended primarily as a tool for integrators rather than publishers. PyPI in particular is currently going through the process of *eliminating* dependencies on external references, as unreliable external services have the effect of slowing down installation operations, as well as reducing PyPI's own apparent reliability. Adding arbitrary equality ------------------------- Arbitrary equality is added as an "escape clause" to handle the case where someone needs to install a project which uses a non compliant version. Although this PEP is able to attain ~97% compatibility with the versions that are already on PyPI there are still ~3% of versions which cannot be parsed. This operator gives a simple and effective way to still depend on them without having to "guess" at the semantics of what they mean (which would be required if anything other than strict string based equality was supported). Adding local version identifiers -------------------------------- It's a fact of life that downstream integrators often need to backport upstream bug fixes to older versions. It's one of the services that gets Linux distro vendors paid, and application developers may also apply patches they need to bundled dependencies. Historically, this practice has been invisible to cross-platform language specific distribution tools - the reported "version" in the upstream metadata is the same as for the unmodified code. This inaccuracy can then cause problems when attempting to work with a mixture of integrator provided code and unmodified upstream code, or even just attempting to identify exactly which version of the software is installed. The introduction of local version identifiers and "local version labels" into the versioning scheme, with the corresponding ``python.integrator`` metadata extension allows this kind of activity to be represented accurately, which should improve interoperability between the upstream tools and various integrated platforms. The exact scheme chosen is largely modeled on the existing behavior of ``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``, with the main distinction being that where ``pkg_resources`` currently always takes the suffix into account when comparing versions for exact matches, the PEP requires that the local version label of the candidate version be ignored when no local version label is present in the version specifier clause. Furthermore, the PEP does not attempt to impose any structure on the local version labels (aside from limiting the set of permitted characters and defining their ordering). This change is designed to ensure that an integrator provided version like ``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version specifier like ``pip>=1.5``. The plus is chosen primarily for readability of local version identifiers. It was chosen instead of the hyphen to prevent ``pkg_resources.parse_version`` from parsing it as a prerelease, which is important for enabling a successful migration to the new, more structured, versioning scheme. The plus was chosen instead of a tilde because of the significance of the tilde in Debian's version ordering algorithm. Providing explicit version normalization rules ---------------------------------------------- Historically, the de facto standard for parsing versions in Python has been the ``pkg_resources.parse_version`` command from the setuptools project. This does not attempt to reject *any* version and instead tries to make something meaningful, with varying levels of success, out of whatever it is given. It has a few simple rules but otherwise it more or less relies largely on string comparison. The normalization rules provided in this PEP exist primarily to either increase the compatability with ``pkg_resources.parse_version``, particularly in documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something more reasonable with versions that already exist on PyPI. All possible normalization rules were weighed against whether or not they were *likely* to cause any ambiguity (e.g. while someone might devise a scheme where ``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone actually doing that, much less on any scale that is noticeable, is fairly low). They were also weighed against how ``pkg_resources.parse_version`` treated a particular version string, especially with regards to how it was sorted. Finally each rule was weighed against the kinds of additional versions it allowed, how "ugly" those versions looked, how hard there were to parse (both mentally and mechanically) and how much additional compatibility it would bring. The breadth of possible normalizations were kept to things that could easily be implemented as part of the parsing of the version and not pre-parsing transformations applied to the versions. This was done to limit the side effects of each transformation as simple search and replace style transforms increase the likelihood of ambiguous or "junk" versions. For an extended discussion on the various types of normalizations that were considered, please see the proof of concept for PEP 440 within pip [4]_. References ========== The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] Reference Implementation of PEP 440 Versions and Specifiers https://github.com/pypa/packaging/pull/1 .. [2] Version compatibility analysis script: https://github.com/pypa/packaging/blob/master/tasks/check.py .. [3] Pessimistic version constraint http://docs.rubygems.org/read/chapter/16 .. [4] Proof of Concept: PEP 440 within pip https://github.com/pypa/pip/pull/1894 Appendix A ========== Metadata v2.0 guidelines versus setuptools:: $ invoke check.pep440 Total Version Compatibility: 245806/250521 (98.12%) Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%) Total Sorting Compatibility (Filtered): 47057/47114 (99.88%) Projects with No Compatible Versions: 498/47114 (1.06%) Projects with Differing Latest Version: 688/47114 (1.46%) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Aug 16, 2014, at 11:37 AM, Jason R. Coombs <jaraco@jaraco.com> wrote:
Thanks Donald for the extensive work on this. It all looks generally good.
One thing that stuck out as slightly surprising – the use of ‘_’ as a separator. I imagine most people consider the underscore to be yet another alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest that the PEP give some examples. As is, I didn’t see any examples that showed ‘_’ as a separator.
It’s not a preferred separator, it’s use is mostly because of various normalization schemes, particularly Wheel, which will convert a ``-`` to a `` _`` in the version. Right now pip considers them equal because it was causing some breakage with Wheels. This rationale should probably be given in the PEP though. The original bug report in pip for this was https://github.com/pypa/pip/issues/1150.
Related, I would suggest a consistent scheme for local version tags. Why not have local version tags have the same syntax as primary version numbers? That is, allow the same character set, pre and post versions, etc, just separated by a +. At the very least, I would expect local versions to allow underscores.
Originally local versions were just integers separated by dots. We wound up where we are so that downstream distributions can put more information inside of it. For instance Instead of Ubuntu taking a version like 1.0, and making it 1.0+1, they can do 1.0+ubuntu.1 or similar so that it’s obvious from the version number that this version is different than say Fedora, who might have also patched it and have 1.0+fedora.1. I’m not sure what the use case is for pre/post/etc versions in the local version. At the point you’re doing more than simple patching you’re probably making a full fledged fork which should have it’s own name and version numbers I think?
These are all just nitpicks, though. A lot of work has gone into this spec, and it’s good to see the community coalescing on an implementable standard. It’s surely acceptable in the current form.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 17 August 2014 02:11, Donald Stufft <donald@stufft.io> wrote:
On Aug 16, 2014, at 11:37 AM, Jason R. Coombs <jaraco@jaraco.com> wrote:
Thanks Donald for the extensive work on this. It all looks generally good.
One thing that stuck out as slightly surprising – the use of ‘_’ as a separator. I imagine most people consider the underscore to be yet another alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest that the PEP give some examples. As is, I didn’t see any examples that showed ‘_’ as a separator.
It’s not a preferred separator, it’s use is mostly because of various normalization schemes, particularly Wheel, which will convert a ``-`` to a `` _`` in the version. Right now pip considers them equal because it was causing some breakage with Wheels. This rationale should probably be given in the PEP though.
The original bug report in pip for this was https://github.com/pypa/pip/issues/1150.
Related, I would suggest a consistent scheme for local version tags. Why not have local version tags have the same syntax as primary version numbers? That is, allow the same character set, pre and post versions, etc, just separated by a +. At the very least, I would expect local versions to allow underscores.
Originally local versions were just integers separated by dots. We wound up where we are so that downstream distributions can put more information inside of it. For instance Instead of Ubuntu taking a version like 1.0, and making it 1.0+1, they can do 1.0+ubuntu.1 or similar so that it’s obvious from the version number that this version is different than say Fedora, who might have also patched it and have 1.0+fedora.1.
Right. In the Fedora (et al) case, the local version ordering scheme also matches the way yum orders the RPM release field, so it becomes possible to map the release field (which tracks patches and spec file changes) directly to the Python level "local version". One big advantage of the more relaxed rules around the local version identifier component is that DVCS hashes are allowed, you just want to include a serial number (e.g. number of commits since the last rebase) first to ensure sensible ordering.
I’m not sure what the use case is for pre/post/etc versions in the local version. At the point you’re doing more than simple patching you’re probably making a full fledged fork which should have it’s own name and version numbers I think?
Agreed. For use of the local version field to be appropriate, we should be looking at full API compatibility with the public version identifier. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17 August 2014 02:11, Donald Stufft <donald@stufft.io> wrote:
It’s not a preferred separator, it’s use is mostly because of various normalization schemes, particularly Wheel, which will convert a ``-`` to a `` _`` in the version. Right now pip considers them equal because it was causing some breakage with Wheels. This rationale should probably be given in the PEP though.
The original bug report in pip for this was https://github.com/pypa/pip/issues/1150.
Donald, I've added this to the list at https://bitbucket.org/pypa/pypi-metadata-formats/issue/49/address-pep-440-v6... (the comments from Marcus and Paul were already noted). If you could prepare a patch to clarify those, we can get it merged and published, and I'll accept the PEP. (Since they're just clarifications rather than real changes, I'm actually prepared to accept the PEP now, but I don't believe there's any pressing need for urgency on that front - may as well get the clarifications in first) Thanks for all your work in finally bringing this to fruition, as well as to all those who contributed to the long migration from the original CPAN style versioning to an approach more in line with the Zen of Python :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Donald, I've added this to the list at
https://bitbucket.org/pypa/pypi-metadata-formats/issue/49/address-pep-440-v6... (the comments from Marcus and Paul were already noted).
one other thing Donald and I discussed was being clear what was compatible with pkg_resources. my understanding is that *only* local versions are incompatible, in that they don't sort correctly relative to public versions. I'd like to see mention of that somewhere.
participants (6)
-
Donald Stufft -
Greg Ewing -
Jason R. Coombs -
Marcus Smith -
Nick Coghlan -
Paul Moore