From cournape at gmail.com  Fri Aug  1 08:45:37 2014
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 1 Aug 2014 15:45:37 +0900
Subject: [Distutils] build a wheel with waf instead of setuptools
In-Reply-To: <CALGmxEJhEN+1nwYpPMG6utV2TGQOOgqLZkWyOAUR8Qto8MZoEg@mail.gmail.com>
References: <CAG8k2+6cOx2857-WA2toTdpXtx0QEAVXjpG+A8ZdRwvFTgJUGA@mail.gmail.com>
 <CAG8k2+4J1ei7uj5VXY4p=4YBr7V4-6PFN84WMi6pz5VFxRJ9ng@mail.gmail.com>
 <etPan.53d267b0.5bd062c2.4c3e@Thor.local>
 <CAG8k2+5nFr8L9gy_5XnoPvA=9ed-TCM__S2A_N=e6O3Facr9-Q@mail.gmail.com>
 <CALGmxEJhEN+1nwYpPMG6utV2TGQOOgqLZkWyOAUR8Qto8MZoEg@mail.gmail.com>
Message-ID: <CAGY4rcUS5ojSjXcXBS=gdZ4WF3nUVY7Yt90RPqVjTEUTMOgqBw@mail.gmail.com>

That's certainly possible, bento does exactly this. You would write a
setup.py that would like like:

import setuptools
from bento.distutils.monkey_patch import monkey_patch
monkey_patch()
setuptools.setup()

then you can do python setup.py install/build/sdist/etc... without actually
using distutils.

David


On Wed, Jul 30, 2014 at 8:46 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Jul 25, 2014 at 7:21 AM, Daniel Holth <dholth at gmail.com> wrote:
>
>> > This kind of thing will require us to implement a flag that tells pip
>> > "setup.py cannot install; go through wheel" which is somewhere in the
>> > plans..
>>
>
> couldn't you write a file called "setup.py", with the core API (i.e
> setup.py build | install), but that called waf instead of distutils to do
> the actual work?
>
> or does pip doe more than simply call the setup.py script?
>
> -Chris
>
>
>
>
>
>>  > I don?t think there is any plans to tell pip *not* to use a setup.py
>> and to
>> > use a Wheel instead. Rather I think the plans are to enable pluggable
>> > builders so that a sdist 2.0 package doesn?t rely on setup.py and could
>> use
>> > a waf builder (for instance) plugin.
>>
>> Just a flag that tells pip it can't use the "install" command and has
>> to do package -> install package on an sdist.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140801/a90c1cc5/attachment.html>

From holger at merlinux.eu  Wed Aug  6 23:41:22 2014
From: holger at merlinux.eu (holger krekel)
Date: Wed, 6 Aug 2014 21:41:22 +0000
Subject: [Distutils] new devpi-2.0.1/2 releases: fixes, new anonymous upload
Message-ID: <20140806214122.GA1894@merlinux.eu>


The new devpi releases (devpi-server-2.0.2, devpi-web-2.0.1, devpi-client-2.0.1)
contain several fixes for the private github-style pypi caching server, 
see the changelog below for details.  Find all the documentation at

    http://doc.devpi.net/latest

with tutorials and quickstart documents for particular scenarios.

Note that no export/import procedure is needed if you are already running
a 2.X devpi-server.  If upgrading from a 1.2 devpi-server install, you need
to upgrade using these instructions:

http://doc.devpi.net/latest/quickstart-server.html#versioning-exporting-and-importing-server-state

many thanks to Florian Schulze who again helped a lot with this release.

have fun,

holger krekel

devpi-2.0.1 (metapackage)
---------------------------------

devpi-server-2.0.2:
   
- fix issue120: link to "upgrade" section from main index page.

- preserve http reason string for setup.py submit through replica proxying

- proper error message when "devpi push X" uses an X that comes from 
  a base index or is not existent

- fix issue121: depend on py-1.4.23 to fix python3.4 compatibility
  for a venusian/py34/py interaction import oddity.

- fix issue126: handle deletion of pypi project cache entries correctly 
  (i.e. ones that are triggered by "refresh" on simple page).

- Add special handling of ":ANONYMOUS:" user in acl_upload to allow anonymous
  submit.

- fix nginx template so that when used in a replica setting the master
  always answers HEAD requests without nginx short-cirtcuiting it.

- increase internal cache size to improve performance when many indexes
  and projects are served.

devpi-web-2.0.1:

- fix issue125: javascript for embedded doc view didn't work correctly.

- fix issue118: rendering of description with unicode.

devpi-client-2.0.1:
 
- fix a test to expect a 403 instead of a 401 from devpi server from
  unauthorized access 

- fix error message on API version client/server mismatch

- fix issue124: package name url matching for the "devpi test" command



From holger at merlinux.eu  Fri Aug  8 15:56:31 2014
From: holger at merlinux.eu (holger krekel)
Date: Fri, 8 Aug 2014 13:56:31 +0000
Subject: [Distutils] devpi-server-2.0.3: fix for plain "setup.py register"
In-Reply-To: <20140806214122.GA1894@merlinux.eu>
References: <20140806214122.GA1894@merlinux.eu>
Message-ID: <20140808135631.GZ1894@merlinux.eu>


Florian Schulze and me just released devpi-server-2.0.3 fixing a regression
from the 1.X series preventing a plain "python setup.py register -r NAME"
to succeed.  

have fun,

holger


On Wed, Aug 06, 2014 at 21:41 +0000, holger krekel wrote:
> The new devpi releases (devpi-server-2.0.2, devpi-web-2.0.1, devpi-client-2.0.1)
> contain several fixes for the private github-style pypi caching server, 
> see the changelog below for details.  Find all the documentation at
> 
>     http://doc.devpi.net/latest
> 
> with tutorials and quickstart documents for particular scenarios.
> 
> Note that no export/import procedure is needed if you are already running
> a 2.X devpi-server.  If upgrading from a 1.2 devpi-server install, you need
> to upgrade using these instructions:
> 
> http://doc.devpi.net/latest/quickstart-server.html#versioning-exporting-and-importing-server-state
> 
> many thanks to Florian Schulze who again helped a lot with this release.
> 
> have fun,
> 
> holger krekel
> 
> devpi-2.0.1 (metapackage)
> ---------------------------------
> 
> devpi-server-2.0.2:
>    
> - fix issue120: link to "upgrade" section from main index page.
> 
> - preserve http reason string for setup.py submit through replica proxying
> 
> - proper error message when "devpi push X" uses an X that comes from 
>   a base index or is not existent
> 
> - fix issue121: depend on py-1.4.23 to fix python3.4 compatibility
>   for a venusian/py34/py interaction import oddity.
> 
> - fix issue126: handle deletion of pypi project cache entries correctly 
>   (i.e. ones that are triggered by "refresh" on simple page).
> 
> - Add special handling of ":ANONYMOUS:" user in acl_upload to allow anonymous
>   submit.
> 
> - fix nginx template so that when used in a replica setting the master
>   always answers HEAD requests without nginx short-cirtcuiting it.
> 
> - increase internal cache size to improve performance when many indexes
>   and projects are served.
> 
> devpi-web-2.0.1:
> 
> - fix issue125: javascript for embedded doc view didn't work correctly.
> 
> - fix issue118: rendering of description with unicode.
> 
> devpi-client-2.0.1:
>  
> - fix a test to expect a 403 instead of a 401 from devpi server from
>   unauthorized access 
> 
> - fix error message on API version client/server mismatch
> 
> - fix issue124: package name url matching for the "devpi test" command
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 

From donald at stufft.io  Fri Aug  8 23:53:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 8 Aug 2014 17:53:00 -0400
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
	Dependency Specification Version
Message-ID: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>

I?m happy to announce the 6th, and hopefully final, draft of PEP 440. The
updates to this PEP take into account feedback from several large projects and
users of various systems (such as Linux package tools) where this would be
expected to interface with. It also takes into account experience gained from
attempting to implement this PEP fully as a proof of concept within pip.

Significant updates to the PEP include:

* Switching the Epoch identifier from : to ! as : is not valid in a directory
  name on Windows.
* Local version identifiers use + as a seperator in order to reduce ambigiuty
  with existing versions on PyPI.
* Allow alpha numerics for local versions.
* Define a sorting algorithm for local versions.
* Moved the source label to PEP 426
* Normalization rules for parsing more versions following along the idea of
  Postel's Law.
* Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also should be
  used for Metadata 1.0 and 1.1.
* Declare how invalid versions should be handled.
* Add the escape hatch "Arbitrary equality" operator === to allow depending on
  versions which cannot be parsed by PEP 440.
* Make specifier syntax match what setuptools uses (foo==1.0 instead of foo (==1.0))
* Remove the default specifier.
* Use @ for direct references instead of "from".
* Create a reference implementation and Link To it.
* Lots more minor changes.

The outcome of these changes is that we were able to raise compatability with
all the versions registered with PyPI up to 98.12% and we sort 99.88% of
projects registered with PyPI the same as pkg_resources does when filtering
invalid versions from the list of versions. The fallout is that 498 projects,
or 1.06%, are no longer installable without using the ``===`` operator and 190
projects, or 0.4%, of projects have a different result for what the "latest"
version is.

Of the 498 projects a number of them are nonsensical versions like ``.`` or
the repr of lazy objects and of the 190 projects projects it's about evenly
split between projects where pkg_resources supported something we didn't and
where PEP 440 just simply does a better job at parsing and sorting versions.

You can see the pip proof of concept and a large discussion about normalization
at https://github.com/pypa/pip/pull/1894 and the reference implementation can
be found at https://github.com/pypa/packaging/pull/1. The Proof of Concept does
not allow using the new specifiers inside of an install_requires in a source
distribution because setuptools itself does not support it, but it does support
them on the command line and in requirements.txt files.

The online view of the PEP can be found at
https://www.python.org/dev/peps/pep-0440/ and the changes since the last posting
can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and
http://hg.python.org/peps/rev/257822378672.

Without further ado, the PEP itself:

Abstract
========

This PEP describes a scheme for identifying versions of Python software
distributions, and declaring dependencies on particular versions.

This document addresses several limitations of the previous attempt at a
standardized approach to versioning, as described in PEP 345 and PEP 386.


Definitions
===========

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.

The following terms are to be interpreted as described in PEP 426:

* "Distributions"
* "Releases"
* "Build tools"
* "Index servers"
* "Publication tools"
* "Installation tools"
* "Automated tools"
* "Projects"


Version scheme
==============

Distributions are identified by a public version identifier which
supports all defined version comparison operations

The version scheme is used both to describe the distribution version
provided by a particular distribution archive, as well as to place
constraints on the version of dependencies needed in order to build or
run the software.


Public version identifiers
--------------------------

Public version identifiers MUST comply with the following scheme::

    [N!]N(.N)*[{a|b|c}N][.postN][.devN]

Public version identifiers MUST NOT include leading or trailing whitespace.

Public version identifiers MUST be unique within a given distribution.

Installation tools SHOULD ignore any public versions which do not comply with
this scheme. Installation tools MAY warn the user when non-compliant
or ambiguous versions are detected.

Public version identifiers are separated into up to five segments:

* Epoch segment: ``N!``
* Release segment: ``N(.N)*``
* Pre-release segment: ``{a|b|c}N``
* Post-release segment: ``.postN``
* Development release segment: ``.devN``

Any given release will be a "final release", "pre-release", "post-release" or
"developmental release" as defined in the following sections.

All numeric components MUST be non-negative integers.

All numeric components MUST be interpreted and ordered according to their
numeric value, not as text strings.

All numeric components MAY be zero. Except as described below for the
release segment, a numeric component of zero has no special significance
aside from always being the lowest possible value in the version ordering.

.. note::

   Some hard to read version identifiers are permitted by this scheme in
   order to better accommodate the wide range of versioning practices
   across existing public and private Python projects.

   Accordingly, some of the versioning practices which are technically
   permitted by the PEP are strongly discouraged for new projects. Where
   this is the case, the relevant details are noted in the following
   sections.


Local version identifiers
-------------------------

Local version identifiers MUST comply with the following scheme::

    <public version identifier>[+<local version label>]

They consist of a normal public version identifier (as defined in the
previous section), along with an arbitrary "local version label", separated
from the public version identifier by a plus. Local version labels have
no specific semantics assigned, but some syntactic restrictions are imposed.

Local version identifiers are used to denote fully API (and, if applicable,
ABI) compatible patched versions of upstream projects. For example, these
may be created by application developers and system integrators by applying
specific backported bug fixes when upgrading to a new upstream release would
be too disruptive to the application or other integrated system (such as a
Linux distribution).

The inclusion of the local version label makes it possible to differentiate
upstream releases from potentially altered rebuilds by downstream
integrators. The use of a local version identifier does not affect the kind
of a release but, when applied to a source distribution, does indicate that
it may not contain the exact same code as the corresponding upstream release.

To ensure local version identifiers can be readily incorporated as part of
filenames and URLs, and to avoid formatting inconsistencies in hexadecimal
hash representations, local version labels MUST be limited to the following
set of permitted characters:

* ASCII letters (``[a-zA-Z]``)
* ASCII digits (``[0-9]``)
* periods (``.``)

Local version labels MUST start and end with an ASCII letter or digit.

Comparison and ordering of local versions considers each segment of the local
version (divided by a ``.``) separately. If a segment consists entirely of
ASCII digits then that section should be considered an integer for comparison
purposes and if a segment contains any ASCII letters than that segment is
compared lexicographically with case insensitivity. When comparing a numeric
and lexicographic segment, the numeric section always compares as greater than
the lexicographic segment. Additionally a local version with a great number of
segments will always compare as greater than a local version with fewer
segments, as long as the shorter local version's segments match the beginning
of the longer local version's segments exactly.

Local version identifiers may be used in most locations where a public
version identifier is expected, with the exception of any version specifiers
that explicitly rely on being able to unambiguously order candidate versions.

Public index servers SHOULD NOT allow the use of local version identifiers
for uploaded distributions.

Source distributions using a local version identifier SHOULD provide the
``python.integrator`` extension metadata (as defined in :pep:`459`).


Final releases
--------------

A version identifier that consists solely of a release segment and optionally
an epoch identifier is termed a "final release".

The release segment consists of one or more non-negative integer
values, separated by dots::

    N[.N]+

Final releases within a project MUST be numbered in a consistently
increasing fashion, otherwise automated tools will not be able to upgrade
them correctly.

Comparison and ordering of release segments considers the numeric value
of each component of the release segment in turn. When comparing release
segments with different numbers of components, the shorter segment is
padded out with additional zeros as necessary.

While any number of additional components after the first are permitted
under this scheme, the most common variants are to use two components
("major.minor") or three components ("major.minor.micro").

For example::

    0.9
    0.9.1
    0.9.2
    ...
    0.9.10
    0.9.11
    1.0
    1.0.1
    1.1
    2.0
    2.0.1
    ...

A release series is any set of final release numbers that start with a
common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all
part of the ``3.3`` release series.

.. note::

   ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as
   the release segment comparison rules implicit expand the two component
   form to ``X.Y.0`` when comparing it to any release segment that includes
   three components.

Date based release segments are also permitted. An example of a date based
release scheme using the year and month of the release::

    2012.04
    2012.07
    2012.10
    2013.01
    2013.06
    ...


Pre-releases
------------

Some projects use an "alpha, beta, release candidate" pre-release cycle to
support testing by their users prior to a final release.

If used as part of a project's development cycle, these pre-releases are
indicated by including a pre-release segment in the version identifier::

    X.YaN  # Alpha release
    X.YbN  # Beta release
    X.YcN  # Candidate release
    X.Y    # Final release

A version identifier that consists solely of a release segment and a
pre-release segment is termed a "pre-release".

The pre-release segment consists of an alphabetical identifier for the
pre-release phase, along with a non-negative integer value. Pre-releases for
a given release are ordered first by phase (alpha, beta, release candidate)
and then by the numerical component within that phase.

Installation tools MAY accept both ``c`` and ``rc`` releases for a common
release segment in order to handle some existing legacy distributions.

Installation tools SHOULD interpret ``rc`` versions as being equivalent to
``c`` versions (that is, ``rc1`` indicates the same version as ``c1``).

Build tools, publication tools and index servers SHOULD disallow the creation
of both ``c`` and ``rc`` releases for a common release segment.


Post-releases
-------------

Some projects use post-releases to address minor errors in a final release
that do not affect the distributed software (for example, correcting an error
in the release notes).

If used as part of a project's development cycle, these post-releases are
indicated by including a post-release segment in the version identifier::

    X.Y.postN    # Post-release

A version identifier that includes a post-release segment without a
developmental release segment is termed a "post-release".

The post-release segment consists of the string ``.post``, followed by a
non-negative integer value. Post-releases are ordered by their
numerical component, immediately following the corresponding release,
and ahead of any subsequent release.

.. note::

   The use of post-releases to publish maintenance releases containing
   actual bug fixes is strongly discouraged. In general, it is better
   to use a longer release number and increment the final component
   for each maintenance release.

Post-releases are also permitted for pre-releases::

    X.YaN.postM  # Post-release of an alpha release
    X.YbN.postM  # Post-release of a beta release
    X.YcN.postM  # Post-release of a release candidate

.. note::

   Creating post-releases of pre-releases is strongly discouraged, as
   it makes the version identifier difficult to parse for human readers.
   In general, it is substantially clearer to simply create a new
   pre-release by incrementing the numeric component.


Developmental releases
----------------------

Some projects make regular developmental releases, and system packagers
(especially for Linux distributions) may wish to create early releases
directly from source control which do not conflict with later project
releases.

If used as part of a project's development cycle, these developmental
releases are indicated by including a developmental release segment in the
version identifier::

    X.Y.devN    # Developmental release

A version identifier that includes a developmental release segment is
termed a "developmental release".

The developmental release segment consists of the string ``.dev``,
followed by a non-negative integer value. Developmental releases are ordered
by their numerical component, immediately before the corresponding release
(and before any pre-releases with the same release segment), and following
any previous release (including any post-releases).

Developmental releases are also permitted for pre-releases and
post-releases::

    X.YaN.devM      # Developmental release of an alpha release
    X.YbN.devM      # Developmental release of a beta release
    X.YcN.devM      # Developmental release of a release candidate
    X.Y.postN.devM  # Developmental release of a post-release

.. note::

   Creating developmental releases of pre-releases is strongly
   discouraged, as it makes the version identifier difficult to parse for
   human readers. In general, it is substantially clearer to simply create
   additional pre-releases by incrementing the numeric component.

   Developmental releases of post-releases are also strongly discouraged,
   but they may be appropriate for projects which use the post-release
   notation for full maintenance releases which may include code changes.


Version epochs
--------------

If included in a version identifier, the epoch appears before all other
components, separated from the release segment by an exclamation mark::

    E!X.Y  # Version identifier with epoch

If no explicit epoch is given, the implicit epoch is ``0``.

Most version identifiers will not include an epoch, as an explicit epoch is
only needed if a project *changes* the way it handles version numbering in
a way that means the normal version ordering rules will give the wrong
answer. For example, if a project is using date based versions like
``2014.04`` and would like to switch to semantic versions like ``1.0``, then
the new releases would be identified as *older* than the date based releases
when using the normal sorting scheme::

    1.0
    1.1
    2.0
    2013.10
    2014.04

However, by specifying an explicit epoch, the sort order can be changed
appropriately, as all versions from a later epoch are sorted after versions
from an earlier epoch::

    2013.10
    2014.04
    1!1.0
    1!1.1
    1!2.0

Normalization
-------------

In order to maintain better compatibility with existing versions there are a
number of "alternative" syntaxes that MUST be taken into account when parsing
versions. These syntaxes MUST be considered when parsing a version, however
they should be "normalized" to the standard syntax defined above.


Case sensitivity
~~~~~~~~~~~~~~~~

All ascii letters should be interpreted case insensitively within a version and
the normal form is lowercase. This allows versions such as ``1.1RC1`` which
would be normalized to ``1.1c1``.


Integer Normalization
~~~~~~~~~~~~~~~~~~~~~

All integers are interpreted via the ``int()`` built in and normalize to the
string form of the output. This means that an integer version of ``00`` would
normalize to ``0`` while ``09000`` would normalize to ``9000``. This does not
hold true for integers inside of an alphanumeric segment of a local version
such as ``1.0+foo0100`` which is already in its normalized form.


Pre-release separators
~~~~~~~~~~~~~~~~~~~~~~

Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the
release segment and the pre-release segment. The normal form for this is
without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1``
which would be normalized to ``1.1a1``. It should also allow a seperator to
be used between the pre-release signifier and the numeral. This allows versions
such as ``1.0a.1`` which would be normalized to ``1.0a1``.


Pre-release spelling
~~~~~~~~~~~~~~~~~~~~

Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``,
``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively.
This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which
normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional
spelling should be considered equivalent to their normal forms.


Implicit pre-release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Pre releases allow omitting the numeral in which case it is implicitly assumed
to be ``0``. The normal form for this is to include the ``0`` explicitly. This
allows versions such as ``1.2a`` which is normalized to ``1.2a0``.


Post release separators
~~~~~~~~~~~~~~~~~~~~~~~

Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the
separator all together. The normal form of this is with the ``.`` separator.
This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to
``1.2.post2``. Like the pre-release seperator this also allows an optional
separator between the post release signifier and the numeral. This allows
versions like ``1.2.post-2`` which would normalize to ``1.2.post2``.


Post release spelling
~~~~~~~~~~~~~~~~~~~~~

Post-releases allow the additional spellings of ``rev`` and ``r``. This allows
versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the
pre-releases the additional spellings should be considered equivalent to their
normal forms.


Implicit post release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Post releases allow omiting the numeral in which case it is implicitly assumed
to be ``0``. The normal form for this is to include the ``0`` explicitly. This
allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``.


Implicit post releases
~~~~~~~~~~~~~~~~~~~~~~

Post releases allow omitting the ``post`` signifier all together. When using
this form the separator MUST be ``-`` and no other form is allowed. This allows
versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular
normalization MUST NOT be used in conjunction with the implicit post release
number rule. In other words ``1.0-`` is *not* a valid version and it does *not*
normalize to ``1.0.post0``.


Development release separators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Development releases allow a ``.``, ``-``, or a ``_`` separator as well as
omitting the separator all together. The normal form of this is with the ``.``
separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which
normalize to ``1.2.dev2``.


Implicit development release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Development releases allow omiting the numeral in which case it is implicitly
assumed to be ``0``. The normal form for this is to include the ``0``
explicitly. This allows versions such as ``1.2.dev`` which is normalized to
``1.2.dev0``.


Local version segments
~~~~~~~~~~~~~~~~~~~~~~

With a local version, in addition to the use of ``.`` as a separator of
segments, the use of ``-`` and ``_`` is also acceptable. The normal form is
using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be
normalized to ``1.0+ubuntu.1``.


Preceding v character
~~~~~~~~~~~~~~~~~~~~~

In order to support the common version notation of ``v1.0`` versions may be
preceded by a single literal ``v`` character. This character MUST be ignored
for all purposes and should be omitted from all normalized forms of the
version. The same version with and without the ``v`` is considered equivalent.


Leading and Trailing Whitespace
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Leading and trailing whitespace must be silently ignored and removed from all
normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``,
``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly,
such as a version like ``1.0\n`` which normalizes to ``1.0``.


Examples of compliant version schemes
-------------------------------------

The standard version scheme is designed to encompass a wide range of
identification practices across public and private Python projects. In
practice, a single project attempting to use the full flexibility offered
by the scheme would create a situation where human users had difficulty
figuring out the relative order of versions, even though the rules above
ensure all compliant tools will order them consistently.

The following examples illustrate a small selection of the different
approaches projects may choose to identify their releases, while still
ensuring that the "latest release" and the "latest stable release" can
be easily determined, both by human users and automated tools.

Simple "major.minor" versioning::

    0.1
    0.2
    0.3
    1.0
    1.1
    ...

Simple "major.minor.micro" versioning::

    1.1.0
    1.1.1
    1.1.2
    1.2.0
    ...

"major.minor" versioning with alpha, beta and candidate
pre-releases::

    0.9
    1.0a1
    1.0a2
    1.0b1
    1.0c1
    1.0
    1.1a1
    ...

"major.minor" versioning with developmental releases, release candidates
and post-releases for minor corrections::

    0.9
    1.0.dev1
    1.0.dev2
    1.0.dev3
    1.0.dev4
    1.0rc1
    1.0rc2
    1.0
    1.0.post1
    1.1.dev1
    ...

Date based releases, using an incrementing serial within each year, skipping
zero::

    2012.1
    2012.2
    2012.3
    ...
    2012.15
    2013.1
    2013.2
    ...


Summary of permitted suffixes and relative ordering
---------------------------------------------------

.. note::

   This section is intended primarily for authors of tools that
   automatically process distribution metadata, rather than developers
   of Python distributions deciding on a versioning scheme.

The epoch segment of version identifiers MUST be sorted according to the
numeric value of the given epoch. If no epoch segment is present, the
implicit numeric value is ``0``.

The release segment of version identifiers MUST be sorted in
the same order as Python's tuple sorting when the release segment is
parsed as follows::

    tuple(map(int, release_segment.split(".")))

All release segments involved in the comparison MUST be converted to a
consistent length by padding shorter segments with zeros as needed.

Within a numeric release (``1.0``, ``2.7.3``), the following suffixes
are permitted and MUST be ordered as shown::

   .devN, aN, bN, cN/rcN, <no suffix>, .postN

Note that `rc` is considered to be semantically equivalent to `c` and must be
sorted as if it were `c`. Tools MAY reject the case of having the same ``N``
for both a ``rc`` and a ``c`` in the same release segment as ambiguous and
remain in compliance with the PEP.

Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate
(``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be
ordered as shown::

   .devN, <no suffix>, .postN

Within a post-release (``1.0.post1``), the following suffixes are permitted
and MUST be ordered as shown::

    .devN, <no suffix>

Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even
when used immediately following a numeric version (e.g. ``1.0.dev456``,
``1.0.post1``).

Within a pre-release, post-release or development release segment with a
shared prefix, ordering MUST be by the value of the numeric component.

The following example covers many of the possible combinations::

    1.0.dev456
    1.0a1
    1.0a2.dev456
    1.0a12.dev456
    1.0a12
    1.0b1.dev456
    1.0b2
    1.0b2.post345.dev456
    1.0b2.post345
    1.0c1.dev456
    1.0c1
    1.0
    1.0+abc.5
    1.0+abc.7
    1.0+5
    1.0.post456.dev34
    1.0.post456
    1.1.dev1


Version ordering across different metadata versions
---------------------------------------------------

Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard
version identification or ordering scheme. However metadata v1.2 (PEP 345)
does specify a scheme which is defined in PEP 386.

Due to the nature of the simple installer API it is not possible for an
installer to be aware of which metadata version a particular distribution was
using. Additionally installers required the ability to create a reasonably
prioritized list that includes all, or as many as possible, versions of
a project to determine which versions it should install. These requirements
necessitate a standardization across one parsing mechanism to be used for all
versions of a project.

Due to the above, this PEP MUST be used for all versions of metadata and
supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions
which cannot be parsed by the rules in this PEP, but MAY fall back to
implementation defined version parsing and ordering schemes if no versions
complying with this PEP are available.

Distribution users may wish to explicitly remove non-compliant versions from
any private package indexes they control.


Compatibility with other version schemes
----------------------------------------

Some projects may choose to use a version scheme which requires
translation in order to comply with the public version scheme defined in
this PEP. In such cases, the project specific version can be stored in the
metadata while the translated public version is published in the version field.

This allows automated distribution tools to provide consistently correct
ordering of published releases, while still allowing developers to use
the internal versioning scheme they prefer for their projects.


Semantic versioning
~~~~~~~~~~~~~~~~~~~

`Semantic versioning`_ is a popular version identification scheme that is
more prescriptive than this PEP regarding the significance of different
elements of a release number. Even if a project chooses not to abide by
the details of semantic versioning, the scheme is worth understanding as
it covers many of the issues that can arise when depending on other
distributions, and when publishing a distribution that others rely on.

The "Major.Minor.Patch" (described in this PEP as "major.minor.micro")
aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification)
are fully compatible with the version scheme defined in this PEP, and abiding
by these aspects is encouraged.

Semantic versions containing a hyphen (pre-releases - clause 10) or a
plus sign (builds - clause 11) are *not* compatible with this PEP
and are not permitted in the public version field.

One possible mechanism to translate such semantic versioning based source
labels to compatible public versions is to use the ``.devN`` suffix to
specify the appropriate version order.

Specific build information may also be included in local version labels.

.. _Semantic versioning: http://semver.org/


DVCS based version labels
~~~~~~~~~~~~~~~~~~~~~~~~~

Many build tools integrate with distributed version control systems like
Git and Mercurial in order to add an identifying hash to the version
identifier. As hashes cannot be ordered reliably such versions are not
permitted in the public version field.

As with semantic versioning, the public ``.devN`` suffix may be used to
uniquely identify such releases for publication, while the original DVCS based
label can be stored in the project metadata.

Identifying hash information may also be included in local version labels.


Olson database versioning
~~~~~~~~~~~~~~~~~~~~~~~~~

The ``pytz`` project inherits its versioning scheme from the corresponding
Olson timezone database versioning scheme: the year followed by a lowercase
character indicating the version of the database within that year.

This can be translated to a compliant public version identifier as
``<year>.<serial>``, where the serial starts at zero or one (for the
'<year>a' release) and is incremented with each subsequent database
update within the year.

As with other translated version identifiers, the corresponding Olson
database version could be recorded in the project metadata.


Version specifiers
==================

A version specifier consists of a series of version clauses, separated by
commas. For example::

   ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0

The comparison operator determines the kind of version clause:

* ``~=``: `Compatible release`_ clause
* ``==``: `Version matching`_ clause
* ``!=``: `Version exclusion`_ clause
* ``<=``, ``>=``: `Inclusive ordered comparison`_ clause
* ``<``, ``>``: `Exclusive ordered comparison`_ clause
* ``===``: `Arbitrary equality`_ clause.

The comma (",") is equivalent to a logical **and** operator: a candidate
version must match all given version clauses in order to match the
specifier as a whole.

Whitespace between a conditional operator and the following version
identifier is optional, as is the whitespace around the commas.

When multiple candidate versions match a version specifier, the preferred
version SHOULD be the latest version as determined by the consistent
ordering defined by the standard `Version scheme`_. Whether or not
pre-releases are considered as candidate versions SHOULD be handled as
described in `Handling of pre-releases`_.

Except where specifically noted below, local version identifiers MUST NOT be
permitted in version specifiers, and local version labels MUST be ignored
entirely when checking if candidate versions match a given version
specifier.


Compatible release
------------------

A compatible release clause consists of either a version identifier without
any comparison operator or else the compatible release operator ``~=``
and a version identifier. It matches any candidate version that is expected
to be compatible with the specified version.

The specified version identifier must be in the standard format described in
`Version scheme`_. Local version identifiers are NOT permitted in this
version specifier.

For a given release identifier ``V.N``, the compatible release clause is
approximately equivalent to the pair of comparison clauses::

    >= V.N, == V.*

This operator MUST NOT be used with a single segment version number such as
``~=1``.

For example, the following groups of version clauses are equivalent::

    2.2
    ~= 2.2
    >= 2.2, == 2.*

    1.4.5
    ~= 1.4.5
    >= 1.4.5, == 1.4.*

If a pre-release, post-release or developmental release is named in a
compatible release clause as ``V.N.suffix``, then the suffix is ignored
when determining the required prefix match::

    2.2.post3
    ~= 2.2.post3
    >= 2.2.post3, == 2.*

    1.4.5a4
    ~= 1.4.5a4
    >= 1.4.5a4, == 1.4.*

The padding rules for release segment comparisons means that the assumed
degree of forward compatibility in a compatible release clause can be
controlled by appending additional zeros to the version specifier::

    2.2.0
    ~= 2.2.0
    >= 2.2.0, == 2.2.*

    1.4.5.0
    ~= 1.4.5.0
    >= 1.4.5.0, == 1.4.5.*


Version matching
----------------

A version matching clause includes the version matching operator ``==``
and a version identifier.

The specified version identifier must be in the standard format described in
`Version scheme`_, but a trailing ``.*`` is permitted on public version
identifiers as described below.

By default, the version matching operator is based on a strict equality
comparison: the specified version must be exactly the same as the requested
version. The *only* substitution performed is the zero padding of the
release segment to ensure the release segments are compared with the same
length.

Whether or not strict version matching is appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when strict version matches are used
inappropriately.

Prefix matching may be requested instead of strict comparison, by appending
a trailing ``.*`` to the version identifier in the version matching clause.
This means that additional trailing segments will be ignored when
determining whether or not a version identifier matches the clause. If the
specified version includes only a release segment, than trailing components
(or the lack thereof) in the release segment are also ignored.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::

    == 1.1        # Not equal, so 1.1.post1 does not match clause
    == 1.1.post1  # Equal, so 1.1.post1 matches clause
    == 1.1.*      # Same prefix, so 1.1.post1 matches clause

For purposes of prefix matching, the pre-release segment is considered to
have an implied preceding ``.``, so given the version ``1.1a1``, the
following clauses would match or not as shown::

    == 1.1        # Not equal, so 1.1a1 does not match clause
    == 1.1a1      # Equal, so 1.1a1 matches clause
    == 1.1.*      # Same prefix, so 1.1a1 matches clause

An exact match is also considered a prefix match (this interpreation is
implied by the usual zero padding rules for the release segment of version
identifiers). Given the version ``1.1``, the following clauses would
match or not as shown::

    == 1.1        # Equal, so 1.1 matches clause
    == 1.1.0      # Zero padding expands 1.1 to 1.1.0, so it matches clause
    == 1.1.dev1   # Not equal (dev-release), so 1.1 does not match clause
    == 1.1a1      # Not equal (pre-release), so 1.1 does not match clause
    == 1.1.post1  # Not equal (post-release), so 1.1 does not match clause
    == 1.1.*      # Same prefix, so 1.1 matches clause

It is invalid to have a prefix match containing a development or local release
such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development release
segment is always the final segment in the public version, and the local version
is ignored for comparison purposes, so using either in a prefix match wouldn't
make any sense.

The use of ``==`` (without at least the wildcard suffix) when defining
dependencies for published distributions is strongly discouraged as it
greatly complicates the deployment of security fixes. The strict version
comparison operator is intended primarily for use when defining
dependencies for repeatable *deployments of applications* while using
a shared distribution index.

If the specified version identifier is a public version identifier (no
local version label), then the local version label of any candidate versions
MUST be ignored when matching versions.

If the specified version identifier is a local version identifier, then the
local version labels of candidate versions MUST be considered when matching
versions, with the public version identifier being matched as described
above, and the local version label being checked for equivalence using a
strict string equality comparison.


Version exclusion
-----------------

A version exclusion clause includes the version exclusion operator ``!=``
and a version identifier.

The allowed version identifiers and comparison semantics are the same as
those of the `Version matching`_ operator, except that the sense of any
match is inverted.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::

    != 1.1        # Not equal, so 1.1.post1 matches clause
    != 1.1.post1  # Equal, so 1.1.post1 does not match clause
    != 1.1.*      # Same prefix, so 1.1.post1 does not match clause


Inclusive ordered comparison
----------------------------

An inclusive ordered comparison clause includes a comparison operator and a
version identifier, and will match any version where the comparison is correct
based on the relative position of the candidate version and the specified
version given the consistent ordering defined by the standard
`Version scheme`_.

The inclusive ordered comparison operators are ``<=`` and ``>=``.

As with version matching, the release segment is zero padded as necessary to
ensure the release segments are compared with the same length.

Local version identifiers are NOT permitted in this version specifier.


Exclusive ordered comparison
----------------------------

Exclusive ordered comparisons are similar to inclusive ordered comparisons,
except that the comparison operators are ``<`` and ``>`` and the clause
MUST be effectively interpreted as implying the prefix based version
exclusion clause ``!= V.*``.

The exclusive ordered comparison ``> V`` MUST NOT match a post-release
or maintenance release of the given version. Maintenance releases can be
permitted by using the clause ``> V.0``, while both post releases and
maintenance releases can be permitted by using the inclusive ordered
comparison ``>= V.post1``.

The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of
the given version, even if acceptance of pre-releases is enabled as
described in the section below.

Local version identifiers are NOT permitted in this version specifier.


Arbitrary equality
------------------

Arbitrary equality comparisons are simple string equality operations which do
not take into account any of the semantic information such as zero padding or
local versions. This operator also does not support prefix matching as the
``==`` operator does.

The primary use case for arbitrary equality is to allow for specifying a
version which cannot otherwise be represented by this PEP. This operator is
special and acts as an escape hatch to allow someone using a tool which
implements this PEP to still install a legacy version which is otherwise
incompatible with this PEP.

An example would be ``===foobar`` which would match a version of ``foobar``.

This operator may also be used to explicitly require an unpatched version
of a project such as ``===1.0`` which would not match for a version
``1.0+downstream1``.

Use of this operator is heavily discouraged and tooling MAY display a warning
when it is used.


Handling of pre-releases
------------------------

Pre-releases of any kind, including developmental releases, are implicitly
excluded from all version specifiers, *unless* they are already present
on the system, explicitly requested by the user, or if the only available
version that satisfies the version specifier is a pre-release.

By default, dependency resolution tools SHOULD:

* accept already installed pre-releases for all version specifiers
* accept remotely available pre-releases for version specifiers where
  there is no final or post release that satisfies the version specifier
* exclude all other pre-releases from consideration

Dependency resolution tools MAY issue a warning if a pre-release is needed
to satisfy a version specifier.

Dependency resolution tools SHOULD also allow users to request the
following alternative behaviours:

* accepting pre-releases for all version specifiers
* excluding pre-releases for all version specifiers (reporting an error or
  warning if a pre-release is already installed locally, or if a
  pre-release is the only way to satisfy a particular specifier)

Dependency resolution tools MAY also allow the above behaviour to be
controlled on a per-distribution basis.

Post-releases and final releases receive no special treatment in version
specifiers - they are always included unless explicitly excluded.


Examples
--------

* ``3.1``: version 3.1 or later, but not version 4.0 or later.
* ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later.
* ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later.
* ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases,
  post releases, developmental releases and any 3.1.x maintenance releases.
* ``== 3.1.*``: any version that starts with 3.1. Equivalent to the
  ``3.1.0`` compatible release clause.
* ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and
  not version 3.2.0 or later.


Direct references
=================

Some automated tools may permit the use of a direct reference as an
alternative to a normal version specifier. A direct reference consists of
the specifier ``@`` and an explicit URL.

Whether or not direct references are appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when direct references are used
inappropriately.

Public index servers SHOULD NOT allow the use of direct references in
uploaded distributions. Direct references are intended as a tool for
software integrators rather than publishers.

Depending on the use case, some appropriate targets for a direct URL
reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or
a wheel binary archive. The exact URLs and targets supported will be tool
dependent.

For example, a local source archive may be referenced directly::

    pip @ file:///localbuilds/pip-1.3.1.zip

Alternatively, a prebuilt archive may also be referenced::

    pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl

All direct references that do not refer to a local file URL SHOULD specify
a secure transport mechanism (such as ``https``) AND include an expected
hash value in the URL for verification purposes. If a direct reference is
specified without any hash information, with hash information that the
tool doesn't understand, or with a selected hash algorithm that the tool
considers too weak to trust, automated tools SHOULD at least emit a warning
and MAY refuse to rely on the URL. If such a direct reference also uses an
insecure transport, automated tools SHOULD NOT rely on the URL.

It is RECOMMENDED that only hashes which are unconditionally provided by
the latest version of the standard library's ``hashlib`` module be used
for source archive hashes. At time of writing, that list consists of
``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and
``'sha512'``.

For source archive and wheel references, an expected hash value may be
specified by including a ``<hash-algorithm>=<expected-hash>`` entry as
part of the URL fragment.

For version control references, the ``VCS+protocol`` scheme SHOULD be
used to identify both the version control system and the secure transport,
and a version control system with hash based commit identifiers SHOULD be
used. Automated tools MAY omit warnings about missing hashes for version
control systems that do not provide hash based commit identifiers.

To handle version control systems that do not support including commit or
tag references directly in the URL, that information may be appended to the
end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>``
notation.

.. note::

   This isn't *quite* the same as the existing VCS reference notation
   supported by pip. Firstly, the distribution name is moved in front rather
   than embedded as part of the URL. Secondly, the commit hash is included
   even when retrieving based on a tag, in order to meet the requirement
   above that *every* link should include a hash to make things harder to
   forge (creating a malicious repo with a particular tag is easy, creating
   one with a specific *hash*, less so).

Remote URL examples::

    pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
    pip @ git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d
    pip @ git+https://github.com/pypa/pip.git at 1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d



Updating the versioning specification
=====================================

The versioning specification may be updated with clarifications without
requiring a new PEP or a change to the metadata version.

Actually changing the version comparison semantics still requires a new
versioning scheme and metadata version defined in new PEPs.


Summary of differences from \PEP 386
====================================

* Moved the description of version specifiers into the versioning PEP

* Added the "direct reference" concept as a standard notation for direct
  references to resources (rather than each tool needing to invent its own)

* Added the "local version identifier" and "local version label" concepts to
  allow system integrators to indicate patched builds in a way that is
  supported by the upstream tools, as well as to allow the incorporation of
  build tags into the versioning of binary distributions.

* Added the "compatible release" clause

* Added the trailing wildcard syntax for prefix based version matching
  and exclusion

* Changed the top level sort position of the ``.devN`` suffix

* Allowed single value version numbers

* Explicit exclusion of leading or trailing whitespace

* Explicit support for date based versions

* Explicit normalisation rules to improve compatibility with
  existing version metadata on PyPI where it doesn't introduce
  ambiguity

* Implicitly exclude pre-releases unless they're already present or
  needed to satisfy a dependency

* Treat post releases the same way as unqualified releases

* Discuss ordering and dependencies across metadata versions

The rationale for major changes is given in the following sections.


Changing the version scheme
---------------------------

One key change in the version scheme in this PEP relative to that in
PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
projects already using both development releases and alphas/betas/release
candidates do not want their developmental releases sorted in
between their release candidates and their final releases. There is no
rationale for using ``dev`` releases in that position rather than
merely creating additional release candidates.

The updated sort order also means the sorting of ``dev`` versions is now
consistent between the metadata standard and the pre-existing behaviour
of ``pkg_resources`` (and hence the behaviour of current installation
tools).

Making this change should make it easier for affected existing projects to
migrate to the latest version of the metadata standard.

Another change to the version scheme is to allow single number
versions, similar to those used by non-Python projects like Mozilla
Firefox, Google Chrome and the Fedora Linux distribution. This is actually
expected to be more useful for version specifiers, but it is easier to
allow it for both version specifiers and release numbers, rather than
splitting the two definitions.

The exclusion of leading and trailing whitespace was made explicit after
a couple of projects with version identifiers differing only in a
trailing ``\n`` character were found on PyPI.

Various other normalisation rules were also added as described in the
separate section on version normalisation below.

`Appendix A` shows detailed results of an analysis of PyPI distribution
version information, as collected on 8th August, 2014. This analysis
compares the behavior of the explicitly ordered version scheme defined in
this PEP with the de facto standard defined by the behavior of setuptools.
These metrics are useful, as the intent of this PEP is to follow existing
setuptools behavior as closely as is feasible, while still throwing
exceptions for unorderable versions (rather than trying to guess an
appropriate order as setuptools does).


A more opinionated description of the versioning scheme
-------------------------------------------------------

As in PEP 386, the primary focus is on codifying existing practices to make
them more amenable to automation, rather than demanding that existing
projects make non-trivial changes to their workflow. However, the
standard scheme allows significantly more flexibility than is needed
for the vast majority of simple Python packages (which often don't even
need maintenance releases - many users are happy with needing to upgrade to a
new feature release to get bug fixes).

For the benefit of novice developers, and for experienced developers
wishing to better understand the various use cases, the specification
now goes into much greater detail on the components of the defined
version scheme, including examples of how each component may be used
in practice.

The PEP also explicitly guides developers in the direction of
semantic versioning (without requiring it), and discourages the use of
several aspects of the full versioning scheme that have largely been
included in order to cover esoteric corner cases in the practices of
existing projects and in repackaging software for Linux distributions.


Describing version specifiers alongside the versioning scheme
-------------------------------------------------------------

The main reason to even have a standardised version scheme in the first place
is to make it easier to do reliable automated dependency analysis. It makes
more sense to describe the primary use case for version identifiers alongside
their definition.


Changing the interpretation of version specifiers
-------------------------------------------------

The previous interpretation of version specifiers made it very easy to
accidentally download a pre-release version of a dependency. This in
turn made it difficult for developers to publish pre-release versions
of software to the Python Package Index, as even marking the package as
hidden wasn't enough to keep automated tools from downloading it, and also
made it harder for users to obtain the test release manually through the
main PyPI web interface.

The previous interpretation also excluded post-releases from some version
specifiers for no adequately justified reason.

The updated interpretation is intended to make it difficult to accidentally
accept a pre-release version as satisfying a dependency, while still
allowing pre-release versions to be retrieved automatically when that's the
only way to satisfy a dependency.

The "some forward compatibility assumed" version constraint is derived from the
Ruby community's "pessimistic version constraint" operator [2]_ to allow
projects to take a cautious approach to forward compatibility promises, while
still easily setting a minimum required version for their dependencies. The
spelling of the compatible release clause (``~=``) is inspired by the Ruby
(``~>``) and PHP (``~``) equivalents.

Further improvements are also planned to the handling of parallel
installation of multiple versions of the same library, but these will
depend on updates to the installation database definition along with
improved tools for dynamic path manipulation.

The trailing wildcard syntax to request prefix based version matching was
added to make it possible to sensibly define both compatible release clauses
and the desired pre- and post-release handling semantics for ``<`` and ``>``
ordered comparison clauses.


Support for date based version identifiers
------------------------------------------

Excluding date based versions caused significant problems in migrating
``pytz`` to the new metadata standards. It also caused concerns for the
OpenStack developers, as they use a date based versioning scheme and would
like to be able to migrate to the new metadata standards without changing
it.


Adding version epochs
---------------------

Version epochs are added for the same reason they are part of other
versioning schemes, such as those of the Fedora and Debian Linux
distributions: to allow projects to gracefully change their approach to
numbering releases, without having a new release appear to have a lower
version number than previous releases and without having to change the name
of the project.

In particular, supporting version epochs allows a project that was previously
using date based versioning to switch to semantic versioning by specifying
a new version epoch.

The ``!`` character was chosen to delimit an epoch version rather than the
``:`` character, which is commonly used in other systems, due to the fact that
``:`` is not a valid character in a Windows directory name.


Adding direct references
------------------------

Direct references are added as an "escape clause" to handle messy real
world situations that don't map neatly to the standard distribution model.
This includes dependencies on unpublished software for internal use, as well
as handling the more complex compatibility issues that may arise when
wrapping third party libraries as C extensions (this is of especial concern
to the scientific community).

Index servers are deliberately given a lot of freedom to disallow direct
references, since they're intended primarily as a tool for integrators
rather than publishers. PyPI in particular is currently going through the
process of *eliminating* dependencies on external references, as unreliable
external services have the effect of slowing down installation operations,
as well as reducing PyPI's own apparent reliability.


Adding arbitrary equality
-------------------------

Arbitrary equality is added as an "escape clause" to handle the case where
someone needs to install a project which uses a non compliant version. Although
this PEP is able to attain ~97% compatibility with the versions that are
already on PyPI there are still ~3% of versions which cannot be parsed. This
operator gives a simple and effective way to still depend on them without
having to "guess" at the semantics of what they mean (which would be required
if anything other than strict string based equality was supported).


Adding local version identifiers
--------------------------------

It's a fact of life that downstream integrators often need to backport
upstream bug fixes to older versions. It's one of the services that gets
Linux distro vendors paid, and application developers may also apply patches
they need to bundled dependencies.

Historically, this practice has been invisible to cross-platform language
specific distribution tools - the reported "version" in the upstream
metadata is the same as for the unmodified code. This inaccuracy can then
cause problems when attempting to work with a mixture of integrator
provided code and unmodified upstream code, or even just attempting to
identify exactly which version of the software is installed.

The introduction of local version identifiers and "local version labels"
into the versioning scheme, with the corresponding ``python.integrator``
metadata extension allows this kind of activity to be represented
accurately, which should improve interoperability between the upstream
tools and various integrated platforms.

The exact scheme chosen is largely modeled on the existing behavior of
``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``,
with the main distinction being that where ``pkg_resources`` currently always
takes the suffix into account when comparing versions for exact matches,
the PEP requires that the local version label of the candidate version be
ignored when no local version label is present in the version specifier
clause. Furthermore, the PEP does not attempt to impose any structure on
the local version labels (aside from limiting the set of permitted
characters and defining their ordering).

This change is designed to ensure that an integrator provided version like
``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version
specifier like ``pip>=1.5``.

The plus is chosen primarily for readability of local version identifiers.
It was chosen instead of the hyphen to prevent
``pkg_resources.parse_version`` from parsing it as a prerelease, which is
important for enabling a successful migration to the new, more structured,
versioning scheme. The plus was chosen instead of a tilde because of the
significance of the tilde in Debian's version ordering algorithm.


Providing explicit version normalization rules
----------------------------------------------

Historically, the de facto standard for parsing versions in Python has been the
``pkg_resources.parse_version`` command from the setuptools project. This does
not attempt to reject *any* version and instead tries to make something
meaningful, with varying levels of success, out of whatever it is given. It has
a few simple rules but otherwise it more or less relies largely on string
comparison.

The normalization rules provided in this PEP exist primarily to either increase
the compatability with ``pkg_resources.parse_version``, particularly in
documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something
more reasonable with versions that already exist on PyPI.

All possible normalization rules were weighed against whether or not they were
*likely* to cause any ambiguity (e.g. while someone might devise a scheme where
``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone
actually doing that, much less on any scale that is noticeable, is fairly low).
They were also weighed against how ``pkg_resources.parse_version`` treated a
particular version string, especially with regards to how it was sorted. Finally
each rule was weighed against the kinds of additional versions it allowed, how
"ugly" those versions looked, how hard there were to parse (both mentally and
mechanically) and how much additional compatibility it would bring.

The breadth of possible normalizations were kept to things that could easily
be implemented as part of the parsing of the version and not pre-parsing
transformations applied to the versions. This was done to limit the side
effects of each transformation as simple search and replace style transforms
increase the likelihood of ambiguous or "junk" versions.

For an extended discussion on the various types of normalizations that were
considered, please see the proof of concept for PEP 440 within pip [4]_.


References
==========

The initial attempt at a standardised version scheme, along with the
justifications for needing such a standard can be found in PEP 386.

.. [1] Reference Implementation of PEP 440 Versions and Specifiers
   https://github.com/pypa/packaging/pull/1

.. [2] Version compatibility analysis script:
   https://github.com/pypa/packaging/blob/master/tasks/check.py

.. [3] Pessimistic version constraint
   http://docs.rubygems.org/read/chapter/16

.. [4] Proof of Concept: PEP 440 within pip
    https://github.com/pypa/pip/pull/1894


Appendix A
==========

Metadata v2.0 guidelines versus setuptools::

    $ invoke check.pep440
    Total Version Compatibility:              245806/250521 (98.12%)
    Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%)
    Total Sorting Compatibility (Filtered):   47057/47114 (99.88%)
    Projects with No Compatible Versions:     498/47114 (1.06%)
    Projects with Differing Latest Version:   688/47114 (1.46%)


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140808/3abb48ba/attachment-0001.html>

From p.f.moore at gmail.com  Sat Aug  9 10:26:21 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 9 Aug 2014 09:26:21 +0100
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
Message-ID: <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>

On 8 August 2014 22:53, Donald Stufft <donald at stufft.io> wrote:
> Direct references
> =================
>
> Some automated tools may permit the use of a direct reference as an
> alternative to a normal version specifier. A direct reference consists of
> the specifier ``@`` and an explicit URL.
>
> Whether or not direct references are appropriate depends on the specific
> use case for the version specifier. Automated tools SHOULD at least issue
> warnings and MAY reject them entirely when direct references are used
> inappropriately.
>
> Public index servers SHOULD NOT allow the use of direct references in
> uploaded distributions. Direct references are intended as a tool for
> software integrators rather than publishers.
>
> Depending on the use case, some appropriate targets for a direct URL
> reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or
> a wheel binary archive. The exact URLs and targets supported will be tool
> dependent.
>
> For example, a local source archive may be referenced directly::
>
>     pip @ file:///localbuilds/pip-1.3.1.zip
>
> Alternatively, a prebuilt archive may also be referenced::
>
>     pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl

One very minor question. Is URL format required here? I guess so. On
Windows, file URLs are confusing and annoying to type - use of /
rather than \ makes tab-completion useless, it's difficult to remember
how many slashes go at the start and where the drive letter goes, and
do UNC paths need 157 or 158 slashes at the start (:-)), and the
documentation is inconsistent and hard to find.

Allowing a pathname here would be convenient for users, but it's 100%
not important enough to need a new version of the spec. A
clarification (presumably, that a URL is required) with a pointer to a
document that explains the relevant filename->URL translation
algorithm, would be good, though.

Paul

From donald at stufft.io  Sat Aug  9 18:02:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 9 Aug 2014 12:02:05 -0400
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
	Dependency Specification Version
In-Reply-To: <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
Message-ID: <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>


> On Aug 9, 2014, at 4:26 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 8 August 2014 22:53, Donald Stufft <donald at stufft.io> wrote:
>> Direct references
>> =================
>> 
>> Some automated tools may permit the use of a direct reference as an
>> alternative to a normal version specifier. A direct reference consists of
>> the specifier ``@`` and an explicit URL.
>> 
>> Whether or not direct references are appropriate depends on the specific
>> use case for the version specifier. Automated tools SHOULD at least issue
>> warnings and MAY reject them entirely when direct references are used
>> inappropriately.
>> 
>> Public index servers SHOULD NOT allow the use of direct references in
>> uploaded distributions. Direct references are intended as a tool for
>> software integrators rather than publishers.
>> 
>> Depending on the use case, some appropriate targets for a direct URL
>> reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or
>> a wheel binary archive. The exact URLs and targets supported will be tool
>> dependent.
>> 
>> For example, a local source archive may be referenced directly::
>> 
>>    pip @ file:///localbuilds/pip-1.3.1.zip
>> 
>> Alternatively, a prebuilt archive may also be referenced::
>> 
>>    pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl
> 
> One very minor question. Is URL format required here? I guess so. On
> Windows, file URLs are confusing and annoying to type - use of /
> rather than \ makes tab-completion useless, it's difficult to remember
> how many slashes go at the start and where the drive letter goes, and
> do UNC paths need 157 or 158 slashes at the start (:-)), and the
> documentation is inconsistent and hard to find.
> 
> Allowing a pathname here would be convenient for users, but it's 100%
> not important enough to need a new version of the spec. A
> clarification (presumably, that a URL is required) with a pointer to a
> document that explains the relevant filename->URL translation
> algorithm, would be good, though.
> 
> Paul


To be clear, the direct reference is mostly for use in the install_requires. On
the CLI pip can still just take a path to a file or whatever. This feature is
intended to replace dependency_links in a way that people can use them for
private packages but that they won't be allowed on PyPI or the like.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140809/f447b067/attachment.html>

From p.f.moore at gmail.com  Sat Aug  9 19:41:07 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 9 Aug 2014 18:41:07 +0100
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
 <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
Message-ID: <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>

On 9 August 2014 17:02, Donald Stufft <donald at stufft.io> wrote:
> To be clear, the direct reference is mostly for use in the install_requires.
> On the CLI pip can still just take a path to a file or whatever. This feature
> is intended to replace dependency_links in a way that people can use
> them for private packages but that they won't be allowed on PyPI or the
> like.

Understood. I'd still like an explicit pointer to the canonical URL
details tools should (or must?) use. See
http://en.wikipedia.org/wiki/File_URI_scheme (specifically the
"Windows" sections) for why I'm bothered - specifically the quote
"Here are *some* examples which *may* be accepted by some
applications" (emphasis mine).

I'd be more than happy if the spec was just "tools will use XXX from
the Python standard library. But note (the first 4 are the supposedly
acceptable URLs from that Wikipedia article):

>py
Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600
64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.request import url2pathname
>>> url2pathname("file://localhost/c|/WINDOWS/clock.avi")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
    raise OSError(error)
OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi
>>> url2pathname("file:///c|/WINDOWS/clock.avi")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
    raise OSError(error)
OSError: Bad URL: file|///c|/WINDOWS/clock.avi
>>> url2pathname("file://localhost/c:/WINDOWS/clock.avi")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
    raise OSError(error)
OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi
>>> url2pathname("file:///c:/WINDOWS/clock.avi")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
    raise OSError(error)
OSError: Bad URL: file|///c|/WINDOWS/clock.avi
>>> url2pathname("file:////WINDOWS/clock.avi")
'E:\\WINDOWS\\clock.avi'
>>> url2pathname("file:///WINDOWS/clock.avi")
'E:\\WINDOWS\\clock.avi'
>>> url2pathname("file://WINDOWS/clock.avi")
'E:\\WINDOWS\\clock.avi'
>>> url2pathname("file:/WINDOWS/clock.avi")
'E:\\WINDOWS\\clock.avi'
>>>

I have no idea where Python got that E drive from - my current drive is C:.

At this point in time, I have no idea how to correctly write a file
URL for the file C:\Windows\clock.avi on my PC. While it's very
definitely a corner case in the PEP, and it's clearly not the PEP's
responsibility to explain the file URL spec, I do think it's relevant
to the usability of the PEP.

Paul

From donald at stufft.io  Sat Aug  9 19:50:11 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 9 Aug 2014 13:50:11 -0400
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
	Dependency Specification Version
In-Reply-To: <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
 <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
 <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>
Message-ID: <803A2729-4B27-47BE-B736-A6E4E08AA8C5@stufft.io>


> On Aug 9, 2014, at 1:41 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 9 August 2014 17:02, Donald Stufft <donald at stufft.io> wrote:
>> To be clear, the direct reference is mostly for use in the install_requires.
>> On the CLI pip can still just take a path to a file or whatever. This feature
>> is intended to replace dependency_links in a way that people can use
>> them for private packages but that they won't be allowed on PyPI or the
>> like.
> 
> Understood. I'd still like an explicit pointer to the canonical URL
> details tools should (or must?) use. See
> http://en.wikipedia.org/wiki/File_URI_scheme (specifically the
> "Windows" sections) for why I'm bothered - specifically the quote
> "Here are *some* examples which *may* be accepted by some
> applications" (emphasis mine).
> 
> I'd be more than happy if the spec was just "tools will use XXX from
> the Python standard library. But note (the first 4 are the supposedly
> acceptable URLs from that Wikipedia article):
> 
>> py
> Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600
> 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from urllib.request import url2pathname
>>>> url2pathname("file://localhost/c|/WINDOWS/clock.avi")
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
>    raise OSError(error)
> OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi
>>>> url2pathname("file:///c|/WINDOWS/clock.avi")
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
>    raise OSError(error)
> OSError: Bad URL: file|///c|/WINDOWS/clock.avi
>>>> url2pathname("file://localhost/c:/WINDOWS/clock.avi")
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
>    raise OSError(error)
> OSError: Bad URL: file|//localhost/c|/WINDOWS/clock.avi
>>>> url2pathname("file:///c:/WINDOWS/clock.avi")
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "C:\Apps\Python34\lib\nturl2path.py", line 26, in url2pathname
>    raise OSError(error)
> OSError: Bad URL: file|///c|/WINDOWS/clock.avi
>>>> url2pathname("file:////WINDOWS/clock.avi")
> 'E:\\WINDOWS\\clock.avi'
>>>> url2pathname("file:///WINDOWS/clock.avi")
> 'E:\\WINDOWS\\clock.avi'
>>>> url2pathname("file://WINDOWS/clock.avi")
> 'E:\\WINDOWS\\clock.avi'
>>>> url2pathname("file:/WINDOWS/clock.avi")
> 'E:\\WINDOWS\\clock.avi'
>>>> 
> 
> I have no idea where Python got that E drive from - my current drive is C:.
> 
> At this point in time, I have no idea how to correctly write a file
> URL for the file C:\Windows\clock.avi on my PC. While it's very
> definitely a corner case in the PEP, and it's clearly not the PEP's
> responsibility to explain the file URL spec, I do think it's relevant
> to the usability of the PEP.
> 
> Paul

Does: file:///c:/WINDOWS/clock.avi work? That?s pointed out in the wikipage you
linked as to what IE has done since IE4 and seems to be the most reasonable
candidate. It also uses file://remotehost/sharename/dir/file.txt for UNC paths
apparently.

I misunderstood the problem though, I thought you were worried about needing to
type out the file:// and the slash order on the CLI, not referencing files at
all. We can absolutely add more details to the PEP.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140809/c375e50e/attachment-0001.html>

From p.f.moore at gmail.com  Sat Aug  9 20:51:28 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 9 Aug 2014 19:51:28 +0100
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <803A2729-4B27-47BE-B736-A6E4E08AA8C5@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
 <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
 <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>
 <803A2729-4B27-47BE-B736-A6E4E08AA8C5@stufft.io>
Message-ID: <CACac1F_tC+2+QvXjdpDDpU+Xg886getiSLOyL+zj7e9Gui_95A@mail.gmail.com>

On 9 August 2014 18:50, Donald Stufft <donald at stufft.io> wrote:
> Does: file:///c:/WINDOWS/clock.avi work?

Never mind, it's my mistake. Reading the docs more closely, "Convert
the path component path from a percent-encoded URL to the local syntax
for a path. This does not accept a complete URL." (Note that second
sentence...)

>>> url2pathname("///c:/WINDOWS/clock.avi")
'C:\\WINDOWS\\clock.avi'

Looks like the PEP would be covered by referring to
pathname2url/url2pathname (with a note that you strip the "file:"
prefix for idiots like me ;-)) and giving a couple of examples.

FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how
you'd refer to \\myserver\share\WINDOWS\clock.avi. See what I mean
about the slashes? I can't rationalise the 4 slashes here resulting in
two backslashes, when the 3 slashes above disappear entirely...

Paul.

From greg.ewing at canterbury.ac.nz  Sun Aug 10 02:03:15 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 10 Aug 2014 12:03:15 +1200
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <CACac1F_tC+2+QvXjdpDDpU+Xg886getiSLOyL+zj7e9Gui_95A@mail.gmail.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
 <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
 <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>
 <803A2729-4B27-47BE-B736-A6E4E08AA8C5@stufft.io>
 <CACac1F_tC+2+QvXjdpDDpU+Xg886getiSLOyL+zj7e9Gui_95A@mail.gmail.com>
Message-ID: <53E6B6C3.3030004@canterbury.ac.nz>

Paul Moore wrote:
> FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how
> you'd refer to \\myserver\share\WINDOWS\clock.avi.

Where did you get that from? According to this it's wrong:

http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx

It should be

    file://myserver/share/WINDOWS/clock.avi

i.e. the hosthame part of the path goes into the hostname
part of the URI, which makes sense.

If the path uses a drive letter instead, there is no
hostname, so that part of the URI is empty, resulting in
three consecutive slashes:

    file:///c:/WINDOWS/clock.avi

-- 
Greg

From p.f.moore at gmail.com  Sun Aug 10 10:01:31 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 10 Aug 2014 09:01:31 +0100
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <53E6B6C3.3030004@canterbury.ac.nz>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CACac1F9ZE9O-V5QzLSSA+303jwFF4yx6Bo5iVfsWnzBkDujrLw@mail.gmail.com>
 <30630E65-345A-462E-A36B-53EB8CA1CB9C@stufft.io>
 <CACac1F8Q4OCsTrJh5UGrGxQMU9FoYvj-rdLtWXhZwsG61dbL=w@mail.gmail.com>
 <803A2729-4B27-47BE-B736-A6E4E08AA8C5@stufft.io>
 <CACac1F_tC+2+QvXjdpDDpU+Xg886getiSLOyL+zj7e9Gui_95A@mail.gmail.com>
 <53E6B6C3.3030004@canterbury.ac.nz>
Message-ID: <CACac1F8mNJcd6M3mkdMqP5NwAAB0ddoX4Zzy8Y3tNx6jiCYOEg@mail.gmail.com>

On 10 August 2014 01:03, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Paul Moore wrote:
>>
>> FWIW, it looks like file:////myserver/share/WINDOWS/clock.avi is how
>> you'd refer to \\myserver\share\WINDOWS\clock.avi.
>
>
> Where did you get that from? According to this it's wrong:
>
> http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx
>
> It should be
>
>    file://myserver/share/WINDOWS/clock.avi

>>> from urllib.request import pathname2url
>>> pathname2url(r'\\myserver\share\WINDOWS\clock.avi')
'////myserver/share/WINDOWS/clock.avi'

Bug in urllib?

Paul

From strawman at astraw.com  Sun Aug 10 17:39:07 2014
From: strawman at astraw.com (Andrew Straw)
Date: Sun, 10 Aug 2014 17:39:07 +0200
Subject: [Distutils] [ANN] stdeb-0.8.1
Message-ID: <2852319C-17DF-4A08-8046-1D7E644F6771@astraw.com>

Hi all,
 
stdeb is a package to produce Debian packages from Python packages. Today I?m announcing release 0.8.1 of stdeb. (Due to bugs in 0.8.0, this release is the first announced from the 0.8 series.) Highlights since 0.7.1:

Full support for Python 3. This includes being run from Python 3 and generating packages for Python 3. The default is to build Python 3 packages when run with Python 3 and to build Python 2 packages when run from Python 2. Command line options can be used to build packages for the other Python interpreter, too.
 
Build .changes file for source package. While this still must be signed for upload to a PPA, for example, it should still be useful in some cases.
 
Switch to Debian source format 3.0 (quilt). Practically speaking, the .diff.gz file that used to come with a source package is now replaced by a .debian.tar.gz file.
 
Verify SSL certificates when talking to PyPI using Requests. (Verification requires Requests >= 0.8.8.)
 
Many bugfixes.
Download: https://pypi.python.org/pypi/stdeb/0.8.1
Development: https://github.com/astraw/stdeb
 
Best regards,
Andrew

From techtonik at gmail.com  Sun Aug 10 20:54:51 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 10 Aug 2014 21:54:51 +0300
Subject: [Distutils] dif.io had failed leaving some shiny scrap for the
	Warehouse
Message-ID: <CAPkN8xJgXt5eJ5KDXvyTtaqhqZ=nYTyYVzy=RcJ2T79SvW79kQ@mail.gmail.com>

Hi,

Just want to make sure you know that this fine Apache 2.0
project is down - https://github.com/difio/difio - so while its
developers are still around, there is a chance to salvage any
good parts for the Warehouse.

Look at some fine stats while it is still online at http://www.dif.io/

Please, CC if you have any feedback.
-- 
anatoly t.

From donald at stufft.io  Mon Aug 11 17:22:45 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 11 Aug 2014 11:22:45 -0400
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
	Dependency Specification Version
In-Reply-To: <CAPYWazoJJ+qrstdRPazRovqnyTg=Pf9trz57L-_oxJ9f1c-1Og@mail.gmail.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CAPYWazoJJ+qrstdRPazRovqnyTg=Pf9trz57L-_oxJ9f1c-1Og@mail.gmail.com>
Message-ID: <08EE2A20-65CA-4624-BDE0-9CB8A16994A3@stufft.io>


> On Aug 11, 2014, at 11:11 AM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> > Public index servers SHOULD NOT allow the use of local version identifiers for uploaded distributions.
> 
> I'm thinking this should just say "PyPI" and not "Public" broadly.
> The point is for local versions not to confused with the one authoritative upstream version sequence, not that it couldn't be "public".
> I can imagine locally versioned distributions needing to be distributed or available "publicly" (e.g. for a specific platform or system).
> Considering the recommendation to use the "python.integrator" extension, which is generally about "downstream" modification and redistribution, it seems inconsistent to say that this redistribution couldn't be public.
> 


We actually have a definition for Public Index Server, It?s ""Public index
servers" are index servers which allow distribution uploads from untrusted third
parties. The Python Package Index [3] is a public index server.?. This is
defined in PEP 426.

But thinking about it, that doesn't particularly match it either, because
something like binstar allows this, but should allow local versions.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140811/af9b6216/attachment.html>

From qwcode at gmail.com  Mon Aug 11 17:11:05 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 11 Aug 2014 08:11:05 -0700
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
Message-ID: <CAPYWazoJJ+qrstdRPazRovqnyTg=Pf9trz57L-_oxJ9f1c-1Og@mail.gmail.com>

> Public index servers SHOULD NOT allow the use of local version
identifiers for uploaded distributions.

I'm thinking this should just say "PyPI" and not "Public" broadly.
The point is for local versions not to confused with the one authoritative
upstream version sequence, not that it couldn't be "public".
I can imagine locally versioned distributions needing to be distributed or
available "publicly" (e.g. for a specific platform or system).
Considering the recommendation to use the "python.integrator" extension,
which is generally about "downstream" modification and redistribution, it
seems inconsistent to say that this redistribution couldn't be public.


On Fri, Aug 8, 2014 at 2:53 PM, Donald Stufft <donald at stufft.io> wrote:

> I?m happy to announce the 6th, and hopefully final, draft of PEP 440. The
> updates to this PEP take into account feedback from several large projects
> and
> users of various systems (such as Linux package tools) where this would be
> expected to interface with. It also takes into account experience gained
> from
> attempting to implement this PEP fully as a proof of concept within pip.
>
> Significant updates to the PEP include:
>
> * Switching the Epoch identifier from : to ! as : is not valid in a
> directory
>   name on Windows.
> * Local version identifiers use + as a seperator in order to reduce
> ambigiuty
>   with existing versions on PyPI.
> * Allow alpha numerics for local versions.
> * Define a sorting algorithm for local versions.
> * Moved the source label to PEP 426
> * Normalization rules for parsing more versions following along the idea of
>   Postel's Law.
> * Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also
> should be
>   used for Metadata 1.0 and 1.1.
> * Declare how invalid versions should be handled.
> * Add the escape hatch "Arbitrary equality" operator === to allow
> depending on
>   versions which cannot be parsed by PEP 440.
> * Make specifier syntax match what setuptools uses (foo==1.0 instead of
> foo (==1.0))
> * Remove the default specifier.
> * Use @ for direct references instead of "from".
> * Create a reference implementation and Link To it.
> * Lots more minor changes.
>
> The outcome of these changes is that we were able to raise compatability
> with
> all the versions registered with PyPI up to 98.12% and we sort 99.88% of
> projects registered with PyPI the same as pkg_resources does when filtering
> invalid versions from the list of versions. The fallout is that 498
> projects,
> or 1.06%, are no longer installable without using the ``===`` operator and
> 190
> projects, or 0.4%, of projects have a different result for what the
> "latest"
> version is.
>
> Of the 498 projects a number of them are nonsensical versions like ``.`` or
> the repr of lazy objects and of the 190 projects projects it's about evenly
> split between projects where pkg_resources supported something we didn't
> and
> where PEP 440 just simply does a better job at parsing and sorting
> versions.
>
> You can see the pip proof of concept and a large discussion about
> normalization
> at https://github.com/pypa/pip/pull/1894 and the reference implementation
> can
> be found at https://github.com/pypa/packaging/pull/1. The Proof of
> Concept does
> not allow using the new specifiers inside of an install_requires in a
> source
> distribution because setuptools itself does not support it, but it does
> support
> them on the command line and in requirements.txt files.
>
> The online view of the PEP can be found at
> https://www.python.org/dev/peps/pep-0440/ and the changes since the last
> posting
> can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and
> http://hg.python.org/peps/rev/257822378672.
>
> Without further ado, the PEP itself:
>
> Abstract
> ========
>
> This PEP describes a scheme for identifying versions of Python software
> distributions, and declaring dependencies on particular versions.
>
> This document addresses several limitations of the previous attempt at a
> standardized approach to versioning, as described in PEP 345 and PEP 386.
>
>
> Definitions
> ===========
>
> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
> "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
> document are to be interpreted as described in RFC 2119.
>
> The following terms are to be interpreted as described in PEP 426:
>
> * "Distributions"
> * "Releases"
> * "Build tools"
> * "Index servers"
> * "Publication tools"
> * "Installation tools"
> * "Automated tools"
> * "Projects"
>
>
> Version scheme
> ==============
>
> Distributions are identified by a public version identifier which
> supports all defined version comparison operations
>
> The version scheme is used both to describe the distribution version
> provided by a particular distribution archive, as well as to place
> constraints on the version of dependencies needed in order to build or
> run the software.
>
>
> Public version identifiers
> --------------------------
>
> Public version identifiers MUST comply with the following scheme::
>
>     [N!]N(.N)*[{a|b|c}N][.postN][.devN]
>
> Public version identifiers MUST NOT include leading or trailing whitespace.
>
> Public version identifiers MUST be unique within a given distribution.
>
> Installation tools SHOULD ignore any public versions which do not comply
> with
> this scheme. Installation tools MAY warn the user when non-compliant
> or ambiguous versions are detected.
>
> Public version identifiers are separated into up to five segments:
>
> * Epoch segment: ``N!``
> * Release segment: ``N(.N)*``
> * Pre-release segment: ``{a|b|c}N``
> * Post-release segment: ``.postN``
> * Development release segment: ``.devN``
>
> Any given release will be a "final release", "pre-release", "post-release"
> or
> "developmental release" as defined in the following sections.
>
> All numeric components MUST be non-negative integers.
>
> All numeric components MUST be interpreted and ordered according to their
> numeric value, not as text strings.
>
> All numeric components MAY be zero. Except as described below for the
> release segment, a numeric component of zero has no special significance
> aside from always being the lowest possible value in the version ordering.
>
> .. note::
>
>    Some hard to read version identifiers are permitted by this scheme in
>    order to better accommodate the wide range of versioning practices
>    across existing public and private Python projects.
>
>    Accordingly, some of the versioning practices which are technically
>    permitted by the PEP are strongly discouraged for new projects. Where
>    this is the case, the relevant details are noted in the following
>    sections.
>
>
> Local version identifiers
> -------------------------
>
> Local version identifiers MUST comply with the following scheme::
>
>     <public version identifier>[+<local version label>]
>
> They consist of a normal public version identifier (as defined in the
> previous section), along with an arbitrary "local version label", separated
> from the public version identifier by a plus. Local version labels have
> no specific semantics assigned, but some syntactic restrictions are
> imposed.
>
> Local version identifiers are used to denote fully API (and, if applicable,
> ABI) compatible patched versions of upstream projects. For example, these
> may be created by application developers and system integrators by applying
> specific backported bug fixes when upgrading to a new upstream release
> would
> be too disruptive to the application or other integrated system (such as a
> Linux distribution).
>
> The inclusion of the local version label makes it possible to differentiate
> upstream releases from potentially altered rebuilds by downstream
> integrators. The use of a local version identifier does not affect the kind
> of a release but, when applied to a source distribution, does indicate that
> it may not contain the exact same code as the corresponding upstream
> release.
>
> To ensure local version identifiers can be readily incorporated as part of
> filenames and URLs, and to avoid formatting inconsistencies in hexadecimal
> hash representations, local version labels MUST be limited to the following
> set of permitted characters:
>
> * ASCII letters (``[a-zA-Z]``)
> * ASCII digits (``[0-9]``)
> * periods (``.``)
>
> Local version labels MUST start and end with an ASCII letter or digit.
>
> Comparison and ordering of local versions considers each segment of the
> local
> version (divided by a ``.``) separately. If a segment consists entirely of
> ASCII digits then that section should be considered an integer for
> comparison
> purposes and if a segment contains any ASCII letters than that segment is
> compared lexicographically with case insensitivity. When comparing a
> numeric
> and lexicographic segment, the numeric section always compares as greater
> than
> the lexicographic segment. Additionally a local version with a great
> number of
> segments will always compare as greater than a local version with fewer
> segments, as long as the shorter local version's segments match the
> beginning
> of the longer local version's segments exactly.
>
> Local version identifiers may be used in most locations where a public
> version identifier is expected, with the exception of any version
> specifiers
> that explicitly rely on being able to unambiguously order candidate
> versions.
>
> Public index servers SHOULD NOT allow the use of local version identifiers
> for uploaded distributions.
>
> Source distributions using a local version identifier SHOULD provide the
> ``python.integrator`` extension metadata (as defined in :pep:`459`).
>
>
> Final releases
> --------------
>
> A version identifier that consists solely of a release segment and
> optionally
> an epoch identifier is termed a "final release".
>
> The release segment consists of one or more non-negative integer
> values, separated by dots::
>
>     N[.N]+
>
> Final releases within a project MUST be numbered in a consistently
> increasing fashion, otherwise automated tools will not be able to upgrade
> them correctly.
>
> Comparison and ordering of release segments considers the numeric value
> of each component of the release segment in turn. When comparing release
> segments with different numbers of components, the shorter segment is
> padded out with additional zeros as necessary.
>
> While any number of additional components after the first are permitted
> under this scheme, the most common variants are to use two components
> ("major.minor") or three components ("major.minor.micro").
>
> For example::
>
>     0.9
>     0.9.1
>     0.9.2
>     ...
>     0.9.10
>     0.9.11
>     1.0
>     1.0.1
>     1.1
>     2.0
>     2.0.1
>     ...
>
> A release series is any set of final release numbers that start with a
> common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all
> part of the ``3.3`` release series.
>
> .. note::
>
>    ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as
>    the release segment comparison rules implicit expand the two component
>    form to ``X.Y.0`` when comparing it to any release segment that includes
>    three components.
>
> Date based release segments are also permitted. An example of a date based
> release scheme using the year and month of the release::
>
>     2012.04
>     2012.07
>     2012.10
>     2013.01
>     2013.06
>     ...
>
>
> Pre-releases
> ------------
>
> Some projects use an "alpha, beta, release candidate" pre-release cycle to
> support testing by their users prior to a final release.
>
> If used as part of a project's development cycle, these pre-releases are
> indicated by including a pre-release segment in the version identifier::
>
>     X.YaN  # Alpha release
>     X.YbN  # Beta release
>     X.YcN  # Candidate release
>     X.Y    # Final release
>
> A version identifier that consists solely of a release segment and a
> pre-release segment is termed a "pre-release".
>
> The pre-release segment consists of an alphabetical identifier for the
> pre-release phase, along with a non-negative integer value. Pre-releases
> for
> a given release are ordered first by phase (alpha, beta, release candidate)
> and then by the numerical component within that phase.
>
> Installation tools MAY accept both ``c`` and ``rc`` releases for a common
> release segment in order to handle some existing legacy distributions.
>
> Installation tools SHOULD interpret ``rc`` versions as being equivalent to
> ``c`` versions (that is, ``rc1`` indicates the same version as ``c1``).
>
> Build tools, publication tools and index servers SHOULD disallow the
> creation
> of both ``c`` and ``rc`` releases for a common release segment.
>
>
> Post-releases
> -------------
>
> Some projects use post-releases to address minor errors in a final release
> that do not affect the distributed software (for example, correcting an
> error
> in the release notes).
>
> If used as part of a project's development cycle, these post-releases are
> indicated by including a post-release segment in the version identifier::
>
>     X.Y.postN    # Post-release
>
> A version identifier that includes a post-release segment without a
> developmental release segment is termed a "post-release".
>
> The post-release segment consists of the string ``.post``, followed by a
> non-negative integer value. Post-releases are ordered by their
> numerical component, immediately following the corresponding release,
> and ahead of any subsequent release.
>
> .. note::
>
>    The use of post-releases to publish maintenance releases containing
>    actual bug fixes is strongly discouraged. In general, it is better
>    to use a longer release number and increment the final component
>    for each maintenance release.
>
> Post-releases are also permitted for pre-releases::
>
>     X.YaN.postM  # Post-release of an alpha release
>     X.YbN.postM  # Post-release of a beta release
>     X.YcN.postM  # Post-release of a release candidate
>
> .. note::
>
>    Creating post-releases of pre-releases is strongly discouraged, as
>    it makes the version identifier difficult to parse for human readers.
>    In general, it is substantially clearer to simply create a new
>    pre-release by incrementing the numeric component.
>
>
> Developmental releases
> ----------------------
>
> Some projects make regular developmental releases, and system packagers
> (especially for Linux distributions) may wish to create early releases
> directly from source control which do not conflict with later project
> releases.
>
> If used as part of a project's development cycle, these developmental
> releases are indicated by including a developmental release segment in the
> version identifier::
>
>     X.Y.devN    # Developmental release
>
> A version identifier that includes a developmental release segment is
> termed a "developmental release".
>
> The developmental release segment consists of the string ``.dev``,
> followed by a non-negative integer value. Developmental releases are
> ordered
> by their numerical component, immediately before the corresponding release
> (and before any pre-releases with the same release segment), and following
> any previous release (including any post-releases).
>
> Developmental releases are also permitted for pre-releases and
> post-releases::
>
>     X.YaN.devM      # Developmental release of an alpha release
>     X.YbN.devM      # Developmental release of a beta release
>     X.YcN.devM      # Developmental release of a release candidate
>     X.Y.postN.devM  # Developmental release of a post-release
>
> .. note::
>
>    Creating developmental releases of pre-releases is strongly
>    discouraged, as it makes the version identifier difficult to parse for
>    human readers. In general, it is substantially clearer to simply create
>    additional pre-releases by incrementing the numeric component.
>
>    Developmental releases of post-releases are also strongly discouraged,
>    but they may be appropriate for projects which use the post-release
>    notation for full maintenance releases which may include code changes.
>
>
> Version epochs
> --------------
>
> If included in a version identifier, the epoch appears before all other
> components, separated from the release segment by an exclamation mark::
>
>     E!X.Y  # Version identifier with epoch
>
> If no explicit epoch is given, the implicit epoch is ``0``.
>
> Most version identifiers will not include an epoch, as an explicit epoch is
> only needed if a project *changes* the way it handles version numbering in
> a way that means the normal version ordering rules will give the wrong
> answer. For example, if a project is using date based versions like
> ``2014.04`` and would like to switch to semantic versions like ``1.0``,
> then
> the new releases would be identified as *older* than the date based
> releases
> when using the normal sorting scheme::
>
>     1.0
>     1.1
>     2.0
>     2013.10
>     2014.04
>
> However, by specifying an explicit epoch, the sort order can be changed
> appropriately, as all versions from a later epoch are sorted after versions
> from an earlier epoch::
>
>     2013.10
>     2014.04
>     1!1.0
>     1!1.1
>     1!2.0
>
> Normalization
> -------------
>
> In order to maintain better compatibility with existing versions there are
> a
> number of "alternative" syntaxes that MUST be taken into account when
> parsing
> versions. These syntaxes MUST be considered when parsing a version, however
> they should be "normalized" to the standard syntax defined above.
>
>
> Case sensitivity
> ~~~~~~~~~~~~~~~~
>
> All ascii letters should be interpreted case insensitively within a
> version and
> the normal form is lowercase. This allows versions such as ``1.1RC1`` which
> would be normalized to ``1.1c1``.
>
>
> Integer Normalization
> ~~~~~~~~~~~~~~~~~~~~~
>
> All integers are interpreted via the ``int()`` built in and normalize to
> the
> string form of the output. This means that an integer version of ``00``
> would
> normalize to ``0`` while ``09000`` would normalize to ``9000``. This does
> not
> hold true for integers inside of an alphanumeric segment of a local version
> such as ``1.0+foo0100`` which is already in its normalized form.
>
>
> Pre-release separators
> ~~~~~~~~~~~~~~~~~~~~~~
>
> Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the
> release segment and the pre-release segment. The normal form for this is
> without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1``
> which would be normalized to ``1.1a1``. It should also allow a seperator to
> be used between the pre-release signifier and the numeral. This allows
> versions
> such as ``1.0a.1`` which would be normalized to ``1.0a1``.
>
>
> Pre-release spelling
> ~~~~~~~~~~~~~~~~~~~~
>
> Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``,
> ``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c``
> respectively.
> This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3``
> which
> normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the
> additional
> spelling should be considered equivalent to their normal forms.
>
>
> Implicit pre-release number
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Pre releases allow omitting the numeral in which case it is implicitly
> assumed
> to be ``0``. The normal form for this is to include the ``0`` explicitly.
> This
> allows versions such as ``1.2a`` which is normalized to ``1.2a0``.
>
>
> Post release separators
> ~~~~~~~~~~~~~~~~~~~~~~~
>
> Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting
> the
> separator all together. The normal form of this is with the ``.``
> separator.
> This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize
> to
> ``1.2.post2``. Like the pre-release seperator this also allows an optional
> separator between the post release signifier and the numeral. This allows
> versions like ``1.2.post-2`` which would normalize to ``1.2.post2``.
>
>
> Post release spelling
> ~~~~~~~~~~~~~~~~~~~~~
>
> Post-releases allow the additional spellings of ``rev`` and ``r``. This
> allows
> versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the
> pre-releases the additional spellings should be considered equivalent to
> their
> normal forms.
>
>
> Implicit post release number
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Post releases allow omiting the numeral in which case it is implicitly
> assumed
> to be ``0``. The normal form for this is to include the ``0`` explicitly.
> This
> allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``.
>
>
> Implicit post releases
> ~~~~~~~~~~~~~~~~~~~~~~
>
> Post releases allow omitting the ``post`` signifier all together. When
> using
> this form the separator MUST be ``-`` and no other form is allowed. This
> allows
> versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This
> particular
> normalization MUST NOT be used in conjunction with the implicit post
> release
> number rule. In other words ``1.0-`` is *not* a valid version and it does
> *not*
> normalize to ``1.0.post0``.
>
>
> Development release separators
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Development releases allow a ``.``, ``-``, or a ``_`` separator as well as
> omitting the separator all together. The normal form of this is with the
> ``.``
> separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which
> normalize to ``1.2.dev2``.
>
>
> Implicit development release number
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Development releases allow omiting the numeral in which case it is
> implicitly
> assumed to be ``0``. The normal form for this is to include the ``0``
> explicitly. This allows versions such as ``1.2.dev`` which is normalized to
> ``1.2.dev0``.
>
>
> Local version segments
> ~~~~~~~~~~~~~~~~~~~~~~
>
> With a local version, in addition to the use of ``.`` as a separator of
> segments, the use of ``-`` and ``_`` is also acceptable. The normal form is
> using the ``.`` character. This allows versions such as ``1.0+ubuntu-1``
> to be
> normalized to ``1.0+ubuntu.1``.
>
>
> Preceding v character
> ~~~~~~~~~~~~~~~~~~~~~
>
> In order to support the common version notation of ``v1.0`` versions may be
> preceded by a single literal ``v`` character. This character MUST be
> ignored
> for all purposes and should be omitted from all normalized forms of the
> version. The same version with and without the ``v`` is considered
> equivalent.
>
>
> Leading and Trailing Whitespace
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Leading and trailing whitespace must be silently ignored and removed from
> all
> normalized forms of a version. This includes ``" "``, ``\t``, ``\n``,
> ``\r``,
> ``\f``, and ``\v``. This allows accidental whitespace to be handled
> sensibly,
> such as a version like ``1.0\n`` which normalizes to ``1.0``.
>
>
> Examples of compliant version schemes
> -------------------------------------
>
> The standard version scheme is designed to encompass a wide range of
> identification practices across public and private Python projects. In
> practice, a single project attempting to use the full flexibility offered
> by the scheme would create a situation where human users had difficulty
> figuring out the relative order of versions, even though the rules above
> ensure all compliant tools will order them consistently.
>
> The following examples illustrate a small selection of the different
> approaches projects may choose to identify their releases, while still
> ensuring that the "latest release" and the "latest stable release" can
> be easily determined, both by human users and automated tools.
>
> Simple "major.minor" versioning::
>
>     0.1
>     0.2
>     0.3
>     1.0
>     1.1
>     ...
>
> Simple "major.minor.micro" versioning::
>
>     1.1.0
>     1.1.1
>     1.1.2
>     1.2.0
>     ...
>
> "major.minor" versioning with alpha, beta and candidate
> pre-releases::
>
>     0.9
>     1.0a1
>     1.0a2
>     1.0b1
>     1.0c1
>     1.0
>     1.1a1
>     ...
>
> "major.minor" versioning with developmental releases, release candidates
> and post-releases for minor corrections::
>
>     0.9
>     1.0.dev1
>     1.0.dev2
>     1.0.dev3
>     1.0.dev4
>     1.0rc1
>     1.0rc2
>     1.0
>     1.0.post1
>     1.1.dev1
>     ...
>
> Date based releases, using an incrementing serial within each year,
> skipping
> zero::
>
>     2012.1
>     2012.2
>     2012.3
>     ...
>     2012.15
>     2013.1
>     2013.2
>     ...
>
>
> Summary of permitted suffixes and relative ordering
> ---------------------------------------------------
>
> .. note::
>
>    This section is intended primarily for authors of tools that
>    automatically process distribution metadata, rather than developers
>    of Python distributions deciding on a versioning scheme.
>
> The epoch segment of version identifiers MUST be sorted according to the
> numeric value of the given epoch. If no epoch segment is present, the
> implicit numeric value is ``0``.
>
> The release segment of version identifiers MUST be sorted in
> the same order as Python's tuple sorting when the release segment is
> parsed as follows::
>
>     tuple(map(int, release_segment.split(".")))
>
> All release segments involved in the comparison MUST be converted to a
> consistent length by padding shorter segments with zeros as needed.
>
> Within a numeric release (``1.0``, ``2.7.3``), the following suffixes
> are permitted and MUST be ordered as shown::
>
>    .devN, aN, bN, cN/rcN, <no suffix>, .postN
>
> Note that `rc` is considered to be semantically equivalent to `c` and must
> be
> sorted as if it were `c`. Tools MAY reject the case of having the same
> ``N``
> for both a ``rc`` and a ``c`` in the same release segment as ambiguous and
> remain in compliance with the PEP.
>
> Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate
> (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be
> ordered as shown::
>
>    .devN, <no suffix>, .postN
>
> Within a post-release (``1.0.post1``), the following suffixes are permitted
> and MUST be ordered as shown::
>
>     .devN, <no suffix>
>
> Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even
> when used immediately following a numeric version (e.g. ``1.0.dev456``,
> ``1.0.post1``).
>
> Within a pre-release, post-release or development release segment with a
> shared prefix, ordering MUST be by the value of the numeric component.
>
> The following example covers many of the possible combinations::
>
>     1.0.dev456
>     1.0a1
>     1.0a2.dev456
>     1.0a12.dev456
>     1.0a12
>     1.0b1.dev456
>     1.0b2
>     1.0b2.post345.dev456
>     1.0b2.post345
>     1.0c1.dev456
>     1.0c1
>     1.0
>     1.0+abc.5
>     1.0+abc.7
>     1.0+5
>     1.0.post456.dev34
>     1.0.post456
>     1.1.dev1
>
>
> Version ordering across different metadata versions
> ---------------------------------------------------
>
> Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a
> standard
> version identification or ordering scheme. However metadata v1.2 (PEP 345)
> does specify a scheme which is defined in PEP 386.
>
> Due to the nature of the simple installer API it is not possible for an
> installer to be aware of which metadata version a particular distribution
> was
> using. Additionally installers required the ability to create a reasonably
> prioritized list that includes all, or as many as possible, versions of
> a project to determine which versions it should install. These requirements
> necessitate a standardization across one parsing mechanism to be used for
> all
> versions of a project.
>
> Due to the above, this PEP MUST be used for all versions of metadata and
> supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions
> which cannot be parsed by the rules in this PEP, but MAY fall back to
> implementation defined version parsing and ordering schemes if no versions
> complying with this PEP are available.
>
> Distribution users may wish to explicitly remove non-compliant versions
> from
> any private package indexes they control.
>
>
> Compatibility with other version schemes
> ----------------------------------------
>
> Some projects may choose to use a version scheme which requires
> translation in order to comply with the public version scheme defined in
> this PEP. In such cases, the project specific version can be stored in the
> metadata while the translated public version is published in the version
> field.
>
> This allows automated distribution tools to provide consistently correct
> ordering of published releases, while still allowing developers to use
> the internal versioning scheme they prefer for their projects.
>
>
> Semantic versioning
> ~~~~~~~~~~~~~~~~~~~
>
> `Semantic versioning`_ is a popular version identification scheme that is
> more prescriptive than this PEP regarding the significance of different
> elements of a release number. Even if a project chooses not to abide by
> the details of semantic versioning, the scheme is worth understanding as
> it covers many of the issues that can arise when depending on other
> distributions, and when publishing a distribution that others rely on.
>
> The "Major.Minor.Patch" (described in this PEP as "major.minor.micro")
> aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1
> specification)
> are fully compatible with the version scheme defined in this PEP, and
> abiding
> by these aspects is encouraged.
>
> Semantic versions containing a hyphen (pre-releases - clause 10) or a
> plus sign (builds - clause 11) are *not* compatible with this PEP
> and are not permitted in the public version field.
>
> One possible mechanism to translate such semantic versioning based source
> labels to compatible public versions is to use the ``.devN`` suffix to
> specify the appropriate version order.
>
> Specific build information may also be included in local version labels.
>
> .. _Semantic versioning: http://semver.org/
>
>
> DVCS based version labels
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Many build tools integrate with distributed version control systems like
> Git and Mercurial in order to add an identifying hash to the version
> identifier. As hashes cannot be ordered reliably such versions are not
> permitted in the public version field.
>
> As with semantic versioning, the public ``.devN`` suffix may be used to
> uniquely identify such releases for publication, while the original DVCS
> based
> label can be stored in the project metadata.
>
> Identifying hash information may also be included in local version labels.
>
>
> Olson database versioning
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The ``pytz`` project inherits its versioning scheme from the corresponding
> Olson timezone database versioning scheme: the year followed by a lowercase
> character indicating the version of the database within that year.
>
> This can be translated to a compliant public version identifier as
> ``<year>.<serial>``, where the serial starts at zero or one (for the
> '<year>a' release) and is incremented with each subsequent database
> update within the year.
>
> As with other translated version identifiers, the corresponding Olson
> database version could be recorded in the project metadata.
>
>
> Version specifiers
> ==================
>
> A version specifier consists of a series of version clauses, separated by
> commas. For example::
>
>    ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0
>
> The comparison operator determines the kind of version clause:
>
> * ``~=``: `Compatible release`_ clause
> * ``==``: `Version matching`_ clause
> * ``!=``: `Version exclusion`_ clause
> * ``<=``, ``>=``: `Inclusive ordered comparison`_ clause
> * ``<``, ``>``: `Exclusive ordered comparison`_ clause
> * ``===``: `Arbitrary equality`_ clause.
>
> The comma (",") is equivalent to a logical **and** operator: a candidate
> version must match all given version clauses in order to match the
> specifier as a whole.
>
> Whitespace between a conditional operator and the following version
> identifier is optional, as is the whitespace around the commas.
>
> When multiple candidate versions match a version specifier, the preferred
> version SHOULD be the latest version as determined by the consistent
> ordering defined by the standard `Version scheme`_. Whether or not
> pre-releases are considered as candidate versions SHOULD be handled as
> described in `Handling of pre-releases`_.
>
> Except where specifically noted below, local version identifiers MUST NOT
> be
> permitted in version specifiers, and local version labels MUST be ignored
> entirely when checking if candidate versions match a given version
> specifier.
>
>
> Compatible release
> ------------------
>
> A compatible release clause consists of either a version identifier without
> any comparison operator or else the compatible release operator ``~=``
> and a version identifier. It matches any candidate version that is expected
> to be compatible with the specified version.
>
> The specified version identifier must be in the standard format described
> in
> `Version scheme`_. Local version identifiers are NOT permitted in this
> version specifier.
>
> For a given release identifier ``V.N``, the compatible release clause is
> approximately equivalent to the pair of comparison clauses::
>
>     >= V.N, == V.*
>
> This operator MUST NOT be used with a single segment version number such as
> ``~=1``.
>
> For example, the following groups of version clauses are equivalent::
>
>     2.2
>     ~= 2.2
>     >= 2.2, == 2.*
>
>     1.4.5
>     ~= 1.4.5
>     >= 1.4.5, == 1.4.*
>
> If a pre-release, post-release or developmental release is named in a
> compatible release clause as ``V.N.suffix``, then the suffix is ignored
> when determining the required prefix match::
>
>     2.2.post3
>     ~= 2.2.post3
>     >= 2.2.post3, == 2.*
>
>     1.4.5a4
>     ~= 1.4.5a4
>     >= 1.4.5a4, == 1.4.*
>
> The padding rules for release segment comparisons means that the assumed
> degree of forward compatibility in a compatible release clause can be
> controlled by appending additional zeros to the version specifier::
>
>     2.2.0
>     ~= 2.2.0
>     >= 2.2.0, == 2.2.*
>
>     1.4.5.0
>     ~= 1.4.5.0
>     >= 1.4.5.0, == 1.4.5.*
>
>
> Version matching
> ----------------
>
> A version matching clause includes the version matching operator ``==``
> and a version identifier.
>
> The specified version identifier must be in the standard format described
> in
> `Version scheme`_, but a trailing ``.*`` is permitted on public version
> identifiers as described below.
>
> By default, the version matching operator is based on a strict equality
> comparison: the specified version must be exactly the same as the requested
> version. The *only* substitution performed is the zero padding of the
> release segment to ensure the release segments are compared with the same
> length.
>
> Whether or not strict version matching is appropriate depends on the
> specific
> use case for the version specifier. Automated tools SHOULD at least issue
> warnings and MAY reject them entirely when strict version matches are used
> inappropriately.
>
> Prefix matching may be requested instead of strict comparison, by appending
> a trailing ``.*`` to the version identifier in the version matching clause.
> This means that additional trailing segments will be ignored when
> determining whether or not a version identifier matches the clause. If the
> specified version includes only a release segment, than trailing components
> (or the lack thereof) in the release segment are also ignored.
>
> For example, given the version ``1.1.post1``, the following clauses would
> match or not as shown::
>
>     == 1.1        # Not equal, so 1.1.post1 does not match clause
>     == 1.1.post1  # Equal, so 1.1.post1 matches clause
>     == 1.1.*      # Same prefix, so 1.1.post1 matches clause
>
> For purposes of prefix matching, the pre-release segment is considered to
> have an implied preceding ``.``, so given the version ``1.1a1``, the
> following clauses would match or not as shown::
>
>     == 1.1        # Not equal, so 1.1a1 does not match clause
>     == 1.1a1      # Equal, so 1.1a1 matches clause
>     == 1.1.*      # Same prefix, so 1.1a1 matches clause
>
> An exact match is also considered a prefix match (this interpreation is
> implied by the usual zero padding rules for the release segment of version
> identifiers). Given the version ``1.1``, the following clauses would
> match or not as shown::
>
>     == 1.1        # Equal, so 1.1 matches clause
>     == 1.1.0      # Zero padding expands 1.1 to 1.1.0, so it matches clause
>     == 1.1.dev1   # Not equal (dev-release), so 1.1 does not match clause
>     == 1.1a1      # Not equal (pre-release), so 1.1 does not match clause
>     == 1.1.post1  # Not equal (post-release), so 1.1 does not match clause
>     == 1.1.*      # Same prefix, so 1.1 matches clause
>
> It is invalid to have a prefix match containing a development or local
> release
> such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development
> release
> segment is always the final segment in the public version, and the local
> version
> is ignored for comparison purposes, so using either in a prefix match
> wouldn't
> make any sense.
>
> The use of ``==`` (without at least the wildcard suffix) when defining
> dependencies for published distributions is strongly discouraged as it
> greatly complicates the deployment of security fixes. The strict version
> comparison operator is intended primarily for use when defining
> dependencies for repeatable *deployments of applications* while using
> a shared distribution index.
>
> If the specified version identifier is a public version identifier (no
> local version label), then the local version label of any candidate
> versions
> MUST be ignored when matching versions.
>
> If the specified version identifier is a local version identifier, then the
> local version labels of candidate versions MUST be considered when matching
> versions, with the public version identifier being matched as described
> above, and the local version label being checked for equivalence using a
> strict string equality comparison.
>
>
> Version exclusion
> -----------------
>
> A version exclusion clause includes the version exclusion operator ``!=``
> and a version identifier.
>
> The allowed version identifiers and comparison semantics are the same as
> those of the `Version matching`_ operator, except that the sense of any
> match is inverted.
>
> For example, given the version ``1.1.post1``, the following clauses would
> match or not as shown::
>
>     != 1.1        # Not equal, so 1.1.post1 matches clause
>     != 1.1.post1  # Equal, so 1.1.post1 does not match clause
>     != 1.1.*      # Same prefix, so 1.1.post1 does not match clause
>
>
> Inclusive ordered comparison
> ----------------------------
>
> An inclusive ordered comparison clause includes a comparison operator and a
> version identifier, and will match any version where the comparison is
> correct
> based on the relative position of the candidate version and the specified
> version given the consistent ordering defined by the standard
> `Version scheme`_.
>
> The inclusive ordered comparison operators are ``<=`` and ``>=``.
>
> As with version matching, the release segment is zero padded as necessary
> to
> ensure the release segments are compared with the same length.
>
> Local version identifiers are NOT permitted in this version specifier.
>
>
> Exclusive ordered comparison
> ----------------------------
>
> Exclusive ordered comparisons are similar to inclusive ordered comparisons,
> except that the comparison operators are ``<`` and ``>`` and the clause
> MUST be effectively interpreted as implying the prefix based version
> exclusion clause ``!= V.*``.
>
> The exclusive ordered comparison ``> V`` MUST NOT match a post-release
> or maintenance release of the given version. Maintenance releases can be
> permitted by using the clause ``> V.0``, while both post releases and
> maintenance releases can be permitted by using the inclusive ordered
> comparison ``>= V.post1``.
>
> The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of
> the given version, even if acceptance of pre-releases is enabled as
> described in the section below.
>
> Local version identifiers are NOT permitted in this version specifier.
>
>
> Arbitrary equality
> ------------------
>
> Arbitrary equality comparisons are simple string equality operations which
> do
> not take into account any of the semantic information such as zero padding
> or
> local versions. This operator also does not support prefix matching as the
> ``==`` operator does.
>
> The primary use case for arbitrary equality is to allow for specifying a
> version which cannot otherwise be represented by this PEP. This operator is
> special and acts as an escape hatch to allow someone using a tool which
> implements this PEP to still install a legacy version which is otherwise
> incompatible with this PEP.
>
> An example would be ``===foobar`` which would match a version of
> ``foobar``.
>
> This operator may also be used to explicitly require an unpatched version
> of a project such as ``===1.0`` which would not match for a version
> ``1.0+downstream1``.
>
> Use of this operator is heavily discouraged and tooling MAY display a
> warning
> when it is used.
>
>
> Handling of pre-releases
> ------------------------
>
> Pre-releases of any kind, including developmental releases, are implicitly
> excluded from all version specifiers, *unless* they are already present
> on the system, explicitly requested by the user, or if the only available
> version that satisfies the version specifier is a pre-release.
>
> By default, dependency resolution tools SHOULD:
>
> * accept already installed pre-releases for all version specifiers
> * accept remotely available pre-releases for version specifiers where
>   there is no final or post release that satisfies the version specifier
> * exclude all other pre-releases from consideration
>
> Dependency resolution tools MAY issue a warning if a pre-release is needed
> to satisfy a version specifier.
>
> Dependency resolution tools SHOULD also allow users to request the
> following alternative behaviours:
>
> * accepting pre-releases for all version specifiers
> * excluding pre-releases for all version specifiers (reporting an error or
>   warning if a pre-release is already installed locally, or if a
>   pre-release is the only way to satisfy a particular specifier)
>
> Dependency resolution tools MAY also allow the above behaviour to be
> controlled on a per-distribution basis.
>
> Post-releases and final releases receive no special treatment in version
> specifiers - they are always included unless explicitly excluded.
>
>
> Examples
> --------
>
> * ``3.1``: version 3.1 or later, but not version 4.0 or later.
> * ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later.
> * ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later.
> * ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all
> pre-releases,
>   post releases, developmental releases and any 3.1.x maintenance releases.
> * ``== 3.1.*``: any version that starts with 3.1. Equivalent to the
>   ``3.1.0`` compatible release clause.
> * ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and
>   not version 3.2.0 or later.
>
>
> Direct references
> =================
>
> Some automated tools may permit the use of a direct reference as an
> alternative to a normal version specifier. A direct reference consists of
> the specifier ``@`` and an explicit URL.
>
> Whether or not direct references are appropriate depends on the specific
> use case for the version specifier. Automated tools SHOULD at least issue
> warnings and MAY reject them entirely when direct references are used
> inappropriately.
>
> Public index servers SHOULD NOT allow the use of direct references in
> uploaded distributions. Direct references are intended as a tool for
> software integrators rather than publishers.
>
> Depending on the use case, some appropriate targets for a direct URL
> reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or
> a wheel binary archive. The exact URLs and targets supported will be tool
> dependent.
>
> For example, a local source archive may be referenced directly::
>
>     pip @ file:///localbuilds/pip-1.3.1.zip
>
> Alternatively, a prebuilt archive may also be referenced::
>
>     pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl
>
> All direct references that do not refer to a local file URL SHOULD specify
> a secure transport mechanism (such as ``https``) AND include an expected
> hash value in the URL for verification purposes. If a direct reference is
> specified without any hash information, with hash information that the
> tool doesn't understand, or with a selected hash algorithm that the tool
> considers too weak to trust, automated tools SHOULD at least emit a warning
> and MAY refuse to rely on the URL. If such a direct reference also uses an
> insecure transport, automated tools SHOULD NOT rely on the URL.
>
> It is RECOMMENDED that only hashes which are unconditionally provided by
> the latest version of the standard library's ``hashlib`` module be used
> for source archive hashes. At time of writing, that list consists of
> ``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and
> ``'sha512'``.
>
> For source archive and wheel references, an expected hash value may be
> specified by including a ``<hash-algorithm>=<expected-hash>`` entry as
> part of the URL fragment.
>
> For version control references, the ``VCS+protocol`` scheme SHOULD be
> used to identify both the version control system and the secure transport,
> and a version control system with hash based commit identifiers SHOULD be
> used. Automated tools MAY omit warnings about missing hashes for version
> control systems that do not provide hash based commit identifiers.
>
> To handle version control systems that do not support including commit or
> tag references directly in the URL, that information may be appended to the
> end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>``
> notation.
>
> .. note::
>
>    This isn't *quite* the same as the existing VCS reference notation
>    supported by pip. Firstly, the distribution name is moved in front
> rather
>    than embedded as part of the URL. Secondly, the commit hash is included
>    even when retrieving based on a tag, in order to meet the requirement
>    above that *every* link should include a hash to make things harder to
>    forge (creating a malicious repo with a particular tag is easy, creating
>    one with a specific *hash*, less so).
>
> Remote URL examples::
>
>     pip @
> https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
>     pip @
> git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d
>     pip @
> git+https://github.com/pypa/pip.git at 1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d
>
>
>
> Updating the versioning specification
> =====================================
>
> The versioning specification may be updated with clarifications without
> requiring a new PEP or a change to the metadata version.
>
> Actually changing the version comparison semantics still requires a new
> versioning scheme and metadata version defined in new PEPs.
>
>
> Summary of differences from \PEP 386
> ====================================
>
> * Moved the description of version specifiers into the versioning PEP
>
> * Added the "direct reference" concept as a standard notation for direct
>   references to resources (rather than each tool needing to invent its own)
>
> * Added the "local version identifier" and "local version label" concepts
> to
>   allow system integrators to indicate patched builds in a way that is
>   supported by the upstream tools, as well as to allow the incorporation of
>   build tags into the versioning of binary distributions.
>
> * Added the "compatible release" clause
>
> * Added the trailing wildcard syntax for prefix based version matching
>   and exclusion
>
> * Changed the top level sort position of the ``.devN`` suffix
>
> * Allowed single value version numbers
>
> * Explicit exclusion of leading or trailing whitespace
>
> * Explicit support for date based versions
>
> * Explicit normalisation rules to improve compatibility with
>   existing version metadata on PyPI where it doesn't introduce
>   ambiguity
>
> * Implicitly exclude pre-releases unless they're already present or
>   needed to satisfy a dependency
>
> * Treat post releases the same way as unqualified releases
>
> * Discuss ordering and dependencies across metadata versions
>
> The rationale for major changes is given in the following sections.
>
>
> Changing the version scheme
> ---------------------------
>
> One key change in the version scheme in this PEP relative to that in
> PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
> of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
> projects already using both development releases and alphas/betas/release
> candidates do not want their developmental releases sorted in
> between their release candidates and their final releases. There is no
> rationale for using ``dev`` releases in that position rather than
> merely creating additional release candidates.
>
> The updated sort order also means the sorting of ``dev`` versions is now
> consistent between the metadata standard and the pre-existing behaviour
> of ``pkg_resources`` (and hence the behaviour of current installation
> tools).
>
> Making this change should make it easier for affected existing projects to
> migrate to the latest version of the metadata standard.
>
> Another change to the version scheme is to allow single number
> versions, similar to those used by non-Python projects like Mozilla
> Firefox, Google Chrome and the Fedora Linux distribution. This is actually
> expected to be more useful for version specifiers, but it is easier to
> allow it for both version specifiers and release numbers, rather than
> splitting the two definitions.
>
> The exclusion of leading and trailing whitespace was made explicit after
> a couple of projects with version identifiers differing only in a
> trailing ``\n`` character were found on PyPI.
>
> Various other normalisation rules were also added as described in the
> separate section on version normalisation below.
>
> `Appendix A` shows detailed results of an analysis of PyPI distribution
> version information, as collected on 8th August, 2014. This analysis
> compares the behavior of the explicitly ordered version scheme defined in
> this PEP with the de facto standard defined by the behavior of setuptools.
> These metrics are useful, as the intent of this PEP is to follow existing
> setuptools behavior as closely as is feasible, while still throwing
> exceptions for unorderable versions (rather than trying to guess an
> appropriate order as setuptools does).
>
>
> A more opinionated description of the versioning scheme
> -------------------------------------------------------
>
> As in PEP 386, the primary focus is on codifying existing practices to make
> them more amenable to automation, rather than demanding that existing
> projects make non-trivial changes to their workflow. However, the
> standard scheme allows significantly more flexibility than is needed
> for the vast majority of simple Python packages (which often don't even
> need maintenance releases - many users are happy with needing to upgrade
> to a
> new feature release to get bug fixes).
>
> For the benefit of novice developers, and for experienced developers
> wishing to better understand the various use cases, the specification
> now goes into much greater detail on the components of the defined
> version scheme, including examples of how each component may be used
> in practice.
>
> The PEP also explicitly guides developers in the direction of
> semantic versioning (without requiring it), and discourages the use of
> several aspects of the full versioning scheme that have largely been
> included in order to cover esoteric corner cases in the practices of
> existing projects and in repackaging software for Linux distributions.
>
>
> Describing version specifiers alongside the versioning scheme
> -------------------------------------------------------------
>
> The main reason to even have a standardised version scheme in the first
> place
> is to make it easier to do reliable automated dependency analysis. It makes
> more sense to describe the primary use case for version identifiers
> alongside
> their definition.
>
>
> Changing the interpretation of version specifiers
> -------------------------------------------------
>
> The previous interpretation of version specifiers made it very easy to
> accidentally download a pre-release version of a dependency. This in
> turn made it difficult for developers to publish pre-release versions
> of software to the Python Package Index, as even marking the package as
> hidden wasn't enough to keep automated tools from downloading it, and also
> made it harder for users to obtain the test release manually through the
> main PyPI web interface.
>
> The previous interpretation also excluded post-releases from some version
> specifiers for no adequately justified reason.
>
> The updated interpretation is intended to make it difficult to accidentally
> accept a pre-release version as satisfying a dependency, while still
> allowing pre-release versions to be retrieved automatically when that's the
> only way to satisfy a dependency.
>
> The "some forward compatibility assumed" version constraint is derived
> from the
> Ruby community's "pessimistic version constraint" operator [2]_ to allow
> projects to take a cautious approach to forward compatibility promises,
> while
> still easily setting a minimum required version for their dependencies. The
> spelling of the compatible release clause (``~=``) is inspired by the Ruby
> (``~>``) and PHP (``~``) equivalents.
>
> Further improvements are also planned to the handling of parallel
> installation of multiple versions of the same library, but these will
> depend on updates to the installation database definition along with
> improved tools for dynamic path manipulation.
>
> The trailing wildcard syntax to request prefix based version matching was
> added to make it possible to sensibly define both compatible release
> clauses
> and the desired pre- and post-release handling semantics for ``<`` and
> ``>``
> ordered comparison clauses.
>
>
> Support for date based version identifiers
> ------------------------------------------
>
> Excluding date based versions caused significant problems in migrating
> ``pytz`` to the new metadata standards. It also caused concerns for the
> OpenStack developers, as they use a date based versioning scheme and would
> like to be able to migrate to the new metadata standards without changing
> it.
>
>
> Adding version epochs
> ---------------------
>
> Version epochs are added for the same reason they are part of other
> versioning schemes, such as those of the Fedora and Debian Linux
> distributions: to allow projects to gracefully change their approach to
> numbering releases, without having a new release appear to have a lower
> version number than previous releases and without having to change the name
> of the project.
>
> In particular, supporting version epochs allows a project that was
> previously
> using date based versioning to switch to semantic versioning by specifying
> a new version epoch.
>
> The ``!`` character was chosen to delimit an epoch version rather than the
> ``:`` character, which is commonly used in other systems, due to the fact
> that
> ``:`` is not a valid character in a Windows directory name.
>
>
> Adding direct references
> ------------------------
>
> Direct references are added as an "escape clause" to handle messy real
> world situations that don't map neatly to the standard distribution model.
> This includes dependencies on unpublished software for internal use, as
> well
> as handling the more complex compatibility issues that may arise when
> wrapping third party libraries as C extensions (this is of especial concern
> to the scientific community).
>
> Index servers are deliberately given a lot of freedom to disallow direct
> references, since they're intended primarily as a tool for integrators
> rather than publishers. PyPI in particular is currently going through the
> process of *eliminating* dependencies on external references, as unreliable
> external services have the effect of slowing down installation operations,
> as well as reducing PyPI's own apparent reliability.
>
>
> Adding arbitrary equality
> -------------------------
>
> Arbitrary equality is added as an "escape clause" to handle the case where
> someone needs to install a project which uses a non compliant version.
> Although
> this PEP is able to attain ~97% compatibility with the versions that are
> already on PyPI there are still ~3% of versions which cannot be parsed.
> This
> operator gives a simple and effective way to still depend on them without
> having to "guess" at the semantics of what they mean (which would be
> required
> if anything other than strict string based equality was supported).
>
>
> Adding local version identifiers
> --------------------------------
>
> It's a fact of life that downstream integrators often need to backport
> upstream bug fixes to older versions. It's one of the services that gets
> Linux distro vendors paid, and application developers may also apply
> patches
> they need to bundled dependencies.
>
> Historically, this practice has been invisible to cross-platform language
> specific distribution tools - the reported "version" in the upstream
> metadata is the same as for the unmodified code. This inaccuracy can then
> cause problems when attempting to work with a mixture of integrator
> provided code and unmodified upstream code, or even just attempting to
> identify exactly which version of the software is installed.
>
> The introduction of local version identifiers and "local version labels"
> into the versioning scheme, with the corresponding ``python.integrator``
> metadata extension allows this kind of activity to be represented
> accurately, which should improve interoperability between the upstream
> tools and various integrated platforms.
>
> The exact scheme chosen is largely modeled on the existing behavior of
> ``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``,
> with the main distinction being that where ``pkg_resources`` currently
> always
> takes the suffix into account when comparing versions for exact matches,
> the PEP requires that the local version label of the candidate version be
> ignored when no local version label is present in the version specifier
> clause. Furthermore, the PEP does not attempt to impose any structure on
> the local version labels (aside from limiting the set of permitted
> characters and defining their ordering).
>
> This change is designed to ensure that an integrator provided version like
> ``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version
> specifier like ``pip>=1.5``.
>
> The plus is chosen primarily for readability of local version identifiers.
> It was chosen instead of the hyphen to prevent
> ``pkg_resources.parse_version`` from parsing it as a prerelease, which is
> important for enabling a successful migration to the new, more structured,
> versioning scheme. The plus was chosen instead of a tilde because of the
> significance of the tilde in Debian's version ordering algorithm.
>
>
> Providing explicit version normalization rules
> ----------------------------------------------
>
> Historically, the de facto standard for parsing versions in Python has
> been the
> ``pkg_resources.parse_version`` command from the setuptools project. This
> does
> not attempt to reject *any* version and instead tries to make something
> meaningful, with varying levels of success, out of whatever it is given.
> It has
> a few simple rules but otherwise it more or less relies largely on string
> comparison.
>
> The normalization rules provided in this PEP exist primarily to either
> increase
> the compatability with ``pkg_resources.parse_version``, particularly in
> documented use cases such as ``rev``, ``r``, ``pre``, etc or to do
> something
> more reasonable with versions that already exist on PyPI.
>
> All possible normalization rules were weighed against whether or not they
> were
> *likely* to cause any ambiguity (e.g. while someone might devise a scheme
> where
> ``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of
> anyone
> actually doing that, much less on any scale that is noticeable, is fairly
> low).
> They were also weighed against how ``pkg_resources.parse_version`` treated
> a
> particular version string, especially with regards to how it was sorted.
> Finally
> each rule was weighed against the kinds of additional versions it allowed,
> how
> "ugly" those versions looked, how hard there were to parse (both mentally
> and
> mechanically) and how much additional compatibility it would bring.
>
> The breadth of possible normalizations were kept to things that could
> easily
> be implemented as part of the parsing of the version and not pre-parsing
> transformations applied to the versions. This was done to limit the side
> effects of each transformation as simple search and replace style
> transforms
> increase the likelihood of ambiguous or "junk" versions.
>
> For an extended discussion on the various types of normalizations that were
> considered, please see the proof of concept for PEP 440 within pip [4]_.
>
>
> References
> ==========
>
> The initial attempt at a standardised version scheme, along with the
> justifications for needing such a standard can be found in PEP 386.
>
> .. [1] Reference Implementation of PEP 440 Versions and Specifiers
>    https://github.com/pypa/packaging/pull/1
>
> .. [2] Version compatibility analysis script:
>    https://github.com/pypa/packaging/blob/master/tasks/check.py
>
> .. [3] Pessimistic version constraint
>    http://docs.rubygems.org/read/chapter/16
>
> .. [4] Proof of Concept: PEP 440 within pip
>     https://github.com/pypa/pip/pull/1894
>
>
> Appendix A
> ==========
>
> Metadata v2.0 guidelines versus setuptools::
>
>     $ invoke check.pep440
>     Total Version Compatibility:              245806/250521 (98.12%)
>     Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%)
>     Total Sorting Compatibility (Filtered):   47057/47114 (99.88%)
>     Projects with No Compatible Versions:     498/47114 (1.06%)
>     Projects with Differing Latest Version:   688/47114 (1.46%)
>
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140811/44d74bc2/attachment-0001.html>

From ncoghlan at gmail.com  Tue Aug 12 00:04:42 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Aug 2014 08:04:42 +1000
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <08EE2A20-65CA-4624-BDE0-9CB8A16994A3@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <CAPYWazoJJ+qrstdRPazRovqnyTg=Pf9trz57L-_oxJ9f1c-1Og@mail.gmail.com>
 <08EE2A20-65CA-4624-BDE0-9CB8A16994A3@stufft.io>
Message-ID: <CADiSq7d7sJ=nSQQGKo0-fnFcpzxRhiz9E+kQJi7ZuRixj0+-FA@mail.gmail.com>

On 12 Aug 2014 01:23, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>> On Aug 11, 2014, at 11:11 AM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> > Public index servers SHOULD NOT allow the use of local version
identifiers for uploaded distributions.
>>
>> I'm thinking this should just say "PyPI" and not "Public" broadly.
>> The point is for local versions not to confused with the one
authoritative upstream version sequence, not that it couldn't be "public".
>> I can imagine locally versioned distributions needing to be distributed
or available "publicly" (e.g. for a specific platform or system).
>> Considering the recommendation to use the "python.integrator" extension,
which is generally about "downstream" modification and redistribution, it
seems inconsistent to say that this redistribution couldn't be public.
>>
>
> We actually have a definition for Public Index Server, It?s ""Public index
> servers" are index servers which allow distribution uploads from
untrusted third
> parties. The Python Package Index [3] is a public index server.?. This is
> defined in PEP 426.
>
> But thinking about it, that doesn't particularly match it either, because
> something like binstar allows this, but should allow local versions.

In this particular case, I think Marcus is right - the restriction is
specific to PyPI as the keeper of the authoritative shared namespace,
rather than applying to public index servers in general.

Cheers,
Nick.

>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140812/402b947c/attachment.html>

From donald at stufft.io  Sat Aug 16 18:11:36 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Aug 2014 12:11:36 -0400
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
	Dependency Specification Version
In-Reply-To: <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>


> On Aug 16, 2014, at 11:37 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
> 
> Thanks Donald for the extensive work on this. It all looks generally good.
>  
> One thing that stuck out as slightly surprising ? the use of ?_? as a separator. I imagine most people consider the underscore to be yet another alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest that the PEP give some examples. As is, I didn?t see any examples that showed ?_? as a separator.

It?s not a preferred separator, it?s use is mostly because of various normalization schemes, particularly Wheel, which will convert a ``-`` to a `` _`` in the version. Right now pip considers them equal because it was causing some breakage with Wheels. This rationale should probably be given in the PEP though.

The original bug report in pip for this was https://github.com/pypa/pip/issues/1150.

>  
> Related, I would suggest a consistent scheme for local version tags. Why not have local version tags have the same syntax as primary version numbers? That is, allow the same character set, pre and post versions, etc, just separated by a +. At the very least, I would expect local versions to allow underscores.

Originally local versions were just integers separated by dots. We wound up where we are so that downstream distributions can put more information inside of it. For instance Instead of Ubuntu taking a version like 1.0, and making it 1.0+1, they can do 1.0+ubuntu.1 or similar so that it?s obvious from the version number that this version is different than say Fedora, who might have also patched it and have 1.0+fedora.1.

I?m not sure what the use case is for pre/post/etc versions in the local version. At the point you?re doing more than simple patching you?re probably making a full fledged fork which should have it?s own name and version numbers I think?

>  
> These are all just nitpicks, though. A lot of work has gone into this spec, and it?s good to see the community coalescing on an implementable standard. It?s surely acceptable in the current form.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140816/97510b55/attachment.html>

From ncoghlan at gmail.com  Sun Aug 17 02:52:46 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Aug 2014 10:52:46 +1000
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>
 <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>
Message-ID: <CADiSq7etRDeUxfSM-NL=Dqd6UXiuc9+D0_z3zRfyihBy9PZigA@mail.gmail.com>

On 17 August 2014 02:11, Donald Stufft <donald at stufft.io> wrote:
>
> On Aug 16, 2014, at 11:37 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
>
> Thanks Donald for the extensive work on this. It all looks generally good.
>
> One thing that stuck out as slightly surprising ? the use of ?_? as a
> separator. I imagine most people consider the underscore to be yet another
> alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest
> that the PEP give some examples. As is, I didn?t see any examples that
> showed ?_? as a separator.
>
>
> It?s not a preferred separator, it?s use is mostly because of various
> normalization schemes, particularly Wheel, which will convert a ``-`` to a
> `` _`` in the version. Right now pip considers them equal because it was
> causing some breakage with Wheels. This rationale should probably be given
> in the PEP though.
>
> The original bug report in pip for this was
> https://github.com/pypa/pip/issues/1150.
>
>
> Related, I would suggest a consistent scheme for local version tags. Why not
> have local version tags have the same syntax as primary version numbers?
> That is, allow the same character set, pre and post versions, etc, just
> separated by a +. At the very least, I would expect local versions to allow
> underscores.
>
>
> Originally local versions were just integers separated by dots. We wound up
> where we are so that downstream distributions can put more information
> inside of it. For instance Instead of Ubuntu taking a version like 1.0, and
> making it 1.0+1, they can do 1.0+ubuntu.1 or similar so that it?s obvious
> from the version number that this version is different than say Fedora, who
> might have also patched it and have 1.0+fedora.1.

Right. In the Fedora (et al) case, the local version ordering scheme
also matches the way yum orders the RPM release field, so it becomes
possible to map the release field (which tracks patches and spec file
changes) directly to the Python level "local version".

One big advantage of the more relaxed rules around the local version
identifier component is that DVCS hashes are allowed, you just want to
include a serial number (e.g. number of commits since the last rebase)
first to ensure sensible ordering.

> I?m not sure what the use case is for pre/post/etc versions in the local
> version. At the point you?re doing more than simple patching you?re probably
> making a full fledged fork which should have it?s own name and version
> numbers I think?

Agreed. For use of the local version field to be appropriate, we
should be looking at full API compatibility with the public version
identifier.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Aug 17 03:00:28 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Aug 2014 11:00:28 +1000
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>
 <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>
Message-ID: <CADiSq7fHv_i6qak0nYbpycRu09nCyFQkvpGj0+innEp-pjXxXA@mail.gmail.com>

On 17 August 2014 02:11, Donald Stufft <donald at stufft.io> wrote:
> It?s not a preferred separator, it?s use is mostly because of various
> normalization schemes, particularly Wheel, which will convert a ``-`` to a
> `` _`` in the version. Right now pip considers them equal because it was
> causing some breakage with Wheels. This rationale should probably be given
> in the PEP though.
>
> The original bug report in pip for this was
> https://github.com/pypa/pip/issues/1150.

Donald, I've added this to the list at
https://bitbucket.org/pypa/pypi-metadata-formats/issue/49/address-pep-440-v6-feedback
(the comments from Marcus and Paul were already noted).

If you could prepare a patch to clarify those, we can get it merged
and published, and I'll accept the PEP. (Since they're just
clarifications rather than real changes, I'm actually prepared to
accept the PEP now, but I don't believe there's any pressing need for
urgency on that front - may as well get the clarifications in first)

Thanks for all your work in finally bringing this to fruition, as well
as to all those who contributed to the long migration from the
original CPAN style versioning to an approach more in line with the
Zen of Python :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From qwcode at gmail.com  Sun Aug 17 19:00:17 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 17 Aug 2014 10:00:17 -0700
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and
 Dependency Specification Version
In-Reply-To: <CADiSq7fHv_i6qak0nYbpycRu09nCyFQkvpGj0+innEp-pjXxXA@mail.gmail.com>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
 <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>
 <D970B507-039B-469D-AADE-CDC42A9FC25D@stufft.io>
 <CADiSq7fHv_i6qak0nYbpycRu09nCyFQkvpGj0+innEp-pjXxXA@mail.gmail.com>
Message-ID: <CAPYWazrTa=ROS98ta68iTkZJD2gE9W6_wJD3K96SBVxZKd4x+Q@mail.gmail.com>

>
>
>
> Donald, I've added this to the list at
>
> https://bitbucket.org/pypa/pypi-metadata-formats/issue/49/address-pep-440-v6-feedback
> (the comments from Marcus and Paul were already noted).
>

one other thing Donald and I discussed was being clear what was compatible
with pkg_resources.
my understanding is that *only* local versions are incompatible, in that
they don't sort correctly relative to public versions.
I'd like to see mention of that somewhere.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140817/05729450/attachment.html>

From jaraco at jaraco.com  Sat Aug 16 17:37:23 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Sat, 16 Aug 2014 15:37:23 +0000
Subject: [Distutils] Round 6 - PEP 440 - Version Identification
	and	Dependency Specification Version
In-Reply-To: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
References: <2512F226-CE95-4609-AFB7-F69398359F74@stufft.io>
Message-ID: <66d23824e8d041a7b103f327e8080569@BLUPR06MB434.namprd06.prod.outlook.com>

Thanks Donald for the extensive work on this. It all looks generally good.

One thing that stuck out as slightly surprising ? the use of ?_? as a separator. I imagine most people consider the underscore to be yet another alpha character, similar to [a-zA-Z]. If it is to be a separator, I suggest that the PEP give some examples. As is, I didn?t see any examples that showed ?_? as a separator.

Related, I would suggest a consistent scheme for local version tags. Why not have local version tags have the same syntax as primary version numbers? That is, allow the same character set, pre and post versions, etc, just separated by a +. At the very least, I would expect local versions to allow underscores.

These are all just nitpicks, though. A lot of work has gone into this spec, and it?s good to see the community coalescing on an implementable standard. It?s surely acceptable in the current form.

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Donald Stufft
Sent: Friday, 08 August, 2014 17:53
To: distutils-sig at python.org
Subject: [Distutils] Round 6 - PEP 440 - Version Identification and Dependency Specification Version

I?m happy to announce the 6th, and hopefully final, draft of PEP 440. The
updates to this PEP take into account feedback from several large projects and
users of various systems (such as Linux package tools) where this would be
expected to interface with. It also takes into account experience gained from
attempting to implement this PEP fully as a proof of concept within pip.

Significant updates to the PEP include:

* Switching the Epoch identifier from : to ! as : is not valid in a directory
  name on Windows.
* Local version identifiers use + as a seperator in order to reduce ambigiuty
  with existing versions on PyPI.
* Allow alpha numerics for local versions.
* Define a sorting algorithm for local versions.
* Moved the source label to PEP 426
* Normalization rules for parsing more versions following along the idea of
  Postel's Law.
* Declare that PEP 440 supercedes PEP 386 for Metadata 1.2, and also should be
  used for Metadata 1.0 and 1.1.
* Declare how invalid versions should be handled.
* Add the escape hatch "Arbitrary equality" operator === to allow depending on
  versions which cannot be parsed by PEP 440.
* Make specifier syntax match what setuptools uses (foo==1.0 instead of foo (==1.0))
* Remove the default specifier.
* Use @ for direct references instead of "from".
* Create a reference implementation and Link To it.
* Lots more minor changes.

The outcome of these changes is that we were able to raise compatability with
all the versions registered with PyPI up to 98.12% and we sort 99.88% of
projects registered with PyPI the same as pkg_resources does when filtering
invalid versions from the list of versions. The fallout is that 498 projects,
or 1.06%, are no longer installable without using the ``===`` operator and 190
projects, or 0.4%, of projects have a different result for what the "latest"
version is.

Of the 498 projects a number of them are nonsensical versions like ``.`` or
the repr of lazy objects and of the 190 projects projects it's about evenly
split between projects where pkg_resources supported something we didn't and
where PEP 440 just simply does a better job at parsing and sorting versions.

You can see the pip proof of concept and a large discussion about normalization
at https://github.com/pypa/pip/pull/1894 and the reference implementation can
be found at https://github.com/pypa/packaging/pull/1. The Proof of Concept does
not allow using the new specifiers inside of an install_requires in a source
distribution because setuptools itself does not support it, but it does support
them on the command line and in requirements.txt files.

The online view of the PEP can be found at
https://www.python.org/dev/peps/pep-0440/ and the changes since the last posting
can be found at http://hg.python.org/peps/rev/59a0d31a1bc2 and
http://hg.python.org/peps/rev/257822378672.

Without further ado, the PEP itself:

Abstract
========

This PEP describes a scheme for identifying versions of Python software
distributions, and declaring dependencies on particular versions.

This document addresses several limitations of the previous attempt at a
standardized approach to versioning, as described in PEP 345 and PEP 386.


Definitions
===========

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.

The following terms are to be interpreted as described in PEP 426:

* "Distributions"
* "Releases"
* "Build tools"
* "Index servers"
* "Publication tools"
* "Installation tools"
* "Automated tools"
* "Projects"


Version scheme
==============

Distributions are identified by a public version identifier which
supports all defined version comparison operations

The version scheme is used both to describe the distribution version
provided by a particular distribution archive, as well as to place
constraints on the version of dependencies needed in order to build or
run the software.


Public version identifiers
--------------------------

Public version identifiers MUST comply with the following scheme::

    [N!]N(.N)*[{a|b|c}N][.postN][.devN]

Public version identifiers MUST NOT include leading or trailing whitespace.

Public version identifiers MUST be unique within a given distribution.

Installation tools SHOULD ignore any public versions which do not comply with
this scheme. Installation tools MAY warn the user when non-compliant
or ambiguous versions are detected.

Public version identifiers are separated into up to five segments:

* Epoch segment: ``N!``
* Release segment: ``N(.N)*``
* Pre-release segment: ``{a|b|c}N``
* Post-release segment: ``.postN``
* Development release segment: ``.devN``

Any given release will be a "final release", "pre-release", "post-release" or
"developmental release" as defined in the following sections.

All numeric components MUST be non-negative integers.

All numeric components MUST be interpreted and ordered according to their
numeric value, not as text strings.

All numeric components MAY be zero. Except as described below for the
release segment, a numeric component of zero has no special significance
aside from always being the lowest possible value in the version ordering.

.. note::

   Some hard to read version identifiers are permitted by this scheme in
   order to better accommodate the wide range of versioning practices
   across existing public and private Python projects.

   Accordingly, some of the versioning practices which are technically
   permitted by the PEP are strongly discouraged for new projects. Where
   this is the case, the relevant details are noted in the following
   sections.


Local version identifiers
-------------------------

Local version identifiers MUST comply with the following scheme::

    <public version identifier>[+<local version label>]

They consist of a normal public version identifier (as defined in the
previous section), along with an arbitrary "local version label", separated
from the public version identifier by a plus. Local version labels have
no specific semantics assigned, but some syntactic restrictions are imposed.

Local version identifiers are used to denote fully API (and, if applicable,
ABI) compatible patched versions of upstream projects. For example, these
may be created by application developers and system integrators by applying
specific backported bug fixes when upgrading to a new upstream release would
be too disruptive to the application or other integrated system (such as a
Linux distribution).

The inclusion of the local version label makes it possible to differentiate
upstream releases from potentially altered rebuilds by downstream
integrators. The use of a local version identifier does not affect the kind
of a release but, when applied to a source distribution, does indicate that
it may not contain the exact same code as the corresponding upstream release.

To ensure local version identifiers can be readily incorporated as part of
filenames and URLs, and to avoid formatting inconsistencies in hexadecimal
hash representations, local version labels MUST be limited to the following
set of permitted characters:

* ASCII letters (``[a-zA-Z]``)
* ASCII digits (``[0-9]``)
* periods (``.``)

Local version labels MUST start and end with an ASCII letter or digit.

Comparison and ordering of local versions considers each segment of the local
version (divided by a ``.``) separately. If a segment consists entirely of
ASCII digits then that section should be considered an integer for comparison
purposes and if a segment contains any ASCII letters than that segment is
compared lexicographically with case insensitivity. When comparing a numeric
and lexicographic segment, the numeric section always compares as greater than
the lexicographic segment. Additionally a local version with a great number of
segments will always compare as greater than a local version with fewer
segments, as long as the shorter local version's segments match the beginning
of the longer local version's segments exactly.

Local version identifiers may be used in most locations where a public
version identifier is expected, with the exception of any version specifiers
that explicitly rely on being able to unambiguously order candidate versions.

Public index servers SHOULD NOT allow the use of local version identifiers
for uploaded distributions.

Source distributions using a local version identifier SHOULD provide the
``python.integrator`` extension metadata (as defined in :pep:`459`).


Final releases
--------------

A version identifier that consists solely of a release segment and optionally
an epoch identifier is termed a "final release".

The release segment consists of one or more non-negative integer
values, separated by dots::

    N[.N]+

Final releases within a project MUST be numbered in a consistently
increasing fashion, otherwise automated tools will not be able to upgrade
them correctly.

Comparison and ordering of release segments considers the numeric value
of each component of the release segment in turn. When comparing release
segments with different numbers of components, the shorter segment is
padded out with additional zeros as necessary.

While any number of additional components after the first are permitted
under this scheme, the most common variants are to use two components
("major.minor") or three components ("major.minor.micro").

For example::

    0.9
    0.9.1
    0.9.2
    ...
    0.9.10
    0.9.11
    1.0
    1.0.1
    1.1
    2.0
    2.0.1
    ...

A release series is any set of final release numbers that start with a
common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all
part of the ``3.3`` release series.

.. note::

   ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as
   the release segment comparison rules implicit expand the two component
   form to ``X.Y.0`` when comparing it to any release segment that includes
   three components.

Date based release segments are also permitted. An example of a date based
release scheme using the year and month of the release::

    2012.04
    2012.07
    2012.10
    2013.01
    2013.06
    ...


Pre-releases
------------

Some projects use an "alpha, beta, release candidate" pre-release cycle to
support testing by their users prior to a final release.

If used as part of a project's development cycle, these pre-releases are
indicated by including a pre-release segment in the version identifier::

    X.YaN  # Alpha release
    X.YbN  # Beta release
    X.YcN  # Candidate release
    X.Y    # Final release

A version identifier that consists solely of a release segment and a
pre-release segment is termed a "pre-release".

The pre-release segment consists of an alphabetical identifier for the
pre-release phase, along with a non-negative integer value. Pre-releases for
a given release are ordered first by phase (alpha, beta, release candidate)
and then by the numerical component within that phase.

Installation tools MAY accept both ``c`` and ``rc`` releases for a common
release segment in order to handle some existing legacy distributions.

Installation tools SHOULD interpret ``rc`` versions as being equivalent to
``c`` versions (that is, ``rc1`` indicates the same version as ``c1``).

Build tools, publication tools and index servers SHOULD disallow the creation
of both ``c`` and ``rc`` releases for a common release segment.


Post-releases
-------------

Some projects use post-releases to address minor errors in a final release
that do not affect the distributed software (for example, correcting an error
in the release notes).

If used as part of a project's development cycle, these post-releases are
indicated by including a post-release segment in the version identifier::

    X.Y.postN    # Post-release

A version identifier that includes a post-release segment without a
developmental release segment is termed a "post-release".

The post-release segment consists of the string ``.post``, followed by a
non-negative integer value. Post-releases are ordered by their
numerical component, immediately following the corresponding release,
and ahead of any subsequent release.

.. note::

   The use of post-releases to publish maintenance releases containing
   actual bug fixes is strongly discouraged. In general, it is better
   to use a longer release number and increment the final component
   for each maintenance release.

Post-releases are also permitted for pre-releases::

    X.YaN.postM  # Post-release of an alpha release
    X.YbN.postM  # Post-release of a beta release
    X.YcN.postM  # Post-release of a release candidate

.. note::

   Creating post-releases of pre-releases is strongly discouraged, as
   it makes the version identifier difficult to parse for human readers.
   In general, it is substantially clearer to simply create a new
   pre-release by incrementing the numeric component.


Developmental releases
----------------------

Some projects make regular developmental releases, and system packagers
(especially for Linux distributions) may wish to create early releases
directly from source control which do not conflict with later project
releases.

If used as part of a project's development cycle, these developmental
releases are indicated by including a developmental release segment in the
version identifier::

    X.Y.devN    # Developmental release

A version identifier that includes a developmental release segment is
termed a "developmental release".

The developmental release segment consists of the string ``.dev``,
followed by a non-negative integer value. Developmental releases are ordered
by their numerical component, immediately before the corresponding release
(and before any pre-releases with the same release segment), and following
any previous release (including any post-releases).

Developmental releases are also permitted for pre-releases and
post-releases::

    X.YaN.devM      # Developmental release of an alpha release
    X.YbN.devM      # Developmental release of a beta release
    X.YcN.devM      # Developmental release of a release candidate
    X.Y.postN.devM  # Developmental release of a post-release

.. note::

   Creating developmental releases of pre-releases is strongly
   discouraged, as it makes the version identifier difficult to parse for
   human readers. In general, it is substantially clearer to simply create
   additional pre-releases by incrementing the numeric component.

   Developmental releases of post-releases are also strongly discouraged,
   but they may be appropriate for projects which use the post-release
   notation for full maintenance releases which may include code changes.


Version epochs
--------------

If included in a version identifier, the epoch appears before all other
components, separated from the release segment by an exclamation mark::

    E!X.Y  # Version identifier with epoch

If no explicit epoch is given, the implicit epoch is ``0``.

Most version identifiers will not include an epoch, as an explicit epoch is
only needed if a project *changes* the way it handles version numbering in
a way that means the normal version ordering rules will give the wrong
answer. For example, if a project is using date based versions like
``2014.04`` and would like to switch to semantic versions like ``1.0``, then
the new releases would be identified as *older* than the date based releases
when using the normal sorting scheme::

    1.0
    1.1
    2.0
    2013.10
    2014.04

However, by specifying an explicit epoch, the sort order can be changed
appropriately, as all versions from a later epoch are sorted after versions
from an earlier epoch::

    2013.10
    2014.04
    1!1.0
    1!1.1
    1!2.0

Normalization
-------------

In order to maintain better compatibility with existing versions there are a
number of "alternative" syntaxes that MUST be taken into account when parsing
versions. These syntaxes MUST be considered when parsing a version, however
they should be "normalized" to the standard syntax defined above.


Case sensitivity
~~~~~~~~~~~~~~~~

All ascii letters should be interpreted case insensitively within a version and
the normal form is lowercase. This allows versions such as ``1.1RC1`` which
would be normalized to ``1.1c1``.


Integer Normalization
~~~~~~~~~~~~~~~~~~~~~

All integers are interpreted via the ``int()`` built in and normalize to the
string form of the output. This means that an integer version of ``00`` would
normalize to ``0`` while ``09000`` would normalize to ``9000``. This does not
hold true for integers inside of an alphanumeric segment of a local version
such as ``1.0+foo0100`` which is already in its normalized form.


Pre-release separators
~~~~~~~~~~~~~~~~~~~~~~

Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the
release segment and the pre-release segment. The normal form for this is
without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1``
which would be normalized to ``1.1a1``. It should also allow a seperator to
be used between the pre-release signifier and the numeral. This allows versions
such as ``1.0a.1`` which would be normalized to ``1.0a1``.


Pre-release spelling
~~~~~~~~~~~~~~~~~~~~

Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``,
``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively.
This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which
normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional
spelling should be considered equivalent to their normal forms.


Implicit pre-release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Pre releases allow omitting the numeral in which case it is implicitly assumed
to be ``0``. The normal form for this is to include the ``0`` explicitly. This
allows versions such as ``1.2a`` which is normalized to ``1.2a0``.


Post release separators
~~~~~~~~~~~~~~~~~~~~~~~

Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the
separator all together. The normal form of this is with the ``.`` separator.
This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to
``1.2.post2``. Like the pre-release seperator this also allows an optional
separator between the post release signifier and the numeral. This allows
versions like ``1.2.post-2`` which would normalize to ``1.2.post2``.


Post release spelling
~~~~~~~~~~~~~~~~~~~~~

Post-releases allow the additional spellings of ``rev`` and ``r``. This allows
versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the
pre-releases the additional spellings should be considered equivalent to their
normal forms.


Implicit post release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Post releases allow omiting the numeral in which case it is implicitly assumed
to be ``0``. The normal form for this is to include the ``0`` explicitly. This
allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``.


Implicit post releases
~~~~~~~~~~~~~~~~~~~~~~

Post releases allow omitting the ``post`` signifier all together. When using
this form the separator MUST be ``-`` and no other form is allowed. This allows
versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular
normalization MUST NOT be used in conjunction with the implicit post release
number rule. In other words ``1.0-`` is *not* a valid version and it does *not*
normalize to ``1.0.post0``.


Development release separators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Development releases allow a ``.``, ``-``, or a ``_`` separator as well as
omitting the separator all together. The normal form of this is with the ``.``
separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which
normalize to ``1.2.dev2``.


Implicit development release number
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Development releases allow omiting the numeral in which case it is implicitly
assumed to be ``0``. The normal form for this is to include the ``0``
explicitly. This allows versions such as ``1.2.dev`` which is normalized to
``1.2.dev0``.


Local version segments
~~~~~~~~~~~~~~~~~~~~~~

With a local version, in addition to the use of ``.`` as a separator of
segments, the use of ``-`` and ``_`` is also acceptable. The normal form is
using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be
normalized to ``1.0+ubuntu.1``.


Preceding v character
~~~~~~~~~~~~~~~~~~~~~

In order to support the common version notation of ``v1.0`` versions may be
preceded by a single literal ``v`` character. This character MUST be ignored
for all purposes and should be omitted from all normalized forms of the
version. The same version with and without the ``v`` is considered equivalent.


Leading and Trailing Whitespace
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Leading and trailing whitespace must be silently ignored and removed from all
normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``,
``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly,
such as a version like ``1.0\n`` which normalizes to ``1.0``.


Examples of compliant version schemes
-------------------------------------

The standard version scheme is designed to encompass a wide range of
identification practices across public and private Python projects. In
practice, a single project attempting to use the full flexibility offered
by the scheme would create a situation where human users had difficulty
figuring out the relative order of versions, even though the rules above
ensure all compliant tools will order them consistently.

The following examples illustrate a small selection of the different
approaches projects may choose to identify their releases, while still
ensuring that the "latest release" and the "latest stable release" can
be easily determined, both by human users and automated tools.

Simple "major.minor" versioning::

    0.1
    0.2
    0.3
    1.0
    1.1
    ...

Simple "major.minor.micro" versioning::

    1.1.0
    1.1.1
    1.1.2
    1.2.0
    ...

"major.minor" versioning with alpha, beta and candidate
pre-releases::

    0.9
    1.0a1
    1.0a2
    1.0b1
    1.0c1
    1.0
    1.1a1
    ...

"major.minor" versioning with developmental releases, release candidates
and post-releases for minor corrections::

    0.9
    1.0.dev1
    1.0.dev2
    1.0.dev3
    1.0.dev4
    1.0rc1
    1.0rc2
    1.0
    1.0.post1
    1.1.dev1
    ...

Date based releases, using an incrementing serial within each year, skipping
zero::

    2012.1
    2012.2
    2012.3
    ...
    2012.15
    2013.1
    2013.2
    ...


Summary of permitted suffixes and relative ordering
---------------------------------------------------

.. note::

   This section is intended primarily for authors of tools that
   automatically process distribution metadata, rather than developers
   of Python distributions deciding on a versioning scheme.

The epoch segment of version identifiers MUST be sorted according to the
numeric value of the given epoch. If no epoch segment is present, the
implicit numeric value is ``0``.

The release segment of version identifiers MUST be sorted in
the same order as Python's tuple sorting when the release segment is
parsed as follows::

    tuple(map(int, release_segment.split(".")))

All release segments involved in the comparison MUST be converted to a
consistent length by padding shorter segments with zeros as needed.

Within a numeric release (``1.0``, ``2.7.3``), the following suffixes
are permitted and MUST be ordered as shown::

   .devN, aN, bN, cN/rcN, <no suffix>, .postN

Note that `rc` is considered to be semantically equivalent to `c` and must be
sorted as if it were `c`. Tools MAY reject the case of having the same ``N``
for both a ``rc`` and a ``c`` in the same release segment as ambiguous and
remain in compliance with the PEP.

Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate
(``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be
ordered as shown::

   .devN, <no suffix>, .postN

Within a post-release (``1.0.post1``), the following suffixes are permitted
and MUST be ordered as shown::

    .devN, <no suffix>

Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even
when used immediately following a numeric version (e.g. ``1.0.dev456``,
``1.0.post1``).

Within a pre-release, post-release or development release segment with a
shared prefix, ordering MUST be by the value of the numeric component.

The following example covers many of the possible combinations::

    1.0.dev456
    1.0a1
    1.0a2.dev456
    1.0a12.dev456
    1.0a12
    1.0b1.dev456
    1.0b2
    1.0b2.post345.dev456
    1.0b2.post345
    1.0c1.dev456
    1.0c1
    1.0
    1.0+abc.5
    1.0+abc.7
    1.0+5
    1.0.post456.dev34
    1.0.post456
    1.1.dev1


Version ordering across different metadata versions
---------------------------------------------------

Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard
version identification or ordering scheme. However metadata v1.2 (PEP 345)
does specify a scheme which is defined in PEP 386.

Due to the nature of the simple installer API it is not possible for an
installer to be aware of which metadata version a particular distribution was
using. Additionally installers required the ability to create a reasonably
prioritized list that includes all, or as many as possible, versions of
a project to determine which versions it should install. These requirements
necessitate a standardization across one parsing mechanism to be used for all
versions of a project.

Due to the above, this PEP MUST be used for all versions of metadata and
supersedes PEP 386 even for metadata v1.2. Tools SHOULD ignore any versions
which cannot be parsed by the rules in this PEP, but MAY fall back to
implementation defined version parsing and ordering schemes if no versions
complying with this PEP are available.

Distribution users may wish to explicitly remove non-compliant versions from
any private package indexes they control.


Compatibility with other version schemes
----------------------------------------

Some projects may choose to use a version scheme which requires
translation in order to comply with the public version scheme defined in
this PEP. In such cases, the project specific version can be stored in the
metadata while the translated public version is published in the version field.

This allows automated distribution tools to provide consistently correct
ordering of published releases, while still allowing developers to use
the internal versioning scheme they prefer for their projects.


Semantic versioning
~~~~~~~~~~~~~~~~~~~

`Semantic versioning`_ is a popular version identification scheme that is
more prescriptive than this PEP regarding the significance of different
elements of a release number. Even if a project chooses not to abide by
the details of semantic versioning, the scheme is worth understanding as
it covers many of the issues that can arise when depending on other
distributions, and when publishing a distribution that others rely on.

The "Major.Minor.Patch" (described in this PEP as "major.minor.micro")
aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification)
are fully compatible with the version scheme defined in this PEP, and abiding
by these aspects is encouraged.

Semantic versions containing a hyphen (pre-releases - clause 10) or a
plus sign (builds - clause 11) are *not* compatible with this PEP
and are not permitted in the public version field.

One possible mechanism to translate such semantic versioning based source
labels to compatible public versions is to use the ``.devN`` suffix to
specify the appropriate version order.

Specific build information may also be included in local version labels.

.. _Semantic versioning: http://semver.org/


DVCS based version labels
~~~~~~~~~~~~~~~~~~~~~~~~~

Many build tools integrate with distributed version control systems like
Git and Mercurial in order to add an identifying hash to the version
identifier. As hashes cannot be ordered reliably such versions are not
permitted in the public version field.

As with semantic versioning, the public ``.devN`` suffix may be used to
uniquely identify such releases for publication, while the original DVCS based
label can be stored in the project metadata.

Identifying hash information may also be included in local version labels.


Olson database versioning
~~~~~~~~~~~~~~~~~~~~~~~~~

The ``pytz`` project inherits its versioning scheme from the corresponding
Olson timezone database versioning scheme: the year followed by a lowercase
character indicating the version of the database within that year.

This can be translated to a compliant public version identifier as
``<year>.<serial>``, where the serial starts at zero or one (for the
'<year>a' release) and is incremented with each subsequent database
update within the year.

As with other translated version identifiers, the corresponding Olson
database version could be recorded in the project metadata.


Version specifiers
==================

A version specifier consists of a series of version clauses, separated by
commas. For example::

   ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0

The comparison operator determines the kind of version clause:

* ``~=``: `Compatible release`_ clause
* ``==``: `Version matching`_ clause
* ``!=``: `Version exclusion`_ clause
* ``<=``, ``>=``: `Inclusive ordered comparison`_ clause
* ``<``, ``>``: `Exclusive ordered comparison`_ clause
* ``===``: `Arbitrary equality`_ clause.

The comma (",") is equivalent to a logical **and** operator: a candidate
version must match all given version clauses in order to match the
specifier as a whole.

Whitespace between a conditional operator and the following version
identifier is optional, as is the whitespace around the commas.

When multiple candidate versions match a version specifier, the preferred
version SHOULD be the latest version as determined by the consistent
ordering defined by the standard `Version scheme`_. Whether or not
pre-releases are considered as candidate versions SHOULD be handled as
described in `Handling of pre-releases`_.

Except where specifically noted below, local version identifiers MUST NOT be
permitted in version specifiers, and local version labels MUST be ignored
entirely when checking if candidate versions match a given version
specifier.


Compatible release
------------------

A compatible release clause consists of either a version identifier without
any comparison operator or else the compatible release operator ``~=``
and a version identifier. It matches any candidate version that is expected
to be compatible with the specified version.

The specified version identifier must be in the standard format described in
`Version scheme`_. Local version identifiers are NOT permitted in this
version specifier.

For a given release identifier ``V.N``, the compatible release clause is
approximately equivalent to the pair of comparison clauses::

    >= V.N, == V.*

This operator MUST NOT be used with a single segment version number such as
``~=1``.

For example, the following groups of version clauses are equivalent::

    2.2
    ~= 2.2
    >= 2.2, == 2.*

    1.4.5
    ~= 1.4.5
    >= 1.4.5, == 1.4.*

If a pre-release, post-release or developmental release is named in a
compatible release clause as ``V.N.suffix``, then the suffix is ignored
when determining the required prefix match::

    2.2.post3
    ~= 2.2.post3
    >= 2.2.post3, == 2.*

    1.4.5a4
    ~= 1.4.5a4
    >= 1.4.5a4, == 1.4.*

The padding rules for release segment comparisons means that the assumed
degree of forward compatibility in a compatible release clause can be
controlled by appending additional zeros to the version specifier::

    2.2.0
    ~= 2.2.0
    >= 2.2.0, == 2.2.*

    1.4.5.0
    ~= 1.4.5.0
    >= 1.4.5.0, == 1.4.5.*


Version matching
----------------

A version matching clause includes the version matching operator ``==``
and a version identifier.

The specified version identifier must be in the standard format described in
`Version scheme`_, but a trailing ``.*`` is permitted on public version
identifiers as described below.

By default, the version matching operator is based on a strict equality
comparison: the specified version must be exactly the same as the requested
version. The *only* substitution performed is the zero padding of the
release segment to ensure the release segments are compared with the same
length.

Whether or not strict version matching is appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when strict version matches are used
inappropriately.

Prefix matching may be requested instead of strict comparison, by appending
a trailing ``.*`` to the version identifier in the version matching clause.
This means that additional trailing segments will be ignored when
determining whether or not a version identifier matches the clause. If the
specified version includes only a release segment, than trailing components
(or the lack thereof) in the release segment are also ignored.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::

    == 1.1        # Not equal, so 1.1.post1 does not match clause
    == 1.1.post1  # Equal, so 1.1.post1 matches clause
    == 1.1.*      # Same prefix, so 1.1.post1 matches clause

For purposes of prefix matching, the pre-release segment is considered to
have an implied preceding ``.``, so given the version ``1.1a1``, the
following clauses would match or not as shown::

    == 1.1        # Not equal, so 1.1a1 does not match clause
    == 1.1a1      # Equal, so 1.1a1 matches clause
    == 1.1.*      # Same prefix, so 1.1a1 matches clause

An exact match is also considered a prefix match (this interpreation is
implied by the usual zero padding rules for the release segment of version
identifiers). Given the version ``1.1``, the following clauses would
match or not as shown::

    == 1.1        # Equal, so 1.1 matches clause
    == 1.1.0      # Zero padding expands 1.1 to 1.1.0, so it matches clause
    == 1.1.dev1   # Not equal (dev-release), so 1.1 does not match clause
    == 1.1a1      # Not equal (pre-release), so 1.1 does not match clause
    == 1.1.post1  # Not equal (post-release), so 1.1 does not match clause
    == 1.1.*      # Same prefix, so 1.1 matches clause

It is invalid to have a prefix match containing a development or local release
such as ``1.0.dev1.*`` or ``1.0+foo1.*``. If present, the development release
segment is always the final segment in the public version, and the local version
is ignored for comparison purposes, so using either in a prefix match wouldn't
make any sense.

The use of ``==`` (without at least the wildcard suffix) when defining
dependencies for published distributions is strongly discouraged as it
greatly complicates the deployment of security fixes. The strict version
comparison operator is intended primarily for use when defining
dependencies for repeatable *deployments of applications* while using
a shared distribution index.

If the specified version identifier is a public version identifier (no
local version label), then the local version label of any candidate versions
MUST be ignored when matching versions.

If the specified version identifier is a local version identifier, then the
local version labels of candidate versions MUST be considered when matching
versions, with the public version identifier being matched as described
above, and the local version label being checked for equivalence using a
strict string equality comparison.


Version exclusion
-----------------

A version exclusion clause includes the version exclusion operator ``!=``
and a version identifier.

The allowed version identifiers and comparison semantics are the same as
those of the `Version matching`_ operator, except that the sense of any
match is inverted.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::

    != 1.1        # Not equal, so 1.1.post1 matches clause
    != 1.1.post1  # Equal, so 1.1.post1 does not match clause
    != 1.1.*      # Same prefix, so 1.1.post1 does not match clause


Inclusive ordered comparison
----------------------------

An inclusive ordered comparison clause includes a comparison operator and a
version identifier, and will match any version where the comparison is correct
based on the relative position of the candidate version and the specified
version given the consistent ordering defined by the standard
`Version scheme`_.

The inclusive ordered comparison operators are ``<=`` and ``>=``.

As with version matching, the release segment is zero padded as necessary to
ensure the release segments are compared with the same length.

Local version identifiers are NOT permitted in this version specifier.


Exclusive ordered comparison
----------------------------

Exclusive ordered comparisons are similar to inclusive ordered comparisons,
except that the comparison operators are ``<`` and ``>`` and the clause
MUST be effectively interpreted as implying the prefix based version
exclusion clause ``!= V.*``.

The exclusive ordered comparison ``> V`` MUST NOT match a post-release
or maintenance release of the given version. Maintenance releases can be
permitted by using the clause ``> V.0``, while both post releases and
maintenance releases can be permitted by using the inclusive ordered
comparison ``>= V.post1``.

The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of
the given version, even if acceptance of pre-releases is enabled as
described in the section below.

Local version identifiers are NOT permitted in this version specifier.


Arbitrary equality
------------------

Arbitrary equality comparisons are simple string equality operations which do
not take into account any of the semantic information such as zero padding or
local versions. This operator also does not support prefix matching as the
``==`` operator does.

The primary use case for arbitrary equality is to allow for specifying a
version which cannot otherwise be represented by this PEP. This operator is
special and acts as an escape hatch to allow someone using a tool which
implements this PEP to still install a legacy version which is otherwise
incompatible with this PEP.

An example would be ``===foobar`` which would match a version of ``foobar``.

This operator may also be used to explicitly require an unpatched version
of a project such as ``===1.0`` which would not match for a version
``1.0+downstream1``.

Use of this operator is heavily discouraged and tooling MAY display a warning
when it is used.


Handling of pre-releases
------------------------

Pre-releases of any kind, including developmental releases, are implicitly
excluded from all version specifiers, *unless* they are already present
on the system, explicitly requested by the user, or if the only available
version that satisfies the version specifier is a pre-release.

By default, dependency resolution tools SHOULD:

* accept already installed pre-releases for all version specifiers
* accept remotely available pre-releases for version specifiers where
  there is no final or post release that satisfies the version specifier
* exclude all other pre-releases from consideration

Dependency resolution tools MAY issue a warning if a pre-release is needed
to satisfy a version specifier.

Dependency resolution tools SHOULD also allow users to request the
following alternative behaviours:

* accepting pre-releases for all version specifiers
* excluding pre-releases for all version specifiers (reporting an error or
  warning if a pre-release is already installed locally, or if a
  pre-release is the only way to satisfy a particular specifier)

Dependency resolution tools MAY also allow the above behaviour to be
controlled on a per-distribution basis.

Post-releases and final releases receive no special treatment in version
specifiers - they are always included unless explicitly excluded.


Examples
--------

* ``3.1``: version 3.1 or later, but not version 4.0 or later.
* ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later.
* ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later.
* ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases,
  post releases, developmental releases and any 3.1.x maintenance releases.
* ``== 3.1.*``: any version that starts with 3.1. Equivalent to the
  ``3.1.0`` compatible release clause.
* ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and
  not version 3.2.0 or later.


Direct references
=================

Some automated tools may permit the use of a direct reference as an
alternative to a normal version specifier. A direct reference consists of
the specifier ``@`` and an explicit URL.

Whether or not direct references are appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when direct references are used
inappropriately.

Public index servers SHOULD NOT allow the use of direct references in
uploaded distributions. Direct references are intended as a tool for
software integrators rather than publishers.

Depending on the use case, some appropriate targets for a direct URL
reference may be a valid ``source_url`` entry (see PEP 426), an sdist, or
a wheel binary archive. The exact URLs and targets supported will be tool
dependent.

For example, a local source archive may be referenced directly::

    pip @ file:///localbuilds/pip-1.3.1.zip<file:///\\localbuilds\pip-1.3.1.zip>

Alternatively, a prebuilt archive may also be referenced::

    pip @ file:///localbuilds/pip-1.3.1-py33-none-any.whl<file:///\\localbuilds\pip-1.3.1-py33-none-any.whl>

All direct references that do not refer to a local file URL SHOULD specify
a secure transport mechanism (such as ``https``) AND include an expected
hash value in the URL for verification purposes. If a direct reference is
specified without any hash information, with hash information that the
tool doesn't understand, or with a selected hash algorithm that the tool
considers too weak to trust, automated tools SHOULD at least emit a warning
and MAY refuse to rely on the URL. If such a direct reference also uses an
insecure transport, automated tools SHOULD NOT rely on the URL.

It is RECOMMENDED that only hashes which are unconditionally provided by
the latest version of the standard library's ``hashlib`` module be used
for source archive hashes. At time of writing, that list consists of
``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and
``'sha512'``.

For source archive and wheel references, an expected hash value may be
specified by including a ``<hash-algorithm>=<expected-hash>`` entry as
part of the URL fragment.

For version control references, the ``VCS+protocol`` scheme SHOULD be
used to identify both the version control system and the secure transport,
and a version control system with hash based commit identifiers SHOULD be
used. Automated tools MAY omit warnings about missing hashes for version
control systems that do not provide hash based commit identifiers.

To handle version control systems that do not support including commit or
tag references directly in the URL, that information may be appended to the
end of the URL using the ``@<commit-hash>`` or the ``@<tag>#<commit-hash>``
notation.

.. note::

   This isn't *quite* the same as the existing VCS reference notation
   supported by pip. Firstly, the distribution name is moved in front rather
   than embedded as part of the URL. Secondly, the commit hash is included
   even when retrieving based on a tag, in order to meet the requirement
   above that *every* link should include a hash to make things harder to
   forge (creating a malicious repo with a particular tag is easy, creating
   one with a specific *hash*, less so).

Remote URL examples::

    pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
    pip @ git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d
    pip @ git+https://github.com/pypa/pip.git at 1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d



Updating the versioning specification
=====================================

The versioning specification may be updated with clarifications without
requiring a new PEP or a change to the metadata version.

Actually changing the version comparison semantics still requires a new
versioning scheme and metadata version defined in new PEPs.


Summary of differences from \PEP 386
====================================

* Moved the description of version specifiers into the versioning PEP

* Added the "direct reference" concept as a standard notation for direct
  references to resources (rather than each tool needing to invent its own)

* Added the "local version identifier" and "local version label" concepts to
  allow system integrators to indicate patched builds in a way that is
  supported by the upstream tools, as well as to allow the incorporation of
  build tags into the versioning of binary distributions.

* Added the "compatible release" clause

* Added the trailing wildcard syntax for prefix based version matching
  and exclusion

* Changed the top level sort position of the ``.devN`` suffix

* Allowed single value version numbers

* Explicit exclusion of leading or trailing whitespace

* Explicit support for date based versions

* Explicit normalisation rules to improve compatibility with
  existing version metadata on PyPI where it doesn't introduce
  ambiguity

* Implicitly exclude pre-releases unless they're already present or
  needed to satisfy a dependency

* Treat post releases the same way as unqualified releases

* Discuss ordering and dependencies across metadata versions

The rationale for major changes is given in the following sections.


Changing the version scheme
---------------------------

One key change in the version scheme in this PEP relative to that in
PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
projects already using both development releases and alphas/betas/release
candidates do not want their developmental releases sorted in
between their release candidates and their final releases. There is no
rationale for using ``dev`` releases in that position rather than
merely creating additional release candidates.

The updated sort order also means the sorting of ``dev`` versions is now
consistent between the metadata standard and the pre-existing behaviour
of ``pkg_resources`` (and hence the behaviour of current installation
tools).

Making this change should make it easier for affected existing projects to
migrate to the latest version of the metadata standard.

Another change to the version scheme is to allow single number
versions, similar to those used by non-Python projects like Mozilla
Firefox, Google Chrome and the Fedora Linux distribution. This is actually
expected to be more useful for version specifiers, but it is easier to
allow it for both version specifiers and release numbers, rather than
splitting the two definitions.

The exclusion of leading and trailing whitespace was made explicit after
a couple of projects with version identifiers differing only in a
trailing ``\n`` character were found on PyPI.

Various other normalisation rules were also added as described in the
separate section on version normalisation below.

`Appendix A` shows detailed results of an analysis of PyPI distribution
version information, as collected on 8th August, 2014. This analysis
compares the behavior of the explicitly ordered version scheme defined in
this PEP with the de facto standard defined by the behavior of setuptools.
These metrics are useful, as the intent of this PEP is to follow existing
setuptools behavior as closely as is feasible, while still throwing
exceptions for unorderable versions (rather than trying to guess an
appropriate order as setuptools does).


A more opinionated description of the versioning scheme
-------------------------------------------------------

As in PEP 386, the primary focus is on codifying existing practices to make
them more amenable to automation, rather than demanding that existing
projects make non-trivial changes to their workflow. However, the
standard scheme allows significantly more flexibility than is needed
for the vast majority of simple Python packages (which often don't even
need maintenance releases - many users are happy with needing to upgrade to a
new feature release to get bug fixes).

For the benefit of novice developers, and for experienced developers
wishing to better understand the various use cases, the specification
now goes into much greater detail on the components of the defined
version scheme, including examples of how each component may be used
in practice.

The PEP also explicitly guides developers in the direction of
semantic versioning (without requiring it), and discourages the use of
several aspects of the full versioning scheme that have largely been
included in order to cover esoteric corner cases in the practices of
existing projects and in repackaging software for Linux distributions.


Describing version specifiers alongside the versioning scheme
-------------------------------------------------------------

The main reason to even have a standardised version scheme in the first place
is to make it easier to do reliable automated dependency analysis. It makes
more sense to describe the primary use case for version identifiers alongside
their definition.


Changing the interpretation of version specifiers
-------------------------------------------------

The previous interpretation of version specifiers made it very easy to
accidentally download a pre-release version of a dependency. This in
turn made it difficult for developers to publish pre-release versions
of software to the Python Package Index, as even marking the package as
hidden wasn't enough to keep automated tools from downloading it, and also
made it harder for users to obtain the test release manually through the
main PyPI web interface.

The previous interpretation also excluded post-releases from some version
specifiers for no adequately justified reason.

The updated interpretation is intended to make it difficult to accidentally
accept a pre-release version as satisfying a dependency, while still
allowing pre-release versions to be retrieved automatically when that's the
only way to satisfy a dependency.

The "some forward compatibility assumed" version constraint is derived from the
Ruby community's "pessimistic version constraint" operator [2]_ to allow
projects to take a cautious approach to forward compatibility promises, while
still easily setting a minimum required version for their dependencies. The
spelling of the compatible release clause (``~=``) is inspired by the Ruby
(``~>``) and PHP (``~``) equivalents.

Further improvements are also planned to the handling of parallel
installation of multiple versions of the same library, but these will
depend on updates to the installation database definition along with
improved tools for dynamic path manipulation.

The trailing wildcard syntax to request prefix based version matching was
added to make it possible to sensibly define both compatible release clauses
and the desired pre- and post-release handling semantics for ``<`` and ``>``
ordered comparison clauses.


Support for date based version identifiers
------------------------------------------

Excluding date based versions caused significant problems in migrating
``pytz`` to the new metadata standards. It also caused concerns for the
OpenStack developers, as they use a date based versioning scheme and would
like to be able to migrate to the new metadata standards without changing
it.


Adding version epochs
---------------------

Version epochs are added for the same reason they are part of other
versioning schemes, such as those of the Fedora and Debian Linux
distributions: to allow projects to gracefully change their approach to
numbering releases, without having a new release appear to have a lower
version number than previous releases and without having to change the name
of the project.

In particular, supporting version epochs allows a project that was previously
using date based versioning to switch to semantic versioning by specifying
a new version epoch.

The ``!`` character was chosen to delimit an epoch version rather than the
``:`` character, which is commonly used in other systems, due to the fact that
``:`` is not a valid character in a Windows directory name.


Adding direct references
------------------------

Direct references are added as an "escape clause" to handle messy real
world situations that don't map neatly to the standard distribution model.
This includes dependencies on unpublished software for internal use, as well
as handling the more complex compatibility issues that may arise when
wrapping third party libraries as C extensions (this is of especial concern
to the scientific community).

Index servers are deliberately given a lot of freedom to disallow direct
references, since they're intended primarily as a tool for integrators
rather than publishers. PyPI in particular is currently going through the
process of *eliminating* dependencies on external references, as unreliable
external services have the effect of slowing down installation operations,
as well as reducing PyPI's own apparent reliability.


Adding arbitrary equality
-------------------------

Arbitrary equality is added as an "escape clause" to handle the case where
someone needs to install a project which uses a non compliant version. Although
this PEP is able to attain ~97% compatibility with the versions that are
already on PyPI there are still ~3% of versions which cannot be parsed. This
operator gives a simple and effective way to still depend on them without
having to "guess" at the semantics of what they mean (which would be required
if anything other than strict string based equality was supported).


Adding local version identifiers
--------------------------------

It's a fact of life that downstream integrators often need to backport
upstream bug fixes to older versions. It's one of the services that gets
Linux distro vendors paid, and application developers may also apply patches
they need to bundled dependencies.

Historically, this practice has been invisible to cross-platform language
specific distribution tools - the reported "version" in the upstream
metadata is the same as for the unmodified code. This inaccuracy can then
cause problems when attempting to work with a mixture of integrator
provided code and unmodified upstream code, or even just attempting to
identify exactly which version of the software is installed.

The introduction of local version identifiers and "local version labels"
into the versioning scheme, with the corresponding ``python.integrator``
metadata extension allows this kind of activity to be represented
accurately, which should improve interoperability between the upstream
tools and various integrated platforms.

The exact scheme chosen is largely modeled on the existing behavior of
``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``,
with the main distinction being that where ``pkg_resources`` currently always
takes the suffix into account when comparing versions for exact matches,
the PEP requires that the local version label of the candidate version be
ignored when no local version label is present in the version specifier
clause. Furthermore, the PEP does not attempt to impose any structure on
the local version labels (aside from limiting the set of permitted
characters and defining their ordering).

This change is designed to ensure that an integrator provided version like
``pip 1.5+1`` or ``pip 1.5+1.git.abc123de`` will still satisfy a version
specifier like ``pip>=1.5``.

The plus is chosen primarily for readability of local version identifiers.
It was chosen instead of the hyphen to prevent
``pkg_resources.parse_version`` from parsing it as a prerelease, which is
important for enabling a successful migration to the new, more structured,
versioning scheme. The plus was chosen instead of a tilde because of the
significance of the tilde in Debian's version ordering algorithm.


Providing explicit version normalization rules
----------------------------------------------

Historically, the de facto standard for parsing versions in Python has been the
``pkg_resources.parse_version`` command from the setuptools project. This does
not attempt to reject *any* version and instead tries to make something
meaningful, with varying levels of success, out of whatever it is given. It has
a few simple rules but otherwise it more or less relies largely on string
comparison.

The normalization rules provided in this PEP exist primarily to either increase
the compatability with ``pkg_resources.parse_version``, particularly in
documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something
more reasonable with versions that already exist on PyPI.

All possible normalization rules were weighed against whether or not they were
*likely* to cause any ambiguity (e.g. while someone might devise a scheme where
``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone
actually doing that, much less on any scale that is noticeable, is fairly low).
They were also weighed against how ``pkg_resources.parse_version`` treated a
particular version string, especially with regards to how it was sorted. Finally
each rule was weighed against the kinds of additional versions it allowed, how
"ugly" those versions looked, how hard there were to parse (both mentally and
mechanically) and how much additional compatibility it would bring.

The breadth of possible normalizations were kept to things that could easily
be implemented as part of the parsing of the version and not pre-parsing
transformations applied to the versions. This was done to limit the side
effects of each transformation as simple search and replace style transforms
increase the likelihood of ambiguous or "junk" versions.

For an extended discussion on the various types of normalizations that were
considered, please see the proof of concept for PEP 440 within pip [4]_.


References
==========

The initial attempt at a standardised version scheme, along with the
justifications for needing such a standard can be found in PEP 386.

.. [1] Reference Implementation of PEP 440 Versions and Specifiers
   https://github.com/pypa/packaging/pull/1

.. [2] Version compatibility analysis script:
   https://github.com/pypa/packaging/blob/master/tasks/check.py

.. [3] Pessimistic version constraint
   http://docs.rubygems.org/read/chapter/16

.. [4] Proof of Concept: PEP 440 within pip
    https://github.com/pypa/pip/pull/1894


Appendix A
==========

Metadata v2.0 guidelines versus setuptools::

    $ invoke check.pep440
    Total Version Compatibility:              245806/250521 (98.12%)
    Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%)
    Total Sorting Compatibility (Filtered):   47057/47114 (99.88%)
    Projects with No Compatible Versions:     498/47114 (1.06%)
    Projects with Differing Latest Version:   688/47114 (1.46%)


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140816/d15a1214/attachment-0001.html>

From agroszer at gmail.com  Fri Aug 22 09:02:57 2014
From: agroszer at gmail.com (Adam GROSZER)
Date: Fri, 22 Aug 2014 09:02:57 +0200
Subject: [Distutils] PyPI CDN problem
Message-ID: <53F6EB21.3080900@gmail.com>

Hi,

On some mornings (today too) I get such errors when trying to access any 
package on PyPI:

Error 503 backend read error

backend read error

Guru Mediation:

Details: cache-iad2123-IAD 1408690502 4046817828

Varnish cache server

-- 
Best regards,
  Adam GROSZER
--
Quote of the day:
Rainy days and Mondays always get me down.

From richard at python.org  Fri Aug 22 09:04:43 2014
From: richard at python.org (Richard Jones)
Date: Fri, 22 Aug 2014 17:04:43 +1000
Subject: [Distutils] PyPI CDN problem
In-Reply-To: <53F6EB21.3080900@gmail.com>
References: <53F6EB21.3080900@gmail.com>
Message-ID: <CAHrZfZBCK8Gj32hWsTdQHUj8f=B-=9LAFkFeNAdS_MKiuTFGxA@mail.gmail.com>

Try clearing your cookies for PyPI and see if that helps. It has helped me
in the past.


On 22 August 2014 17:02, Adam GROSZER <agroszer at gmail.com> wrote:

> Hi,
>
> On some mornings (today too) I get such errors when trying to access any
> package on PyPI:
>
> Error 503 backend read error
>
> backend read error
>
> Guru Mediation:
>
> Details: cache-iad2123-IAD 1408690502 4046817828
>
> Varnish cache server
>
> --
> Best regards,
>  Adam GROSZER
> --
> Quote of the day:
> Rainy days and Mondays always get me down.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140822/49e5297b/attachment.html>

From agroszer at gmail.com  Fri Aug 22 09:22:00 2014
From: agroszer at gmail.com (Adam GROSZER)
Date: Fri, 22 Aug 2014 09:22:00 +0200
Subject: [Distutils] PyPI CDN problem
In-Reply-To: <CAHrZfZBCK8Gj32hWsTdQHUj8f=B-=9LAFkFeNAdS_MKiuTFGxA@mail.gmail.com>
References: <53F6EB21.3080900@gmail.com>
 <CAHrZfZBCK8Gj32hWsTdQHUj8f=B-=9LAFkFeNAdS_MKiuTFGxA@mail.gmail.com>
Message-ID: <53F6EF98.4050103@gmail.com>

Yep, that helped. Thanks.

Tho... why-oh-why?


On 08/22/2014 09:04 AM, Richard Jones wrote:
> Try clearing your cookies for PyPI and see if that helps. It has helped
> me in the past.
>
>
> On 22 August 2014 17:02, Adam GROSZER <agroszer at gmail.com
> <mailto:agroszer at gmail.com>> wrote:
>
>     Hi,
>
>     On some mornings (today too) I get such errors when trying to access
>     any package on PyPI:
>
>     Error 503 backend read error
>
>     backend read error
>
>     Guru Mediation:
>
>     Details: cache-iad2123-IAD 1408690502 4046817828
>
>     Varnish cache server
>
>     --
>     Best regards,
>       Adam GROSZER
>     --
>     Quote of the day:
>     Rainy days and Mondays always get me down.
>     _________________________________________________
>     Distutils-SIG maillist  - Distutils-SIG at python.org
>     <mailto:Distutils-SIG at python.org>
>     https://mail.python.org/__mailman/listinfo/distutils-sig
>     <https://mail.python.org/mailman/listinfo/distutils-sig>
>
>

-- 
Best regards,
  Adam GROSZER
--
Quote of the day:
Time is an illusion perpetrated by the manufacturers of space.  -  Graffiti

From ncoghlan at gmail.com  Fri Aug 22 14:34:39 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Aug 2014 22:34:39 +1000
Subject: [Distutils] Accepting PEP 440: Version Identification and
	Dependency Specification
Message-ID: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>

I just pushed Donald's final round of edits in response to the
feedback on the last PEP 440 thread, and as such I'm happy to announce
that I am accepting PEP 440 as the recommended approach to identifying
versions and specifying dependencies when distributing Python
software.

The PEP is available in the usual place at
http://www.python.org/dev/peps/pep-0440/

It's been a long road to get to an implementation independent
versioning standard that has a feasible migration path from the
current pkg_resources defined de facto standard, and I'd like to thank
a few folks:

* Donald Stufft for his extensive work on PEP 440 itself, especially
the proof of concept integration into pip
* Vinay Sajip for his efforts in validating earlier versions of the PEP
* Tarek Ziad? for starting us down the road to an implementation
independent versioning standard with the initial creation of PEP 386
back in June 2009, more than five years ago!

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From richard at python.org  Fri Aug 22 15:09:57 2014
From: richard at python.org (Richard Jones)
Date: Fri, 22 Aug 2014 23:09:57 +1000
Subject: [Distutils] Accepting PEP 440: Version Identification and
 Dependency Specification
In-Reply-To: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>
References: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>
Message-ID: <CAHrZfZDJqhoW1W4Q40WaMsvZA_t1SYrWTdXYwyPwzxaEmbyejg@mail.gmail.com>

Wow, a huge thanks to everyone named (as well as you, Nick ;) for
persevering and getting this through.


On 22 August 2014 22:34, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I just pushed Donald's final round of edits in response to the
> feedback on the last PEP 440 thread, and as such I'm happy to announce
> that I am accepting PEP 440 as the recommended approach to identifying
> versions and specifying dependencies when distributing Python
> software.
>
> The PEP is available in the usual place at
> http://www.python.org/dev/peps/pep-0440/
>
> It's been a long road to get to an implementation independent
> versioning standard that has a feasible migration path from the
> current pkg_resources defined de facto standard, and I'd like to thank
> a few folks:
>
> * Donald Stufft for his extensive work on PEP 440 itself, especially
> the proof of concept integration into pip
> * Vinay Sajip for his efforts in validating earlier versions of the PEP
> * Tarek Ziad? for starting us down the road to an implementation
> independent versioning standard with the initial creation of PEP 386
> back in June 2009, more than five years ago!
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140822/efb282df/attachment.html>

From david.genest at ubisoft.com  Fri Aug 22 16:07:18 2014
From: david.genest at ubisoft.com (David Genest)
Date: Fri, 22 Aug 2014 14:07:18 +0000
Subject: [Distutils] windows permissions on bdist egg scripts, data?
In-Reply-To: <CFC0BCE0.26182%erik.purins@imgtec.com>
References: <CFC0BCE0.26182%erik.purins@imgtec.com>
Message-ID: <f27956accd7d4d5c892804a86da8b2bf@MSR-MAIL-EXCH01.ubisoft.org>


> Hey there,

> I'm using python 2.7/setuptools 3.6 to build an egg of my code (checked into perforce) using my setup.py. This egg gets installed into a virtualenv.

> On osx/darwin this works fine.

> On windows, the bdist creates scripts and data whose permissions are read-only (attrib+r), and they cannot overwrite themselves in a subsequent build of that egg. Similarly, >re-installing a new egg using virtualenv fails because the installed egg's scripts and data are not writable.

> I thought it might be that when checked out from perforce, the permissions stayed read-only during the build of the egg, so I set all the files in my module to be read-write before building. Even when the original source files' permissions are all set to read-write, the generated bdist egg script/data content end up becoming read-only.

Yes, the default copy_file implementation from distutils keeps attributes on
copy (preserve_mode parameter is non zero). In our setup.py we monkey patch
the copy_file to get around this situation

> Couple of questions.
> 1. Since it works fine on osx, it seems like a bug in the windows implementation of setuptools. Is this known behavior, is there a reason for setting this on scripts/data on windows?

This is not a setuptools problem, but a (probably wrong) choice of defaults in
distutils

> 2. Is there a hook in the bdist egg build process I can use to set the attributes of the files correctly when the bdist egg is made? How/where should I define this in my setup.py?
Kind regards,

In your setup.py, use a monkey patch like the one below:

#monkey-patch the copy_file function beacause it has the wrong default value for preserve_mode
from distutils.file_util import copy_file as old_copy_file
def non_preserve_copy_file(src, dst, preserve_mode=0, preserve_times=1, update=0,
                           link=None, verbose=1, dry_run=0):
    return old_copy_file(src, dst, preserve_mode=0, preserve_times=preserve_times, update=update, link=link, verbose=verbose, dry_run=dry_run)

distutils.file_util.__dict__['copy_file'] = non_preserve_copy_file
#end monkey patching

Regards, 
David.

From antoine at python.org  Sun Aug 24 16:31:42 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 14:31:42 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?Accepting_PEP_440=3A_Version_Identification?=
	=?utf-8?q?_and=09Dependency_Specification?=
References: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>
Message-ID: <loom.20140824T163029-616@post.gmane.org>


Hi,

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> I just pushed Donald's final round of edits in response to the
> feedback on the last PEP 440 thread, and as such I'm happy to announce
> that I am accepting PEP 440 as the recommended approach to identifying
> versions and specifying dependencies when distributing Python
> software.

It would be nice if such accepted PEPs were also announced on python-dev.
Always good to keep people informed :-)

cheers (and congrats for the PEP)

Antoine.



From Erik.Purins at imgtec.com  Mon Aug 25 18:28:56 2014
From: Erik.Purins at imgtec.com (Erik Purins)
Date: Mon, 25 Aug 2014 16:28:56 +0000
Subject: [Distutils] windows permissions on bdist egg scripts, data?
In-Reply-To: <f27956accd7d4d5c892804a86da8b2bf@MSR-MAIL-EXCH01.ubisoft.org>
Message-ID: <D01F7BCF.288C9%erik.purins@imgtec.com>

Kind thanks for your pointing me to the problem area, and providing an example fix!

I?ll give this a shot tomorrow.

-e

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140825/954af175/attachment.html>

From techtonik at gmail.com  Tue Aug 26 11:25:03 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 26 Aug 2014 12:25:03 +0300
Subject: [Distutils] Accepting PEP 440: Version Identification and
 Dependency Specification
In-Reply-To: <CAHrZfZDJqhoW1W4Q40WaMsvZA_t1SYrWTdXYwyPwzxaEmbyejg@mail.gmail.com>
References: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>
 <CAHrZfZDJqhoW1W4Q40WaMsvZA_t1SYrWTdXYwyPwzxaEmbyejg@mail.gmail.com>
Message-ID: <CAPkN8xJURfh7DtKMjijz6SD=pQ1bygxhd91epwEWoh5vb_DNaQ@mail.gmail.com>

It is not compatible with
http://legacy.python.org/dev/peps/pep-0440/#semantic-versioning
Does that mean that packages that choose this way of versioning will
not be supported by
Python tools anymore?

On Fri, Aug 22, 2014 at 4:09 PM, Richard Jones <richard at python.org> wrote:
> Wow, a huge thanks to everyone named (as well as you, Nick ;) for
> persevering and getting this through.
>
>
> On 22 August 2014 22:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> I just pushed Donald's final round of edits in response to the
>> feedback on the last PEP 440 thread, and as such I'm happy to announce
>> that I am accepting PEP 440 as the recommended approach to identifying
>> versions and specifying dependencies when distributing Python
>> software.
>>
>> The PEP is available in the usual place at
>> http://www.python.org/dev/peps/pep-0440/
>>
>> It's been a long road to get to an implementation independent
>> versioning standard that has a feasible migration path from the
>> current pkg_resources defined de facto standard, and I'd like to thank
>> a few folks:
>>
>> * Donald Stufft for his extensive work on PEP 440 itself, especially
>> the proof of concept integration into pip
>> * Vinay Sajip for his efforts in validating earlier versions of the PEP
>> * Tarek Ziad? for starting us down the road to an implementation
>> independent versioning standard with the initial creation of PEP 386
>> back in June 2009, more than five years ago!
>>
>> Regards,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
anatoly t.

From donald at stufft.io  Tue Aug 26 13:32:53 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Aug 2014 07:32:53 -0400
Subject: [Distutils] Accepting PEP 440: Version Identification and
	Dependency Specification
In-Reply-To: <CAPkN8xJURfh7DtKMjijz6SD=pQ1bygxhd91epwEWoh5vb_DNaQ@mail.gmail.com>
References: <CADiSq7eXATd2hzC8=3QO6vgGMmUXKzcSqzPGqDQT1O1MJ+j-kw@mail.gmail.com>
 <CAHrZfZDJqhoW1W4Q40WaMsvZA_t1SYrWTdXYwyPwzxaEmbyejg@mail.gmail.com>
 <CAPkN8xJURfh7DtKMjijz6SD=pQ1bygxhd91epwEWoh5vb_DNaQ@mail.gmail.com>
Message-ID: <9F50D8BE-B312-4EC4-8617-84D502D12DB8@stufft.io>


> On Aug 26, 2014, at 5:25 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> It is not compatible with
> http://legacy.python.org/dev/peps/pep-0440/#semantic-versioning
> Does that mean that packages that choose this way of versioning will
> not be supported by
> Python tools anymore?


I?m not sure what this is saying, what part of PEP 440 isn?t compatible with that section you linked?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140826/68926c05/attachment.html>

From jerome.fuselier at free.fr  Tue Aug 26 23:54:54 2014
From: jerome.fuselier at free.fr (Jerome Fuselier)
Date: Tue, 26 Aug 2014 23:54:54 +0200
Subject: [Distutils] Undefined symbol when loading a library using dlopen
Message-ID: <53FD022E.4070100@free.fr>

Hello,

I'm trying to wrap a C++ library which uses dlopen to load a dynamic 
library and I don't manage to get rid of an "undefined symbol" error.

I tried to simulate my problem with a simpler example so hopefully 
someone may be able to tell me what I'm doing wrong.

I have a file func.cpp which provides 2 functions :

func.cpp:

int simple(void) {
     return 12;
}

int call_lib(void) {
     int res;
     void *handle = dlopen("/home/jerome/Test/libDyn.so", RTLD_NOW);

     if (!handle) {
         return -1;
     }
     dlerror();
     char* err = 0;
     int (*fun_dyn)() = (int (*)())dlsym(handle, "fun_dyn");
     if ( (err = dlerror()) != 0 ) {
         dlclose(handle);
         return -1;
     }
     res = (*fun_dyn)();
     dlclose(handle);
     return res;
}

It is created like that:
$ g++ -Wall -g -fPIC -c -o func.o func.cpp
$ ar rcs libFun.a func.o

I have a dynamic library which is called from call_lib() which depends 
on libFun.a:

libDyn.cpp:

#include "func.hpp"

int fun_dyn(void) {
     return simple();
}

It is created like that:
$ g++ -Wall -g -fPIC -c -o libDyn.o libDyn.cpp
$ g++ -Wall -g -fPIC -shared -o libDyn.so libDyn.o

I'm using swig to wrap the libFun.a library with this mylib.i interface 
file:

%module mylib

int simple(void);
int call_lib(void);

Then I'm using this setup.py file to create the Python module:

from distutils.core import setup, Extension

mylib = Extension("_mylib",
                   sources=['mylib.i'],
                   language="c++",
                   extra_link_args=['-rdynamic'],
                   libraries=["Fun", "dl"],
                  )

setup(name="MyLib",
       platforms=["Linux"],
       ext_modules=[mylib],
       py_modules = ["mylib"])


It's a simplified example but when I compile everything I get this error 
in Python:
 >>> import _mylib
 >>> _mylib.call_lib()
failed to open shared object file
/home/jerome/Test/libDyn.so: undefined symbol: simple

If I do a nm _mylib.so I see that the symbol simple is present in the 
library so I'm wondering why dlopen doesn't see it when it tries to load 
the library.

I know I can recompile libDyn.so to link libFun.a but this would means I 
would need to recompile all the plugins provided for the library I'm 
trying to wrap and that's not really a solution for my project.

Sorry for the long email. I can provide a tar.gz of the source code if 
needed.

Thanks for your help
Jerome


From p.f.moore at gmail.com  Wed Aug 27 08:50:27 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Aug 2014 07:50:27 +0100
Subject: [Distutils] Packages that have problems being installed from wheels
Message-ID: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>

tl; dr; If you know of a project that can't be successfully installed
with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
me know the details?

One of the longer-term goals of the introduction of wheels was to
split the build and install steps for a package - specifically, in
pip, to restrict "pip install" to installing from wheels, and when no
wheel is available, to (transparently) run "pip wheel" on the sdist
followed by "pip install on the generated wheel". During a discussion
yesterday, I realised that I don't know how close we are to that goal.
So I'm looking for information on packages which don't install
properly from wheels at the moment. If anyone has examples of packages
where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip
install /tmp/xxx/*.whl" does not result in an equivalent install,
could they post them here with details of how & why they fail?

Things I already know about (but would like specific examples):

1. Post-install steps included in setup.py. That should be covered by
the support in Metadata 2.0. I'd also be interested in how much of an
issue omitting the postinstall would be in practice (for instance many
such steps just set up "Start Menu" type shortcuts, which are not
essential for the package to be usable).
2. Actually, that's the only one :-)

Things that should not be a problem (but might be):

1. Numpy (and the scipy stack) need better tagging facilities for
wheels - but that wouldn't matter for a wheel that's built, used, then
thrown away.
2. Some things are complex to build - but I don't know of any cases
where building a wheel is *more* complex than installing, and I don't
see how it could be, in theory.
3. Projects that customise setup.py so much that they aren't
compatible with setuptools and bdist_wheel. Such projects are quite
probably already incompatible with pip (which injects setuptools when
running setup.py anyway) and so not relevant for this discussion. But
if any do work with "pip install" but not "pip wheel", I'd like to
hear about them.

Thanks in advance for any information.
Paul

From donald at stufft.io  Wed Aug 27 09:29:20 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Aug 2014 03:29:20 -0400
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
Message-ID: <2EC78938-FA64-4B1C-A6D5-FBD9141E00FE@stufft.io>


> On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> tl; dr; If you know of a project that can't be successfully installed
> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
> me know the details?
> 
> One of the longer-term goals of the introduction of wheels was to
> split the build and install steps for a package - specifically, in
> pip, to restrict "pip install" to installing from wheels, and when no
> wheel is available, to (transparently) run "pip wheel" on the sdist
> followed by "pip install on the generated wheel". During a discussion
> yesterday, I realised that I don't know how close we are to that goal.
> So I'm looking for information on packages which don't install
> properly from wheels at the moment. If anyone has examples of packages
> where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip
> install /tmp/xxx/*.whl" does not result in an equivalent install,
> could they post them here with details of how & why they fail?
> 
> Things I already know about (but would like specific examples):
> 
> 1. Post-install steps included in setup.py. That should be covered by
> the support in Metadata 2.0. I'd also be interested in how much of an
> issue omitting the postinstall would be in practice (for instance many
> such steps just set up "Start Menu" type shortcuts, which are not
> essential for the package to be usable).
> 2. Actually, that's the only one :-)
> 
> Things that should not be a problem (but might be):
> 
> 1. Numpy (and the scipy stack) need better tagging facilities for
> wheels - but that wouldn't matter for a wheel that's built, used, then
> thrown away.
> 2. Some things are complex to build - but I don't know of any cases
> where building a wheel is *more* complex than installing, and I don't
> see how it could be, in theory.
> 3. Projects that customise setup.py so much that they aren't
> compatible with setuptools and bdist_wheel. Such projects are quite
> probably already incompatible with pip (which injects setuptools when
> running setup.py anyway) and so not relevant for this discussion. But
> if any do work with "pip install" but not "pip wheel", I'd like to
> hear about them.
> 
> Thanks in advance for any information.
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Numpy (and other SciPy stack) has the problem too that it uses numpy.distutils
so I'm not even sure if ``pip wheel`` actually works on it at all or not. They
might have their own support for ``setup.py bdist_wheel`` though I don't know.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/a47734f0/attachment-0001.html>

From aragilar at gmail.com  Wed Aug 27 10:44:42 2014
From: aragilar at gmail.com (James Tocknell)
Date: Wed, 27 Aug 2014 18:44:42 +1000
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <2EC78938-FA64-4B1C-A6D5-FBD9141E00FE@stufft.io>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <2EC78938-FA64-4B1C-A6D5-FBD9141E00FE@stufft.io>
Message-ID: <CABNXP=TRzr47ARqrgESd4R9C_XvWc93MC2sVyaWUveWfGDC5ng@mail.gmail.com>

Scipy (the package) doesn't work due to the build requirement on numpy (and
the lack of metadata stating this), however some of the others have
included https://bitbucket.org/dholth/setup-requires (or something
equivalent) to install using numpy. Effectively, if the package can be pip
installed, then wheeling should work (I created a script a while ago which
uses distlib to modify scipy wheels such that they contain the correct
metadata, and I haven't run into any issues). Matplotlib works, however
some of the graphics toolkits aren't wheelable (e.g. pygtk and pyqt).

One case that I know of where wheels do not replace a straight install is
PySide (there needs to be rpath modification), however the PySide
developers are aware of this and in their installation instructions they
explicitly mention to run a post install script if installing from a wheel
(The post install probably could be got rid of if $ORIGIN was used, but
that would involve digging into the cmake install, and checking that it
doesn't break anything. Also, I'm not sure how widely $ORIGIN is
supported). I'd guess that any package which creates shared libraries which
link to each other would have this problem also, though how many of those
are pip installable is probably almost none.

James


On 27 August 2014 17:29, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> tl; dr; If you know of a project that can't be successfully installed
> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
> me know the details?
>
> One of the longer-term goals of the introduction of wheels was to
> split the build and install steps for a package - specifically, in
> pip, to restrict "pip install" to installing from wheels, and when no
> wheel is available, to (transparently) run "pip wheel" on the sdist
> followed by "pip install on the generated wheel". During a discussion
> yesterday, I realised that I don't know how close we are to that goal.
> So I'm looking for information on packages which don't install
> properly from wheels at the moment. If anyone has examples of packages
> where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip
> install /tmp/xxx/*.whl" does not result in an equivalent install,
> could they post them here with details of how & why they fail?
>
> Things I already know about (but would like specific examples):
>
> 1. Post-install steps included in setup.py. That should be covered by
> the support in Metadata 2.0. I'd also be interested in how much of an
> issue omitting the postinstall would be in practice (for instance many
> such steps just set up "Start Menu" type shortcuts, which are not
> essential for the package to be usable).
> 2. Actually, that's the only one :-)
>
> Things that should not be a problem (but might be):
>
> 1. Numpy (and the scipy stack) need better tagging facilities for
> wheels - but that wouldn't matter for a wheel that's built, used, then
> thrown away.
> 2. Some things are complex to build - but I don't know of any cases
> where building a wheel is *more* complex than installing, and I don't
> see how it could be, in theory.
> 3. Projects that customise setup.py so much that they aren't
> compatible with setuptools and bdist_wheel. Such projects are quite
> probably already incompatible with pip (which injects setuptools when
> running setup.py anyway) and so not relevant for this discussion. But
> if any do work with "pip install" but not "pip wheel", I'd like to
> hear about them.
>
> Thanks in advance for any information.
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> Numpy (and other SciPy stack) has the problem too that it uses
> numpy.distutils
> so I'm not even sure if ``pip wheel`` actually works on it at all or not.
> They
> might have their own support for ``setup.py bdist_wheel`` though I don't
> know.
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Don't send me files in proprietary formats (.doc(x), .xls, .ppt etc.). It
isn't good enough for Tim Berners-Lee
<http://opendotdotdot.blogspot.com/2010/04/rms-and-tim-berners-lee-separated-at.html>,
and it isn't good enough for me either. For more information visit
http://www.gnu.org/philosophy/no-word-attachments.html.

Truly great madness cannot be achieved without significant intelligence.
 - Henrik Tikkanen

If you're not messing with your sanity, you're not having fun.
 - James Tocknell

In theory, there is no difference between theory and practice; In practice,
there is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/514cb17b/attachment.html>

From skip at pobox.com  Wed Aug 27 17:34:52 2014
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 27 Aug 2014 10:34:52 -0500
Subject: [Distutils] Pip enhancement?
Message-ID: <CANc-5Uz=+_M5u3EMR-xfAdd-hX6ue2ytTZ8G+G2irrLcjhfrvw@mail.gmail.com>

Over on python-dev, Neal Becker asked:

> On systems where os-level packaging is available (e.g., fedora
> linux), it is not unusual to want a newer python package installed
> than available from the vendor.  pip install --user can be used for
> this.

> But then there is the danger that these pip installed packages are
> not maintained.

> At least, pip should have the ability to alert the user to potential
> updates,

> pip update

> could list which packages need updating, and offer to perform the
> update.  I think this would go a long way to helping with this
> problem.

I responded:

> How? I have exactly this problem with nose. We actually get it
> bundled (currently at ancient 1.1.2, trying to get to 1.3.4) with a
> bunch of other open source software from an outside packaging
> company, and even though I add the --user flag, it still complains
> that a version is already installed. When I add the --upgrade flag
> it tries to uninstall the global version.

> then in a follow-up reply to Paul Moore's reply:

> On Wed, Aug 27, 2014 at 8:24 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > Do you mean something like "pip list --outdated"?

> I was unaware of that command, as we were stuck at pip 1.2.1. I just
> updated pip manually to 1.5.6. That is a very helpful command. It
> would be even better if it understood --user so it could restrict
> it's view to user-installed stuff.

> Also, given that packages can be found in multiple places on a
> system, for me:

> * the OpenSuSE system packages
> * TWW-provided system-wide packages
> * our own system-wide packages in /opt/local
> * my private stuff in ~/.local

> it would be great if there was a way for it to tell me where on my
> system it found outdated package X. The --verbose flag tells me all
> sorts of other stuff I'm not really interested in, but not the
> installed location of the outdated package.

Paul also added:

> On 27 August 2014 14:46, Skip Montanaro <skip at pobox.com> wrote:
> > it would be great if there was a way for it to tell me where on my
> > system it found outdated package X. The --verbose flag tells me
> > all sorts of other stuff I'm not really interested in, but not the
> > installed location of the outdated package.

> There's also packaged environments like conda. It would be nice if
> pip could distinguish between conda-managed packages and ones I
> installed myself.

> Really, though, this is what the PEP 376 "INSTALLER" file was
> intended for. As far as I know, though, it was never implemented
> (and you'd also need to persuade the Linux vendors, the conda
> people, etc, to use it as well if it were to be of any practical
> use).

> Agreed about reporting the installed location, though. Specific
> suggestions like this would be good things to add to the pip issue
> tracker.

The entire thread is here:

https://mail.python.org/pipermail/python-dev/2014-August/136004.html

So, think of this as a multi-faceted feature request for pip. :-)

Thx,

Skip

From david.genest at ubisoft.com  Wed Aug 27 17:42:21 2014
From: david.genest at ubisoft.com (David Genest)
Date: Wed, 27 Aug 2014 15:42:21 +0000
Subject: [Distutils] Packages that have problems being installed from
 wheels
In-Reply-To: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
Message-ID: <1bee9d0996924d94b8d114aa0cc1dd1e@MSR-MAIL-EXCH01.ubisoft.org>

 
> 1. Post-install steps included in setup.py. That should be covered by the
> support in Metadata 2.0. I'd also be interested in how much of an issue
> omitting the postinstall would be in practice (for instance many such steps
> just set up "Start Menu" type shortcuts, which are not essential for the
> package to be usable).

On windows, we use wheel convert to convert binary installers like pywin32 to wheels. But the post-install step in the setup is mandatory  for pywin32 to work (it is not a mere Start Menu customization).

So this is what we do:
wheel convert  pywin32-217.win32-py2.7.exe
pip install pywin32-217-cp27-none-win32.whl

In this case pywin32_postinstall.py is not run and we need to run it manually.

I do not know the specifics, but pywin32 predates wheels, I don't know if it can be made to be installed so simply.

D. 


From p.f.moore at gmail.com  Wed Aug 27 17:48:40 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Aug 2014 16:48:40 +0100
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <1bee9d0996924d94b8d114aa0cc1dd1e@MSR-MAIL-EXCH01.ubisoft.org>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <1bee9d0996924d94b8d114aa0cc1dd1e@MSR-MAIL-EXCH01.ubisoft.org>
Message-ID: <CACac1F_ZVJHnJo97n1bhQ8V-_AdEuQ7W8qgCE_awj5GT3ZszOA@mail.gmail.com>

On 27 August 2014 16:42, David Genest <david.genest at ubisoft.com> wrote:
> On windows, we use wheel convert to convert binary installers like pywin32 to wheels. But the post-install step in the setup is mandatory  for pywin32 to work (it is not a mere Start Menu customization).

For pywin32, my understanding is that (ignoring start menu stuff) the
postinstall registers the DLLs for COM and things like that. Which is
essential, but only for the system installation (you don't want every
virtualenv registering as "the" Python COM server). So pywin32 would
need work to make it virtualenv-friendly anyway, and while that's
being done, it can probably be made wheel-friendly too. For now,
though, my experience is that using converted wheels in virtualenvs
works fine as long as you don't write services or COM servers.

But anyway, pywin32 isn't hosted anywhere pip can find, so "pip
install" doesn't work from sdist either.

Thanks for the information, nevertheless.
Paul

From p.f.moore at gmail.com  Wed Aug 27 17:50:55 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Aug 2014 16:50:55 +0100
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
Message-ID: <CACac1F9-UKHwL6u9_v+Mg-tZubL=ZDkMO9OZvAcTXFZ+gG_X9g@mail.gmail.com>

On 27 August 2014 07:50, Paul Moore <p.f.moore at gmail.com> wrote:
> tl; dr; If you know of a project that can't be successfully installed
> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
> me know the details?

I should have added: but can be installed from source with "pip install proj"
Paul

From qwcode at gmail.com  Wed Aug 27 20:09:31 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 27 Aug 2014 11:09:31 -0700
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
Message-ID: <CAPYWazqkjhmvv3Z72=FHspVZROAAm2+rkJ3H0xiYxK1gdo1gbg@mail.gmail.com>

There are cases where projects do this in setup.py:

for scheme in INSTALL_SCHEMES.values():
    scheme['data'] = scheme['purelib']

e.g.  https://github.com/celery/django-celery/blob/master/setup.py#L80

so for the sdist install, data ends up relative to site-packages, whereas
for the whl install, it ends up relative to sys.prefix




On Tue, Aug 26, 2014 at 11:50 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> tl; dr; If you know of a project that can't be successfully installed
> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
> me know the details?
>
> One of the longer-term goals of the introduction of wheels was to
> split the build and install steps for a package - specifically, in
> pip, to restrict "pip install" to installing from wheels, and when no
> wheel is available, to (transparently) run "pip wheel" on the sdist
> followed by "pip install on the generated wheel". During a discussion
> yesterday, I realised that I don't know how close we are to that goal.
> So I'm looking for information on packages which don't install
> properly from wheels at the moment. If anyone has examples of packages
> where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip
> install /tmp/xxx/*.whl" does not result in an equivalent install,
> could they post them here with details of how & why they fail?
>
> Things I already know about (but would like specific examples):
>
> 1. Post-install steps included in setup.py. That should be covered by
> the support in Metadata 2.0. I'd also be interested in how much of an
> issue omitting the postinstall would be in practice (for instance many
> such steps just set up "Start Menu" type shortcuts, which are not
> essential for the package to be usable).
> 2. Actually, that's the only one :-)
>
> Things that should not be a problem (but might be):
>
> 1. Numpy (and the scipy stack) need better tagging facilities for
> wheels - but that wouldn't matter for a wheel that's built, used, then
> thrown away.
> 2. Some things are complex to build - but I don't know of any cases
> where building a wheel is *more* complex than installing, and I don't
> see how it could be, in theory.
> 3. Projects that customise setup.py so much that they aren't
> compatible with setuptools and bdist_wheel. Such projects are quite
> probably already incompatible with pip (which injects setuptools when
> running setup.py anyway) and so not relevant for this discussion. But
> if any do work with "pip install" but not "pip wheel", I'd like to
> hear about them.
>
> Thanks in advance for any information.
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/30ee4261/attachment.html>

From donald at stufft.io  Wed Aug 27 20:38:10 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Aug 2014 14:38:10 -0400
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CAPYWazqkjhmvv3Z72=FHspVZROAAm2+rkJ3H0xiYxK1gdo1gbg@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAPYWazqkjhmvv3Z72=FHspVZROAAm2+rkJ3H0xiYxK1gdo1gbg@mail.gmail.com>
Message-ID: <2DA61E3A-7F8B-4A4B-82A6-5139C76FECEE@stufft.io>


> On Aug 27, 2014, at 2:09 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> There are cases where projects do this in setup.py:
> 
> for scheme in INSTALL_SCHEMES.values():
>     scheme['data'] = scheme['purelib']
> 
> e.g.  https://github.com/celery/django-celery/blob/master/setup.py#L80 <https://github.com/celery/django-celery/blob/master/setup.py#L80>
> 
> so for the sdist install, data ends up relative to site-packages, whereas for the whl install, it ends up relative to sys.prefix

Oh right I forgot about that. Older versions of Django did it (not newer versions) and some stuff in the django ecosystem copied it.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/9b1aca54/attachment.html>

From dholth at gmail.com  Wed Aug 27 21:55:50 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 27 Aug 2014 15:55:50 -0400
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CABNXP=TRzr47ARqrgESd4R9C_XvWc93MC2sVyaWUveWfGDC5ng@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <2EC78938-FA64-4B1C-A6D5-FBD9141E00FE@stufft.io>
 <CABNXP=TRzr47ARqrgESd4R9C_XvWc93MC2sVyaWUveWfGDC5ng@mail.gmail.com>
Message-ID: <CAG8k2+44GMG7fEhtTshn4jzvrJ5C2ixZhLidJ_9qvMzH=iG45A@mail.gmail.com>

Tell me more about setup-requires. It's nice to hear it has users. Should
we promote it to a pypa project?
On Aug 27, 2014 7:45 AM, "James Tocknell" <aragilar at gmail.com> wrote:

> Scipy (the package) doesn't work due to the build requirement on numpy
> (and the lack of metadata stating this), however some of the others have
> included https://bitbucket.org/dholth/setup-requires (or something
> equivalent) to install using numpy. Effectively, if the package can be pip
> installed, then wheeling should work (I created a script a while ago which
> uses distlib to modify scipy wheels such that they contain the correct
> metadata, and I haven't run into any issues). Matplotlib works, however
> some of the graphics toolkits aren't wheelable (e.g. pygtk and pyqt).
>
> One case that I know of where wheels do not replace a straight install is
> PySide (there needs to be rpath modification), however the PySide
> developers are aware of this and in their installation instructions they
> explicitly mention to run a post install script if installing from a wheel
> (The post install probably could be got rid of if $ORIGIN was used, but
> that would involve digging into the cmake install, and checking that it
> doesn't break anything. Also, I'm not sure how widely $ORIGIN is
> supported). I'd guess that any package which creates shared libraries which
> link to each other would have this problem also, though how many of those
> are pip installable is probably almost none.
>
> James
>
>
> On 27 August 2014 17:29, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> tl; dr; If you know of a project that can't be successfully installed
>> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
>> me know the details?
>>
>> One of the longer-term goals of the introduction of wheels was to
>> split the build and install steps for a package - specifically, in
>> pip, to restrict "pip install" to installing from wheels, and when no
>> wheel is available, to (transparently) run "pip wheel" on the sdist
>> followed by "pip install on the generated wheel". During a discussion
>> yesterday, I realised that I don't know how close we are to that goal.
>> So I'm looking for information on packages which don't install
>> properly from wheels at the moment. If anyone has examples of packages
>> where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip
>> install /tmp/xxx/*.whl" does not result in an equivalent install,
>> could they post them here with details of how & why they fail?
>>
>> Things I already know about (but would like specific examples):
>>
>> 1. Post-install steps included in setup.py. That should be covered by
>> the support in Metadata 2.0. I'd also be interested in how much of an
>> issue omitting the postinstall would be in practice (for instance many
>> such steps just set up "Start Menu" type shortcuts, which are not
>> essential for the package to be usable).
>> 2. Actually, that's the only one :-)
>>
>> Things that should not be a problem (but might be):
>>
>> 1. Numpy (and the scipy stack) need better tagging facilities for
>> wheels - but that wouldn't matter for a wheel that's built, used, then
>> thrown away.
>> 2. Some things are complex to build - but I don't know of any cases
>> where building a wheel is *more* complex than installing, and I don't
>> see how it could be, in theory.
>> 3. Projects that customise setup.py so much that they aren't
>> compatible with setuptools and bdist_wheel. Such projects are quite
>> probably already incompatible with pip (which injects setuptools when
>> running setup.py anyway) and so not relevant for this discussion. But
>> if any do work with "pip install" but not "pip wheel", I'd like to
>> hear about them.
>>
>> Thanks in advance for any information.
>> Paul
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> Numpy (and other SciPy stack) has the problem too that it uses
>> numpy.distutils
>> so I'm not even sure if ``pip wheel`` actually works on it at all or not.
>> They
>> might have their own support for ``setup.py bdist_wheel`` though I don't
>> know.
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
> Don't send me files in proprietary formats (.doc(x), .xls, .ppt etc.). It
> isn't good enough for Tim Berners-Lee
> <http://opendotdotdot.blogspot.com/2010/04/rms-and-tim-berners-lee-separated-at.html>,
> and it isn't good enough for me either. For more information visit
> http://www.gnu.org/philosophy/no-word-attachments.html.
>
> Truly great madness cannot be achieved without significant intelligence.
>  - Henrik Tikkanen
>
> If you're not messing with your sanity, you're not having fun.
>  - James Tocknell
>
> In theory, there is no difference between theory and practice; In
> practice, there is.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/3cec9de8/attachment-0001.html>

From braingram720 at gmail.com  Wed Aug 27 23:59:28 2014
From: braingram720 at gmail.com (braingram)
Date: Wed, 27 Aug 2014 17:59:28 -0400
Subject: [Distutils] requirements on pypi: what is egg.info/requires.txt?
Message-ID: <CAHYXAX6vZKifVWRYy2qStxLVjQ0U=Xb35L0jLa1skZN6hq1uQw@mail.gmail.com>

Hi,

I feel like these are stupid questions but I cannot seem to find a straight
answer.

In brief,

1) what is egg-info/requires.txt used for?
2) how do I properly define requirements for pypi?

The details are:

I'm updating some packages on pypi and am having difficulty defining
requirements.

One of the packages in question is: pypi.python.org/pypi/jsui

I'm initially defining the requirements in a requirements.txt that then
gets parsed in setup.py and install_requires gets set to the contents of
requirements.txt. Some of the output from python setup.py sdist build is
below. The resulting requires.txt in jsui.egg-info is:

flask
wsrpc

However, when I upload this to pypi with "python setup.py sdist upload" I'm
not seeing these requirements listed nor does pip installing the package
install the requirements.

Thanks for your help.

==== python setup.py sdist build partial output ====

running sdist
running egg_info
writing requirements to jsui.egg-info/requires.txt
writing jsui.egg-info/PKG-INFO
writing top-level names to jsui.egg-info/top_level.txt
writing dependency_links to jsui.egg-info/dependency_links.txt
reading manifest file 'jsui.egg-info/SOURCES.txt'
writing manifest file 'jsui.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url

creating jsui-0.0.1
creating jsui-0.0.1/jsui
creating jsui-0.0.1/jsui.egg-info
making hard links in jsui-0.0.1...
hard linking README -> jsui-0.0.1
hard linking setup.py -> jsui-0.0.1
hard linking jsui/__init__.py -> jsui-0.0.1/jsui
hard linking jsui/serve.py -> jsui-0.0.1/jsui
hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
Writing jsui-0.0.1/setup.cfg
Creating tar archive
removing 'jsui-0.0.1' (and everything under it)
running build
running build_py
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/bbe0e663/attachment.html>

From brettgraham at gmail.com  Wed Aug 27 23:51:58 2014
From: brettgraham at gmail.com (Brett Graham)
Date: Wed, 27 Aug 2014 17:51:58 -0400
Subject: [Distutils] defining requirements on pypi
Message-ID: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>

Hi,

I feel like these are stupid questions but I cannot seem to find a straight
answer.

In brief,

1) what is egg-info/requires.txt used for?
2) how do I properly define requirements for pypi?

The details are:

I'm updating some packages on pypi and am having difficulty defining
requirements.

One of the packages in question is: pypi.python.org/pypi/jsui

I'm initially defining the requirements in a requirements.txt that then
gets parsed in setup.py and install_requires gets set to the contents of
requirements.txt. Some of the output from python setup.py sdist build is
below. The resulting requires.txt in jsui.egg-info is:

flask
wsrpc

However, when I upload this to pypi with "python setup.py sdist upload" I'm
not seeing these requirements listed nor does pip installing the package
install the requirements.

Thanks for your help.

==== python setup.py sdist build partial output ====

running sdist
running egg_info
writing requirements to jsui.egg-info/requires.txt
writing jsui.egg-info/PKG-INFO
writing top-level names to jsui.egg-info/top_level.txt
writing dependency_links to jsui.egg-info/dependency_links.txt
reading manifest file 'jsui.egg-info/SOURCES.txt'
writing manifest file 'jsui.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url

creating jsui-0.0.1
creating jsui-0.0.1/jsui
creating jsui-0.0.1/jsui.egg-info
making hard links in jsui-0.0.1...
hard linking README -> jsui-0.0.1
hard linking setup.py -> jsui-0.0.1
hard linking jsui/__init__.py -> jsui-0.0.1/jsui
hard linking jsui/serve.py -> jsui-0.0.1/jsui
hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
Writing jsui-0.0.1/setup.cfg
Creating tar archive
removing 'jsui-0.0.1' (and everything under it)
running build
running build_py
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140827/b145a2bc/attachment.html>

From ncoghlan at gmail.com  Thu Aug 28 14:35:59 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Aug 2014 22:35:59 +1000
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CAG8k2+44GMG7fEhtTshn4jzvrJ5C2ixZhLidJ_9qvMzH=iG45A@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <2EC78938-FA64-4B1C-A6D5-FBD9141E00FE@stufft.io>
 <CABNXP=TRzr47ARqrgESd4R9C_XvWc93MC2sVyaWUveWfGDC5ng@mail.gmail.com>
 <CAG8k2+44GMG7fEhtTshn4jzvrJ5C2ixZhLidJ_9qvMzH=iG45A@mail.gmail.com>
Message-ID: <CADiSq7eExJWNuo9LnCb9sqy8Mu7k6Xnk+66BiStmp2u_+7xBog@mail.gmail.com>

On 28 Aug 2014 05:56, "Daniel Holth" <dholth at gmail.com> wrote:
>
> Tell me more about setup-requires. It's nice to hear it has users. Should
we promote it to a pypa project?

That would be cool - bootstrapping as much as we can *without* metadata 2.0
has the virtue of working in many more environments :)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/ccec2a19/attachment.html>

From qwcode at gmail.com  Thu Aug 28 20:41:45 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 28 Aug 2014 11:41:45 -0700
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
Message-ID: <CAPYWazqgHX-QsQm1onDCfpr9nNcxW77RWEjmxaccTHTUk8JGwQ@mail.gmail.com>

On Wed, Aug 27, 2014 at 2:51 PM, Brett Graham <brettgraham at gmail.com> wrote:

> Hi,
>
> I feel like these are stupid questions but I cannot seem to find a
> straight answer.
>
> In brief,
>
> 1) what is egg-info/requires.txt used for?
> 2) how do I properly define requirements for pypi?
>
> The details are:
>
> I'm updating some packages on pypi and am having difficulty defining
> requirements.
>
> One of the packages in question is: pypi.python.org/pypi/jsui
>
> I'm initially defining the requirements in a requirements.txt that then
> gets parsed in setup.py and install_requires gets set to the contents of
> requirements.txt. Some of the output from python setup.py sdist build is
> below. The resulting requires.txt in jsui.egg-info is:
>
> flask
> wsrpc
>
> However, when I upload this to pypi with "python setup.py sdist upload"
> I'm not seeing these requirements listed nor does pip installing the
> package install the requirements
>

when I install jsui from pypi, it finds and installs flask and wsrpc.
what you've done is working.

in general though, I wouldn't parse "install_requires" from a requirements
file.
It's mixing use cases.

see the "install_requires vs requirements files" section in the Python
Packaging User Guide
https://packaging.python.org/en/latest/technical.html#install-requires-vs-requirements-files
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/64a72361/attachment-0001.html>

From donald at stufft.io  Thu Aug 28 20:58:46 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 14:58:46 -0400
Subject: [Distutils] Handling Case/Normalization Differences
Message-ID: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>

Right now the ?canonical? page for a particular project on PyPI is whatever the
author happened to name their package (e.g. Django). This requires PyPI to have
some "smarts" so that it can redirect things like /simple/django/ to
/simple/Django/ otherwise someone doing ``pip install django`` would fall back
to a much worse behavior.

If this redirect doesn't happen, then pip will issue a request for just
/simple/ and look for a link that, when both sides are normalized, compares
equal to the name it's looking for. It will then follow the link, get
/simple/Django/ and everything works... Except it doesn't. The problem here
comes from the external link classification that we have now. Pip sees the
link to /simple/Django/ as an external link (because it lacks the required
rels) and the installation finally fails.

The /simple/ case rarely happens when installing from PyPI itself because of
the redirect, however it happens quite often when someone is attempting to
instal from a mirror instead. Even when everything works correctly the penality
for not knowing exactly what name to type in results in at least 1 extra http
request, one of which (/simple/) requires pulling down a 2.1MB file.

To fix this I'm going to modify PyPI so that it uses the normalized name in
the /simple/ URL and redirects everything else to the non-normalized name. I'm
also going to submit a PR to bandersnatch so that it will use normalized names
for it's directories and such as well. These two changes will make it so that
the client side will know ahead of time exactly what form the server expects
any given name to be in. This will allow a change in pip to happen which
will pre-normalize all names which will make the interaction with mirrors better
and will reduce the number of HTTP requests that a single ``pip install`` needs
to make.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/14079c01/attachment.html>

From j.j.molenaar at gmail.com  Thu Aug 28 21:24:35 2014
From: j.j.molenaar at gmail.com (Joost Molenaar)
Date: Thu, 28 Aug 2014 21:24:35 +0200
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
Message-ID: <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>

On Aug 27, 2014 8:50 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> tl; dr; If you know of a project that can't be successfully installed
> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
> me know the details?

Both Ansible and Fabric want to put stuff in <virtualenv>/usr/share,
which doesn't work from a .whl file but does from the sdist.

AFAICT, you need to spell out the data files redundantly in
MANIFEST.in *and* setup(data_files=...), or the wheel won't contain
the files.

I've been meaning to write up a test case to demonstrate the
problem for a good while, but haven't got around to it yet.

HTH,

Joost Molenaar

From donald at stufft.io  Thu Aug 28 21:35:08 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 15:35:08 -0400
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
 <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>
Message-ID: <B9B3FF03-5094-48CA-BB53-E36AD3293CBE@stufft.io>


> On Aug 28, 2014, at 3:24 PM, Joost Molenaar <j.j.molenaar at gmail.com> wrote:
> 
> On Aug 27, 2014 8:50 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>> 
>> tl; dr; If you know of a project that can't be successfully installed
>> with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let
>> me know the details?
> 
> Both Ansible and Fabric want to put stuff in <virtualenv>/usr/share,
> which doesn't work from a .whl file but does from the sdist.

This should be possible I think, that?s just the ?data? scheme. Maybe
the tooling isn?t quite set up for it?

> 
> AFAICT, you need to spell out the data files redundantly in
> MANIFEST.in *and* setup(data_files=...), or the wheel won't contain
> the files.

This is correct, because these two things mean different things, MANIFEST.in
is the files that get added to the sdist itself, it has no control over what gets
installed whereas data_files tells distills what files to install into the data
location.

You need both because if you only have MANIFEST.in, then distills won?t
actually install the files, and if you only have data_files then the sdist won?t
actually contain the files.

> 
> I've been meaning to write up a test case to demonstrate the
> problem for a good while, but haven't got around to it yet.
> 
> HTH,
> 
> Joost Molenaar
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/2e7ae881/attachment.html>

From p.f.moore at gmail.com  Thu Aug 28 21:54:42 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 28 Aug 2014 20:54:42 +0100
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <B9B3FF03-5094-48CA-BB53-E36AD3293CBE@stufft.io>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
 <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>
 <B9B3FF03-5094-48CA-BB53-E36AD3293CBE@stufft.io>
Message-ID: <CACac1F-ZXzRztKOC0rYoyvj3m=BzWW1rF0gnTooxRzwY5sY_kA@mail.gmail.com>

On 28 August 2014 20:35, Donald Stufft <donald at stufft.io> wrote:
>> Both Ansible and Fabric want to put stuff in <virtualenv>/usr/share,
>> which doesn't work from a .whl file but does from the sdist.
>
> This should be possible I think, that?s just the ?data? scheme. Maybe
> the tooling isn?t quite set up for it?

There is an issue here, though. I don't know if it's what Ansible and
Fabric try to do, but I have seen it in the past (MoinMoin, for
instance, used to do this). Packages want to install files to
arbitrary locations, maybe places like /etc or other FHS-mandated
locations on Unix. They can do this in setup.py by hacking locations
or using absolute paths. But the packaging standards only allow for
the various scheme locations.

For example, the Ansible setup.py does:

# always install in /usr/share/ansible if specified
# otherwise use the first module path listed
if '/usr/share/ansible' in module_paths:
    install_path = '/usr/share/ansible'
else:
    install_path = module_paths[0]
dirs=os.listdir("./library/")
data_files = []
for i in dirs:
    data_files.append((os.path.join(install_path, i),
glob('./library/' + i + '/*')))

That seems to be using absolute paths in data_files, which is
basically what I mean (a hack to force installation to an absolute
path). Joost is right that this won't install to
<virtualenv>/usr/share in a venv, though. That's just not how the data
scheme works.

I think we need to establish a "proper" solution for this situation.

My view is that Python packaging should not support installation of
files to anywhere other than subdirectories of the scheme locations.
Doing so simply won't work with wheels as things stand, and it'll
never be cross-platform (there is no /usr on Windows, for example).
But that does mean that we explicitly don't support hacks like the
above, so we break a certain proportion of packages.

For packages that need to install to absolute locations, I would
suggest that this be handled by a post-install script, that copies the
relevant files from the package data location to the final
destination. Such a script can check if it's running on an unsupported
platform, or in a virtualenv, and handle that case appropriately.

The files will not be removed on uninstall with this proposal. But the
distribution could include a pre-uninstall script for that (are
pre-uninstall scripts a thing?) or just accept that this is a
limitation of trying to use Python packaging tools in a non-standard
manner and advise users to clean up those files manually.

Comments? Is my proposal reasonable? Does anyone have an alternative
to suggest? Would Ansible, Fabric and any other projects using the
current hacks be willing to migrate to post-install scripts?

Paul.

PS Actually, installing Windows start menu shortcuts is logically the
same issue, and Windows projects have used post-install scripts to do
this for some time now - see pywin32 for an example.

From j.j.molenaar at gmail.com  Thu Aug 28 22:33:21 2014
From: j.j.molenaar at gmail.com (Joost Molenaar)
Date: Thu, 28 Aug 2014 22:33:21 +0200
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CACac1F-ZXzRztKOC0rYoyvj3m=BzWW1rF0gnTooxRzwY5sY_kA@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
 <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>
 <B9B3FF03-5094-48CA-BB53-E36AD3293CBE@stufft.io>
 <CACac1F-ZXzRztKOC0rYoyvj3m=BzWW1rF0gnTooxRzwY5sY_kA@mail.gmail.com>
Message-ID: <CAG07R1AAmz8zzVOQpxL5NhHwWjq_vJ41sJjpTPEYaS3ddfKLRw@mail.gmail.com>

FWIW, I did whip up a test case just now, I put it up at github[1].
The second commit works correctly, but not the first or the third.

For Ansible and Fabric, IMO it's not necessary to put the files in /usr/share.
It's just that they need somewhere to put those things and be able to get at
them at runtime.

Question; would it help to use pkg_resources, or being able to specify locations
relative to the python interpreter prefix? Or to put it in some
'standard' directory
such as <name>.egg-info?

I run into this stuff in my own project as well. It annoys me to have to specify
the same data twice. :-) All I want to say is: these non-python are important
and I need to be able to find them somewhere at runtime. I don't really care
much where exactly they are.


Joost

[1] https://github.com/j0057/pydatafiles

From p.f.moore at gmail.com  Thu Aug 28 22:46:37 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 28 Aug 2014 21:46:37 +0100
Subject: [Distutils] Packages that have problems being installed from
	wheels
In-Reply-To: <CAG07R1AAmz8zzVOQpxL5NhHwWjq_vJ41sJjpTPEYaS3ddfKLRw@mail.gmail.com>
References: <CACac1F8x55W3kV0R-t_dxP=xw8hWfDTT1aoa=+HBeR0TSieuhg@mail.gmail.com>
 <CAG07R1BEQ9KGKThbJqa5hsYoHKCokrECSu3YoDWhjv5FspJOkQ@mail.gmail.com>
 <CAG07R1CDg04h8WVSKsWhhXTCwuDF5dHj84v=AgHkSbaY1Ecr+w@mail.gmail.com>
 <B9B3FF03-5094-48CA-BB53-E36AD3293CBE@stufft.io>
 <CACac1F-ZXzRztKOC0rYoyvj3m=BzWW1rF0gnTooxRzwY5sY_kA@mail.gmail.com>
 <CAG07R1AAmz8zzVOQpxL5NhHwWjq_vJ41sJjpTPEYaS3ddfKLRw@mail.gmail.com>
Message-ID: <CACac1F8G_ijsJx6iR46BC=jzLYjB53UiysDAy4B-H9n83VRKvA@mail.gmail.com>

On 28 August 2014 21:33, Joost Molenaar <j.j.molenaar at gmail.com> wrote:
> Question; would it help to use pkg_resources, or being able to specify locations
> relative to the python interpreter prefix?

Generally, I'd recommend putting resource files in the package
directory and reading them via pkgutil.get_data (from the stdlib). The
advantage of that is that it supports non-filesystem importers (such
as zipfiles). pkg_resources lets you be a bit more flexible but at the
cost of a runtime dependency on setuptools (which I personally find
unacceptable, but YMMV). For projects that don't mind failing when run
from anything other than the filesystem, you can use package.__file__
to get the filesystem location of your package and work from there -
but that's zipfile-hostile and won't work with tools like
cx_Freeze/py2exe/bbfreeze.

If you are willing to restrict your code to only working when
installed to the actual filesystem, you could also use
sysconfig.get_path(name) to get the filesystem location of the various
distutils locations (name='data' is where data_files are located, for
example) and use that to locate your files. I'm not entirely sure why
this would be better than using pkgutil, though.

The key thing is to *not* try to force your data files into a fixed
location that you then hard-code in your application.

Paul

From dholth at gmail.com  Fri Aug 29 00:02:39 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 28 Aug 2014 18:02:39 -0400
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
Message-ID: <CAG8k2+5tftthZ5Jj34g-c+xZ8mMKEG545S6ftvpxTPXz3raLmw@mail.gmail.com>

Pypi only parses requirements from wheels for display but this doesn't
affect installation.
On Aug 28, 2014 7:49 AM, "Brett Graham" <brettgraham at gmail.com> wrote:

> Hi,
>
> I feel like these are stupid questions but I cannot seem to find a
> straight answer.
>
> In brief,
>
> 1) what is egg-info/requires.txt used for?
> 2) how do I properly define requirements for pypi?
>
> The details are:
>
> I'm updating some packages on pypi and am having difficulty defining
> requirements.
>
> One of the packages in question is: pypi.python.org/pypi/jsui
>
> I'm initially defining the requirements in a requirements.txt that then
> gets parsed in setup.py and install_requires gets set to the contents of
> requirements.txt. Some of the output from python setup.py sdist build is
> below. The resulting requires.txt in jsui.egg-info is:
>
> flask
> wsrpc
>
> However, when I upload this to pypi with "python setup.py sdist upload"
> I'm not seeing these requirements listed nor does pip installing the
> package install the requirements.
>
> Thanks for your help.
>
> ==== python setup.py sdist build partial output ====
>
> running sdist
> running egg_info
> writing requirements to jsui.egg-info/requires.txt
> writing jsui.egg-info/PKG-INFO
> writing top-level names to jsui.egg-info/top_level.txt
> writing dependency_links to jsui.egg-info/dependency_links.txt
> reading manifest file 'jsui.egg-info/SOURCES.txt'
> writing manifest file 'jsui.egg-info/SOURCES.txt'
> running check
> warning: check: missing required meta-data: url
>
> creating jsui-0.0.1
> creating jsui-0.0.1/jsui
> creating jsui-0.0.1/jsui.egg-info
> making hard links in jsui-0.0.1...
> hard linking README -> jsui-0.0.1
> hard linking setup.py -> jsui-0.0.1
> hard linking jsui/__init__.py -> jsui-0.0.1/jsui
> hard linking jsui/serve.py -> jsui-0.0.1/jsui
> hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
> Writing jsui-0.0.1/setup.cfg
> Creating tar archive
> removing 'jsui-0.0.1' (and everything under it)
> running build
> running build_py
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/514e8da1/attachment.html>

From donald at stufft.io  Fri Aug 29 00:11:12 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 18:11:12 -0400
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <CAG8k2+5tftthZ5Jj34g-c+xZ8mMKEG545S6ftvpxTPXz3raLmw@mail.gmail.com>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
 <CAG8k2+5tftthZ5Jj34g-c+xZ8mMKEG545S6ftvpxTPXz3raLmw@mail.gmail.com>
Message-ID: <9EF7A89A-B572-4DC4-BC0F-B2D8A6C9D2A3@stufft.io>

PyPI doesn?t parse anything, metadata is passed alongside the file upload. This means that for sdists it would just need modified to communicate that information to PyPI. PyPI won?t show metadata for Wheels either unless you upload them with Twine.


> On Aug 28, 2014, at 6:02 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> Pypi only parses requirements from wheels for display but this doesn't affect installation.
> 
> On Aug 28, 2014 7:49 AM, "Brett Graham" <brettgraham at gmail.com <mailto:brettgraham at gmail.com>> wrote:
> Hi,
> 
> I feel like these are stupid questions but I cannot seem to find a straight answer.
> 
> In brief,
> 
> 1) what is egg-info/requires.txt used for?
> 2) how do I properly define requirements for pypi?
> 
> The details are:
> 
> I'm updating some packages on pypi and am having difficulty defining requirements.
> 
> One of the packages in question is: pypi.python.org/pypi/jsui <http://pypi.python.org/pypi/jsui>
> 
> I'm initially defining the requirements in a requirements.txt that then gets parsed in setup.py and install_requires gets set to the contents of requirements.txt. Some of the output from python setup.py sdist build is below. The resulting requires.txt in jsui.egg-info is:
> 
> flask
> wsrpc
> 
> However, when I upload this to pypi with "python setup.py sdist upload" I'm not seeing these requirements listed nor does pip installing the package install the requirements.
>  
> Thanks for your help.
> 
> ==== python setup.py sdist build partial output ====
> 
> running sdist
> running egg_info
> writing requirements to jsui.egg-info/requires.txt
> writing jsui.egg-info/PKG-INFO
> writing top-level names to jsui.egg-info/top_level.txt
> writing dependency_links to jsui.egg-info/dependency_links.txt
> reading manifest file 'jsui.egg-info/SOURCES.txt'
> writing manifest file 'jsui.egg-info/SOURCES.txt'
> running check
> warning: check: missing required meta-data: url
> 
> creating jsui-0.0.1
> creating jsui-0.0.1/jsui
> creating jsui-0.0.1/jsui.egg-info
> making hard links in jsui-0.0.1...
> hard linking README -> jsui-0.0.1
> hard linking setup.py -> jsui-0.0.1
> hard linking jsui/__init__.py -> jsui-0.0.1/jsui
> hard linking jsui/serve.py -> jsui-0.0.1/jsui
> hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
> hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
> Writing jsui-0.0.1/setup.cfg
> Creating tar archive
> removing 'jsui-0.0.1' (and everything under it)
> running build
> running build_py
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/b994feb2/attachment-0001.html>

From donald at stufft.io  Fri Aug 29 00:09:51 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 18:09:51 -0400
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
Message-ID: <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>


> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> Right now the ?canonical? page for a particular project on PyPI is whatever the
> author happened to name their package (e.g. Django). This requires PyPI to have
> some "smarts" so that it can redirect things like /simple/django/ to
> /simple/Django/ otherwise someone doing ``pip install django`` would fall back
> to a much worse behavior.
> 
> If this redirect doesn't happen, then pip will issue a request for just
> /simple/ and look for a link that, when both sides are normalized, compares
> equal to the name it's looking for. It will then follow the link, get
> /simple/Django/ and everything works... Except it doesn't. The problem here
> comes from the external link classification that we have now. Pip sees the
> link to /simple/Django/ as an external link (because it lacks the required
> rels) and the installation finally fails.
> 
> The /simple/ case rarely happens when installing from PyPI itself because of
> the redirect, however it happens quite often when someone is attempting to
> instal from a mirror instead. Even when everything works correctly the penality
> for not knowing exactly what name to type in results in at least 1 extra http
> request, one of which (/simple/) requires pulling down a 2.1MB file.
> 
> To fix this I'm going to modify PyPI so that it uses the normalized name in
> the /simple/ URL and redirects everything else to the non-normalized name. I'm
> also going to submit a PR to bandersnatch so that it will use normalized names
> for it's directories and such as well. These two changes will make it so that
> the client side will know ahead of time exactly what form the server expects
> any given name to be in. This will allow a change in pip to happen which
> will pre-normalize all names which will make the interaction with mirrors better
> and will reduce the number of HTTP requests that a single ``pip install`` needs
> to make.
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Hm, so here?s the problem.

I have this implemented and deployed to TestPyPI, it works great!

However, the next step is to make the change to bandersnatch so that it saves
things using their normalized name instead of using their "proper" name. Doing
this will trigger it so that everyone using pip 1.5 won't be able to install
anything from that mirror unless it's name is specified as the normalized name
(e.g. ``pip install Django`` will fail without --allow-unverified but
``pip install django`` will work). This would be fixed with pip 1.6 (since
it would know to "normalize" the name before fetching the URL).

The same thing will occur if we make the change in pip first, it would
normalize names so you'd need to use --allow-unverified for everything because
it would act as if you typed ``pip install django`` instead of ``pip install
Django``.

To my knowledge, this *only* will affect pip 1.5.x.

So the only way forward I can see to make this change, which I think is a good
change and will remove a big "gotcha" from using a mirror, is to coordinate
a release of bandersnatch that coincides with pip 1.6, and tell people they
need to upgrade in lockstep.

Does anyone have any other ideas?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/279a0cd4/attachment.html>

From donald at stufft.io  Fri Aug 29 00:26:33 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 18:26:33 -0400
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
 <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
Message-ID: <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>


> On Aug 28, 2014, at 6:09 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> 
>> Right now the ?canonical? page for a particular project on PyPI is whatever the
>> author happened to name their package (e.g. Django). This requires PyPI to have
>> some "smarts" so that it can redirect things like /simple/django/ to
>> /simple/Django/ otherwise someone doing ``pip install django`` would fall back
>> to a much worse behavior.
>> 
>> If this redirect doesn't happen, then pip will issue a request for just
>> /simple/ and look for a link that, when both sides are normalized, compares
>> equal to the name it's looking for. It will then follow the link, get
>> /simple/Django/ and everything works... Except it doesn't. The problem here
>> comes from the external link classification that we have now. Pip sees the
>> link to /simple/Django/ as an external link (because it lacks the required
>> rels) and the installation finally fails.
>> 
>> The /simple/ case rarely happens when installing from PyPI itself because of
>> the redirect, however it happens quite often when someone is attempting to
>> instal from a mirror instead. Even when everything works correctly the penality
>> for not knowing exactly what name to type in results in at least 1 extra http
>> request, one of which (/simple/) requires pulling down a 2.1MB file.
>> 
>> To fix this I'm going to modify PyPI so that it uses the normalized name in
>> the /simple/ URL and redirects everything else to the non-normalized name. I'm
>> also going to submit a PR to bandersnatch so that it will use normalized names
>> for it's directories and such as well. These two changes will make it so that
>> the client side will know ahead of time exactly what form the server expects
>> any given name to be in. This will allow a change in pip to happen which
>> will pre-normalize all names which will make the interaction with mirrors better
>> and will reduce the number of HTTP requests that a single ``pip install`` needs
>> to make.
>> 
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 
> Hm, so here?s the problem.
> 
> I have this implemented and deployed to TestPyPI, it works great!
> 
> However, the next step is to make the change to bandersnatch so that it saves
> things using their normalized name instead of using their "proper" name. Doing
> this will trigger it so that everyone using pip 1.5 won't be able to install
> anything from that mirror unless it's name is specified as the normalized name
> (e.g. ``pip install Django`` will fail without --allow-unverified but
> ``pip install django`` will work). This would be fixed with pip 1.6 (since
> it would know to "normalize" the name before fetching the URL).
> 
> The same thing will occur if we make the change in pip first, it would
> normalize names so you'd need to use --allow-unverified for everything because
> it would act as if you typed ``pip install django`` instead of ``pip install
> Django``.
> 
> To my knowledge, this *only* will affect pip 1.5.x.
> 
> So the only way forward I can see to make this change, which I think is a good
> change and will remove a big "gotcha" from using a mirror, is to coordinate
> a release of bandersnatch that coincides with pip 1.6, and tell people they
> need to upgrade in lockstep.
> 
> Does anyone have any other ideas?
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Just thought of this, if the normalized name doesn?t match the "real" name,
then add entries for both. This will make it so that pip 1.5 continues to work
and pip 1.6+.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/af7419a4/attachment-0001.html>

From erik.m.bray at gmail.com  Fri Aug 29 00:39:03 2014
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 28 Aug 2014 18:39:03 -0400
Subject: [Distutils] Undefined symbol when loading a library using dlopen
In-Reply-To: <53FD022E.4070100@free.fr>
References: <53FD022E.4070100@free.fr>
Message-ID: <CAOTD34ZdQRO2RKWRssT+HjOA03vEobgCvrysuoyMKpS6H-o2Qw@mail.gmail.com>

On Tue, Aug 26, 2014 at 5:54 PM, Jerome Fuselier
<jerome.fuselier at free.fr> wrote:
> Hello,
>
> I'm trying to wrap a C++ library which uses dlopen to load a dynamic library
> and I don't manage to get rid of an "undefined symbol" error.
>
> I tried to simulate my problem with a simpler example so hopefully someone
> may be able to tell me what I'm doing wrong.
>
> I have a file func.cpp which provides 2 functions :
>
> func.cpp:
>
> int simple(void) {
>     return 12;
> }
>
> int call_lib(void) {
>     int res;
>     void *handle = dlopen("/home/jerome/Test/libDyn.so", RTLD_NOW);
>
>     if (!handle) {
>         return -1;
>     }
>     dlerror();
>     char* err = 0;
>     int (*fun_dyn)() = (int (*)())dlsym(handle, "fun_dyn");
>     if ( (err = dlerror()) != 0 ) {
>         dlclose(handle);
>         return -1;
>     }
>     res = (*fun_dyn)();
>     dlclose(handle);
>     return res;
> }
>
> It is created like that:
> $ g++ -Wall -g -fPIC -c -o func.o func.cpp
> $ ar rcs libFun.a func.o
>
> I have a dynamic library which is called from call_lib() which depends on
> libFun.a:
>
> libDyn.cpp:
>
> #include "func.hpp"
>
> int fun_dyn(void) {
>     return simple();
> }
>
> It is created like that:
> $ g++ -Wall -g -fPIC -c -o libDyn.o libDyn.cpp
> $ g++ -Wall -g -fPIC -shared -o libDyn.so libDyn.o
>
> I'm using swig to wrap the libFun.a library with this mylib.i interface
> file:
>
> %module mylib
>
> int simple(void);
> int call_lib(void);
>
> Then I'm using this setup.py file to create the Python module:
>
> from distutils.core import setup, Extension
>
> mylib = Extension("_mylib",
>                   sources=['mylib.i'],
>                   language="c++",
>                   extra_link_args=['-rdynamic'],
>                   libraries=["Fun", "dl"],
>                  )
>
> setup(name="MyLib",
>       platforms=["Linux"],
>       ext_modules=[mylib],
>       py_modules = ["mylib"])
>
>
> It's a simplified example but when I compile everything I get this error in
> Python:
>>>> import _mylib
>>>> _mylib.call_lib()
> failed to open shared object file
> /home/jerome/Test/libDyn.so: undefined symbol: simple
>
> If I do a nm _mylib.so I see that the symbol simple is present in the
> library so I'm wondering why dlopen doesn't see it when it tries to load the
> library.
>
> I know I can recompile libDyn.so to link libFun.a but this would means I
> would need to recompile all the plugins provided for the library I'm trying
> to wrap and that's not really a solution for my project.

I don't think this question really has to do with distutils' role in
this.  But have you tried declaring

extern "C" {
    int simple(void);
}

in func.cpp?  I think that should do it (to get around C++ name mangling).

Best of luck,

Erik

From brettgraham at gmail.com  Thu Aug 28 22:09:43 2014
From: brettgraham at gmail.com (Brett Graham)
Date: Thu, 28 Aug 2014 16:09:43 -0400
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <CAPYWazqgHX-QsQm1onDCfpr9nNcxW77RWEjmxaccTHTUk8JGwQ@mail.gmail.com>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
 <CAPYWazqgHX-QsQm1onDCfpr9nNcxW77RWEjmxaccTHTUk8JGwQ@mail.gmail.com>
Message-ID: <CAHYXAX6W21c=ix1s9dED00YbvLdFKfb3uLisYcbWfP54GXvN-w@mail.gmail.com>

Thanks for testing it and extra info!

It sounds like I 'fixed' the problem in the not-recommended way of
including requirements.txt in the MANIFEST.in.

I've been using a pretty hefty setup.py:
https://github.com/braingram/simple_setup that attempts to auto-parse and
detect 'everything' (finding packages, data, scripts, author info, version,
requirements) with the hope of dropping it in a directory with a python
module and magically having it pypi compatible. It sounds like it may be a
pipe dream :)


On Thu, Aug 28, 2014 at 2:41 PM, Marcus Smith <qwcode at gmail.com> wrote:

>
>
>
> On Wed, Aug 27, 2014 at 2:51 PM, Brett Graham <brettgraham at gmail.com>
> wrote:
>
>> Hi,
>>
>> I feel like these are stupid questions but I cannot seem to find a
>> straight answer.
>>
>> In brief,
>>
>> 1) what is egg-info/requires.txt used for?
>> 2) how do I properly define requirements for pypi?
>>
>> The details are:
>>
>> I'm updating some packages on pypi and am having difficulty defining
>> requirements.
>>
>> One of the packages in question is: pypi.python.org/pypi/jsui
>>
>> I'm initially defining the requirements in a requirements.txt that then
>> gets parsed in setup.py and install_requires gets set to the contents of
>> requirements.txt. Some of the output from python setup.py sdist build is
>> below. The resulting requires.txt in jsui.egg-info is:
>>
>> flask
>> wsrpc
>>
>> However, when I upload this to pypi with "python setup.py sdist upload"
>> I'm not seeing these requirements listed nor does pip installing the
>> package install the requirements
>>
>
> when I install jsui from pypi, it finds and installs flask and wsrpc.
> what you've done is working.
>
> in general though, I wouldn't parse "install_requires" from a requirements
> file.
> It's mixing use cases.
>
> see the "install_requires vs requirements files" section in the Python
> Packaging User Guide
>
> https://packaging.python.org/en/latest/technical.html#install-requires-vs-requirements-files
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/e7239c57/attachment.html>

From yasumoto7 at gmail.com  Fri Aug 29 00:39:43 2014
From: yasumoto7 at gmail.com (Joe Smith)
Date: Thu, 28 Aug 2014 15:39:43 -0700
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
 <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
 <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
Message-ID: <CA+y6w6rzg=RJMKd=9YEdTuowt_GUGFM=0j7OhONT4Y1RhA=o2g@mail.gmail.com>

Naive question- does pip send over a UserAgent (or something) that contains
a version number the server can use to determine which behavior to default
to?

That would allow a deprecation cycle of N months or so that will let people
upgrade from 1.5 to 1.6. We could then watch usage of 1.5 decrease over
time until it's a non-factor.


On Thu, Aug 28, 2014 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 28, 2014, at 6:09 PM, Donald Stufft <donald at stufft.io> wrote:
>
>
> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io> wrote:
>
> Right now the ?canonical? page for a particular project on PyPI is
> whatever the
> author happened to name their package (e.g. Django). This requires PyPI to
> have
> some "smarts" so that it can redirect things like /simple/django/ to
> /simple/Django/ otherwise someone doing ``pip install django`` would fall
> back
> to a much worse behavior.
>
> If this redirect doesn't happen, then pip will issue a request for just
> /simple/ and look for a link that, when both sides are normalized, compares
> equal to the name it's looking for. It will then follow the link, get
> /simple/Django/ and everything works... Except it doesn't. The problem here
> comes from the external link classification that we have now. Pip sees the
> link to /simple/Django/ as an external link (because it lacks the required
> rels) and the installation finally fails.
>
> The /simple/ case rarely happens when installing from PyPI itself because
> of
> the redirect, however it happens quite often when someone is attempting to
> instal from a mirror instead. Even when everything works correctly the
> penality
> for not knowing exactly what name to type in results in at least 1 extra
> http
> request, one of which (/simple/) requires pulling down a 2.1MB file.
>
> To fix this I'm going to modify PyPI so that it uses the normalized name in
> the /simple/ URL and redirects everything else to the non-normalized name.
> I'm
> also going to submit a PR to bandersnatch so that it will use normalized
> names
> for it's directories and such as well. These two changes will make it so
> that
> the client side will know ahead of time exactly what form the server
> expects
> any given name to be in. This will allow a change in pip to happen which
> will pre-normalize all names which will make the interaction with mirrors
> better
> and will reduce the number of HTTP requests that a single ``pip install``
> needs
> to make.
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> Hm, so here?s the problem.
>
> I have this implemented and deployed to TestPyPI, it works great!
>
> However, the next step is to make the change to bandersnatch so that it
> saves
> things using their normalized name instead of using their "proper" name.
> Doing
> this will trigger it so that everyone using pip 1.5 won't be able to
> install
> anything from that mirror unless it's name is specified as the normalized
> name
> (e.g. ``pip install Django`` will fail without --allow-unverified but
> ``pip install django`` will work). This would be fixed with pip 1.6 (since
> it would know to "normalize" the name before fetching the URL).
>
> The same thing will occur if we make the change in pip first, it would
> normalize names so you'd need to use --allow-unverified for everything
> because
> it would act as if you typed ``pip install django`` instead of ``pip
> install
> Django``.
>
> To my knowledge, this *only* will affect pip 1.5.x.
>
> So the only way forward I can see to make this change, which I think is a
> good
> change and will remove a big "gotcha" from using a mirror, is to coordinate
> a release of bandersnatch that coincides with pip 1.6, and tell people they
> need to upgrade in lockstep.
>
> Does anyone have any other ideas?
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> Just thought of this, if the normalized name doesn?t match the "real" name,
> then add entries for both. This will make it so that pip 1.5 continues to
> work
> and pip 1.6+.
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/39ea65a7/attachment-0001.html>

From ncoghlan at gmail.com  Fri Aug 29 00:40:59 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Aug 2014 08:40:59 +1000
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
 <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
 <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
Message-ID: <CADiSq7dWJqMnyDk0c9cGSTRN+LxE8_yRzzkqh5qBKU8uZKOFxg@mail.gmail.com>

On 29 Aug 2014 08:27, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> Just thought of this, if the normalized name doesn?t match the "real"
name,
> then add entries for both. This will make it so that pip 1.5 continues to
work
> and pip 1.6+.

Having bandersnatch mirrors publish under both names sounds like a good
approach.

Then the pip 1.6 release notes can just be explicit that using older
mirrors will need the extra option - earlier versions won't have a problem.

Cheers,
Nick.

>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140829/8d1d402b/attachment.html>

From donald at stufft.io  Fri Aug 29 00:41:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 18:41:05 -0400
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <CA+y6w6rzg=RJMKd=9YEdTuowt_GUGFM=0j7OhONT4Y1RhA=o2g@mail.gmail.com>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
 <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
 <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
 <CA+y6w6rzg=RJMKd=9YEdTuowt_GUGFM=0j7OhONT4Y1RhA=o2g@mail.gmail.com>
Message-ID: <09F1A8B9-AC9F-42DB-AEC3-2B3BED922393@stufft.io>

Since pip 1.4 it does yes, however the problem here is that typically bandersnatch
mirrors are simply hosted by plain static web servers and don?t require any sort of runtime logic.

> On Aug 28, 2014, at 6:39 PM, Joe Smith <yasumoto7 at gmail.com> wrote:
> 
> Naive question- does pip send over a UserAgent (or something) that contains a version number the server can use to determine which behavior to default to?
> 
> That would allow a deprecation cycle of N months or so that will let people upgrade from 1.5 to 1.6. We could then watch usage of 1.5 decrease over time until it's a non-factor.
> 
> 
> On Thu, Aug 28, 2014 at 3:26 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Aug 28, 2014, at 6:09 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> 
>> 
>>> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>>> 
>>> Right now the ?canonical? page for a particular project on PyPI is whatever the
>>> author happened to name their package (e.g. Django). This requires PyPI to have
>>> some "smarts" so that it can redirect things like /simple/django/ to
>>> /simple/Django/ otherwise someone doing ``pip install django`` would fall back
>>> to a much worse behavior.
>>> 
>>> If this redirect doesn't happen, then pip will issue a request for just
>>> /simple/ and look for a link that, when both sides are normalized, compares
>>> equal to the name it's looking for. It will then follow the link, get
>>> /simple/Django/ and everything works... Except it doesn't. The problem here
>>> comes from the external link classification that we have now. Pip sees the
>>> link to /simple/Django/ as an external link (because it lacks the required
>>> rels) and the installation finally fails.
>>> 
>>> The /simple/ case rarely happens when installing from PyPI itself because of
>>> the redirect, however it happens quite often when someone is attempting to
>>> instal from a mirror instead. Even when everything works correctly the penality
>>> for not knowing exactly what name to type in results in at least 1 extra http
>>> request, one of which (/simple/) requires pulling down a 2.1MB file.
>>> 
>>> To fix this I'm going to modify PyPI so that it uses the normalized name in
>>> the /simple/ URL and redirects everything else to the non-normalized name. I'm
>>> also going to submit a PR to bandersnatch so that it will use normalized names
>>> for it's directories and such as well. These two changes will make it so that
>>> the client side will know ahead of time exactly what form the server expects
>>> any given name to be in. This will allow a change in pip to happen which
>>> will pre-normalize all names which will make the interaction with mirrors better
>>> and will reduce the number of HTTP requests that a single ``pip install`` needs
>>> to make.
>>> 
>>> ---
>>> Donald Stufft
>>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>> 
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
>>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
>> 
>> 
>> Hm, so here?s the problem.
>> 
>> I have this implemented and deployed to TestPyPI, it works great!
>> 
>> However, the next step is to make the change to bandersnatch so that it saves
>> things using their normalized name instead of using their "proper" name. Doing
>> this will trigger it so that everyone using pip 1.5 won't be able to install
>> anything from that mirror unless it's name is specified as the normalized name
>> (e.g. ``pip install Django`` will fail without --allow-unverified but
>> ``pip install django`` will work). This would be fixed with pip 1.6 (since
>> it would know to "normalize" the name before fetching the URL).
>> 
>> The same thing will occur if we make the change in pip first, it would
>> normalize names so you'd need to use --allow-unverified for everything because
>> it would act as if you typed ``pip install django`` instead of ``pip install
>> Django``.
>> 
>> To my knowledge, this *only* will affect pip 1.5.x.
>> 
>> So the only way forward I can see to make this change, which I think is a good
>> change and will remove a big "gotcha" from using a mirror, is to coordinate
>> a release of bandersnatch that coincides with pip 1.6, and tell people they
>> need to upgrade in lockstep.
>> 
>> Does anyone have any other ideas?
>> 
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 
> Just thought of this, if the normalized name doesn?t match the "real" name,
> then add entries for both. This will make it so that pip 1.5 continues to work
> and pip 1.6+.
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/b7747c81/attachment.html>

From donald at stufft.io  Fri Aug 29 00:47:19 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 28 Aug 2014 18:47:19 -0400
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <1409266005.79644.YahooMailNeo@web172401.mail.ir2.yahoo.com>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
 <CAG8k2+5tftthZ5Jj34g-c+xZ8mMKEG545S6ftvpxTPXz3raLmw@mail.gmail.com>
 <9EF7A89A-B572-4DC4-BC0F-B2D8A6C9D2A3@stufft.io>
 <1409266005.79644.YahooMailNeo@web172401.mail.ir2.yahoo.com>
Message-ID: <AF7AB5EB-C4BF-4EE1-A74D-5D528E1F4C07@stufft.io>

Send the data when they upload.

> On Aug 28, 2014, at 6:46 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> 
> > PyPI won?t show metadata for Wheels either unless you upload them with Twine
> 
> Why is that, exactly? Is there something alternate tools would need to do to achieve the same thing?
> 
> Regards,
> 
> Vinay Sajip
> 
> From: Donald Stufft <donald at stufft.io>
> To: Daniel Holth <dholth at gmail.com> 
> Cc: Brett Graham <brettgraham at gmail.com>; DistUtils mailing list <distutils-sig at python.org> 
> Sent: Thursday, 28 August 2014, 23:11
> Subject: Re: [Distutils] defining requirements on pypi
> 
> PyPI doesn?t parse anything, metadata is passed alongside the file upload. This means that for sdists it would just need modified to communicate that information to PyPI. PyPI won?t show metadata for Wheels either unless you upload them with Twine.
> 
> 
>> On Aug 28, 2014, at 6:02 PM, Daniel Holth <dholth at gmail.com <mailto:dholth at gmail.com>> wrote:
>> 
>> Pypi only parses requirements from wheels for display but this doesn't affect installation.
>> On Aug 28, 2014 7:49 AM, "Brett Graham" <brettgraham at gmail.com <mailto:brettgraham at gmail.com>> wrote:
>> Hi,
>> 
>> I feel like these are stupid questions but I cannot seem to find a straight answer.
>> 
>> In brief,
>> 
>> 1) what is egg-info/requires.txt used for?
>> 2) how do I properly define requirements for pypi?
>> 
>> The details are:
>> 
>> I'm updating some packages on pypi and am having difficulty defining requirements.
>> 
>> One of the packages in question is: pypi.python.org/pypi/jsui <http://pypi.python.org/pypi/jsui>
>> 
>> I'm initially defining the requirements in a requirements.txt that then gets parsed in setup.py and install_requires gets set to the contents of requirements.txt. Some of the output from python setup.py sdist build is below. The resulting requires.txt in jsui.egg-info is:
>> 
>> flask
>> wsrpc
>> 
>> However, when I upload this to pypi with "python setup.py sdist upload" I'm not seeing these requirements listed nor does pip installing the package install the requirements.
>>  
>> Thanks for your help.
>> 
>> ==== python setup.py sdist build partial output ====
>> 
>> running sdist
>> running egg_info
>> writing requirements to jsui.egg-info/requires.txt
>> writing jsui.egg-info/PKG-INFO
>> writing top-level names to jsui.egg-info/top_level.txt
>> writing dependency_links to jsui.egg-info/dependency_links.txt
>> reading manifest file 'jsui.egg-info/SOURCES.txt'
>> writing manifest file 'jsui.egg-info/SOURCES.txt'
>> running check
>> warning: check: missing required meta-data: url
>> 
>> creating jsui-0.0.1
>> creating jsui-0.0.1/jsui
>> creating jsui-0.0.1/jsui.egg-info
>> making hard links in jsui-0.0.1...
>> hard linking README -> jsui-0.0.1
>> hard linking setup.py -> jsui-0.0.1
>> hard linking jsui/__init__.py -> jsui-0.0.1/jsui
>> hard linking jsui/serve.py -> jsui-0.0.1/jsui
>> hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
>> hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
>> hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
>> hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
>> hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
>> Writing jsui-0.0.1/setup.cfg
>> Creating tar archive
>> removing 'jsui-0.0.1' (and everything under it)
>> running build
>> running build_py
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/809234c5/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Fri Aug 29 00:46:45 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 28 Aug 2014 23:46:45 +0100
Subject: [Distutils] defining requirements on pypi
In-Reply-To: <9EF7A89A-B572-4DC4-BC0F-B2D8A6C9D2A3@stufft.io>
References: <CAHYXAX45+f5N41S3=HDFsV0xfkOjNVLr+v9krrY8f0y+1fGfeg@mail.gmail.com>
 <CAG8k2+5tftthZ5Jj34g-c+xZ8mMKEG545S6ftvpxTPXz3raLmw@mail.gmail.com>
 <9EF7A89A-B572-4DC4-BC0F-B2D8A6C9D2A3@stufft.io>
Message-ID: <1409266005.79644.YahooMailNeo@web172401.mail.ir2.yahoo.com>

> PyPI won?t show metadata for Wheels either unless you upload them with Twine

Why is that, exactly? Is there something alternate tools would need to do to achieve the same thing?

Regards,

Vinay Sajip


________________________________
 From: Donald Stufft <donald at stufft.io>
To: Daniel Holth <dholth at gmail.com> 
Cc: Brett Graham <brettgraham at gmail.com>; DistUtils mailing list <distutils-sig at python.org> 
Sent: Thursday, 28 August 2014, 23:11
Subject: Re: [Distutils] defining requirements on pypi
 


PyPI doesn?t parse anything, metadata is passed alongside the file upload. This means that for sdists it would just need modified to communicate that information to PyPI. PyPI won?t show metadata for Wheels either unless you upload them with Twine.


On Aug 28, 2014, at 6:02 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>Pypi only parses requirements from wheels for display but this doesn't affect installation.
>On Aug 28, 2014 7:49 AM, "Brett Graham" <brettgraham at gmail.com> wrote:
>
>Hi,
>>
>>
>>I feel like these are stupid questions but I cannot seem to find a straight answer.
>>
>>
>>In brief,
>>
>>
>>1) what is egg-info/requires.txt used for?
>>2) how do I properly define requirements for pypi?
>>
>>
>>The details are:
>>
>>
>>I'm updating some packages on pypi and am having difficulty defining requirements.
>>
>>
>>One of the packages in question is: pypi.python.org/pypi/jsui
>>
>>
>>I'm initially defining the requirements in a requirements.txt that then gets parsed in setup.py and install_requires gets set to the contents of requirements.txt. Some of the output from python setup.py sdist build is below. The resulting requires.txt in jsui.egg-info is:
>>
>>
>>flask
>>wsrpc
>>
>>
>>However, when I upload this to pypi with "python setup.py sdist upload" I'm not seeing these requirements listed nor does pip installing the package install the requirements.
>> 
>>Thanks for your help.
>>
>>
>>==== python setup.py sdist build partial output ====
>>
>>
>>running sdist
>>running egg_info
>>writing requirements to jsui.egg-info/requires.txt
>>writing jsui.egg-info/PKG-INFO
>>writing top-level names to jsui.egg-info/top_level.txt
>>writing dependency_links to jsui.egg-info/dependency_links.txt
>>reading manifest file 'jsui.egg-info/SOURCES.txt'
>>writing manifest file 'jsui.egg-info/SOURCES.txt'
>>running check
>>warning: check: missing required meta-data: url
>>
>>
>>creating jsui-0.0.1
>>creating jsui-0.0.1/jsui
>>creating jsui-0.0.1/jsui.egg-info
>>making hard links in jsui-0.0.1...
>>hard linking README -> jsui-0.0.1
>>hard linking setup.py -> jsui-0.0.1
>>hard linking jsui/__init__.py -> jsui-0.0.1/jsui
>>hard linking jsui/serve.py -> jsui-0.0.1/jsui
>>hard linking jsui.egg-info/PKG-INFO -> jsui-0.0.1/jsui.egg-info
>>hard linking jsui.egg-info/SOURCES.txt -> jsui-0.0.1/jsui.egg-info
>>hard linking jsui.egg-info/dependency_links.txt -> jsui-0.0.1/jsui.egg-info
>>hard linking jsui.egg-info/requires.txt -> jsui-0.0.1/jsui.egg-info
>>hard linking jsui.egg-info/top_level.txt -> jsui-0.0.1/jsui.egg-info
>>Writing jsui-0.0.1/setup.cfg
>>Creating tar archive
>>removing 'jsui-0.0.1' (and everything under it)
>>running build
>>running build_py
>>_______________________________________________
>>Distutils-SIG maillist  -  Distutils-SIG at python.org
>>https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
_______________________________________________
>Distutils-SIG maillist  -  Distutils-SIG at python.org
>https://mail.python.org/mailman/listinfo/distutils-sig
>

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/c5f18001/attachment.html>

From yasumoto7 at gmail.com  Fri Aug 29 07:41:18 2014
From: yasumoto7 at gmail.com (Joe Smith)
Date: Thu, 28 Aug 2014 22:41:18 -0700
Subject: [Distutils] Handling Case/Normalization Differences
In-Reply-To: <09F1A8B9-AC9F-42DB-AEC3-2B3BED922393@stufft.io>
References: <26B92C5C-FE81-4D7D-BEC1-7E89D7791451@stufft.io>
 <25F9EB0D-6893-4BB6-AB74-8B13F29FDF64@stufft.io>
 <1862EC8F-69C3-4FAF-A582-2679146B9DF2@stufft.io>
 <CA+y6w6rzg=RJMKd=9YEdTuowt_GUGFM=0j7OhONT4Y1RhA=o2g@mail.gmail.com>
 <09F1A8B9-AC9F-42DB-AEC3-2B3BED922393@stufft.io>
Message-ID: <CA+y6w6oJ49+TgBomVEvaJQw7pqZOt=BhN7rfKcAap=5gdgHHUA@mail.gmail.com>

Ah, I didn't think of that- good point. +1 to your suggested approach.


On Thu, Aug 28, 2014 at 3:41 PM, Donald Stufft <donald at stufft.io> wrote:

> Since pip 1.4 it does yes, however the problem here is that typically
> bandersnatch
> mirrors are simply hosted by plain static web servers and don?t require
> any sort of runtime logic.
>
> On Aug 28, 2014, at 6:39 PM, Joe Smith <yasumoto7 at gmail.com> wrote:
>
> Naive question- does pip send over a UserAgent (or something) that
> contains a version number the server can use to determine which behavior to
> default to?
>
> That would allow a deprecation cycle of N months or so that will let
> people upgrade from 1.5 to 1.6. We could then watch usage of 1.5 decrease
> over time until it's a non-factor.
>
>
> On Thu, Aug 28, 2014 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 28, 2014, at 6:09 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>> Right now the ?canonical? page for a particular project on PyPI is
>> whatever the
>> author happened to name their package (e.g. Django). This requires PyPI
>> to have
>> some "smarts" so that it can redirect things like /simple/django/ to
>> /simple/Django/ otherwise someone doing ``pip install django`` would fall
>> back
>> to a much worse behavior.
>>
>> If this redirect doesn't happen, then pip will issue a request for just
>> /simple/ and look for a link that, when both sides are normalized,
>> compares
>> equal to the name it's looking for. It will then follow the link, get
>> /simple/Django/ and everything works... Except it doesn't. The problem
>> here
>> comes from the external link classification that we have now. Pip sees the
>> link to /simple/Django/ as an external link (because it lacks the required
>> rels) and the installation finally fails.
>>
>> The /simple/ case rarely happens when installing from PyPI itself because
>> of
>> the redirect, however it happens quite often when someone is attempting to
>> instal from a mirror instead. Even when everything works correctly the
>> penality
>> for not knowing exactly what name to type in results in at least 1 extra
>> http
>> request, one of which (/simple/) requires pulling down a 2.1MB file.
>>
>> To fix this I'm going to modify PyPI so that it uses the normalized name
>> in
>> the /simple/ URL and redirects everything else to the non-normalized
>> name. I'm
>> also going to submit a PR to bandersnatch so that it will use normalized
>> names
>> for it's directories and such as well. These two changes will make it so
>> that
>> the client side will know ahead of time exactly what form the server
>> expects
>> any given name to be in. This will allow a change in pip to happen which
>> will pre-normalize all names which will make the interaction with mirrors
>> better
>> and will reduce the number of HTTP requests that a single ``pip install``
>> needs
>> to make.
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> Hm, so here?s the problem.
>>
>> I have this implemented and deployed to TestPyPI, it works great!
>>
>> However, the next step is to make the change to bandersnatch so that it
>> saves
>> things using their normalized name instead of using their "proper" name.
>> Doing
>> this will trigger it so that everyone using pip 1.5 won't be able to
>> install
>> anything from that mirror unless it's name is specified as the normalized
>> name
>> (e.g. ``pip install Django`` will fail without --allow-unverified but
>> ``pip install django`` will work). This would be fixed with pip 1.6 (since
>> it would know to "normalize" the name before fetching the URL).
>>
>> The same thing will occur if we make the change in pip first, it would
>> normalize names so you'd need to use --allow-unverified for everything
>> because
>> it would act as if you typed ``pip install django`` instead of ``pip
>> install
>> Django``.
>>
>> To my knowledge, this *only* will affect pip 1.5.x.
>>
>> So the only way forward I can see to make this change, which I think is a
>> good
>> change and will remove a big "gotcha" from using a mirror, is to
>> coordinate
>> a release of bandersnatch that coincides with pip 1.6, and tell people
>> they
>> need to upgrade in lockstep.
>>
>> Does anyone have any other ideas?
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> Just thought of this, if the normalized name doesn?t match the "real"
>> name,
>> then add entries for both. This will make it so that pip 1.5 continues to
>> work
>> and pip 1.6+.
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/92ec178c/attachment.html>