[Distutils] Local version identifiers from PEP 440 in practice

Maurits van Rees m.van.rees at zestsoftware.nl
Wed Dec 17 00:53:08 CET 2014


I wanted to try out local version identifiers for packages.  Or
actually, I already used them, but with a wrong notation, and ran into
trouble, also after fixing the notation.  So I did some testing and
made notes on what does or does not work.  See conclusions at the
bottom.


Local version identifiers: general idea
---------------------------------------

PEP440 is here, specifically the local version identifiers:
http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers

Use case in short, for those who do not know what local version
identifiers are.  Upstream community package largetrout has version
1.0.  As integrator you notice a problem, fix it in the code, and
create a pull request.  While waiting for a new official release you
already want to use the new code in production.  If you release a
version 1.1 yourself, this will conflict with a later official
version.  This is where local version identifiers come in.  You create
largetrout version 1.0+local.1 that you put in a directory on a
webserver.  You add the url of the package as find-link for
pip/zc.buildout and can now use your local version.

That is the general idea.  How does it work in practice?  Two things
need to work: creating a distribution, and installing the distribution
with pip or buildout.


Creating a distribution
-----------------------

I have created a very basic python project called 'myproject'.  It
does nothing.  I have released a few versions here:
http://pypi.zestsoftware.nl/public/packagingtest/

The various versions only differ in their version numbers and in the
way in which the distributions were created.  All were created with
Python 2.7.8. on Mac OSX 10.9.5, with ``python setup.py sdist 
--formats=zip``.

- setup.py version = 1.0
   * Created with: setuptools 2.2
   * Distribution file: myproject-1.0.zip
   * PKG-INFO version: 1.0
   * works everywhere

- setup.py version = 1.1+maurits.1
   * Created with: setuptools 2.2
   * Distribution file: myproject-1.1-maurits.1.zip
   * PKG-INFO version: 1.1-maurits.1
   * NOT installable with zc.buildout 2.3 and setuptools 8 (see below)

- setup.py version = 1.1+maurits.2
   * Created with: setuptools 7.0
   * Distribution file: myproject-1.1-maurits.2.zip
   * PKG-INFO version: 1.1-maurits.2
   * NOT installable with zc.buildout 2.3 and setuptools 8 (see below)

- setup.py version = 1.1+maurits.3
   * Created with: setuptools 8.0.4
   * Distribution file: myproject-1.1+maurits.3.zip
   * PKG-INFO version: 1.1+maurits.3
   * NOT installable with current pip (see below)
   * hard to install with development pip (BUG, see below)
   * NOT installable with zc.buildout 2.2 and setuptools<8

So: with any setuptools version you can specify a local version
identifier (``+something``) in the setup.py, but you need setuptools
8.0+ to correctly handle it.  In earlier versions, the plus sign is
replaced by a dash.  This may lead to problems while installing; we
will see that below.

Note: ``python2.7 setup.py --version`` always correctly returns the
version number as specified in setup.py.


Installing with pip
-------------------

Now let's see how installing goes.  Again we are using Python 2.7.

Below, some warnings or errors are expected (possibly with a
workaround, marked with TIP), some may be surprising
(marked with DANGER), others may be bugs (marked with BUG).

I picked two combinations of pip and setuptools versions.  Other 
combinations may give other results.


current pip, setuptools 7.0
---------------------------

A basic virtualenv:

$ pip list
pip (1.5.6)
setuptools (7.0)
wsgiref (0.1.2)
$ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ 
myproject==1.0
...
$ python -c "import myproject" && echo "importing works"
importing works

We try ``pip install`` for the available versions.

- myproject==1.0
   * importing works

- myproject=1.1-maurits.1
   * importing works

- myproject==1.1-maurits.2
   * importing works

- myproject==1.1-maurits.3
   * pip says:
     Could not find a version that satisfies the requirement
     myproject==1.1-maurits.3 (from versions: 1.0, 1.1, 1.1-maurits.1,
     1.1-maurits.2)

- myproject==1.1+maurits.3
   * pip says:
     Exception:
     Traceback (most recent call last):
       ...
       File ".../pip/_vendor/pkg_resources.py", line 2583, in scan_list
         "Expected ',' or end-of-list in",line,"at",line[p:]
     ValueError: ("Expected ',' or end-of-list in", 
'myproject==1.1+maurits.3', 'at', '+maurits.3')
   * DANGER: older pip versions cannot install packages with local 
version identifiers.

- pip install 
http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip
   * importing works


development pip, setuptools 8.0
-------------------------------

A basic virtualenv:

$ pip list
pip (6.0.dev1)
setuptools (8.0.4)
$ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.0
...
$ python -c "import myproject" && echo "importing works"
importing works

We try ``pip install`` for the available versions.

- myproject==1.0
   * importing works

- myproject==1.1-maurits.1
   * pip warns:
     Requested myproject==1.1-maurits.1, but installing version 
1.1+maurits.1
   * importing works

- myproject==1.1-maurits.2
   * pip warns:
     Requested myproject==1.1-maurits.2, but installing version 
1.1+maurits.2
   * importing works

- myproject==1.1-maurits.3
   * pip says:
     Collecting myproject==1.1+maurits.3
       Could not find a version that satisfies the requirement 
myproject==1.1-maurits.3 (from versions: )
       No distributions matching the version for myproject==1.1-maurits.3

- myproject==1.1+maurits.3
   * pip says:
     Collecting myproject==1.1+maurits.3
       Could not find a version that satisfies the requirement 
myproject==1.1+maurits.3 (from versions: )
       No distributions matching the version for myproject==1.1+maurits.3
   * BUG: this should work, right?

- pip install 
http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip
   * importing works


Installing with buildout
------------------------

We try the same with buildout.  I use this basic buildout config:

[buildout]
find-links = http://pypi.zestsoftware.nl/public/packagingtest/
eggs-directory = eggs
download-cache = downloads
parts = mypy test
show-picked-versions = true

[mypy]
recipe = zc.recipe.egg
eggs = myproject
interpreter = mypy

[test]
recipe = plone.recipe.command
command = grep myproject bin/mypy && bin/mypy -c "import myproject" && 
echo "importing works"
update-command = ${:command}

[versions]
myproject = 1.0
plone.recipe.command = 1.1
#setuptools = ...
#zc.buildout = ...
zc.recipe.egg = 2.0.1


This buildout creates a bin/mypy python interpreter with the myproject
package installed.  And it greps bin/mypy for 'myproject' so I can
easily see whether this has the version I expect.  And it checks to
see it importing the module works.


Using previous zc.buildout and setuptools
-----------------------------------------

zc.buildout = 2.2.5
setuptools = 7.0

- myproject = 1.0
   * all is well

- myproject = 1.1-maurits.1
   * all is well

- myproject = 1.1-maurits.2
   * all is well

- myproject = 1.1-maurits.3
   * all is well

- myproject = 1.1+maurits.3
   * buildout fails:
     Traceback (most recent call last):
       ...
       File ".../setuptools-7.0-py2.7.egg/pkg_resources.py", line 2690, 
in scan_list
         raise ValueError(msg, line, "at", line[p:])
     ValueError: ("Expected ',' or end-of-list in", 
'myproject[]==1.1+maurits.3', 'at', '+maurits.3')
   * TIP: you can use a dash instead of a plus in the version pin.


Using latest zc.buildout and setuptools
---------------------------------------

zc.buildout = 2.3.1
setuptools = 8.0.2

- myproject = 1.0
   * all is well

- myproject = 1.1-maurits.1
   * buildout fails:
     Installing myproject 1.1-maurits.1
     Caused installation of a distribution:
     myproject 1.1+maurits.1
     with a different version.
     Got None.
     While:
       Updating mypy.
     Error: There is a version conflict.
     We already have: myproject 1.1+maurits.1
     We require myproject==1.1-maurits.1

   * when I do not specify a version in the buildout config, but the
     latest on server is 1.1-maurits.1, buildout succeeds and importing
     works.  buildout reports this (note the plus-sign):

     Versions had to be automatically picked.
     The following part definition lists the versions picked:
     [versions]
     myproject = 1.1+maurits.1

   * BUG?  pip only warns, buildout may want to do the same.  Maybe
     somehow normalize versions before comparison.

- myproject = 1.1-maurits.2
   * buildout fails:
     Installing myproject 1.1-maurits.2
     Caused installation of a distribution:
     myproject 1.1+maurits.2
     with a different version.
     ...

   * So: same remarks as for 1.1-maurits.1 apply.

- myproject = 1.1-maurits.3
   * buildout fails:
     Error: Couldn't find a distribution for 'myproject==1.1-maurits.3'.

- myproject = 1.1+maurits.3
   * importing works


Conclusions
-----------

- BUG in pip: development version of pip cannot find local version
   identifiers (version numbers with a plus sign).

- TIP: do NOT use setuptools 7.0 or older to create a distribution
   with a local version identifier.  Installation of this may fail,
   because the version according to the filename differs from the
   actual version.

- BUG in buildout?  Where pip only warns about non-matching versions
   from the above tip, buildout actually quits with an error.  Maybe
   somehow normalize versions before comparison.

- TIP: only use local version identifiers when you use setuptools 8 or
   higher both when creating and when using the distribution.  Then
   installing with zc.buildout will work.  Installing with pip can be
   done by pointing to the exact file url.



-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl



More information about the Distutils-SIG mailing list