[Distutils] version scheme: a case for dropping ".devNNN" and ".postNNN"

Thu Jun 11 11:29:31 CEST 2009

PEP 386 is proposing a new versioning scheme for distutils. One that
currently includes ways to spell development releases (with a ".devNNN"
suffix) and "post-releases" (with a ".postNNN" suffix).

This is a long email. You can jump to the bottom for conclusions.

First a survey of what "the other guys" do.

# Perl

CPAN. A free for all. A couple quotes from a Perl core developer here at
ActiveState:

> There is no standardized version numbering scheme.  This module is supposed
> to be it:
>
>    http://search.cpan.org/~jpeacock/version-0.76/lib/version.pod
>
> But I think it probably created as many problems as it solved.

# Tcl

http://www.tcl.tk/man/tcl8.5/TclCmd/package.htm#M22

Strict version scheme, and because this is part of a core language feature
it is effectively enforced. The scheme is:

    N[.N]+[(a|b)N]

For examples of versions, ActiveState has a repo of binary builds of Tcl
extensions called "teapot" (the "TEA" stands for Tcl Extension Architecture)
here: http://teapot.activestate.com/index

# PHP

PEAR, http://pear.php.net/packages.php
535 packages: relatively small

Strict version scheme (http://pear.php.net/group/docs/20040226-vn.php)
"which will be mandatory for all packages from now on". They *do* support
a "dev" state -- "0.9.5 (dev)" -- that is less than an "alpha" state,
but NOT with an attached revision/date numeric value.

Note that I did some poking around on the PEAR site and I couldn't find
any packages with "dev"-state release.

# Others?

Anybody?

# Some current PyPI numbers

Conclusions are **bolded** for skimming.

Using the PyPI XML-RPC API I got a dump of all current project names
and the list of versions of uploaded packages for that project. A line
like this for each project:

    Markdown: 1.7, 1.7, 1.7, 2.0, 2.0, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.1

This shows the order in which packages were uploaded... duplicates because
separate package types for the same version (sdist, bdist_egg, etc.).

    script: http://dl.getdropbox.com/u/1301040/distutilsver/gatherpypi.py
    data: http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name.txt

The current verlib.py has a "suggest_rational_version" that has pretty
good coverage on getting the equivalent rational version number from
a pypi version (16133 out of 17089 in my count). I counted uses of ".devNNN"
and ".postNNN" in all the currently used PyPI versions:

**Out of a total 17089 versions released to PyPI there are 281 (in 90
of about 6726 projects, 1.3%) that use a post-release tag.** These are:
http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_using_post.txt

**Out of a total 17089 versions released to PyPI there are 738 (in 292
of about 6726 projects, 4.3%) that use a "dev" tag.** These are:
http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_using_dev.txt

Looking at those that do there is a much lower percentage that *frequently*
use the post- and pre-release tags. For example, two of SQLObject's
*many* releases using a pre-release "dev-rNNNN" tag:

    SQLObject: 0.9.5, 0.7.8, 0.8.5, 0.9.1, 0.8.6b1, 0.9.2b1, 0.7.9b1,
        0.8.6b2, 0.7.9b2, 0.9.2b2, 0.7.9b3, 0.8.6b3, 0.9.2b3, 0.7.9, 0.8.6,
        0.9.2, 0.7.10, 0.9.3, 0.8.7, 0.10.0b1, 0.10.0b2, 0.9.4b1, 0.10.0b3,
        0.10.0b4, 0.9.4, 0.10.0, 0.10.1, 0.9.6, 0.10.2b1, 0.9.7b1,
        0.10.2b1dev-r3466, 0.9.7, 0.10dev-r3481, 0.10.2, 0.10.3, 0.9.8, 0.9.9,
        0.10.4, 0.9.10, 0.10.5, 0.10.6, 0.9.11

**I.e. some might have just been a one off trying out the related setuptools
command-line options for this. For others it might have just been a
mistake.** For example, notice that the SQLObject "0.10.2b1dev-r3466"
release happened *after* the "0.10.2b1". That "0.10.2b1dev-r3466" SQLObject
release has subsequently been deleted from PyPI (indicating a mistake),
though older releases are still there (e.g.
http://pypi.python.org/pypi/SQLObject/0.9.3).

There are indicates that the setuptools "-rNNNN" *post-release* tag is
being accidentally mis-interpreted. According to setuptools:

    >>> from pkg_resources import parse_version as V
    >>> V("0.7") < V("0.7-r15094")
    True

However, looking at the "Chandler-EVDBPlugin" project: "0.7-r15094" was
uploaded 2007-07-24, and "0.7" was uploaded 2007-09-11 (later). Basically
I suspect the maintainer of "Chandler-EVDBPlugin" misinterpreted how to
use the setuptools "--tag-svn-revision" facility. **I.e. I suspect there
are significantly fewer than 1% of projects uploading packages to PyPI
that need/want a "post-release" tag facility**.

It would be interesting to get numbers of whether this setuptools
dependency pattern is at all frequently used:

    install_requires = ["OtherProject>=0.2a1.dev-r143"]

I.e. a dependency on an uploaded-to-PyPI "dev" release.

# The "Twisted" example

The Twisted example was a justifier for including ".postNNNN" in the versioning
scheme. See my previous email:
http://mail.python.org/pipermail/distutils-sig/2009-June/012161.html

However, while it appears to be true that Twisted doesn't
appear to increment the version number in the source code until very
close to release [1] (opposite to what Python itself, for example, does),
I'm not sure this constitutes "using post-releases".

- Twisted doesn't release packages to PyPI
(http://pypi.python.org/pypi/Twisted/)
- The archive of Twisted releases doesn't include post-releases.
  http://tmrc.mit.edu/mirror/twisted/Twisted/
- Perhaps not relevant, but Twisted's "Version" class seems to indicate a
  leaning towards "*pre*releases"
  http://twistedmatrix.com/trac/browser/trunk/twisted/python/versions.py#L38

[1] When correlating changes to Twisted's version file
    http://twistedmatrix.com/trac/browser/trunk/twisted/_version.py
    to Twisted release dates.

# My conclusion/opinion

In so far as the proposed new distutils versioning scheme is to be used
for enforcing/encouraging versions for packages uploaded to PyPI, I don't
believe there is a strong case for including ".devNNN" and certainly not
".postNNN".

- It just isn't used often enough to justify the complexity in the version
  scheme for packages on PyPI.
- Setuptools' facility to do `install_requires = ["OtherProject==dev"]`
  is a good thing, but this doesn't require the ".devNNNN" and post-release
  tags on sdist/bdist packages uploaded to PyPI.
- The other language repo systems don't demonstrate a need for this.
- Pressure on project authors to do normal releases (rather than having a
quick "dev sorta-release") when user's want it is a *good thing*.
- Dropping this complexity in the version scheme documentation for users
would allow us to spend more time educating Python module authors on
more important best practices like incrementing the major version number
for backward incompatible changes.

Cheers,
Trent

-- 
Trent Mick
trentm at gmail.com