Here-in the second part of my recollections of the Pycon distutils versioning
discussions. This first part is here:
http://mail.python.org/pipermail/distutils-sig/2009-June/012143.html
The first part left us with this pseudo-pattern:
N.N[.N]*[(abc)N[.N]*]
Some examples to help illustrate:
1.0
1.4.2.45.2.2.5
1.2.3a4
1.34.5.6c1
3.0b2.1
That leaves the ".devN" and ".postN" business of the current version
proposal (PEP 386).
# A brief setuptools lesson
(Mainly this is because I needed it -- I'm not intimately familiar with
setuptools. Skip this section if you know this already.)
Setuptools defines
(http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-…)
"pre-release tags" and "post-release tags" on a version number. In setuptools
one part of the "pre-release tags" are the "a", "b", "c" (for alpha, beta,
candidate) parts of the proposed pattern that we've already discussed. The
remaining part of "pre-release tags" is the use of "dev" or "pre", e.g.:
1.0dev
2.5.2pre
3.1dev459259
(Strictly speaking also "preview" and any string that sorts less than
"final", but "dev" and "pre" are the common ones.)
"Post-release tags" are any non-numeric in the version string that sorts
greater than "final" or the special case "-". Practically speaking it is
almost always the latter because setuptools provides command-line
options (http://peak.telecommunity.com/DevCenter/setuptools#egg-info) to
automatically append an svn rev ("-rNNNNN") or a date ("-YYYYMMDD").
Examples:
1.0-r2345
5.4.3-20090107
# ".devNNNN" in the new versioning proposal
Reminder: I'm just trying to describe my recollections of the Pycon
discussions here. I'll post my current opinions in a separate email. I.e.,
don't shoot the messenger. :)
In general, people tended to agree that development builds are a reasonable
thing. Your project is in the run-up to 1.0 and you'll be doing regular
builds at incremental revisions of your source code control system.
Setuptools encourages (by offering certain command-line options) spelling
this kind of thing this way (though, of course there are other spellings of
the equivalent):
1.0dev-r342
1.0dev-r343
1.0dev-r350
1.0dev-r355
At this point in the Pycon discussions I vaguely recall that there was some
opposition to having support for spelling development builds in the
versioning scheme at all. However, I don't fully recall. If so, they were won
over (or drowned out) by (a) others giving examples of doing daily/nightly
builds of product X -- including me describing nightlies of the Komodo
product (not a
Python project) I work on -- and (b) the start of the debate on how to spell
"1.0dev-r342", those presuming that its inclusion was a forgone conclusion.
I'll touch more on this in a followup email.
Ensuing debate, pleading, fisticuffs, cajoling ... leading to
N.N[.N]*[(abc)N[.N]*][.devN]
1.0.dev456
2.5.0b2.dev523
Why "dev" at all and not "-", "_", or "~" or just another "."? None of the
alternatives were explicit that this is a release that should sort *earlier*
than the version without it. Also, '-' is used to setuptools to mean a
*post*-release; '_' causes problems for Debian packaging; '~' is used in
Debian package versioning (IIRC, best to avoid using it); just '.' (e.g.
'1.0a1.456') can be ambiguous.
Why "dev"? Because it is by far the most common string used for this in
current PyPI versions. "pre" a far second.
Why ".devN" (1.0a1.dev456) instead of just "devN" (1.0a1dev456)? Partly a
feeling that the '.' helped separate from the leading part -- if "1.0.a1"
didn't look ridiculous we probably would have chosen that too. :)
Also partly because there is justifiably little point in quibbling for
hours over a single '.' here.
# ".postNNNN" in the new versioning proposal
Amid fading sounds of hands being dusted off, backs being patted and
rising discussions of what's for dinner ("Salad bar or steak?" "What's
the special tonight?") the question:
"What about post-releases?" -- Zooko
Groans. "Who uses that?" "Twisted, for example." "Really? Hrm, okay,
let's tack on a '.postN' (like the '.devN') and call it a night."
The discussion *was* longer than that, but not a whole lot longer. Mostly the
justification for ".postN" in the proposal rested on the example of at least
a couple significant Python projects using this (e.g. Twisted). An example
of the resulting sorting order:
1.0a1
1.0a2.dev456
1.0a2
1.0a2.1.dev456
1.0a2.1
1.0b1.dev456
1.0b2
1.0c1.dev456
1.0c1
1.0.dev456
1.0
1.0.post456
and the final pseudo-pattern:
N.N[.N]*[(a|b|c)N[.N]*][(.devN|.postN)]
Those are my recollections of the Pycon versioning discussions. I want to
following up (in a separate email) with my current opinions.
Cheers,
Trent
--
Trent Mick
trentm(a)gmail.com
I've reworked the APIs using all the feedback:
- now using basic types (dict + set) to make the code simpler
- renamed the classes using the "Distribution" root name
- the global functions are also now sys.path-based and there's no more
global variable
- egg_info -> egginfo
- ...
Please comment !
On Tue, Jun 9, 2009 at 12:07 AM, Kevin Teague<kevin(a)bud.ca> wrote:
> Glancing over the EggInfo, EggInfoDirectories, and EggInfoDirectory
> APIs, it might be interesting if these were able to match with the PEP
> 3119 interfaces (http://www.python.org/dev/peps/pep-3119/).
>
> For example, the EggInfoDirectories has __iter__, append, clear and
> remove methods, but the source code comments list them as 'container
> APIs'. Technically though, that's an Iterable interface and a partial
> implementation of the MutableSequence interface.
>
> The EggInfoDirectories.get_egg_infos(project_name) is also suggestive
> of the Mapping interface - if this method was shortened from
> get_egg_infos() to just get().
>
> Maybe it would be confusing if an EggInfoDirectories was a
> MutableSequence which could only have paths added to it, but returned
> an iterable of EggInfoDirectory, but could also be accessed as a
> Mapping where the keys are project names and those values are
> EggInfos ... but maybe something along those lines would work?
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG(a)python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
--
Tarek Ziadé | http://ziade.org
I've run into a problem where easy_install/setuptools continues to
recommend upgrading itself even after the upgrade has been installed. Steps
to reproduce on CentOS 5.2:
include ~/python in PYTHONPATH
include ~/bin in PATH
create ~/.pydistutils.cfg
[install]
install_lib = ~/python
install_scripts = ~/bin
$ easy_install flup
...
Running flup-1.0.3.dev-20090608/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-rPyZ6a/flup-1.0.3.dev-20090608/egg-dist-tmp-pm0o4x
The required version of setuptools (>=0.6c9) is not available, and
can't be installed while this script is running. Please install
a more recent version first, using 'easy_install -U setuptools'.
(Currently using setuptools 0.6c5 (/usr/lib/python2.4/site-packages))
error: Setup script exited with 2
$ easy_install -U setuptools
...
Processing dependencies for setuptools
$ which easy_install
~/bin/easy_install
$ python -c 'import setuptools; print setuptools.__version__'
0.6c9
$ easy_install flup
...
Running flup-1.0.3.dev-20090608/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-WoxTch/flup-1.0.3.dev-20090608/egg-dist-tmp-oFotRt
The required version of setuptools (>=0.6c9) is not available, and
can't be installed while this script is running. Please install
a more recent version first, using 'easy_install -U setuptools'.
(Currently using setuptools 0.6c5 (/usr/lib/python2.4/site-packages))
error: Setup script exited with 2
At this point I thought it might be a problem with flup's setup.py or
ez_setup.py, but manually running python setup.py bdist_egg outside of
easy_install works fine.
Python svn builds (http://pyvm.sourceforge.net) is a project to provide
python binary builds ready to use.
The main features for these builds are:
+ install under /opt/opt-python-svnXXXXX-2.7a0 (XXXXX is the svn revision)
+ multiple interpreters can be installed side by side
+ installed in parallel to the system python (administrator friendly)
+ they're build for several different platforms (rpm based)
+ the build host is provided by the OpenSUSE buildserver
+ the builds can be deployed using yum/yast/urpmi (OpenSUSE buildserver)
+ each build runs a small smoke test and it is unattented
+ the test results are reported
The aim is:
+ implementing a continuous integration practice
+ to provide python developer with reliable and repeatable builds
+ modules developer can test their modules against the latest python
News for this version:
Few modules (imaging, numpy, pexpect and setuptools) have been added and
they're build for each interpreter.
These are still in testing but the aim here is to have a full complete build
chain with a test suite (next step is providing a test driven development).
Download and documentation from:
http://pyvm.sourceforge.net
Regards,
Antonio
References:
OpenSUSE buildserver: https://build.opensuse.org
At 03:16 PM 6/11/2009 +0100, Paul Moore wrote:
>2009/6/11 P.J. Eby <pje(a)telecommunity.com>:
> > PyPI uploads aren't a suitable basis for analyzing "dev" use cases, since
> > the whole point of having a "dev" tag is for *non-released* versions.
> > (E.g., in-progress development via SVN.)
>
>If it's non-released, I've yet to see a clear explanation of why the
>PEP is relevant. Who is going to use an API from the PEP to parse your
>"version number", and why?
>
> > Dev tags are so that while you're
> > doing development, your locally-installed versions can be
> distinguished from
> > one another.
>
>Distinguished by what? What code (that you didn't write yourself,
>purely for internal use) needs to parse your dev tag?
Distinguished by setuptools for processing version requirements of
scripts, or require() statements in code, and installation
requirements of newly-installed code.
For example, if I'm working on two projects that are distributed via
SVN and one depends on the other, if I update one, it may require an
update of the other; the failure of the .dev#### version requirement
in the first one will inform me of the need to "svn up" the second
project and rerun "setup.py develop" on it.
This is a routine circumstance in at least my development cycle; I
would expect that it's the case in other open source development
workflows as well as proprietary ones.
At 03:48 PM 6/11/2009 +0100, Paul Moore wrote:
>Please note, in case I seem excessively aggressive - my assumption is
>that the standard defined by the PEP doesn't have to be used by
>everyone, in all cases. You presumably have code that works at the
>moment, and you are perfectly entitled to continue using it.
That depends on whether this PEP's version scheme is also going to be
embedded in the code implied by PEP 376. If so, then nobody will
have any reason to switch from using pkg_resources to the PEP 376
pkgutil code, because they won't be able to manage their test, dev,
and collaboration environments that way.
> If you
>switch to the APIs proposed in the PEP, then presumably you see some
>benefits in following the standard.
That's precisely it - there aren't any visible benefits to doing so,
but there *are* visible problems.
We need to have a way to translate existing version schemes to the
new one, if we're to be able to have a reasonable upgrade path --
considering that these PEPs' only benefit to adopters (at least among
those using setuptools) is that the code will be in some future
stdlib version, and support simple uninstalls.
At 02:29 AM 6/11/2009 -0700, Trent Mick wrote:
>It would be interesting to get numbers of whether this setuptools
>dependency pattern is at all frequently used:
>
> install_requires = ["OtherProject>=0.2a1.dev-r143"]
>
>I.e. a dependency on an uploaded-to-PyPI "dev" release.
PyPI uploads aren't a suitable basis for analyzing "dev" use cases,
since the whole point of having a "dev" tag is for *non-released*
versions. (E.g., in-progress development via SVN.) Dev tags are so
that while you're doing development, your locally-installed versions
can be distinguished from one another.
PEP 386 is proposing a new versioning scheme for distutils. One that
currently includes ways to spell development releases (with a ".devNNN"
suffix) and "post-releases" (with a ".postNNN" suffix).
This is a long email. You can jump to the bottom for conclusions.
First a survey of what "the other guys" do.
# Perl
CPAN. A free for all. A couple quotes from a Perl core developer here at
ActiveState:
> There is no standardized version numbering scheme. This module is supposed
> to be it:
>
> http://search.cpan.org/~jpeacock/version-0.76/lib/version.pod
>
> But I think it probably created as many problems as it solved.
# Tcl
http://www.tcl.tk/man/tcl8.5/TclCmd/package.htm#M22
Strict version scheme, and because this is part of a core language feature
it is effectively enforced. The scheme is:
N[.N]+[(a|b)N]
For examples of versions, ActiveState has a repo of binary builds of Tcl
extensions called "teapot" (the "TEA" stands for Tcl Extension Architecture)
here: http://teapot.activestate.com/index
# PHP
PEAR, http://pear.php.net/packages.php
535 packages: relatively small
Strict version scheme (http://pear.php.net/group/docs/20040226-vn.php)
"which will be mandatory for all packages from now on". They *do* support
a "dev" state -- "0.9.5 (dev)" -- that is less than an "alpha" state,
but NOT with an attached revision/date numeric value.
Note that I did some poking around on the PEAR site and I couldn't find
any packages with "dev"-state release.
# Others?
Anybody?
# Some current PyPI numbers
Conclusions are **bolded** for skimming.
Using the PyPI XML-RPC API I got a dump of all current project names
and the list of versions of uploaded packages for that project. A line
like this for each project:
Markdown: 1.7, 1.7, 1.7, 2.0, 2.0, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.1
This shows the order in which packages were uploaded... duplicates because
separate package types for the same version (sdist, bdist_egg, etc.).
script: http://dl.getdropbox.com/u/1301040/distutilsver/gatherpypi.py
data: http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name.txt
The current verlib.py has a "suggest_rational_version" that has pretty
good coverage on getting the equivalent rational version number from
a pypi version (16133 out of 17089 in my count). I counted uses of ".devNNN"
and ".postNNN" in all the currently used PyPI versions:
**Out of a total 17089 versions released to PyPI there are 281 (in 90
of about 6726 projects, 1.3%) that use a post-release tag.** These are:
http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_usi…
**Out of a total 17089 versions released to PyPI there are 738 (in 292
of about 6726 projects, 4.3%) that use a "dev" tag.** These are:
http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_usi…
Looking at those that do there is a much lower percentage that *frequently*
use the post- and pre-release tags. For example, two of SQLObject's
*many* releases using a pre-release "dev-rNNNN" tag:
SQLObject: 0.9.5, 0.7.8, 0.8.5, 0.9.1, 0.8.6b1, 0.9.2b1, 0.7.9b1,
0.8.6b2, 0.7.9b2, 0.9.2b2, 0.7.9b3, 0.8.6b3, 0.9.2b3, 0.7.9, 0.8.6,
0.9.2, 0.7.10, 0.9.3, 0.8.7, 0.10.0b1, 0.10.0b2, 0.9.4b1, 0.10.0b3,
0.10.0b4, 0.9.4, 0.10.0, 0.10.1, 0.9.6, 0.10.2b1, 0.9.7b1,
0.10.2b1dev-r3466, 0.9.7, 0.10dev-r3481, 0.10.2, 0.10.3, 0.9.8, 0.9.9,
0.10.4, 0.9.10, 0.10.5, 0.10.6, 0.9.11
**I.e. some might have just been a one off trying out the related setuptools
command-line options for this. For others it might have just been a
mistake.** For example, notice that the SQLObject "0.10.2b1dev-r3466"
release happened *after* the "0.10.2b1". That "0.10.2b1dev-r3466" SQLObject
release has subsequently been deleted from PyPI (indicating a mistake),
though older releases are still there (e.g.
http://pypi.python.org/pypi/SQLObject/0.9.3).
There are indicates that the setuptools "-rNNNN" *post-release* tag is
being accidentally mis-interpreted. According to setuptools:
>>> from pkg_resources import parse_version as V
>>> V("0.7") < V("0.7-r15094")
True
However, looking at the "Chandler-EVDBPlugin" project: "0.7-r15094" was
uploaded 2007-07-24, and "0.7" was uploaded 2007-09-11 (later). Basically
I suspect the maintainer of "Chandler-EVDBPlugin" misinterpreted how to
use the setuptools "--tag-svn-revision" facility. **I.e. I suspect there
are significantly fewer than 1% of projects uploading packages to PyPI
that need/want a "post-release" tag facility**.
It would be interesting to get numbers of whether this setuptools
dependency pattern is at all frequently used:
install_requires = ["OtherProject>=0.2a1.dev-r143"]
I.e. a dependency on an uploaded-to-PyPI "dev" release.
# The "Twisted" example
The Twisted example was a justifier for including ".postNNNN" in the versioning
scheme. See my previous email:
http://mail.python.org/pipermail/distutils-sig/2009-June/012161.html
However, while it appears to be true that Twisted doesn't
appear to increment the version number in the source code until very
close to release [1] (opposite to what Python itself, for example, does),
I'm not sure this constitutes "using post-releases".
- Twisted doesn't release packages to PyPI
(http://pypi.python.org/pypi/Twisted/)
- The archive of Twisted releases doesn't include post-releases.
http://tmrc.mit.edu/mirror/twisted/Twisted/
- Perhaps not relevant, but Twisted's "Version" class seems to indicate a
leaning towards "*pre*releases"
http://twistedmatrix.com/trac/browser/trunk/twisted/python/versions.py#L38
[1] When correlating changes to Twisted's version file
http://twistedmatrix.com/trac/browser/trunk/twisted/_version.py
to Twisted release dates.
# My conclusion/opinion
In so far as the proposed new distutils versioning scheme is to be used
for enforcing/encouraging versions for packages uploaded to PyPI, I don't
believe there is a strong case for including ".devNNN" and certainly not
".postNNN".
- It just isn't used often enough to justify the complexity in the version
scheme for packages on PyPI.
- Setuptools' facility to do `install_requires = ["OtherProject==dev"]`
is a good thing, but this doesn't require the ".devNNNN" and post-release
tags on sdist/bdist packages uploaded to PyPI.
- The other language repo systems don't demonstrate a need for this.
- Pressure on project authors to do normal releases (rather than having a
quick "dev sorta-release") when user's want it is a *good thing*.
- Dropping this complexity in the version scheme documentation for users
would allow us to spend more time educating Python module authors on
more important best practices like incrementing the major version number
for backward incompatible changes.
Cheers,
Trent
--
Trent Mick
trentm(a)gmail.com
At 06:01 PM 6/11/2009 +0100, Paul Moore wrote:
>2009/6/11 P.J. Eby <pje(a)telecommunity.com>:
> >> > Dev tags are so that while you're
> >> > doing development, your locally-installed versions can be distinguished
> >> > from one another.
> >>
> >> Distinguished by what? What code (that you didn't write yourself,
> >> purely for internal use) needs to parse your dev tag?
> >
> > Distinguished by setuptools for processing version requirements of scripts,
> > or require() statements in code, and installation requirements of
> > newly-installed code.
>
>So will setuptools be modified to use the new code?
It doesn't have to be; the RationalVersion scheme is both a strict
subset of setuptools' scheme, and also has the property that most
in-field uses of setuptools-based versioning schemes can be
*mechanically* translated to RationalVersion instances, using an
appropriate implementation of suggest_version().
This means that if somebody chooses to adopt RationalVersion (as
currently defined in the PEP), setuptools will read and process it as
correctly as it does more exotic numbering schemes.
IOW, setuptools doesn't need to be changed, unless it was to say, add
warnings for non-convertible versions... and that could actually be
done with a plugin.
I tried to build java2python using bdist_rpm on Fedora 10 and got the
following errors:
RPM build errors:
Installed (but unpackaged) file(s) found:
/usr/lib/python2.5/site-packages/java2python/__init__.pyo
/usr/lib/python2.5/site-packages/java2python/lib/__init__.pyo
/usr/lib/python2.5/site-packages/java2python/lib/defaultconfig.pyo
/usr/lib/python2.5/site-packages/java2python/lib/lexer.pyo
/usr/lib/python2.5/site-packages/java2python/lib/parser.pyo
/usr/lib/python2.5/site-packages/java2python/lib/sourcetypes.pyo
/usr/lib/python2.5/site-packages/java2python/lib/walker.pyo
error: command 'rpmbuild' failed with exit status 1
This had been an issue in the past, and the recommended solution was to
prepare a setup.cfg containing
[install]
optimize = 1
This had worked in the past, but it had no effect this time.
I found some discussion of the issue and some aspect of it is supposedly
addressed in a change to Fedora. I upgraded to the latest rpm packages
(version -4.6.1-1.fc10), but got the same behavior.
How can I proceed to do the build?
Thanks.
Stan Klein