Here-in my recollections of the Pycon distutils versioning
discussions... but only
up to the part where reasoning for the ".dev" and ".post" parts are added to the
Hopefully the following will be helpful for reference in the current
- as self-explanatory and clear versioning as reasonable. A whole dog's
breakfast of versions is just a pain.
- deterministic translation btwn version string and version tuple. Bonus
points if two reasonable humans would sort them the same way.
- capable dependency specs and reasonably readable
When I say things like "this isn't common on PyPI" below, this is from
analysing a dump of all the version strings currently in use on PyPI. This
list was produced by Martin von Loewis during Pycon.
# Super simple
A very simple versioning scheme for released packages would be:
where those are all numeric fields. Update the "patch"-level when making
a small change without compatibility issues. Update the "minor" field
when adding a feature. Update the "major" field when making backward
incompat changes. Easy.
Incidentally this is what Ruby suggests for Gem authors ("Gems" are Ruby
packages), though they call the last field "build":
> Any "public" release of a gem should have a different version. Normally
> that means incrementing the build number. This means a developer can
> generate builds all day long for himself, but as soon as he/she makes a
> public release, the version must be updated.
# "But I want to do an alpha release!"
I don't think I'm overstating in saying that most of us (those that care to
help in defining Python packaging tools) would want to allow alpha/beta
releases. Certainly this was true at the Pycon discussions. This gives us:
N.N.NaN # e.g. "1.0.0a2"
N.N.NbN # e.g. "2.6.0b1"
Alternatives like the following were discarded:
- "1.0.0.a2" The '.' before the 'a' separator, while nice for parsing
is not common practice at all.
- "1.0.0alpha2" While this *does* appear on PyPI, it is less common than
just using the single character. As well, Python itself uses 'a'.
- "1.0.0a" No alpha version. The concensus was to not support this. Reasonable
people disagreed on whether this would imply "a0" or "a1". It was felt
that explicit was better than implicit here. As well, Python itself always
has a version on the alpha/beta.
# To "c", or not to "rc"?
Doing a release candidate is reasonable too, it was felt. That gives us:
N.N.NaN # e.g. "1.0.0a2"
N.N.NbN # e.g. "2.6.0b1"
N.N.NcN # e.g. "2.6.0c1"
Why "c" instead of "rc"? All of "c", "rc", and "candidate" are in use on
PyPI (I don't have percentages right now). "c" won because:
- that's what Python itself uses
- the one-character symmetry with 'a' and 'b' is nice
- it was felt that 'c' clearly enough indicated "release candidate"
So far I don't expect anything to be too controversial (but I'm probably
# Just three N's?
The current PyPI versions include quite a few versions with just two
"N's" -- e.g. "0.1", "3.5a2" -- as well as a some, though fewer, with
four N's -- e.g. "18.104.22.168". This gives us (this is just a pseudo-pattern):
It was felt that just a single N -- e.g. "1" -- should be disallowed.
However, the upper limit was left unbounded, i.e. this is allowed:
An example of where more N's is useful is for a Python module that wraps
a third-party library. Say that library ("libfoo") version is 2.5.2, a
reasonable version for "python-libfoo" might be "22.214.171.124.0" where the
first three bits track the "libfoo" version.
# Multiple N's after the "abc".
"1.2.0a3.4" or in the pseudo-pattern I've been using:
This was discussed and added. I don't recall who supported this.
Personally I've not had a need for this. 29 out of 4975 PyPI versions (in
MvL's list generated during Pycon) use this -- and in 7 of those that last
".N" is a date stamp (e.g. "1.0a2.20070215") where I think the datestamp is
meant as a sort of ".dev" or "pre-release" tag or build number.
# ... the rest
I'll try to post tomorrow night with my recollections and current
understanding of the rational for the ".dev" and ".post" parts of the
proposed version scheme.
I have some feedback on PEP376, both the pep itself and the pkgutil
code. I'll start with comments on the PEP.
* I don't like the definition of a project: this seems to define what
distutils calls a distribution, and that is not necessarily and
* The description of the status quo is not entirely acurate w.r.t.
setuptools, you describe only one of the possible ways setuptools can
install a project (pip seems to use another one of setuptools' ways of
installing a project, and setuptools can also install projects inside
of zipfiles in multiple ways).
* Regarding the RECORD file: it is not a "CSV-like" file, it is a real
CSV file. I'd specify the exact options for the 'csv' module that will
be used, rather than writing that the default options are used and
then explaining what those are.
* Should the PEP specify the encoding of text-files? PEP314 doesn't
seem to specify the encoding of PKG-INFO files, which can cause
problems when a field contains data that isn't ASCII.
All of these are minor issues. The following are imho more serious.
* The PEP doesn't describe how this PEP interacts with PEP302. That
is, how should the "egg-info" machinery work when a project is not
installed on the filesystem but in a zipfile. I'm not primarily
interested in the ".egg" zipfiles that setuptools uses, but I am
worried about how this will affect tools like py2exe and py2app that
bundle the python code used by an application into a zipfile.
* Why is there a "paths" argument for the global functions (such as
"get_file_users")? The description claims the functions will use
sys.path and hence it is not necessary to have an argument to specify
* There is no API to list the files in the egg-info directory, you
could build one yourself on top of the get_installed_files method but
should IMO be part of the public API.
And now on to the implementation:
* EggInfo.get_file: the implementation seems to want to load a file
from the .egg-info directory, the PEP itself is unclear about what
this method is intended to do.
If get_file should open a file in the egg-info directory I'd raise an
exception if the path argument specifies an absolute path.
I wonder if it wouldn't be better to have a function that returns the
contents of the file instead of one that returns a file-like object.
Especially when thinking of PEP302 integration.
Wouldn't it be better to have two separate methods instead of "binary"
argument. In 3.x file-like objects behave slightly different w.r.t
binary and text stream ("bytes" vs. "str" as the result of read).
* EggInfo.get_installed_files: if 'local' is True the yielded paths
are made absolute w.r.t. the egg-info directory rather than the
directory containing the egg-info directory.
* The global functions seem to maintain and modify global state,
wouldn't this cause problems if I specify different values of the path
arguments in different pieces of code?
I want to port pyobjc to python 3.1 and need a py3k port of setuptools
for that. I'm slowy moving forward with the latter, partly based on
earlier discussions on this list. I don't have anything useful to
share at the moment, beyond a setup3.py file (see below).
But first a question: some of the tests use docutils and I'm having a
hard time getting those to run with both python3 and python2. The bit
that's particularly annoying is this bit of api_tests.txt:
Note that asking for a conflicting version of a distribution
already in a
working set triggers a ``pkg_resources.VersionConflict`` error:
>>> ws.find(Requirement.parse("Bar==1.0")) # doctest:
Traceback (most recent call last):
VersionConflict: (Bar 0.9 (http://example.com/something),
This passes with 2.6, but fails in 3.1 because it expects
'pkg_resources.VersionConflict' rather than just 'VersionConflict'.
What's the best way to work around this issue?
The attached file "setup3.py" is a script that incrementally copies
the setuptools source-tree to a temporary directory, runs 2to3 on that
copy and exec-s the contents of the translated 'setup.py' file. This
seems to be good enough to be able to install setuptools for python2
and python3 using the same code-base. That said, this is just the
first step of a python3 port there need to be changes to the
setuptools sources to be able to actually use the translated sources.
New submission from Zooko O'Whielacronx <zooko(a)zooko.com>:
A user of Tahoe-LAFS encountered an error in which pyOpenSSL emitted:
exceptions.UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-5:
unsupported Unicode code range
It took some effort on the part of the user and the Tahoe-LAFS devs to delve
into the code and figure out how an invalid string got into a certificate inside
pyOpenSSL. Eventually the user (David Abrahams) figured out that the issue was
He had installed a pyOpenSSL egg which had been built with UCS4, but his Python
interpreter was UCS2. According to the thread linked above, the best way to fix
this is for distutils get_platform() to include the unicode variant in its
output, and then for setuptools to test the compatibility of that field when
choosing an egg. Is that right?
What's the next step?
title: egg platform names don't reflect unicode variant (UCS2, UCS4)
Setuptools tracker <setuptools(a)bugs.python.org>
2009/6/9 Eric Smith <eric(a)trueblade.com>:
>> 2009/6/8 Tarek Ziadé <ziade.tarek(a)gmail.com>:
> < plans and reasons to drop some commands from distutils, which I agree
> with >
>>> - we need to detect for each existing command (rpm, etc) a project
>>> that can take over, prior to remove it from Distutils
>>> when appliable .
>> And a policy on what to do if no such project can be found.
> I don't think we need a policy at all. If no one wants to support it, it
> gets dropped. You can't tell volunteers what they have to do.
No, it could be kept in the stdlib as is. If no-one wants to do the
work to handle the proposed change (from in the stdlib to externally
supported) then why not keep the status quo?
You seem to be saying that the correct approach should be to spend
effort removing bdist_wininst and others from the stdlib regardless of
whether someone is willing to commit to maintaining them externally.
I'm saying that unless someone wants to take on the job of being an
external maintainer, we should do nothing.
If bdist_wininst gets a special exception and is kept in the stdlib,
I'll shut up - I have no vested interest in any of the other bdist
forms. Otherwise, I fail to see the point of this purge -
bdist_wininst is important to Windows users, and I think it's
important that it stays in the stdlib.
To provide some concrete facts, here's the top of hg log for bdist_wininst.py:
date: Wed May 28 03:54:55 2008 +0200
summary: [svn r63755] bdist_wininst now works correctly when both
--skip-build and --plat-name are specified.
date: Fri May 02 14:48:15 2008 +0200
summary: [svn r62636] #2581: Vista UAC/elevation support for bdist_wininst
date: Mon Apr 07 03:53:39 2008 +0200
summary: [svn r62197] Issue #2513: enable 64bit cross compilation
date: Mon Dec 31 15:47:07 2007 +0100
summary: [svn r59620] Added wininst-9.0.exe executable for VS 2008
date: Wed Mar 23 19:54:36 2005 +0100
summary: [svn r38697] Make dist_files a triple, with the Python
target version included,
date: Tue Mar 22 23:23:29 2005 +0100
summary: [svn r38692] Fix registration of output file.
date: Mon Mar 21 21:56:35 2005 +0100
summary: [svn r38684] Add the upload command. Make all dist
commands register their
Doesn't look like a huge burden - since 2005, there have been 7
changes - 3 contributed by Mark Hammond, 3 by Martin von Loewis, and
one by Christian Heimes. All of whom I would assume would continue to
provide such changes if bdist_wininst stays in the stdlib, but who may
well not if it becomes externally maintained.
Note: If I seem to be making a lot of fuss about bdist_wininst, that's
because the only bdist modules in the stdlib right now are bdist_msi,
bdist_wininst, bdist_rpm and bdist_dumb. Either this policy is aimed
to a substantial extent at Windows, or it should be stated as "remove
bdist_rpm from the stdlib".
Can someone clearly articulate what is proposed here? I can see 4 main options:
1. Remove all 4 bdist modules (wininst, msi, rpm, dumb) from the
stdlib, and let them be picked up by external maintainers, or die.
2. Remove bdist_rpm from the stdlib, so that RPMs don't have special
status out of the Linux packaging formats, but retain bdist_msi,
bdist_wininst and bdist_dumb (effectively, the "windows is a special
3. Solicit offers from the community to maintain the 4 bdist_xxx
modules as external packages on PyPI. If no such offers are
forthcoming, continue as at present.
4. Simply make a formal policy that no new bdist_ commands will be
implemented in the stdlib, and leave open the possibility that someone
comes along and offers to pull one or more of the existing commands
out into an external module on PyPI (hardly any different from the
current status quo, except in being articulated a bit more
The only one which seriously concerns me is (1) out of the above. If
someone (Tarek?) comes out and says that (1) is not what is being
proposed, I'm happy to let the rest of the discussions take their
PS I have a paranoid feeling that there's a subtext somewhere here to
the effect that participants in the discussion don't see why there's
such a big deal, as everyone should be using
setuptools/easy_install/eggs anyway. Please let's avoid getting into
that here. Can we assume for the purposes of this discussion that
easy_install and eggs are not relevant, as the target audience for the
bdist_ commands is *precisely* those people who don't want to use
I have gathered all the documentation for the versioning lib in a
It explains how the existing tools work and how "verlib", that we
built during Pycon, works.
It still needs to be completed by some people that were present, but a
new feedback round
is more than welcome.
The goal here is to provide a version comparison standard to be
included in Distutils.
We do need such a standard to be able to include the
"install_requires" metadata as planned
(that will be a change in PEP 345) to be able to provide a standard
for version comparisons.
Tarek Ziadé | http://ziade.org
At 04:35 PM 6/8/2009 +0100, Paul Moore wrote:
>2009/6/8 Ronald Oussoren <ronaldoussoren(a)mac.com>:
> > The thingy we're getting is called an "EggInfo", which would IMHO mean that
> > "get_egg_infos" is technically the correct name for something
> that returns a
> > list of them. I'd interpret the singular form as a function that
> returns one
> > EggInfo object.
>That's the place I mean. But I'm not sure I like the idea of calling
>it an "EggInfo". I'll see if I can think of a better name (but not
>being familiar with the domain, I'm not sure I'll be able to).
FWIW, pkg_resources uses "Distribution", and it has functions such as
-----BEGIN PGP SIGNED MESSAGE-----
our project uses epydoc to create it's documentation.
is there a way to let build do the documentation generation?
and how to get clean into the mix to remove the generated documentation?
is there a way to build the doc without introducing a new command?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
-----END PGP SIGNATURE-----