Mailman 3 June 2009 - Distutils-SIG

recollections of Pycon distutils versioning discussion (part 2)
by Trent Mick 14 Jun '09

14 Jun '09

Here-in the second part of my recollections of the Pycon distutils versioning discussions. This first part is here: http://mail.python.org/pipermail/distutils-sig/2009-June/012143.html The first part left us with this pseudo-pattern: N.N[.N]*[(abc)N[.N]*] Some examples to help illustrate: 1.0 1.4.2.45.2.2.5 1.2.3a4 1.34.5.6c1 3.0b2.1 That leaves the ".devN" and ".postN" business of the current version proposal (PEP 386). # A brief setuptools lesson (Mainly this is because I needed it -- I'm not intimately familiar with setuptools. Skip this section if you know this already.) Setuptools defines (http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-…) "pre-release tags" and "post-release tags" on a version number. In setuptools one part of the "pre-release tags" are the "a", "b", "c" (for alpha, beta, candidate) parts of the proposed pattern that we've already discussed. The remaining part of "pre-release tags" is the use of "dev" or "pre", e.g.: 1.0dev 2.5.2pre 3.1dev459259 (Strictly speaking also "preview" and any string that sorts less than "final", but "dev" and "pre" are the common ones.) "Post-release tags" are any non-numeric in the version string that sorts greater than "final" or the special case "-". Practically speaking it is almost always the latter because setuptools provides command-line options (http://peak.telecommunity.com/DevCenter/setuptools#egg-info) to automatically append an svn rev ("-rNNNNN") or a date ("-YYYYMMDD"). Examples: 1.0-r2345 5.4.3-20090107 # ".devNNNN" in the new versioning proposal Reminder: I'm just trying to describe my recollections of the Pycon discussions here. I'll post my current opinions in a separate email. I.e., don't shoot the messenger. :) In general, people tended to agree that development builds are a reasonable thing. Your project is in the run-up to 1.0 and you'll be doing regular builds at incremental revisions of your source code control system. Setuptools encourages (by offering certain command-line options) spelling this kind of thing this way (though, of course there are other spellings of the equivalent): 1.0dev-r342 1.0dev-r343 1.0dev-r350 1.0dev-r355 At this point in the Pycon discussions I vaguely recall that there was some opposition to having support for spelling development builds in the versioning scheme at all. However, I don't fully recall. If so, they were won over (or drowned out) by (a) others giving examples of doing daily/nightly builds of product X -- including me describing nightlies of the Komodo product (not a Python project) I work on -- and (b) the start of the debate on how to spell "1.0dev-r342", those presuming that its inclusion was a forgone conclusion. I'll touch more on this in a followup email. Ensuing debate, pleading, fisticuffs, cajoling ... leading to N.N[.N]*[(abc)N[.N]*][.devN] 1.0.dev456 2.5.0b2.dev523 Why "dev" at all and not "-", "_", or "~" or just another "."? None of the alternatives were explicit that this is a release that should sort *earlier* than the version without it. Also, '-' is used to setuptools to mean a *post*-release; '_' causes problems for Debian packaging; '~' is used in Debian package versioning (IIRC, best to avoid using it); just '.' (e.g. '1.0a1.456') can be ambiguous. Why "dev"? Because it is by far the most common string used for this in current PyPI versions. "pre" a far second. Why ".devN" (1.0a1.dev456) instead of just "devN" (1.0a1dev456)? Partly a feeling that the '.' helped separate from the leading part -- if "1.0.a1" didn't look ridiculous we probably would have chosen that too. :) Also partly because there is justifiably little point in quibbling for hours over a single '.' here. # ".postNNNN" in the new versioning proposal Amid fading sounds of hands being dusted off, backs being patted and rising discussions of what's for dinner ("Salad bar or steak?" "What's the special tonight?") the question: "What about post-releases?" -- Zooko Groans. "Who uses that?" "Twisted, for example." "Really? Hrm, okay, let's tack on a '.postN' (like the '.devN') and call it a night." The discussion *was* longer than that, but not a whole lot longer. Mostly the justification for ".postN" in the proposal rested on the example of at least a couple significant Python projects using this (e.g. Twisted). An example of the resulting sorting order: 1.0a1 1.0a2.dev456 1.0a2 1.0a2.1.dev456 1.0a2.1 1.0b1.dev456 1.0b2 1.0c1.dev456 1.0c1 1.0.dev456 1.0 1.0.post456 and the final pseudo-pattern: N.N[.N]*[(a|b|c)N[.N]*][(.devN|.postN)] Those are my recollections of the Pycon versioning discussions. I want to following up (in a separate email) with my current opinions. Cheers, Trent -- Trent Mick trentm(a)gmail.com

5 11

PEP 376 APIs reworked (was: Re: PEP 376 comments)
by Tarek Ziadé 13 Jun '09

13 Jun '09

I've reworked the APIs using all the feedback: - now using basic types (dict + set) to make the code simpler - renamed the classes using the "Distribution" root name - the global functions are also now sys.path-based and there's no more global variable - egg_info -> egginfo - ... Please comment ! On Tue, Jun 9, 2009 at 12:07 AM, Kevin Teague<kevin(a)bud.ca> wrote: > Glancing over the EggInfo, EggInfoDirectories, and EggInfoDirectory > APIs, it might be interesting if these were able to match with the PEP > 3119 interfaces (http://www.python.org/dev/peps/pep-3119/). > > For example, the EggInfoDirectories has __iter__, append, clear and > remove methods, but the source code comments list them as 'container > APIs'. Technically though, that's an Iterable interface and a partial > implementation of the MutableSequence interface. > > The EggInfoDirectories.get_egg_infos(project_name) is also suggestive > of the Mapping interface - if this method was shortened from > get_egg_infos() to just get(). > > Maybe it would be confusing if an EggInfoDirectories was a > MutableSequence which could only have paths added to it, but returned > an iterable of EggInfoDirectory, but could also be accessed as a > Mapping where the keys are project names and those values are > EggInfos ... but maybe something along those lines would work? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziadé | http://ziade.org

2 4

easy_install self upgrade/version check bug
by jason 12 Jun '09

12 Jun '09

I've run into a problem where easy_install/setuptools continues to recommend upgrading itself even after the upgrade has been installed. Steps to reproduce on CentOS 5.2: include ~/python in PYTHONPATH include ~/bin in PATH create ~/.pydistutils.cfg [install] install_lib = ~/python install_scripts = ~/bin $ easy_install flup ... Running flup-1.0.3.dev-20090608/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rPyZ6a/flup-1.0.3.dev-20090608/egg-dist-tmp-pm0o4x The required version of setuptools (>=0.6c9) is not available, and can't be installed while this script is running. Please install a more recent version first, using 'easy_install -U setuptools'. (Currently using setuptools 0.6c5 (/usr/lib/python2.4/site-packages)) error: Setup script exited with 2 $ easy_install -U setuptools ... Processing dependencies for setuptools $ which easy_install ~/bin/easy_install $ python -c 'import setuptools; print setuptools.__version__' 0.6c9 $ easy_install flup ... Running flup-1.0.3.dev-20090608/setup.py -q bdist_egg --dist-dir /tmp/easy_install-WoxTch/flup-1.0.3.dev-20090608/egg-dist-tmp-oFotRt The required version of setuptools (>=0.6c9) is not available, and can't be installed while this script is running. Please install a more recent version first, using 'easy_install -U setuptools'. (Currently using setuptools 0.6c5 (/usr/lib/python2.4/site-packages)) error: Setup script exited with 2 At this point I thought it might be a problem with flup's setup.py or ez_setup.py, but manually running python setup.py bdist_egg outside of easy_install works fine.

3 4

Python svn builds
by A. Cavallo 12 Jun '09

12 Jun '09

Python svn builds (http://pyvm.sourceforge.net) is a project to provide python binary builds ready to use. The main features for these builds are: + install under /opt/opt-python-svnXXXXX-2.7a0 (XXXXX is the svn revision) + multiple interpreters can be installed side by side + installed in parallel to the system python (administrator friendly) + they're build for several different platforms (rpm based) + the build host is provided by the OpenSUSE buildserver + the builds can be deployed using yum/yast/urpmi (OpenSUSE buildserver) + each build runs a small smoke test and it is unattented + the test results are reported The aim is: + implementing a continuous integration practice + to provide python developer with reliable and repeatable builds + modules developer can test their modules against the latest python News for this version: Few modules (imaging, numpy, pexpect and setuptools) have been added and they're build for each interpreter. These are still in testing but the aim here is to have a full complete build chain with a test suite (next step is providing a test driven development). Download and documentation from: http://pyvm.sourceforge.net Regards, Antonio References: OpenSUSE buildserver: https://build.opensuse.org

1 0

Re: [Distutils] version scheme: a case for dropping ".devNNN" and ".postNNN"
by P.J. Eby 12 Jun '09

12 Jun '09

At 03:16 PM 6/11/2009 +0100, Paul Moore wrote: >2009/6/11 P.J. Eby <pje(a)telecommunity.com>: > > PyPI uploads aren't a suitable basis for analyzing "dev" use cases, since > > the whole point of having a "dev" tag is for *non-released* versions. > > (E.g., in-progress development via SVN.) > >If it's non-released, I've yet to see a clear explanation of why the >PEP is relevant. Who is going to use an API from the PEP to parse your >"version number", and why? > > > Dev tags are so that while you're > > doing development, your locally-installed versions can be > distinguished from > > one another. > >Distinguished by what? What code (that you didn't write yourself, >purely for internal use) needs to parse your dev tag? Distinguished by setuptools for processing version requirements of scripts, or require() statements in code, and installation requirements of newly-installed code. For example, if I'm working on two projects that are distributed via SVN and one depends on the other, if I update one, it may require an update of the other; the failure of the .dev#### version requirement in the first one will inform me of the need to "svn up" the second project and rerun "setup.py develop" on it. This is a routine circumstance in at least my development cycle; I would expect that it's the case in other open source development workflows as well as proprietary ones.

8 8

Re: [Distutils] post-release revisions
by P.J. Eby 12 Jun '09

12 Jun '09

At 03:48 PM 6/11/2009 +0100, Paul Moore wrote: >Please note, in case I seem excessively aggressive - my assumption is >that the standard defined by the PEP doesn't have to be used by >everyone, in all cases. You presumably have code that works at the >moment, and you are perfectly entitled to continue using it. That depends on whether this PEP's version scheme is also going to be embedded in the code implied by PEP 376. If so, then nobody will have any reason to switch from using pkg_resources to the PEP 376 pkgutil code, because they won't be able to manage their test, dev, and collaboration environments that way. > If you >switch to the APIs proposed in the PEP, then presumably you see some >benefits in following the standard. That's precisely it - there aren't any visible benefits to doing so, but there *are* visible problems. We need to have a way to translate existing version schemes to the new one, if we're to be able to have a reasonable upgrade path -- considering that these PEPs' only benefit to adopters (at least among those using setuptools) is that the code will be in some future stdlib version, and support simple uninstalls.

3 3

Re: [Distutils] version scheme: a case for dropping ".devNNN" and ".postNNN"
by P.J. Eby 12 Jun '09

12 Jun '09

At 02:29 AM 6/11/2009 -0700, Trent Mick wrote: >It would be interesting to get numbers of whether this setuptools >dependency pattern is at all frequently used: > > install_requires = ["OtherProject>=0.2a1.dev-r143"] > >I.e. a dependency on an uploaded-to-PyPI "dev" release. PyPI uploads aren't a suitable basis for analyzing "dev" use cases, since the whole point of having a "dev" tag is for *non-released* versions. (E.g., in-progress development via SVN.) Dev tags are so that while you're doing development, your locally-installed versions can be distinguished from one another.

4 5

version scheme: a case for dropping ".devNNN" and ".postNNN"
by Trent Mick 12 Jun '09

12 Jun '09

PEP 386 is proposing a new versioning scheme for distutils. One that currently includes ways to spell development releases (with a ".devNNN" suffix) and "post-releases" (with a ".postNNN" suffix). This is a long email. You can jump to the bottom for conclusions. First a survey of what "the other guys" do. # Perl CPAN. A free for all. A couple quotes from a Perl core developer here at ActiveState: > There is no standardized version numbering scheme. This module is supposed > to be it: > > http://search.cpan.org/~jpeacock/version-0.76/lib/version.pod > > But I think it probably created as many problems as it solved. # Tcl http://www.tcl.tk/man/tcl8.5/TclCmd/package.htm#M22 Strict version scheme, and because this is part of a core language feature it is effectively enforced. The scheme is: N[.N]+[(a|b)N] For examples of versions, ActiveState has a repo of binary builds of Tcl extensions called "teapot" (the "TEA" stands for Tcl Extension Architecture) here: http://teapot.activestate.com/index # PHP PEAR, http://pear.php.net/packages.php 535 packages: relatively small Strict version scheme (http://pear.php.net/group/docs/20040226-vn.php) "which will be mandatory for all packages from now on". They *do* support a "dev" state -- "0.9.5 (dev)" -- that is less than an "alpha" state, but NOT with an attached revision/date numeric value. Note that I did some poking around on the PEAR site and I couldn't find any packages with "dev"-state release. # Others? Anybody? # Some current PyPI numbers Conclusions are **bolded** for skimming. Using the PyPI XML-RPC API I got a dump of all current project names and the list of versions of uploaded packages for that project. A line like this for each project: Markdown: 1.7, 1.7, 1.7, 2.0, 2.0, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.1 This shows the order in which packages were uploaded... duplicates because separate package types for the same version (sdist, bdist_egg, etc.). script: http://dl.getdropbox.com/u/1301040/distutilsver/gatherpypi.py data: http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name.txt The current verlib.py has a "suggest_rational_version" that has pretty good coverage on getting the equivalent rational version number from a pypi version (16133 out of 17089 in my count). I counted uses of ".devNNN" and ".postNNN" in all the currently used PyPI versions: **Out of a total 17089 versions released to PyPI there are 281 (in 90 of about 6726 projects, 1.3%) that use a post-release tag.** These are: http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_usi… **Out of a total 17089 versions released to PyPI there are 738 (in 292 of about 6726 projects, 4.3%) that use a "dev" tag.** These are: http://dl.getdropbox.com/u/1301040/distutilsver/file_versions_from_name_usi… Looking at those that do there is a much lower percentage that *frequently* use the post- and pre-release tags. For example, two of SQLObject's *many* releases using a pre-release "dev-rNNNN" tag: SQLObject: 0.9.5, 0.7.8, 0.8.5, 0.9.1, 0.8.6b1, 0.9.2b1, 0.7.9b1, 0.8.6b2, 0.7.9b2, 0.9.2b2, 0.7.9b3, 0.8.6b3, 0.9.2b3, 0.7.9, 0.8.6, 0.9.2, 0.7.10, 0.9.3, 0.8.7, 0.10.0b1, 0.10.0b2, 0.9.4b1, 0.10.0b3, 0.10.0b4, 0.9.4, 0.10.0, 0.10.1, 0.9.6, 0.10.2b1, 0.9.7b1, 0.10.2b1dev-r3466, 0.9.7, 0.10dev-r3481, 0.10.2, 0.10.3, 0.9.8, 0.9.9, 0.10.4, 0.9.10, 0.10.5, 0.10.6, 0.9.11 **I.e. some might have just been a one off trying out the related setuptools command-line options for this. For others it might have just been a mistake.** For example, notice that the SQLObject "0.10.2b1dev-r3466" release happened *after* the "0.10.2b1". That "0.10.2b1dev-r3466" SQLObject release has subsequently been deleted from PyPI (indicating a mistake), though older releases are still there (e.g. http://pypi.python.org/pypi/SQLObject/0.9.3). There are indicates that the setuptools "-rNNNN" *post-release* tag is being accidentally mis-interpreted. According to setuptools: >>> from pkg_resources import parse_version as V >>> V("0.7") < V("0.7-r15094") True However, looking at the "Chandler-EVDBPlugin" project: "0.7-r15094" was uploaded 2007-07-24, and "0.7" was uploaded 2007-09-11 (later). Basically I suspect the maintainer of "Chandler-EVDBPlugin" misinterpreted how to use the setuptools "--tag-svn-revision" facility. **I.e. I suspect there are significantly fewer than 1% of projects uploading packages to PyPI that need/want a "post-release" tag facility**. It would be interesting to get numbers of whether this setuptools dependency pattern is at all frequently used: install_requires = ["OtherProject>=0.2a1.dev-r143"] I.e. a dependency on an uploaded-to-PyPI "dev" release. # The "Twisted" example The Twisted example was a justifier for including ".postNNNN" in the versioning scheme. See my previous email: http://mail.python.org/pipermail/distutils-sig/2009-June/012161.html However, while it appears to be true that Twisted doesn't appear to increment the version number in the source code until very close to release [1] (opposite to what Python itself, for example, does), I'm not sure this constitutes "using post-releases". - Twisted doesn't release packages to PyPI (http://pypi.python.org/pypi/Twisted/) - The archive of Twisted releases doesn't include post-releases. http://tmrc.mit.edu/mirror/twisted/Twisted/ - Perhaps not relevant, but Twisted's "Version" class seems to indicate a leaning towards "*pre*releases" http://twistedmatrix.com/trac/browser/trunk/twisted/python/versions.py#L38 [1] When correlating changes to Twisted's version file http://twistedmatrix.com/trac/browser/trunk/twisted/_version.py to Twisted release dates. # My conclusion/opinion In so far as the proposed new distutils versioning scheme is to be used for enforcing/encouraging versions for packages uploaded to PyPI, I don't believe there is a strong case for including ".devNNN" and certainly not ".postNNN". - It just isn't used often enough to justify the complexity in the version scheme for packages on PyPI. - Setuptools' facility to do `install_requires = ["OtherProject==dev"]` is a good thing, but this doesn't require the ".devNNNN" and post-release tags on sdist/bdist packages uploaded to PyPI. - The other language repo systems don't demonstrate a need for this. - Pressure on project authors to do normal releases (rather than having a quick "dev sorta-release") when user's want it is a *good thing*. - Dropping this complexity in the version scheme documentation for users would allow us to spend more time educating Python module authors on more important best practices like incrementing the major version number for backward incompatible changes. Cheers, Trent -- Trent Mick trentm(a)gmail.com

4 5

Re: [Distutils] version scheme: a case for dropping ".devNNN" and ".postNNN"
by P.J. Eby 11 Jun '09

11 Jun '09

At 06:01 PM 6/11/2009 +0100, Paul Moore wrote: >2009/6/11 P.J. Eby <pje(a)telecommunity.com>: > >> > Dev tags are so that while you're > >> > doing development, your locally-installed versions can be distinguished > >> > from one another. > >> > >> Distinguished by what? What code (that you didn't write yourself, > >> purely for internal use) needs to parse your dev tag? > > > > Distinguished by setuptools for processing version requirements of scripts, > > or require() statements in code, and installation requirements of > > newly-installed code. > >So will setuptools be modified to use the new code? It doesn't have to be; the RationalVersion scheme is both a strict subset of setuptools' scheme, and also has the property that most in-field uses of setuptools-based versioning schemes can be *mechanically* translated to RationalVersion instances, using an appropriate implementation of suggest_version(). This means that if somebody chooses to adopt RationalVersion (as currently defined in the PEP), setuptools will read and process it as correctly as it does more exotic numbering schemes. IOW, setuptools doesn't need to be changed, unless it was to say, add warnings for non-convertible versions... and that could actually be done with a plugin.

1 0

Problem using bdist_rpm with Fedora 10
by Stanley A. Klein 11 Jun '09

11 Jun '09

I tried to build java2python using bdist_rpm on Fedora 10 and got the following errors: RPM build errors: Installed (but unpackaged) file(s) found: /usr/lib/python2.5/site-packages/java2python/__init__.pyo /usr/lib/python2.5/site-packages/java2python/lib/__init__.pyo /usr/lib/python2.5/site-packages/java2python/lib/defaultconfig.pyo /usr/lib/python2.5/site-packages/java2python/lib/lexer.pyo /usr/lib/python2.5/site-packages/java2python/lib/parser.pyo /usr/lib/python2.5/site-packages/java2python/lib/sourcetypes.pyo /usr/lib/python2.5/site-packages/java2python/lib/walker.pyo error: command 'rpmbuild' failed with exit status 1 This had been an issue in the past, and the recommended solution was to prepare a setup.cfg containing [install] optimize = 1 This had worked in the past, but it had no effect this time. I found some discussion of the issue and some aspect of it is supposedly addressed in a change to Fedora. I upgraded to the latest rpm packages (version -4.6.1-1.fc10), but got the same behavior. How can I proceed to do the build? Thanks. Stan Klein

3 2