Hello, For a long time, the idiom I've used to specify version information in my distutils-based packages has been this simple one: put version information into the Python package somewhere, structure the source layout so that the package can be imported directly from an unpacked source tarball (or vcs wc), and import the version into setup.py. This worked great. And I only had to define my version once. Increasingly, it seems that new distutils-related tools don't support this idiom and break in various ways. (eg http://divmod.org/trac/ticket/2629 http://divmod.org/trac/ticket/2831 ) What is the recommendation for specifying version information in a way which is compatible with all these new tools? Jean-Paul
On May 19, 2009, at 12:21 PM, Jean-Paul Calderone wrote:
What is the recommendation for specifying version information in a way which is compatible with all these new tools?
My personal recommendation is to put the version information in a flat text file and have your setup.py read that flat text file instead of importing a Python package, e.g.: http://allmydata.org/trac/zfec/browser/zfec/setup.py?rev=285#L88 http://allmydata.org/trac/pyutil/browser/pyutil/setup.py?rev=141#L44 Another approach is to have the revision control be the single store of versioning information, and read it out of revision control whenever your setup.py runs. I believe I've already submitted one patch for each of these techniques for Nevow: http://divmod.org/trac/ticket/2699 A third approach is require setuptools and rely on setuptools's implementation of the second approach -- reading versioning information from your revision control tool. That one has obvious disadvantages (e.g. The Big Two: http://bugs.python.org/setuptools/ issue54 (be more like distutils with regard to --prefix=) and http:// bugs.python.org/setuptools/issue53 (respect the PYTHONPATH)), but it has the advantage that the implementation of this feature is not in your setup.py but is in setuptools or in a setuptools plugin such as setuptools_hg or setuptools_bzr. :-) Regards, Zooko
Jean-Paul Calderone <exarkun@divmod.com> writes:
For a long time, the idiom I've used to specify version information in my distutils-based packages has been this simple one: put version information into the Python package somewhere, structure the source layout so that the package can be imported directly from an unpacked source tarball (or vcs wc), and import the version into setup.py. This worked great. And I only had to define my version once.
Increasingly, it seems that new distutils-related tools don't support this idiom and break in various ways. (eg
http://divmod.org/trac/ticket/2629 http://divmod.org/trac/ticket/2831 )
Those breakages seem to be caused in part because the module containing version information is doing too much.
What is the recommendation for specifying version information in a way which is compatible with all these new tools?
I would recommend having a module in your package whose *only* responsibility is version information for the package, so that importing it won't break in different environments. That module can the be imported both by the package itself when it needs to know its version string, and by the ‘setup.py’ to get that same version string for package building. -- \ “The only tyrant I accept in this world is the still voice | `\ within.” —Mahatma Gandhi | _o__) | Ben Finney
On Wed, 20 May 2009 10:17:31 +1000, Ben Finney <ben+python@benfinney.id.au> wrote:
Jean-Paul Calderone <exarkun@divmod.com> writes:
For a long time, the idiom I've used to specify version information in my distutils-based packages has been this simple one: put version information into the Python package somewhere, structure the source layout so that the package can be imported directly from an unpacked source tarball (or vcs wc), and import the version into setup.py. This worked great. And I only had to define my version once.
Increasingly, it seems that new distutils-related tools don't support this idiom and break in various ways. (eg
http://divmod.org/trac/ticket/2629 http://divmod.org/trac/ticket/2831 )
Those breakages seem to be caused in part because the module containing version information is doing too much.
The only thing the version module does is define the version. For example, $ cat _version.py # This is an auto-generated file. Use Epsilon/bin/release-divmod to update. from twisted.python import versions version = versions.Version(__name__[:__name__.rfind('.')], 0, 9, 33) $ Of course, the "too much" here is the use of a structured version object. Doing anything simpler would be much like the approach Phillip's pointed out, to have a simple text file and parse its contents without getting Python involved at all.
What is the recommendation for specifying version information in a way which is compatible with all these new tools?
I would recommend having a module in your package whose *only* responsibility is version information for the package, so that importing it won't break in different environments.
That module can the be imported both by the package itself when it needs to know its version string, and by the ‘setup.py’ to get that same version string for package building.
This has issues. For examples, sometimes a system-wide installation of the package will get used instead (generally not when running setup.py directly, but it seems to happen when other tools are used). Otherwise, the existing _version.py from above works fine. Jean-Paul
Jean-Paul Calderone <exarkun@divmod.com> writes:
On Wed, 20 May 2009 10:17:31 +1000, Ben Finney <ben+python@benfinney.id.au> wrote:
Those breakages seem to be caused in part because the module containing version information is doing too much.
The only thing the version module does is define the version. For example,
$ cat _version.py # This is an auto-generated file. Use Epsilon/bin/release-divmod to update. from twisted.python import versions version = versions.Version(__name__[:__name__.rfind('.')], 0, 9, 33) $
Of course, the "too much" here is the use of a structured version object.
No, I would say “too much” here is importing a module not guaranteed to be installed, for creating something as simple as a version. I'd prefer the above to be:: $ cat _version.py # This is an auto-generated file. Use magicbean to update. version = "0.9.33" $ Anything that wants to have that structured into something more complex than a delimited string can use the funky Version type; but this particular module should import without any dependencies, for exactly the reasons that started this thread.
Doing anything simpler would be much like the approach Phillip's pointed out, to have a simple text file and parse its contents without getting Python involved at all.
I still like the approach of having it in a module that can be imported to automatically have the name bound to an appropriate value. -- \ “If you pick up a starving dog and make him prosperous, he will | `\ not bite you. This is the principal difference between a dog | _o__) and a man.” —Mark Twain, _Pudd'n'head Wilson_ | Ben Finney
No, I would say “too much” here is importing a module not guaranteed to be installed, for creating something as simple as a version. I'd prefer the above to be::
$ cat _version.py # This is an auto-generated file. Use magicbean to update. version = "0.9.33" $
Or my preferred way: $cat foobar/__init__.py __version__ = "0.9.33" and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33
I still like the approach of having it in a module that can be imported to automatically have the name bound to an appropriate value.
Yes, agree 100% the KISS approach is always the best. Regards, Antonio
A. Cavallo wrote:
No, I would say “too much” here is importing a module not guaranteed to be installed, for creating something as simple as a version. I'd prefer the above to be::
$ cat _version.py # This is an auto-generated file. Use magicbean to update. version = "0.9.33" $
Or my preferred way: $cat foobar/__init__.py __version__ = "0.9.33"
and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33
That doesn't work in all cases. Your example is of an external query of the version of an installed module. You also need to query the version -before- it is installed (during the packaging phase) on sys.path, and also from -within- the foobar module itself. Your code does not handle those cases. An attempt to 'import foobar; print foobar.__version__' from a setup.py inside foobar won't find foobar. -Jeff
Jeff Rush wrote:
A. Cavallo wrote:
No, I would say “too much” here is importing a module not guaranteed to be installed, for creating something as simple as a version. I'd prefer the above to be::
$ cat _version.py # This is an auto-generated file. Use magicbean to update. version = "0.9.33" $
Or my preferred way: $cat foobar/__init__.py __version__ = "0.9.33"
and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33
That doesn't work in all cases. Your example is of an external query of the version of an installed module. You also need to query the version -before- it is installed (during the packaging phase) on sys.path, and also from -within- the foobar module itself. Your code does not handle those cases. An attempt to 'import foobar; print foobar.__version__' from a setup.py inside foobar won't find foobar.
That's my experience as well. I am a bit delighted to see that other people struggled on this, I felt very stupid the first time I tried to setup this consistently without success. My current approach is to set up the version in setup.py, but to always generated a trivial version.py within the package (for the installed version). The setup.py version is used by paver, the installed version is used by the sphinx build inside a bootstrapped environment. I think the situation where you want to build documentation at the same time as the software happens relatively often, I would be glad to hear of a better alternative. cheers, David
Or my preferred way: $cat foobar/__init__.py __version__ = "0.9.33"
and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33
That doesn't work in all cases.
No it is not that the case and I'm going to bore you probably (sorry). The "source" dir where the module foobar is located: ~someuser/foobar-1.0/setup.py ~someuser/foobar-1.0/foobar/__init__.py $> cat ~someuser/foobar-1.0/foobar/__init__.py __version__ = "1.0" Assume the previous module is installed already under: /usr/lib/python2.5/site-lib/foobar/ /usr/lib/python2.5/site-lib/foobar/__init__.py $> cat /usr/lib/python2.5/site-lib/foobar/__init__.py __version__ = "0.9" (please note the site-lib subidr, where all the module not belonging to the python standard library are located). $~someuser/foobar-1.0/> python -c 'import foobar; print foobar.__version__' This return (or it does on my python interpreter): "1.0" not "0.9" The case is different for the standard libraries, in fact: $~someuser> touch os.py $~someuser> python -c 'import os; print os.__file__' this will return: /usr/lib/python2.5/os.py(c) There's no need to have foobar "installed" to reflect the correct __version__. Then we need to agree on what do we mean for install and packaging.... "python setup.py install", "python setup.py bdist_rpm/wininst" or during a complete deb/rpm package build? In the latter the version the "version" is the one available in the spec/deb files and it cannot be reflected from the sources anyway, no matter how hard one try to do it.
Your example is of an external query of the version of an installed module. You also need to query the version -before- it is installed (during the packaging phase) on sys.path, and also from -within- the foobar module itself. Your code does not handle those cases. An attempt to 'import foobar; print foobar.__version__' from a setup.py inside foobar won't find foobar.
Finding modules can always be forced using PYTHONPATH (like in case of foobar-1.0/src/foobar layout) or from setup.py inserting into sys.path the subdir: that's the easiest and in my opinion the best way to do it during build/install stages. Regards, Antonio
On Wed, 20 May 2009 12:37:19 +0100, "A. Cavallo" <a.cavallo@cavallinux.eu> wrote:
Or my preferred way: $cat foobar/__init__.py __version__ = "0.9.33"
and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33
That doesn't work in all cases.
No it is not that the case and I'm going to bore you probably (sorry).
[snip - explanation of how you can import from the package you're about to install]
This is true, but it is not the complete story. You explained the case where setup.py is being run directly to install the package. My first post in the thread explained that this case works fine. It's other cases, such as easy_install and buildout which do not work with this strategy. (Someone who has experience with these tools might be able to explain why this is; I haven't investigated the details yet, I only know that people are reporting bugs against my packages.)
[snip]
Then we need to agree on what do we mean for install and packaging.... "python setup.py install", "python setup.py bdist_rpm/wininst" or during a complete deb/rpm package build? In the latter the version the "version" is the one available in the spec/deb files and it cannot be reflected from the sources anyway, no matter how hard one try to do it.
The only version I'm talking about here is the one I assign to my package. If someone else is assigning a different version, then it becomes their problem to make things work.
[snip]
Finding modules can always be forced using PYTHONPATH (like in case of foobar-1.0/src/foobar layout) or from setup.py inserting into sys.path the subdir: that's the easiest and in my opinion the best way to do it during build/install stages.
Maybe so, but it can be made compatible with all (or most, or any?) of the tools people are using to install distutils-based packages these days? Jean-Paul
To chime in here with my two cents, The version information for a Python distribution should be specified in setup.py (or setup.py can read this information from somewhere in the project's files). It should *not* be specified inside a Python package or module that is part of the distribution itself. One area where embedding the version inside the module is problematic is a distribution which provides multiple modules or pacakges. Remember, one package (or module) does not have to equal one distribution! A distribution can provide many modules and packages. Let's say you have the Python project 'foobar'. A 'foobar' distribution provides the 'foo' package and the 'bar' package. If you are putting the distribution version in the package, should it be in foo.__version__ or bar.__version__? Does foo.__version__ and bar.__version__ both get updated to reflect the version of the foobar disribution, or are they for more fine grained versioning where they each specify additional version's beyond the distribution version? (whew!) Also it's worth mentioning that PEP 376 (http://www.python.org/dev/ peps/pep-0376/) proposes to solve some of this problem, by providing an API for reading the version of an installed distribution - and does so without needing to import any part of that installed distribution. For the 'foobar' distribution this would be: import pkgutil foobar_metadata = pkgutil.get_metadata('foobar') print foobar_metadata.version With this API one will be able to consistently query for the version of an installed *distribution*, and you don't run into the nasty chicken-and-egg problem of needing to import a package before you install it to determine it's version, or other weirdness or corner cases that this approach can cause.
Wheat <wheatix@gmail.com> writes:
The version information for a Python distribution should be specified in setup.py (or setup.py can read this information from somewhere in the project's files).
Yes, agreed completely.
It should *not* be specified inside a Python package or module that is part of the distribution itself.
Why not? It certainly makes sense for the programs to be able to discover their own version. I hope you don't advocate duplicating the version in another place for this purpose?
One area where embedding the version inside the module is problematic is a distribution which provides multiple modules or pacakges. Remember, one package (or module) does not have to equal one distribution! A distribution can provide many modules and packages.
In this case, I'm of the opinion that each Python package should know its own version (via a simple module that declares or discovers it, with no dependency on anything but standard Python), and the package distribution tools should choose one of those Python packages to be the one which determines the distribution version.
Let's say you have the Python project 'foobar'. A 'foobar' distribution provides the 'foo' package and the 'bar' package. If you are putting the distribution version in the package, should it be in foo.__version__ or bar.__version__?
I'd say the distribution version should be identical to one of the packages, as chosen by whoever's maintaining the distribution configuration.
Also it's worth mentioning that PEP 376 (http://www.python.org/dev/ peps/pep-0376/) proposes to solve some of this problem, by providing an API for reading the version of an installed distribution - and does so without needing to import any part of that installed distribution. For the 'foobar' distribution this would be:
import pkgutil foobar_metadata = pkgutil.get_metadata('foobar') print foobar_metadata.version
Sure. This fits my constraint that the version should be discoverable by importing a Python module without any dependencies on anything except standard Python. It just means that “standard Python” would need to include that ‘pkgutil’ module. -- \ “I don't care to belong to a club that accepts people like me | `\ as members.” —Groucho Marx | _o__) | Ben Finney
participants (7)
-
A. Cavallo -
Ben Finney -
David Cournapeau -
Jean-Paul Calderone -
Jeff Rush -
Wheat -
Zooko Wilcox-O'Hearn