PEP 396, Module Version Numbers
I just checked in PEP 396, Module Version Numbers. This is an informational PEP describing how to specify version numbers using the __version__ attribute. This has already made one round through distutils-sig so it's time to post it here. Comments welcome of course!
Cheers, -Barry
P.S. Yeah, I know the $Keyword$ strings here are wrong. I'm not sure what to do about them though.
PEP: 396 Title: Module Version Numbers Version: $Revision: 65628 $ Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $ Author: Barry Warsaw barry@python.org Status: Draft Type: Informational Content-Type: text/x-rst Created: 2011-03-16 Post-History: 2011-04-05
Abstract ========
Given that it is useful and common to specify version numbers for Python modules, and given that different ways of doing this have grown organically within the Python community, it is useful to establish standard conventions for module authors to adhere to and reference. This informational PEP describes best practices for Python module authors who want to define the version number of their Python module.
Conformance with this PEP is optional, however other Python tools (such as ``distutils2`` [1]_) may be adapted to use the conventions defined here.
User Stories ============
Alice is writing a new module, called ``alice``, which she wants to share with other Python developers. ``alice`` is a simple module and lives in one file, ``alice.py``. Alice wants to specify a version number so that her users can tell which version they are using. Because her module lives entirely in one file, she wants to add the version number to that file.
Bob has written a module called ``bob`` which he has shared with many users. ``bob.py`` contains a version number for the convenience of his users. Bob learns about the Cheeseshop [2]_, and adds some simple packaging using classic distutils so that he can upload *The Bob Bundle* to the Cheeseshop. Because ``bob.py`` already specifies a version number which his users can access programmatically, he wants the same API to continue to work even though his users now get it from the Cheeseshop.
Carole maintains several namespace packages, each of which are independently developed and distributed. In order for her users to properly specify dependencies on the right versions of her packages, she specifies the version numbers in the namespace package's ``setup.py`` file. Because Carol wants to have to update one version number per package, she specifies the version number in her module and has the ``setup.py`` extract the module version number when she builds the *sdist* archive.
David maintains a package in the standard library, and also produces standalone versions for other versions of Python. The standard library copy defines the version number in the module, and this same version number is used for the standalone distributions as well.
Rationale =========
Python modules, both in the standard library and available from third parties, have long included version numbers. There are established de-facto standards for describing version numbers, and many ad-hoc ways have grown organically over the years. Often, version numbers can be retrieved from a module programmatically, by importing the module and inspecting an attribute. Classic Python distutils ``setup()`` functions [3]_ describe a ``version`` argument where the release's version number can be specified. PEP 8 [4]_ describes the use of a module attribute called ``__version__`` for recording "Subversion, CVS, or RCS" version strings using keyword expansion. In the PEP author's own email archives, the earliest example of the use of an ``__version__`` module attribute by independent module developers dates back to 1995.
Another example of version information is the sqlite3 [5]_ library with its ``sqlite_version_info``, ``version``, and ``version_info`` attributes. It may not be immediately obvious which attribute contains a version number for the module, and which contains a version number for the underlying SQLite3 library.
This informational PEP codifies established practice, and recommends standard ways of describing module version numbers, along with some use cases for when -- and when *not* -- to include them. Its adoption by module authors is purely voluntary; packaging tools in the standard library will provide optional support for the standards defined herein, and other tools in the Python universe may comply as well.
Specification =============
#. In general, modules in the standard library SHOULD NOT have version numbers. They implicitly carry the version number of the Python release they are included in.
#. On a case-by-case basis, standard library modules which are also released in standalone form for other Python versions MAY include a module version number when included in the standard library, and SHOULD include a version number when packaged separately.
#. When a module includes a version number, it SHOULD be available in the ``__version__`` attribute on that module.
#. For modules which are also packages, the module namespace SHOULD include the ``__version__`` attribute.
#. For modules which live inside a namespace package, the sub-package name SHOULD include the ``__version__`` attribute. The namespace module itself SHOULD NOT include its own ``__version__`` attribute.
#. The ``__version__`` attribute's value SHOULD be a string.
#. Module version numbers SHOULD conform to the normalized version format specified in PEP 386 [6]_.
#. Module version numbers SHOULD NOT contain version control system supplied revision numbers, or any other semantically different version numbers (e.g. underlying library version number).
#. Wherever a ``__version__`` attribute exists, a module MAY also include a ``__version_info__`` attribute, containing a tuple representation of the module version number, for easy comparisons.
#. ``__version_info__`` SHOULD be of the format returned by PEP 386's ``parse_version()`` function.
#. The ``version`` attribute in a classic distutils ``setup.py`` file, or the PEP 345 [7]_ ``Version`` metadata field SHOULD be derived from the ``__version__`` field, or vice versa.
Examples ========
Retrieving the version number from a third party package::
>>> import bzrlib >>> bzrlib.__version__ '2.3.0'
Retrieving the version number from a standard library package that is also distributed as a standalone module::
>>> import email >>> email.__version__ '5.1.0'
Version numbers for namespace packages::
>>> import flufl.i18n >>> import flufl.enum >>> import flufl.lock
>>> print flufl.i18n.__version__ 1.0.4 >>> print flufl.enum.__version__ 3.1 >>> print flufl.lock.__version__ 2.1
>>> import flufl >>> flufl.__version__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute '__version__' >>>
Deriving ========
Module version numbers can appear in at least two places, and sometimes more. For example, in accordance with this PEP, they are available programmatically on the module's ``__version__`` attribute. In a classic distutils ``setup.py`` file, the ``setup()`` function takes a ``version`` argument, while the distutils2 ``setup.cfg`` file has a ``version`` key. The version number must also get into the PEP 345 metadata, preferably when the *sdist* archive is built. It's desirable for module authors to only have to specify the version number once, and have all the other uses derive from this single definition.
While there are any number of ways this could be done, this section describes one possible approach, for each scenario.
Let's say Elle adds this attribute to her module file ``elle.py``::
__version__ = '3.1.1'
Classic distutils -----------------
In classic distutils, the simplest way to add the version string to the ``setup()`` function in ``setup.py`` is to do something like this::
from elle import __version__ setup(name='elle', version=__version__)
In the PEP author's experience however, this can fail in some cases, such as when the module uses automatic Python 3 conversion via the ``2to3`` program (because ``setup.py`` is executed by Python 3 before the ``elle`` module has been converted).
In that case, it's not much more difficult to write a little code to parse the ``__version__`` from the file rather than importing it::
import re DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+.\d(?:.\d+)?)')
def get_version(filename, pattern=None): if pattern is None: cre = DEFAULT_VERSION_RE else: cre = re.compile(pattern) with open(filename) as fp: for line in fp: if line.startswith('__version__'): mo = cre.search(line) assert mo, 'No valid __version__ string found' return mo.group('version') raise AssertionError('No __version__ assignment found')
setup(name='elle', version=get_version('elle.py'))
Distutils2 ----------
Because the distutils2 style ``setup.cfg`` is declarative, we can't run any code to extract the ``__version__`` attribute, either via import or via parsing. This PEP suggests a special key be added to the ``[metadata]`` section of the ``setup.cfg`` file to indicate "get the version from this file". Something like this might work::
[metadata] version-from-file: elle.py
where ``parse`` means to use a parsing method similar to the above, on the file named after the colon. The exact recipe for doing this will be discussed in the appropriate distutils2 development forum.
An alternative is to only define the version number in ``setup.cfg`` and use the ``pkgutil`` module [8]_ to make it available programmatically. E.g. in ``elle.py``::
from distutils2._backport import pkgutil __version__ = pkgutil.get_distribution('elle').metadata['version']
PEP 376 metadata ================
PEP 376 [9]_ defines a standard for static metadata, but doesn't describe the process by which this metadata gets created. It is highly desirable for the derived version information to be placed into the PEP 376 ``.dist-info`` metadata at build-time rather than install-time. This way, the metadata will be available for introspection even when the code is not installed.
References ==========
.. [1] Distutils2 documentation (http://distutils2.notmyidea.org/)
.. [2] The Cheeseshop (Python Package Index) (http://pypi.python.org)
.. [3] http://docs.python.org/distutils/setupscript.html
.. [4] PEP 8, Style Guide for Python Code (http://www.python.org/dev/peps/pep-0008)
.. [5] sqlite3 module documentation (http://docs.python.org/library/sqlite3.html)
.. [6] PEP 386, Changing the version comparison module in Distutils (http://www.python.org/dev/peps/pep-0386/)
.. [7] PEP 345, Metadata for Python Software Packages 1.2 (http://www.python.org/dev/peps/pep-0345/#version)
.. [8] pkgutil - Package utilities (http://distutils2.notmyidea.org/library/pkgutil.html)
.. [9] PEP 376, Database of Installed Python Distributions (http://www.python.org/dev/peps/pep-0376/)
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
#. Module version numbers SHOULD conform to the normalized version format specified in PEP 386 [6]_.
From PEP 386:
Roadmap http://www.python.org/dev/peps/pep-0386/#id21
Distutils will deprecate its existing versions class in favor of NormalizedVersion. The verlib module presented in this PEP will be renamed to version and placed into the distutils package.
With more standardization of versions, should the version module be promoted to stdlib directly?
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']
The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Nor will it match the code shown and quoted for the alternative distutils2 case.
Other comments:
Are there issues for finding and loading multiple versions of the same module?
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Linderman v+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Indeed, I really don't like the RE suggestion - better to tell people to just move the version info into the static config file and use pkgutil to make it available as shown. That solves the build time vs install time problem as well.
Nor will it match the code shown and quoted for the alternative distutils2 case.
Other comments:
Are there issues for finding and loading multiple versions of the same module?
No, you simply can't do it. Python's import semantics are already overly complicated even without opening that particular can of worms.
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
Yep, this is why the version information should be in the setup.cfg file, and hence available via pkgutil without loading the module first.
Cheers, Nick.
On 4/6/2011 7:26 AM, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
I thought that might be part of the answer :) But that, and below, seem to indicate that use of "packaging" suddenly becomes a requirement for all modules that want to include versions. The packaging of "version" inside a version of "packaging" implies more dependencies on a larger body of code for a simple function.
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Indeed, I really don't like the RE suggestion - better to tell people to just move the version info into the static config file and use pkgutil to make it available as shown. That solves the build time vs install time problem as well.
Nor will it match the code shown and quoted for the alternative distutils2 case.
Other comments:
Are there issues for finding and loading multiple versions of the same module?
No, you simply can't do it. Python's import semantics are already overly complicated even without opening that particular can of worms.
OK, I just recalled some discussion about multiple coexisting versions in past time, not that they produced any conclusion that such should or would ever be implemented.
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
Yep, this is why the version information should be in the setup.cfg file, and hence available via pkgutil without loading the module first.
So, no support for single .py file modules, then?
If "packaging" truly is the only thing that knows the version of something, and "version" lives in "packaging", then perhaps packaging "__version__" as part of the module is inappropriate, and the API to obtain the version of a module should be inside "packaging" with the module (or its name) as a parameter, rather than asking the module, which may otherwise not need a dependency on the internals of "packaging" except to obtain its own version, which, it doesn't likely need for its own use anyway, except to report it.
Which is likely why Barry offered so many choices as to where the version of a package or module might live in the first place.
Perhaps a different technique would be that if packaging is in use, that it could somehow inject the version from setup.cfg into the module, either by tweaking the source as it gets packaged, or installed, or tweaking the module as/after it gets loaded (the latter still required some runtime dependency on code from the packaging system). A line like the following in some designated-to-"packaging" source file could be replaced during packaging:
__version__ = "7.9.7xxxx" # replaced by "packaging"
could be used for source codes that use "packaging" which would replace it by the version from setup.cfg during the packaging process, whereas a module that doesn't use "packaging" would put in the real version, and avoid the special comment. The reason the fake version would have a (redundant) number would be to satisfy dependencies during pre-"packaging" testing. (The above would add a new parsing requirement to "version" for "xxxx" at the end. Something different than "dev" so that development releases that still go through the packaging process are still different than developers test code. "packaging" should probably complain if the versions are numerically different and the version in the source file doesn't have "xxxx" or doesn't exactly match the version in setup.cfg, and if the special comment is not found.)
Caveat: I'm not 100% clear on when/how any of "distutils", "setuptools", or "packaging" are invoked (I was sort of waiting for the dust to settle on "packaging" to learn how to use this latest way of doing things), but based on historical experience with other systems, and expectations about how things "should" work, I would expect that a packaging system is something that should be used after a module is complete, to wrap it up for distribution and installation, but that the module itself should not have significant knowledge of or dependency on such a packaging system, so that when the module is invoked at runtime, it doesn't bring overhead of such packaging systems to the runtime. I've seen the setuptools "egg" where the module stays inside the egg after being installed, and while that might have benefits for reducing the installed file count, and perhaps other benefits, I've not tried to figure out whether the module is dependent on "setuptools" or whether the "setuptools" just provides the benefits mentioned, without the module needing to be dependent on it, but still it seems the application winds up being dependent on some "setuptools" stuff at runtime, when it uses such a module.
On Thu, Apr 7, 2011 at 7:58 AM, Glenn Linderman v+python@g.nevcal.com wrote:
Perhaps a different technique would be that if packaging is in use, that it could somehow inject the version from setup.cfg into the module, either by tweaking the source as it gets packaged, or installed, or tweaking the module as/after it gets loaded (the latter still required some runtime dependency on code from the packaging system). A line like the following in some designated-to-"packaging" source file could be replaced during packaging:
__version__ = "7.9.7xxxx" # replaced by "packaging"
If you don't upload your module to PyPI, then you can do whatever you want with your versioning info. If you *do* upload it to PyPI, then part of doing so properly is to package it so that your metadata is where other utilities expect it to be. At that point, you can move the version info over to setup.cfg and add the code into the module to read it from the metadata store.
The guidelines in 396 really only apply to distributed packages, so it doesn't make sense to obfuscate by catering to esoteric use cases. If prviate modules don't work with the standard tools, who is going to care? The module author clearly doesn't, and they aren't distributing it to anyone else. Once they *do* start distributing it, then their new users will help bring them into line. Having the recommended practice clearly documented just makes it easier for those users to point new module distributors in the right direction.
(Also, tsk, tsk, Barry for including Standards track proposals in an Informational PEP!)
Cheers, Nick.
P.S. A nice coincidental progression: PEP 376, 386 and 396 are all related to versioning and package metadata
On 4/6/2011 9:08 PM, Nick Coghlan wrote:
On Thu, Apr 7, 2011 at 7:58 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
Perhaps a different technique would be that if packaging is in use, that it could somehow inject the version from setup.cfg into the module, either by tweaking the source as it gets packaged, or installed, or tweaking the module as/after it gets loaded (the latter still required some runtime dependency on code from the packaging system). A line like the following in some designated-to-"packaging" source file could be replaced during packaging:
__version__ = "7.9.7xxxx" # replaced by "packaging"
If you don't upload your module to PyPI, then you can do whatever you want with your versioning info. If you *do* upload it to PyPI, then part of doing so properly is to package it so that your metadata is where other utilities expect it to be. At that point, you can move the version info over to setup.cfg and add the code into the module to read it from the metadata store.
The PEP doesn't mention PyPI, and at present none of the modules there use "packaging" :) So it wasn't obvious to me that the PEP applies only to PyPI, and I have used modules that were not available from PyPI yet were still distributed and packaged somehow (not using "packaging" clearly).
While there has been much effort (discussion by many) to make "packaging" useful to many, and that is probably a good thing, I still wonder why a packaging system should be loaded into applications when all the code has already been installed. Or is the runtime of "packaging" packaged so that only a small amount of code has to be loaded to obtain "version" and "__version__"? I don't recall that being discussed on this list, but maybe it has been on more focused lists, sorry for my ignorance... but I also read about embedded people complaining about how many files Python opens at start up, and see no need for a full packaging system to be loaded, just to do version checking.
The guidelines in 396 really only apply to distributed packages, so it doesn't make sense to obfuscate by catering to esoteric use cases. If prviate modules don't work with the standard tools, who is going to care? The module author clearly doesn't, and they aren't distributing it to anyone else. Once they *do* start distributing it, then their new users will help bring them into line. Having the recommended practice clearly documented just makes it easier for those users to point new module distributors in the right direction.
Oh, I fully agree that there be a PEP with guidelines, and yesterday converted my private versioning system to conform with the names in the PEP, and the style of version string in the referenced PEP. And I distribute my modules -- so far only in a private group, and so far as straight .py files... no use of "packaging". And even if I never use "packaging", it seems like a good thing to conform to this PEP, if I can. Version checking is useful.
(Also, tsk, tsk, Barry for including Standards track proposals in an Informational PEP!)
Cheers, Nick.
P.S. A nice coincidental progression: PEP 376, 386 and 396 are all related to versioning and package metadata
On Thu, Apr 7, 2011 at 2:55 PM, Glenn Linderman v+python@g.nevcal.com wrote:
__version__ = "7.9.7xxxx" # replaced by "packaging"
If you don't upload your module to PyPI, then you can do whatever you want with your versioning info. If you *do* upload it to PyPI, then part of doing so properly is to package it so that your metadata is where other utilities expect it to be. At that point, you can move the version info over to setup.cfg and add the code into the module to read it from the metadata store.
The PEP doesn't mention PyPI, and at present none of the modules there use "packaging" :)
They all use distutils (or setuptools or distutils2) though, which is what packaging replaces.
(Sorry for not making that clear - it's easy to forget which aspects of these issues aren't common knowledge as yet)
So it wasn't obvious to me that the PEP applies only to PyPI, and I have used modules that were not available from PyPI yet were still distributed and packaged somehow (not using "packaging" clearly).
packaging is the successor to the current distutils package. Distribution via PyPI is the main reason to bother with creating a correctly structured package - for internal distribution, people use all sorts of ad hoc schemes (often just the packaging systems of their internal target platforms). I'll grant that some people do use properly structured packages for purely internal use, but I'd also be willing to bet that they're the exception rather than the rule.
What I would like to see the PEP say is that if you don't *have* a setup.cfg file, then go ahead and embed the version directly in your Python source file. If you *do* have one, then put the version there and retrieve it with "pkgutil" if you want to provide a __version__ attribute.
Barry is welcome to make a feature request to allow that dependency to go the other way, with the packaging system reading the version number out of the source file, but such a suggestion doesn't belong in an Informational PEP. If such a feature is ever accepted, then the recommendation in the PEP could be updated.
While there has been much effort (discussion by many) to make "packaging" useful to many, and that is probably a good thing, I still wonder why a packaging system should be loaded into applications when all the code has already been installed. Or is the runtime of "packaging" packaged so that only a small amount of code has to be loaded to obtain "version" and "__version__"? I don't recall that being discussed on this list, but maybe it has been on more focused lists, sorry for my ignorance... but I also read about embedded people complaining about how many files Python opens at start up, and see no need for a full packaging system to be loaded, just to do version checking.
pkgutil will be able to read the metadata - it is a top level standard library module, *not* a submodule of distutils/packaging.
It may make sense for the version parsing support to be in pkgutil as well, since PEP 345 calls for it to be stored as a string in the package metadata, but it needs to be converted with NormalizedVersion to be safe to use in arbitrary version range checks. That's Tarek's call as to whether to provide it that way, or as a submodule of packaging. As you say, the fact that distutils/packaging are usually first on the chopping block when distros are looking to save space is a strong point in favour of having that particular functionality somewhere else.
That said, I've seen people have problems because a Python 2.6 redistributor decided "contextlib" wasn't important and left it out, so YMMV regardless of where the code ends up.
Cheers, Nick.
On 4/6/2011 11:53 PM, Nick Coghlan wrote:
On Thu, Apr 7, 2011 at 2:55 PM, Glenn Lindermanv+python@g.nevcal.com wrote:
__version__ = "7.9.7xxxx" # replaced by "packaging"
If you don't upload your module to PyPI, then you can do whatever you want with your versioning info. If you *do* upload it to PyPI, then part of doing so properly is to package it so that your metadata is where other utilities expect it to be. At that point, you can move the version info over to setup.cfg and add the code into the module to read it from the metadata store.
The PEP doesn't mention PyPI, and at present none of the modules there use "packaging" :)
They all use distutils (or setuptools or distutils2) though, which is what packaging replaces.
(Sorry for not making that clear - it's easy to forget which aspects of these issues aren't common knowledge as yet)
I knew that packaging replaced those others, but was unaware that those were the only two methods used on PyPI. Not that I'd heard of or experienced any others from that source, but there are many packages there.
So it wasn't obvious to me that the PEP applies only to PyPI, and I have used modules that were not available from PyPI yet were still distributed and packaged somehow (not using "packaging" clearly).
packaging is the successor to the current distutils package. Distribution via PyPI is the main reason to bother with creating a correctly structured package - for internal distribution, people use all sorts of ad hoc schemes (often just the packaging systems of their internal target platforms). I'll grant that some people do use properly structured packages for purely internal use, but I'd also be willing to bet that they're the exception rather than the rule.
What I would like to see the PEP say is that if you don't *have* a setup.cfg file, then go ahead and embed the version directly in your Python source file. If you *do* have one, then put the version there and retrieve it with "pkgutil" if you want to provide a __version__ attribute.
Barry is welcome to make a feature request to allow that dependency to go the other way, with the packaging system reading the version number out of the source file, but such a suggestion doesn't belong in an Informational PEP. If such a feature is ever accepted, then the recommendation in the PEP could be updated.
While there has been much effort (discussion by many) to make "packaging" useful to many, and that is probably a good thing, I still wonder why a packaging system should be loaded into applications when all the code has already been installed. Or is the runtime of "packaging" packaged so that only a small amount of code has to be loaded to obtain "version" and "__version__"? I don't recall that being discussed on this list, but maybe it has been on more focused lists, sorry for my ignorance... but I also read about embedded people complaining about how many files Python opens at start up, and see no need for a full packaging system to be loaded, just to do version checking.
pkgutil will be able to read the metadata - it is a top level standard library module, *not* a submodule of distutils/packaging.
It may make sense for the version parsing support to be in pkgutil as well, since PEP 345 calls for it to be stored as a string in the package metadata, but it needs to be converted with NormalizedVersion to be safe to use in arbitrary version range checks. That's Tarek's call as to whether to provide it that way, or as a submodule of packaging. As you say, the fact that distutils/packaging are usually first on the chopping block when distros are looking to save space is a strong point in favour of having that particular functionality somewhere else.
This sounds more practical; if I recall prior discussions correctly, pkgutil reads a standard set of metadata data packaging systems should provide, and version would seem to be part of that, more so than of packaging itself... seems things would have a better (smaller at runtime) dependency tree that way, from what I understand about it.
That said, I've seen people have problems because a Python 2.6 redistributor decided "contextlib" wasn't important and left it out, so YMMV regardless of where the code ends up.
:)
Cheers, Nick
Thanks Nick, for the info in this thread. This is mostly a thank you note for helping me understand better.
On Thu, Apr 7, 2011 at 5:05 PM, Glenn Linderman v+python@g.nevcal.com wrote:
On 4/6/2011 11:53 PM, Nick Coghlan wrote: They all use distutils (or setuptools or distutils2) though, which is what packaging replaces.
(Sorry for not making that clear - it's easy to forget which aspects of these issues aren't common knowledge as yet)
I knew that packaging replaced those others, but was unaware that those were the only two methods used on PyPI. Not that I'd heard of or experienced any others from that source, but there are many packages there.
I believe it is possible to get stuff up onto PyPI without actually using one of the various packaging utilities, but such entries generally won't play well with others (including automated tools like pip and cheesecake).
Cheers, Nick.
On Apr 07, 2011, at 04:53 PM, Nick Coghlan wrote:
What I would like to see the PEP say is that if you don't *have* a setup.cfg file, then go ahead and embed the version directly in your Python source file. If you *do* have one, then put the version there and retrieve it with "pkgutil" if you want to provide a __version__ attribute.
I'm not convinced there's consensus on that, i.e. that the version string should go in setup.cfg if it exists. It doesn't help that the pkgutil API for doing that is pretty ugly.
Barry is welcome to make a feature request to allow that dependency to go the other way, with the packaging system reading the version number out of the source file, but such a suggestion doesn't belong in an Informational PEP. If such a feature is ever accepted, then the recommendation in the PEP could be updated.
Note that there's really no reason why packaging has to grow a method to do this. It would be a convenience, but not a requirement. For example, I have my own helper function (something like the now elided get_version() code) that digs version strings out of files for my own packages just fine. True, it doesn't handle the full normalized version specification, but I don't care because my version numbers will never look that complex. If yours does, and you don't want to rely on the pkgutil API, or you need it to work even when your module isn't installed, well, write your own code!
The Deriving section of the PEP is not the most important part of it, and is not making specific recommendations. If it's not clear that it's only providing examples, or it's distracting, then maybe it's better off being removed, cut down or rewritten.
Cheers, -Barry
On Apr 06, 2011, at 09:55 PM, Glenn Linderman wrote:
The PEP doesn't mention PyPI, and at present none of the modules there use "packaging" :) So it wasn't obvious to me that the PEP applies only to PyPI, and I have used modules that were not available from PyPI yet were still distributed and packaged somehow (not using "packaging" clearly).
The core of the PEP does not require packaging or PyPI. The Specification section is the most important part of the PEP. Yes, that does mention parse_version() from PEP 386, and the Version metadata field from PEP 345, but I think those cross-references are fine, because it's just referring to the information contained there.
-Barry
On Apr 07, 2011, at 02:08 PM, Nick Coghlan wrote:
(Also, tsk, tsk, Barry for including Standards track proposals in an Informational PEP!)
Is that really illegal? :)
P.S. A nice coincidental progression: PEP 376, 386 and 396 are all related to versioning and package metadata
time-machine-ly y'rs, -Barry
Hi,
Le 06/04/2011 23:58, Glenn Linderman a écrit :
On 4/6/2011 7:26 AM, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
I thought that might be part of the answer :) But that, and below, seem to indicate that use of "packaging" suddenly becomes a requirement for all modules that want to include versions. The packaging of "version" inside a version of "packaging" implies more dependencies on a larger body of code for a simple function.
Not really. Some parts of distutils2/packaging are standalone modules that are deliberately exposed publicly for third parties to use: version, metadata, markers, etc. packaging.version is just the full name of the module implementing PEP 386. (The first implementation, called verlib, has not been explicitly ended, nor the references in the PEP updated.)
So, no support for single .py file modules, then?
From packaging’s viewpoint, a project (something with a name and a version) is a directory with a setup.cfg file. The directory can contain zero or more Python modules, Python packages, extension modules or data files.
Caveat: I'm not 100% clear on when/how any of "distutils", "setuptools", or "packaging" are invoked
FTR: setuptools is a monkey-patching set of extensions to distutils: packaging is a full replacement of distutils. packaging does not depend on distutils nor setuptools; it is a fork of distutils with some ideas and code taken from setuptools.
Regards
On 4/9/2011 9:23 AM, Éric Araujo wrote:
Hi,
Le 06/04/2011 23:58, Glenn Linderman a écrit :
On 4/6/2011 7:26 AM, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
I thought that might be part of the answer :) But that, and below, seem to indicate that use of "packaging" suddenly becomes a requirement for all modules that want to include versions. The packaging of "version" inside a version of "packaging" implies more dependencies on a larger body of code for a simple function.
Not really. Some parts of distutils2/packaging are standalone modules that are deliberately exposed publicly for third parties to use: version, metadata, markers, etc. packaging.version is just the full name of the module implementing PEP 386. (The first implementation, called verlib, has not been explicitly ended, nor the references in the PEP updated.)
Glad for the clarification here. As long as packaging.version doesn't have dependencies on large amounts of code to be loaded from the rest of packaging, then I have no serious objection to that structure.
Although, writing "packaging.version" just now, it seems like the result could be perceived as the "packaging version" which might be distinct from the "real version" or "actual version" or "code version". That's just a terminology thing that could be overcome with adequate documentation.
Then there is the question Nick raised about distributions that want to cut back on space, and provide alternate mechanisms than Python's packaging module to distribute code... producing the perception that they could avoid including packaging in their smallest distribution (but would probably package a packaging package using their packaging mechanism). If that dropped packaging.version, it could be problematic for doing version checking in applications. So distributions might have to split apart the packaging package.
Finally, if there are other competing historical packaging mechanisms, or new ones develop in the future, packaging packaging.version separately from the rest of packaging might avoid having different versions of version: when included a replacement system might think it needs to also replace that component for completeness.
So I still favor a top-level version module that isn't included in packaging, and implementation that doesn't depend on packaging. But only at +0.
So, no support for single .py file modules, then?
From packaging’s viewpoint, a project (something with a name and a version) is a directory with a setup.cfg file. The directory can contain zero or more Python modules, Python packages, extension modules or data files.
From packaging's viewpoint, there is nothing wrong with that. But from a non-packaging viewpoint, a user of other distribution mechanisms wouldn't necessarily want to or need to create a setup.cfg file, nor be forced to write code to obtain __version__ by calling a packaging method to obtain it. Not that the PEP as written currently requires that.
Caveat: I'm not 100% clear on when/how any of "distutils", "setuptools", or "packaging" are invoked
FTR: setuptools is a monkey-patching set of extensions to distutils: packaging is a full replacement of distutils. packaging does not depend on distutils nor setuptools; it is a fork of distutils with some ideas and code taken from setuptools.
Thanks, but that part I actually knew from cecent discussion on this list. What I'm not clear on is whether modules packaged by any of distutils, setuptools, or packaging, because of being packaged by them, wind up including 0%, 10%, 90%, or 100% of the code from distutils, setuptools, or packaging at runtime. My favored percentage would be 0%, as I believe a packaging system should do its stuff at the time of making, distributing, and installing code, but should get out of the way of the runtime code, but I would find maybe 10% acceptable, if it was only one or two files to be searched for and included, to achieve significant benefits to the packaged code.
On 06/04/2011 15:26, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']
I really dislike this way of specifying the version. For a start it is really ugly.
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'
Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
What I may have to do is use a python script that will generate the static metadata, which is not such a bad thing I guess as it will only need to be executed at package build time. I won't be switching to that horrible technique for specifying versions within my packages though.
All the best,
Michael
The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Indeed, I really don't like the RE suggestion - better to tell people to just move the version info into the static config file and use pkgutil to make it available as shown. That solves the build time vs install time problem as well.
Nor will it match the code shown and quoted for the alternative distutils2 case.
Other comments:
Are there issues for finding and loading multiple versions of the same module?
No, you simply can't do it. Python's import semantics are already overly complicated even without opening that particular can of worms.
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
Yep, this is why the version information should be in the setup.cfg file, and hence available via pkgutil without loading the module first.
Cheers, Nick.
On 07/04/2011 12:10, Michael Foord wrote:
On 06/04/2011 15:26, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']I really dislike this way of specifying the version. For a start it is really ugly.
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
And in fact it would make the module itself unimportable unless installed by "packaging", so not compatible with other installation methods (including the ever-loved 'just drop it somewhere on sys.path) or earlier versions of Python that don't have the required apis (or don't have packaging installed).
So I don't think recommending "pkgutil.get_distribution('elle').metadata['version']" as a way for packages to provide version information is good advice.
All the best,
Michael Foord
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
What I may have to do is use a python script that will generate the static metadata, which is not such a bad thing I guess as it will only need to be executed at package build time. I won't be switching to that horrible technique for specifying versions within my packages though.
All the best,
Michael
The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Indeed, I really don't like the RE suggestion - better to tell people to just move the version info into the static config file and use pkgutil to make it available as shown. That solves the build time vs install time problem as well.
Nor will it match the code shown and quoted for the alternative distutils2 case.
Other comments:
Are there issues for finding and loading multiple versions of the same module?
No, you simply can't do it. Python's import semantics are already overly complicated even without opening that particular can of worms.
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
Yep, this is why the version information should be in the setup.cfg file, and hence available via pkgutil without loading the module first.
Cheers, Nick.
On Apr 07, 2011, at 12:51 PM, Michael Foord wrote:
So I don't think recommending "pkgutil.get_distribution('elle').metadata['version']" as a way for packages to provide version information is good advice.
I want to make it clear that this section of the PEP is intended only to provide some choices and examples, not to be definitive. I've added this text:
This could be done in any number of ways, a few of which are outlined below. These are included for illustrative purposes only and are not intended to be definitive, complete, or all-encompassing. Other approaches are possible, and some included below may have limitations that prevent their use in some situations.
Does that help? -Barry
On Thu, 07 Apr 2011 12:10:59 +0100 Michael Foord fuzzyman@voidspace.org.uk wrote:
On 06/04/2011 15:26, Nick Coghlan wrote:
On Wed, Apr 6, 2011 at 6:22 AM, Glenn Lindermanv+python@g.nevcal.com wrote:
With more standardization of versions, should the version module be promoted to stdlib directly?
When Tarek lands "packaging" (i.e. what distutils2 becomes in the Python 3.3 stdlib), the standardised version handling will come with it.
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']I really dislike this way of specifying the version. For a start it is really ugly.
Agreed, it is incredibly obscure and unpleasantly opaque.
Regards
Antoine.
On Thu, Apr 7, 2011 at 9:10 PM, Michael Foord fuzzyman@voidspace.org.uk wrote:
I really dislike this way of specifying the version. For a start it is really ugly.
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'
Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
What I may have to do is use a python script that will generate the static metadata, which is not such a bad thing I guess as it will only need to be executed at package build time. I won't be switching to that horrible technique for specifying versions within my packages though.
It sounds like part of the PEP needs another trip through distutils-sig. An informational PEP really shouldn't be advocating standard library changes, but it would make sense for this point of view to inform any updates to PEP 386 (the version handling standardisation PEP).
As I see it, there appear to be two main requests: 1. Normalised version parsing and comparison should be available even if packaging itself is not installed (e.g. as part of pkgutil) 2. packaging should support extraction of the version metadata from the source files when bundling a package for distribution
On point 2, rather than requiring that it be explicitly requested, I would suggest the following semantics for determining the version when bundling a package ready for distribution:
- if present in the metadata, use that - if not present in the metadata, look for __version__ in the module source code (or the __init__ source code for an actual package) - otherwise warn the developer that no version information has been provided so it is automatically being set to "0.0.0a0"
Cheers, Nick.
On 07/04/2011 12:59, Nick Coghlan wrote:
On Thu, Apr 7, 2011 at 9:10 PM, Michael Foordfuzzyman@voidspace.org.uk wrote:
I really dislike this way of specifying the version. For a start it is really ugly.
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
What I may have to do is use a python script that will generate the static metadata, which is not such a bad thing I guess as it will only need to be executed at package build time. I won't be switching to that horrible technique for specifying versions within my packages though.
It sounds like part of the PEP needs another trip through distutils-sig. An informational PEP really shouldn't be advocating standard library changes, but it would make sense for this point of view to inform any updates to PEP 386 (the version handling standardisation PEP).
As I see it, there appear to be two main requests:
- Normalised version parsing and comparison should be available even
if packaging itself is not installed (e.g. as part of pkgutil) 2. packaging should support extraction of the version metadata from the source files when bundling a package for distribution
On point 2, rather than requiring that it be explicitly requested, I would suggest the following semantics for determining the version when bundling a package ready for distribution:
- if present in the metadata, use that
- if not present in the metadata, look for __version__ in the module
source code (or the __init__ source code for an actual package)
- otherwise warn the developer that no version information has been
provided so it is automatically being set to "0.0.0a0"
This sounds good to me.
As an added consideration the suggested technique may not work for tools like py2exe / py2app, embedded python and alternative implementations - which may not have the full "pacakging" machinery available.
All the best,
Michael Foord
Cheers, Nick.
On Apr 07, 2011, at 09:59 PM, Nick Coghlan wrote:
It sounds like part of the PEP needs another trip through distutils-sig. An informational PEP really shouldn't be advocating standard library changes, but it would make sense for this point of view to inform any updates to PEP 386 (the version handling standardisation PEP).
I'm certainly open to any suggestions from distutils-sigsters, though I'm not sure the PEP needs to be discussed there exclusively at this point.
As I see it, there appear to be two main requests:
- Normalised version parsing and comparison should be available even
if packaging itself is not installed (e.g. as part of pkgutil) 2. packaging should support extraction of the version metadata from the source files when bundling a package for distribution
On point 2, rather than requiring that it be explicitly requested, I would suggest the following semantics for determining the version when bundling a package ready for distribution:
- if present in the metadata, use that
- if not present in the metadata, look for __version__ in the module
source code (or the __init__ source code for an actual package)
- otherwise warn the developer that no version information has been
provided so it is automatically being set to "0.0.0a0"
I like that. Given the recommendations in PEP 396, I think it's more in scope of the distutils-sig, and the various related PEPs to define the details of how that would work. I'd be happy to update the Deriving section of PEP 396 with any such recommendations. That section isn't meant to be definitive or even all-encompassing. It's just meant to give some examples of how you could do things in your own modules.
-Barry
Hi,
Le 07/04/2011 13:10, Michael Foord a écrit :
On 4/5/2011 11:52 AM, Barry Warsaw wrote: __version__ = pkgutil.get_distribution('elle').metadata['version']
I really dislike this way of specifying the version. For a start it is really ugly.
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
This is similar to my own comment on distutils-sig:
One of the main complaint against setuptools is that having to change your application code because of the packaging tool used was not a good idea. Having to use require instead of import or resource_whatever instead of open (or get_data, the most sadly underused function in the stdlib) because you use setuptools instead of distutils was a bad thing.
As stated in the PEP, having a __version__ attribute in the module is common, so my opinion is that making the packaging tool use that info is the Right Thing™, and having the dependency in the reverse sense is wrong. I don’t see a problem with having harmless duplication in the *installed* system, once in elle.__version__ and once in the pkgutil metadata database.
Barry’s reply:
I'm including this section because at Pycon, some people did express an interest in deriving the version number in this direction. I wanted to capture what that might look like. Since this is an informational PEP, I think it makes sense to include alternative approaches, but I tend to agree with you that it will be much more common to define module.__version__ and derive the metadata from that.
IOW, you can define the version only once, either in your source file or in the setup.cfg file, and the PEP describes how to get that info from the other place. My personal opinion is that the approach using pkgutil.get_distribution should be much less prominent than the one putting the version in the Python code.
Regards
On Apr 09, 2011, at 06:23 PM, Éric Araujo wrote:
One of the main complaint against setuptools is that having to change your application code because of the packaging tool used was not a > good idea. Having to use require instead of import or resource_whatever instead of open (or get_data, the most sadly underused function in > the stdlib) because you use setuptools instead of distutils was a bad > thing.
As stated in the PEP, having a __version__ attribute in the module is common, so my opinion is that making the packaging tool use that info > is the Right Thing™, and having the dependency in the reverse sense is wrong. I don’t see a problem with having harmless duplication in the *installed* system, once in elle.__version__ and once in the pkgutil metadata database.
Barry’s reply:
I'm including this section because at Pycon, some people did express > an interest in deriving the version number in this direction. I wanted > to capture what that might look like. Since this is an informational > PEP, I think it makes sense to include alternative approaches, but I tend to > agree with you that it will be much more common to define > module.__version__ and derive the metadata from that.
IOW, you can define the version only once, either in your source file or in the setup.cfg file, and the PEP describes how to get that info from the other place. My personal opinion is that the approach using pkgutil.get_distribution should be much less prominent than the one putting the version in the Python code.
It is already though, right? To me anyway the PEP does emphasize setting __version__, but I'm open to specific suggestions.
-Barry
On Apr 07, 2011, at 12:10 PM, Michael Foord wrote:
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']I really dislike this way of specifying the version. For a start it is really ugly.
Agreed! There should be a higher level API for this, e.g.:
__version__ = pkgutil.get_version('elle')
More importantly it means the version information is *only* available if the package has been installed by "packaging", and so isn't available for the parts of my pre-build process like building the documentation (which import the version number to put into the docs).
That would have to be an important feature of .get_version() I think.
Currently all my packages have the canonical version number information in the package itself using:
__version__ = '1.2.3'Anything that needs the version number, including setup.py for upload to pypi, has one place to look for it and it doesn't depend on any other tools or processes. If switching to "packaging" prevents me from doing this then it will inhibit me using "packaging".
It definitely shouldn't prevent this. I personally do the same thing, and it seems the least bad way of doing it. I think the clear version string assigned to __version__ is the best recommendation (though not the only one) that the PEP makes.
-Barry
On Apr 07, 2011, at 12:26 AM, Nick Coghlan wrote:
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
Indeed, I really don't like the RE suggestion - better to tell people to just move the version info into the static config file and use pkgutil to make it available as shown. That solves the build time vs install time problem as well.
I'm actually going to remove the regexp example from the PEP. It's distracting, incorrect, and unnecessary (give that `packaging` will have such an API).
Yep, this is why the version information should be in the setup.cfg file, and hence available via pkgutil without loading the module first.
If the version information is in the setup.cfg, then the question is, what's the code look like to get that stuffed into a module's __version__ attribute? If it's not the pkgutil ugliness, what is it? And does it work whether your in say the source tree of your uninstalled module, or in a Python where the package was installed via they OS?
-Barry
On Apr 05, 2011, at 01:22 PM, Glenn Linderman wrote:
On 4/5/2011 11:52 AM, Barry Warsaw wrote:
DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') __version__ = pkgutil.get_distribution('elle').metadata['version']The RE as given won't match alpha, beta, rc, dev, and post suffixes that are discussed in POP 386.
It really wasn't intended to. I'm torn about even including this code sample in the PEP. I'm highly tempted to rip this out and hand-wave over the implementation of get_version(). It's not a critical part of the PEP and might just be distracting.
Are there issues for finding and loading multiple versions of the same module?
Out of scope for this PEP I think.
Should it be possible to determine a version before loading a module? If yes, the version module would have to be able to find a parse version strings in any of the many places this PEP suggests they could be... so that would be somewhat complex, but the complexity shouldn't be used to change the answer... but if the answer is yes, it might encourage fewer variant cases to be supported for acceptable version definition locations for this PEP.
I think the answer can be "yes", but only through distutils2/packaging APIs. If there's no metadata for a module available, then I don't have a problem saying the version information can't be determined without importing it.
-Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
...
#. ``__version_info__`` SHOULD be of the format returned by PEP 386's ``parse_version()`` function.
The only reference to parse_version in PEP 386 I could find was the setuptools implementation which is pretty odd:
In other words, parse_version will return a tuple for each version string, that is compatible with StrictVersion but also accept arbitrary version and deal with them so they can be compared:
from pkg_resources import parse_version as V V('1.2')
('00000001', '00000002', '*final')
V('1.2b2')
('00000001', '00000002', '*b', '00000002', '*final')
V('FunkyVersion')
('*funkyversion', '*final')
bzrlib has certainly used 'version_info' as a tuple indication such as:
version_info = (2, 4, 0, 'dev', 2)
and
version_info = (2, 4, 0, 'beta', 1)
and
version_info = (2, 3, 1, 'final', 0)
etc.
This is mapping what we could sort out from Python's "sys.version_info".
The *really* nice bit is that you can do:
if sys.version_info >= (2, 6): # do stuff for python 2.6(.0) and beyond
Doing that as:
if sys.version_info >= ('000000002', '000000006'):
is pretty ugly.
John =:->
On Wed, Apr 06, 2011 at 11:04:08AM +0200, John Arbash Meinel wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
...
#. ``__version_info__`` SHOULD be of the format returned by PEP 386's ``parse_version()`` function.
The only reference to parse_version in PEP 386 I could find was the setuptools implementation which is pretty odd:
In other words, parse_version will return a tuple for each version string, that is compatible with StrictVersion but also accept arbitrary version and deal with them so they can be compared:
from pkg_resources import parse_version as V V('1.2')
('00000001', '00000002', '*final')
V('1.2b2')
('00000001', '00000002', '*b', '00000002', '*final')
V('FunkyVersion')
('*funkyversion', '*final')
Barry -- I think we want to talk about NormalizedVersion.from_parts() rather than parse_version().
bzrlib has certainly used 'version_info' as a tuple indication such as:
version_info = (2, 4, 0, 'dev', 2)
and
version_info = (2, 4, 0, 'beta', 1)
and
version_info = (2, 3, 1, 'final', 0)
etc.
This is mapping what we could sort out from Python's "sys.version_info".
The *really* nice bit is that you can do:
if sys.version_info >= (2, 6): # do stuff for python 2.6(.0) and beyond
<nod> People like to compare versions and the tuple forms allow that. Note that the tuples you give don't compare correctly. This is the order that they sort:
(2, 4, 0) (2, 4, 0, 'beta', 1) (2, 4, 0, 'dev', 2) (2, 4, 0, 'final', 0)
So that means, snapshot releases will always sort after the alpha and beta releases (and release candidate if you use 'c' to mean release candidate). Since the simple (2, 4, 0) tuple sorts before everything else, a comparison that doesn't work with the 2.4.0-alpha (or beta or arbitrary dev snapshots) would need to specify something like:
(2, 4, 0, 'z')
NormalizedVersion.from_parts() uses nested tuples to handle this better. But I think that even with nested tuples a naive comparison fails since most of the suffixes are prerelease strings. ie: ((2, 4, 0),) < ((2, 4, 0), ('beta', 1))
So you can't escape needing a function to compare versions. (NormalizedVersion does this by letting you compare NormalizedVersions together). Barry if this is correct, maybe __version_info__ is useless and I shouldn't have brought it up at pycon?
-Toshio
On Apr 07, 2011, at 09:13 AM, Toshio Kuratomi wrote:
Barry -- I think we want to talk about NormalizedVersion.from_parts() rather than parse_version().
See my previous follow up. It probably makes sense to be explicit in one PEP or the other, but...
So you can't escape needing a function to compare versions. (NormalizedVersion does this by letting you compare NormalizedVersions together). Barry if this is correct, maybe __version_info__ is useless and I shouldn't have brought it up at pycon?
...yikes! You might be right about that. Unless there are any counter arguments, I think I'll have to remove it from PEP 396.
(Makes me like hexversion even more :).
-Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Apr 06, 2011, at 11:04 AM, John Arbash Meinel wrote:
In other words, parse_version will return a tuple for each version string, that is compatible with StrictVersion but also accept arbitrary version and deal with them so they can be compared:
from pkg_resources import parse_version as V V('1.2')
('00000001', '00000002', '*final')
V('1.2b2')
('00000001', '00000002', '*b', '00000002', '*final')
V('FunkyVersion')
('*funkyversion', '*final')
bzrlib has certainly used 'version_info' as a tuple indication such as:
version_info = (2, 4, 0, 'dev', 2)
and
version_info = (2, 4, 0, 'beta', 1)
and
version_info = (2, 3, 1, 'final', 0)
etc.
This is mapping what we could sort out from Python's "sys.version_info".
It's probably worth specifying the __version_info__ tuple in more detail in either PEP 386 or 396. I think more detail should go *somewhere*, and it feels like it could go in either PEP. Maybe Tarek can chime in on that.
The *really* nice bit is that you can do:
if sys.version_info >= (2, 6): # do stuff for python 2.6(.0) and beyond
Doing that as:
if sys.version_info >= ('000000002', '000000006'):
is pretty ugly.
I personally often do tests against sys.hexversion, which is a little less ugly (maybe ;).
if sys.hexversion >= 0x20600f0: # 2.6 or 2.7
- -Barry
Howdy Barry,
Nitpick: Please call these “version strings”. A version string is hardly ever just one number, and not in the general case anyway.
I'd like to suggest another user story:
Barry Warsaw barry@python.org writes:
User Stories
Emily maintains a package consisting of programs and modules in several languages that inter-operate; several are Python, but some are Unix shell, Perl, and there are some C modules. Emily decides the simplest API for all these modules to get the package version string is a single text file named ``version`` at the root of the project tree. All the programs and modules, including the ``setup.py`` file, simply read the contents of ``version`` to get the version string.
This is an often-overlooked case, I think. The unspoken assumption is often that ``setup.py`` is a suitable place for the overall version string, but this is not the case when that string must be read by non-Python programs.
At 08:52 AM 4/10/2011 +1000, Ben Finney wrote:
This is an often-overlooked case, I think. The unspoken assumption is often that ``setup.py`` is a suitable place for the overall version string, but this is not the case when that string must be read by non-Python programs.
If you haven't used the distutils a lot, you might not realize that you can do this:
$ python setup.py --version 0.6c12
(The --name option also works, and they can be used together -- the answers will be on two separate lines.)
On 04:02 am, pje@telecommunity.com wrote:
At 08:52 AM 4/10/2011 +1000, Ben Finney wrote:
This is an often-overlooked case, I think. The unspoken assumption is often that ``setup.py`` is a suitable place for the overall version string, but this is not the case when that string must be read by non-Python programs.
If you haven't used the distutils a lot, you might not realize that you can do this:
$ python setup.py --version 0.6c12
(The --name option also works, and they can be used together -- the answers will be on two separate lines.)
This only works as long as setup.py is around - which it typically no longer is after installation is complete.
And though it's common and acceptable enough to launch a child process in a shell script in order to get some piece of information, it isn't as pleasant in a Python program. Can you get this version information out of setup.py without running a child process and without monkey-patching sys.argv and sys.stdout?
Jean-Paul
On Apr 10, 2011, at 08:52 AM, Ben Finney wrote:
Nitpick: Please call these “version strings”. A version string is hardly ever just one number, and not in the general case anyway.
The PEP title does say version *numbers* (plural), and that seems more general than using 'strings' here.
Emily maintains a package consisting of programs and modules in several languages that inter-operate; several are Python, but some are Unix shell, Perl, and there are some C modules. Emily decides the simplest API for all these modules to get the package version string is a single text file named ``version`` at the root of the project tree. All the programs and modules, including the ``setup.py`` file, simply read the contents of ``version`` to get the version string.
This is an often-overlooked case, I think. The unspoken assumption is often that ``setup.py`` is a suitable place for the overall version string, but this is not the case when that string must be read by non-Python programs.
I'm not certain that the additional story informs any recommendations made by the PEP. In the case where the version number is kept in some external file, then you'd likely see something like this in setup.py:
setup(version=open('version.txt').read())
or this in foo/__init__.py:
__version__ = open('version.txt').read()
The details aren't that important, but the fact that the version is kept in an external file doesn't change any of the recommendations the PEP is already making.
Cheers, -Barry
participants (11)
-
Antoine Pitrou -
Barry Warsaw -
Ben Finney -
exarkun＠twistedmatrix.com -
Glenn Linderman -
John Arbash Meinel -
Michael Foord -
Nick Coghlan -
P.J. Eby -
Toshio Kuratomi -
Éric Araujo