Revive PEP 396 -- Module Version Numbers ?
I've already asked on python-ideas, but it was suggested that as there's a PEP already, it's time to move this along. So here's the pitch. Over the years, I've seen __version__ used very broadly but not *quite* in all packages. I've always known it was a convention, not a requirement. But it turns out it's not even a "official" convention. But it really would be nice if there was a consistent way that I could count on to get the version of a package at run time from the package itself. Turns out this was suggested in PEP 396 -- and deferred almost 13 years ago! https://www.python.org/dev/peps/pep-0396/ In the status note, it says: """ Further exploration of the concepts covered in this PEP has been deferred for lack of a current champion interested in promoting the goals of the PEP and collecting and incorporating feedback, and with sufficient available time to do so effectively. """ Well, I may be willing to be that champion, if a core dev is willing to sponsor. And, well, after 13 years, we've seen __version__ be very broadly, though certainly not universally used. Honestly, I haven't looked to see to what extent setuptools supports it, but will, of course, do so if folks think this is worth pursuing. And the PyPA has moved toward "distribution" meta data, preliminarily supported by importlib.metadata.version. So maybe this is a settled issue, and we just need to change the status of the PEP. But for my part, I FAR prefer the version info to be embedded in the code of the package in a simple way, rather than hiding among the metadata, and requiring importlib.metadata.version to get a version at runtime. I note that PEP8 uses __version__ as an example of a "module level dunder" -- but only suggests where it should be put, not how it be used :-) Of course, there will be a need to update the PEP to match current practice, and if it is me doing it, I'd make it very simple So what do you'all think? After thirteen years, it would be nice to put this to bed. I think the next step is to see if I can find a sponsor :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Apr 14, 2021 at 7:48 AM Christopher Barker <pythonchb@gmail.com> wrote:
So what do you'all think? After thirteen years, it would be nice to put this to bed.
There are two main use cases for versions: * Display them to the user * Compare versions to check if one is newer, older or the same I dislike using strings for comparison. You need to use packaging.version for that: https://packaging.pypa.io/en/latest/version.html Many C libraries provide the version as a number of as 3 numbers (major, minor, micro). In its C API, Python provides all of them: * PY_VERSION_HEX: single number * (PY_MAJOR_VERSION, PY_MINOR_VERSION, PY_MICRO_VERSION, PY_RELEASE_LEVEL, PY_RELEASE_SERIAL): as 5 numbers * PY_VERSION: string In my Python projects, I like to provide the version as a tuple which can be used directly for comparison: version_a <= version_b. Example: VERSION = (2, 2, 1) __version__ = '.'.join(map(str, VERSION)) The tuple might contain strings like "beta" or "rc", as soon as comparison makes sense ;-) Sadly, such tuple is no standardized. Which part is the major version? How to format it as a string? Good luck with trying to standardize that ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, 14 Apr 2021 at 13:28, Victor Stinner <vstinner@python.org> wrote:
Agreed, programmatic use should really conform to PEP 440 versioning, as implemented in packaging.version, if it's to align with existing packaging standards (which currently focus on *project* versions rather than package versions, but the distinction is not something we should be making a big deal of).
Agreed. We should stick to PEP 440 format strings, and not try to dictate other formats like (named) tuples. But does the PEP want to cover programmatic use *at all*? It would be possible to define __version__ as "a human-readable string in PEP 440 format" and note that any use other than for display to a person is outside the scope of the PEP. That's closer in spirit to the 13-year old version we currently have, but it does miss an opportunity as we now do have a standard version class ("standard" in the sense of "a packaging standard", but not "in the stdlib"). Paul
In my Python projects, I like to provide the version as a tuple which can be used directly for comparison
To add to this, comparing tuples doesn't always work well for projects where multiple release lines are maintained simultaneously, e.g. user-facing changes introduced in minor/point releases across several major versions. People use version numbers in wildly different ways. Barney On Wed, 14 Apr 2021 at 13:26, Victor Stinner <vstinner@python.org> wrote:
I recently encountered this, which is very useful, but only for a human-readable perspective.
On Wed, Apr 14, 2021 at 2:01 PM Barney Gale <barney.gale@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, Apr 14, 2021 at 7:48 AM David Mertz <mertz@gnosis.cx> wrote:
Well, THAT is a great argument for some official standardization! There is sometimes a need for that sort of thing, but I think it's best handled by either putting __version__ in each sub_package, or having a different attribute altogether. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Agreed! I was trying to figure out why an API from Vaex 3.x was no longer working, and to my human eye, this quickly pointed me at the issue. However as a way to automatically check for versions, this is a mess. I have no idea what additional keys the next version might add or remove, for example. I'm not even sure if this is dynamically determined based on optional components being installed or not. Providing this kind of information *somehow* feels like a useful thing to do. But .__version__ is probably not the right way to do it. On Wed, Apr 14, 2021 at 5:08 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, Apr 14, 2021 at 5:24 AM Victor Stinner <vstinner@python.org> wrote:
Yes, though it looks a little too heavyweight to be required, nifty features like ``is_prerelease` is going pretty far. On the other hand, maybe no harm in all that, as long as: - It's easy to do the easy stuff (which is looks like it is) - It would be added to the stdlib However, there is a lot of code out there that's already using strings for __version__, so it would be a real shame if we had to ask everyone to upgrade. And Version isn't going to be in older versions of the Python stdlib, so hard to make code compatible. But: If we make a Version object that can be compared to (confoming) strings then there might be a path forward.
In my Python projects, I like to provide the version as a tuple which can be used directly for comparison: version_a <= version_b. Example:
Yes, and tuples are already built in to Python, so an easier lift than adding the Version object. Another possible issue: using Version would require an extra import in many module initializations -- is that a performance issue that would matter? In any case, I think the PEP should specify a standard for what is in __version__ And ideally it would be backward compatible with strings, which are the most commonly used currently. And conform to PEP 440 Sadly, such tuple is no standardized. Which
part is the major version? How to format it as a string?
Good luck with trying to standardize that ;-)
well, PEP 440 gets us close -- and I *think* compliant strings could be unambiguously converted to-from tuples. Anyway, the question now for me is whether this is worth any more of my time. So: - Is there someone willing to sponsor? - Do folks generally think there's a chance of it being accepted without a huge debate and expansion of scope. If the answers are not yes, I can better spend my Python time of other things. BTW: one source of hope: __version__ strings are pretty common, but, as evidenced already by this brief discussion, it's not by any means a universal standard -- so we really can't break anything that already works in any universal way. What i mean by that is that anything done here might break someone's special use case of __version__ (Like the one David Mertz identified), but that scheme only works with that one package's code anyway. There are no general tools that expect that scheme. And it could be preserved by making a subclass of Version (Or str) that was also a mapping :-) - CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Apr 14, 2021 at 5:44 PM Christopher Barker <pythonchb@gmail.com> wrote:
I like having a `Version` object that is easily importable in the standard library. I think it should relatively polymorphic. I.e. it should accept a module or package as an argument, but also should accept a string or a tuple. Maybe other objects from which one could reasonably extract a version. In particular, I think initializing this object with a module object should at least look for a .__version__ attribute, and appropriately case either a string (that looks sufficiently version-like) or a tuple. I think that if it doesn't succeed, it should become some sort of "cannot determine" object that is neither less than nor greater than any other Version object. In particular, the Vaex example with a dictionary of versions of each component should probably just become this "cannot determine" value (but as an instance of Version). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, 14 Apr 2021 at 18:01, David Mertz <mertz@gnosis.cx> wrote:
If it's not basically equivalent to packaging.version.Version (and based on PEP 440) then we'll be creating a nightmare of confusion, because PEP 440 versions are fundamental to packaging.
In particular, I think initializing this object with a module object should at least look for a .__version__ attribute, and appropriately case either a string (that looks sufficiently version-like) or a tuple. I think that if it doesn't succeed, it should become some sort of "cannot determine" object that is neither less than nor greater than any other Version object. In particular, the Vaex example with a dictionary of versions of each component should probably just become this "cannot determine" value (but as an instance of Version).
What's wrong with Version(module.__version__)? And if the __version__ attribute isn't a valid version, raise an exception? That's what packaging.version does, and it's worked fine for the packaging ecosystem. Is there a benefit that justifies being different here? Can I remind people that the packaging community have done a *huge* amount of work designing and standardising concepts like version identifiers, version comparisons and constraints, etc. (PEPs 440 and 508). Understanding those standards should really be the basic starting point for any discussion like this. Paul
On Wed, Apr 14, 2021 at 9:12 PM Paul Moore <p.f.moore@gmail.com> wrote:
Are you suggesting that users should have to install an external module to tell what version of packages they are using?! What's wrong with Version(module.__version__)? And if the __version__
Doesn't that seem really, really painful to use in an interactive shell? This is honestly (a simplified version of) what I tried in trying to learn packaging.version.Version. It was not a good experience: % conda create -n test python=3.9 pandas [...] % conda activate test % python
I cannot find a case where the more verbose spelling is ever nicer than just referencing `mod.__version__` directly for such exploration. Yes, I get that formal packaging has different needs. But here it just seems like a lot more work to get a lot less information back. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, 14 Apr 2021 at 21:59, David Mertz <mertz@gnosis.cx> wrote:
No. To tell what version of a package they are using, a string is sufficient. They only need a version object if they want to do additional processing (like comparing versions, or checking whether a version meets a constraint). Given that the packaging ecosystem already has a canonical version object (provided by the packaging library), which has been used and tested extensively in many environments, inventing a different API seems at best ill-advised. Whether the stdlib needs a version object. rather than leaving that functionality to a 3rd party library, is the same question that comes up for *any* functionality that's proposed for the stdlib, and I have no particular opinion in this case.
It's designed for programmatic use, not interactive use, yes. But that's sort of my point, why do you want anything more than the bare string in the REPL? What are you planning on doing with it? Paul
On Wed, 14 Apr 2021 22:23:44 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
Tangentially, until now projects could use distutils's LooseVersion if they wanted to compare version numbers reliably. With distutils being deprecated, they'll have to either depending on packaging (which is a large dependency just for comparison version numbers) or vendor packaging's Version class (which is doable but still some bothersome additional work). Regards Antoine.
Paul Bryan:
Seems like this is something that should make its way into stdlib?
In the last 10 years, the trend is more to remove anything related to packaging *outside* the stdlib :-) Since it's evolving way faster than Python releases and the stdlib cannot be updated once installed, I don't think that it's a good idea. On Thu, Apr 15, 2021 at 12:22 AM Antoine Pitrou <antoine@python.org> wrote:
If packaging is too big and if packaging maintainters like the idea, maybe packaging.version could be moved into a dedicated package? I didn't look if it makes sense from a technical point of view. $ wc -l packaging/version.py packaging/_structures.py 556 packaging/version.py 86 packaging/_structures.py 642 total version.py uses _structures.py (InfinityType, NegativeInfinityType). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 15 Apr 2021 11:37:28 +0200 Victor Stinner <vstinner@python.org> wrote:
What is "evolving way faster"? packaging.version implements PEP 440, which has been approved in 2014. Recent changes have been really minor: https://github.com/pypa/packaging/commits/main/packaging/version.py
On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner <vstinner@python.org> wrote:
Well, maybe all of us don't think that's a good idea ;-) But anyway, I would say removing anything *related* to packaging outside the stdlib is a bad idea -- requiring an external tool to build a package is OK, but requireing an external use packages, not so much. Presumably that's why importlib.metadata exists in the stdlib. Version is arguably useful from the package user side. As I believe Victor mentioned, there are two uses for version information: display to the user -- for which version strings are fine, or programmatic comparison -- for which something like the Version object is very helpful. Do we only need to use version information programmatically when we are creating (or installing) packages? I don't think so -- I know I have code that (poorly) does version checking programmatically. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 4:55 PM Christopher Barker <pythonchb@gmail.com> wrote:
Presumably that's why importlib.metadata exists in the stdlib.
I was so hopeful about this, but in the end... not really. I have not used this capability before. Here are a few different situations I know of: pandas 1.2.4 vaex {'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0', 'vaex-hdf5': '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0', 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.11.1'} bs4 4.9.3 pandas 1.2.4 vaex 4.1.0 bs4 PackageNotFoundError('bs4') It seems like (somehow or another) importlib.metadata is doing something perhaps more useful for vaex. But it is doing distinctly worse for re, statistics, and bs4. Version is arguably useful from the package user side. As I believe Victor
I think the key thing that Chris and I are pointing to is that there is a definite use for versions that is NOT for package maintainers/creators. Interactive use is definitely one such case, and eyeballing something, even if it is the oddball use that Vaex has, can help with figuring out an issue. But in actual code, I relatively often do something like this. Or rather, the below is what I would find really nice to be able to do. ver = robust_version(module) if ver >= (5, 2, 1): doit_modern_style() elif ver < (5, 2, 1): doit_old_style else: doit_unversioned_style() I'd like code like that to ALWAYS succeed. No exceptions raised. And it would *usually* use reasonable heuristics to figure out some sort of structured version info as well as is possible. I.e. probably look in several places and return some custom object that represents a "best effort." That object might be like NaN in being neither less than nor more than other things, in the fallback case. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Apr 15, 2021 at 9:36 AM David Mertz <mertz@gnosis.cx> wrote:
I don't see how importlib.metadata plays into your code sample since you didn't import it to use it?
And is installing a dependency that much of a burden for that code?
All doable with importlib.metadata and 'packaging'. -Brett
On 4/15/2021 12:35 PM, David Mertz wrote:
re 2.2.1
re.__version__ was last modified 2001-10-07 by F. Lundh in bec95b9d8825b39cff46a8c645fa0eeb8409854e. What does '2.2.1' mean? Only Fredrik knows. The commit message says that he rewrote the pattern.sub and pattern.subn methods in C. That should have been user invisible except for speed. I believe that it is dead data left only for back compatibility and because of an inability to give a deprecation warning.
re PackageNotFoundError('re')
It seems like (somehow or another) importlib.metadata is doing something perhaps more useful for vaex. But it is doing distinctly worse for re,
Is an effectively meaningless and misleading value better than none? IDLE once had a version last set, I believe, in 2.5 or 2.6, by K. Kaiser, with whatever meaning it had to him. I removed it in 3.6 (allowed by PEP 434 + RM decision). No one complained. The real version of any stdlib module is the Python version it is released with, plus and external app versions it is associated with (sqlite and ssl, for instance). It has been suggested, if not decided, that other versions should go. -- Terry Jan Reedy
On Thu, Apr 15, 2021 at 9:36 AM David Mertz <mertz@gnosis.cx> wrote:
I was so hopeful about this, but in the end... not really. I have not used this capability before. Here are a few different situations I know of:
... ... re PackageNotFoundError('re') statistics PackageNotFoundError('statistics') pandas 1.2.4 vaex 4.1.0 bs4 PackageNotFoundError('bs4')
funny you should try bs4, which I also used as an example in another post. But what you have found is that there is a difference between a pacakge (which you can import) and a "distribution" which is something you install. importlib.metadata is looking for distributions. re and statistics are not distributions, but rather, built in pacakges (modules). bs4 is not a distributiuion, it is a package that is provided by the "beautifulsoup4" distribution. In [3]: importlib.metadata.version("beautifulsoup4") Out[3]: '4.9.3' Frankly, I'm still a bit confused about the distiction, but I do know for sure that a single distribution (maybe somethign that you can install via PyPi) can install more than one top-level package -- and certainly the packages/modules installed can have a different name than the distribution name. And if a distribution installs more than one package, they may have different version numbers. I'm not sure what to make of all this, though I'm leaning toward better suporting the distiction by asking for __version__ strings in top-level packages -- and maybe making importlib.metadata.version a bit smarter about looking for packages, and not just distributions. If you look at the docstring of metadata.version: """ In [2]: importlib.metadata.version? Signature: importlib.metadata.version(distribution_name) Docstring: Get the version string for the named package. :param distribution_name: The name of the distribution package to query. :return: The version string for the package as defined in the package's "Version" metadata key. File: ~/miniconda3/envs/py3/lib/python3.9/importlib/metadata.py Type: function """ It's a bit inconsistent with the use of the term "distribution" vs "package". That should get cleaned up if nothing else. Also, the Exception raised is "PackageNotFoundError" -- which should maybe be "DistributionNotFoundError"? Version is arguably useful from the package user side. As I believe Victor mentioned, there are two uses for version information: display to the user -- for which version strings are fine, or programmatic comparison -- for which something like the Version object is very helpful. Do we only need to use version information programmatically when we are creating (or installing) packages? I don't think so -- I know I have code that (poorly) does version checking programmatically.
Exactly -- and I htink we are close, if pacakges(modules) had compliant __version__ strings, thenyou could do (with the Version object from packaging -- maybe why it should be in the stdlib) ver = Version(module.__version__) if ver >= Version("5.2.1"): doit_modern_style() elif ver < Version("5.2.1"): doit_old_style else: doit_unversioned_style() And if my PR is accepted (which looks unlikley) to allow camparison between Version objects and strings: ver = Version(module.__version__) if ver >= "5.2.1": doit_modern_style() elif ver < "5.2.1": doit_old_style else: doit_unversioned_style() A key distiction here from the importlib.metadata approach is the level of indirection -- I like being able to ask a module itself what version it is, rather than asking some other part of teh system to go look it up for me. So I could do: import numpy as np print("using np version:", np.__version) And this is pretty consitent with the rest of Python, where many objects (functions, classes, modules) have a __name__ attribute if things "know" their name, shouldn't they "know" their version as well? -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 9:54 PM Christopher Barker <pythonchb@gmail.com> wrote:
This wouldn't be bad. A little more verbose, but fine. The thing is, I almost never care about the version of a *distribution* in my programs. I care about a version of the *package*. Since, as Chris mentions, a distribution could contain multiple packages, the same package could be vendorized (in different versions) by different distributions. The question I want to ask above is always because "version such-and-such changed an API," and that's about a package. When I care about distribution versions, it is because there are dependencies stated in requirements.txt or environment.yaml or whatever.deb. Asking for the `.__version__` attributes feels like needless redundancy in Chris' variation. The constructor should be able to ask "Is this a module? If so, does it have that attribute". But whatever, I can learn to do that easily enough. Someone else mentioned that the versions of standard library modules are an odd collection of relics. I do not disagree, of course. But sometimes the stdlib modules *do* change. I thought of the `re` example because I was doing a tutorial on it, and something or another depended on a change between 3.8 and 3.9 (I think, I forget the detail). For training material, this can just be approached as describing the Python version... but for running code, a version check could be relevant. Therefore, I think this `Version` constructor (or `robust_version()` function, or whatever) should have access to that. I.e. if not an automated update to `.__version__` inside each stdlib module during builds, then a fallback to "If this is a stdlib module, report sys.version_info (within the right kind of "Version" object). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Apr 15, 2021 at 8:50 AM Christopher Barker <pythonchb@gmail.com> wrote:
As an active participant in the PyPA side of the packaging ecosystem and one of the maintainers of 'packaging', I am definitely NOT in favour of moving any of 'packaging' into the stdlib, nor breaking it up. We are taking distutils out of the stdlib for a reason. Backsliding on plans that have been in place for years is not something I would want to see happen as the motivations have not changed.
You can totally use a package already as things exist today and will in the future; you're just wanting a specific way to interpret a package's metadata which I don't view as the same thing.
Presumably that's why importlib.metadata exists in the stdlib.
It's there because of how tightly it binds to the import system; same goes for importlib.resources.
No one said that's the *only* use-case, but just because there's more than that does not mean we should move any packaging code into the stdlib to support a zero-install use-case. A key point I want to make here is the version metadata is in every project's .dist-info/METADATA file. That's what importlib.metadata knows how to read and hence why people have been suggesting to use it. The version information is fundamental to the wheel format and in fact you can't have a wheel without it. So asking every Python project to set a __version__ isn't going to change accessibility of version numbers when it comes to installed projects.
On Thu, 15 Apr 2021 10:28:53 -0700 Brett Cannon <brett@python.org> wrote:
This seems gratuitously dogmatic. Version numbers are a simple feature that is stably encoded in PEP 440. It's purely computational, does not involve reading or saving any persistent state in the filesystem, making any network access, etc. Why wouldn't it have its place in the stdlib?
That specific way is a Python standard (PEP 440). Having the functionality in the stdlib would encourage people to use it. Not having it in the stdlib encourages people to use adhoc version parsing, or worse, naive string comparison.
That doesn't have much to do with the suggestion of shipping a Version class in the stdlib, though. Many projects already provide a __version__, and that field is often inspected programmatically in dependent packages (to work around behaviour changes, for example). Regards Antoine.
On Thu, Apr 15, 2021 at 10:53 AM Antoine Pitrou <antoine@python.org> wrote:
Technically PyPA specs might start as PEPs but their official home is packaging.python.org and updates can occur there outside of the PEP process. In the case of version specifiers ( https://packaging.python.org/specifications/version-specifiers/), they point back to the original PEP, but that is not universally true nor guaranteed to hold in the future. In other words I personally don't view the fact that it's a PEP as a reason to have it in the stdlib (else pretty much every single packaging PEP is missing). We also have other things standardized in PEPs and not covered in the stdlib, e.g. @ as an operator. So I don't think version comparison occurs often enough to be in the stdlib, and the fact that an external project exists which isn't interested in being put in the stdlib suggests to me it isn't worth doing. But that's a separate topic to discuss at the language summit. :)
I don't know if I draw the same line since the packaging community is relying on 'packaging' without issue for this exact thing.
But this thread is about standardizing on __version__ which then led to wanting to bring in some Version object into the stdlib to make that more useful. So to me they are tied together, else a separate thread should probably be started if a new module *just *for version parsing support is desired.
On Thu, 15 Apr 2021 16:12:57 -0700 Brett Cannon <brett@python.org> wrote:
Ah, I should submit a question then.
Right, but the functionality is useful generally, not only for the packaging generally.
You're right, this thread was hijacked somewhat for the Version object discussion. Regards Antoine.
On Wed, Apr 14, 2021 at 2:23 PM Paul Moore <p.f.moore@gmail.com> wrote:
exactly -- if we do anything at all here, it should probably be at least. if you provide a __version__ attribute, it should be a PEP 440 compliant string. That would at least make the use __version__ a touch more consistent. The next, more controversial, step would be to suggest that people SHOULD provide a __version__string on all top-level packages. I would like that -- the current situation where many, but not all packages have __version__ is really annoying. They only need a version object if they want to do additional
processing (like comparing versions, or checking whether a version meets a constraint).
And indeed, we could add the ability for packaging.version.Version objects to be able to compare themselves with compatible strings -- I think that would be pretty handy. Given that the packaging ecosystem already has a canonical version
absolutely.
I don't think it's the same as any functionality -- if we do want to better standardize package versioning in Python, and I think we should, then the Version object may become something useful to, and maybe even needed by, virtually every third party package. Which makes it a pretty prime candidate for the stdlib. Alternatively, the packaging package is pretty small, but if it grows, it might be good to split out the run-time vs build-time pieces. It's designed for programmatic use, not interactive use, yes. But
that's sort of my point, why do you want anything more than the bare string in the REPL? What are you planning on doing with it?
there is something in between the REPL and full on system development -- something simple for quickly scripts is nice too. But a simple standardised version string is fine for that. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Paul Moore wrote:
What's wrong with Version(module.__version__)? And if the __version__ attribute isn't a valid version, raise an exception?
I don't have a deep answer, but I do think __version__ should be specified (or at least mentioned) at https://docs.python.org/3/reference/datamodel.html At the moment, I can't even find a listing of possible __dunder__ attributes, though I'm almost sure I've seen one in the past. -jJ
https://docs.python.org/3/reference/datamodel.html On Wed, 2021-04-14 at 22:56 +0000, Jim J. Jewett wrote:
On Wed, Apr 14, 2021 at 7:04 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
Given the intent to reject PEP 394, I can't imagine any good would come of that. Drop a __version__ in your modules if you find it valuable, but otherwise, there's nothing to do here. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
On 4/14/2021 12:38 PM, Christopher Barker wrote:
I think that before these can be answered, you need to decide why this needs to be standardized. I don't see any reason to standardize it unless there's some programmatic use for these version strings. If it's just the user poking around on the REPL, then I think the status quo is fine. I've read the PEP, and the User Stories section talks about user convenience, not programmatic access. I also think the distribution version is more useful than any __version__ information in each module/package. I realize this information might not be available, depending on how the code was installed (say, just by copying some files into the right place). Eric
On Wed, 14 Apr 2021 at 18:04, Eric V. Smith <eric@trueblade.com> wrote:
Agreed. The original PEP was very limited, just proposing a string value for user convenience, with no programmatic interface. Doing anything more than that is opening up a whole load of complexity and compatibility (with packaging) that no-one has expressed any need for. The distribution version (exposed via importlib.metadata) is the correct thing to use for anything programmatic. Paul PS I see Barry plans on rejecting the PEP, which I think is probably the right decision, given the way this thread has developed.
On Wed, Apr 14, 2021 at 4:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
PS I see Barry plans on rejecting the PEP, which I think is probably the right decision, given the way this thread has developed.
Barry makes good plans. Putting the version into the sources is a bit of an anti-pattern. IMO. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
On Apr 13, 2021, at 22:40, Christopher Barker <pythonchb@gmail.com> wrote:
Turns out this was suggested in PEP 396 -- and deferred almost 13 years ago!
I’d forgotten that this PEP was in Deferred state. I think it should be rejected and I plan on making that change. importlib.metadata is a much better approach to programmatically determining package versions. https://docs.python.org/3/library/importlib.metadata.html#distribution-versi... -Barry
On Wed, Apr 14, 2021 at 12:10 PM Barry Warsaw <barry@python.org> wrote: I’d forgotten that this PEP was in Deferred state. I think it should be
Barry, You wrote the original PEP, so of course you can withdraw it (or reject it), but... Are you sure? See this discussion, I don't think it's as simple as all that. Honestly, I'm still kind of confused about a "distribution" vs a package or module, but in general, it seems to me that the packaging community has done a great job of building a system that can accommodate a lot of complexity, but as they say: The easy things should be easy, and the hard things should be possible. And the full process of using packaging and distributions, and a Version object and all that is not so easy anymore --at least compared to putting: __version__ = "1.2.3" in a package __init__.py. Anyway, a couple points: 1) it seems self-evident that it would be easier for the whole community if "how to declare", and "how to find" the version of a package was standardized: "there's only one way to do it" 1b) It seems like a really good idea that the (or one of the) official recommended way to do it is supported in the standard library itself, and using a string is pretty easy to support :-) -- do we really want people to have to install a third party package to set or get the version of a simple module? 2) __version__ strings are already pretty darn common -- maybe more common than proper distribution metadata? -- are we saying that everyone should stop doing that? 3) I don't know that having __version__ strings is incompatible with packging.version.Version or importlib.metadata.version 4) Yes, there are real advantages to the more complex system for programmatically working with versions. But there are real advantages to something simple for human readability / interactive use -- and, frankly, simple scripting. See David Mertz's recent post -- I went through a similar process in iPython. Consider this analogy: if I want to know what version of git I have installed, I can run: $ git --version And that's a convention adhered to by many *nix command line tools. Would we all really be happier if we had to go use an rpm or deb or homebrew, or ?? command to see what package i have installed? I think this is similar -- while a good distribution management system is a good thing, it's also good if I can tell something about the version of a package/module from the package itself. With regard to having both __version__ and importlib.metadata.version, see beautiful soup for example: In [10]: import beautifulsoup4 ... ModuleNotFoundError: No module named 'beautifulsoup4' oops, that's not what the importable module is called. So: In [11]: import bs4 That worked -- now, what version is it? In [12]: importlib.metadata.version('bs4') --------------------------------------------------------------------------- PackageNotFoundError Traceback (most recent call last) ... PackageNotFoundError: bs4 [*] That didn't work -- I guess I need to use the distribution name: In [13]: importlib.metadata.version('beautifulsoup4') Out[13]: '4.9.3' OK, that worked. But I imported "bs4", so what if I do this? In [14]: bs4.__version__ Out[14]: '4.9.3' Ah yes, that worked too. So I guess what I'm suggesting is that we officially recommend that folks do what the BeautifulSoup folks, and many others, are already doing. In many (most) cases, the distribution name and the importable package name are the same, but this does accommodate both. [*] One other annoyance to importlib.metadata.version is that in addition to having to import importlib to use it, I also need to use importlib.metadata.PackageNotFoundError if I want to trap the exception. BTW: I'm not making that last point up -- I recently updated some in-house code (that someone else had written) in a web app that presented the version of various packages core to the system. We had just refactor and one of these packages what now optional, but the whole system was crashing on that darn PackageNotFoundError. I ended up jsut commenting out that package, rather than dealing with how to catch the exception and deal with it properly. I would have so much rathered a simple: try: import the_package the_pacakge_version = the_package.__version__ except the_package_version = None Granted, it's not that hard to "do it right", but I still like things simple :-) Anyway, as the original PEP author is no longer supportive, this is dead in the water unless another core dev is interested in sponsoring this (or a new) PEP. If not, I can stop wasting my and everyone else's time. -Chris -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Apr 14, 2021, at 23:11, Christopher Barker <pythonchb@gmail.com> wrote:
You wrote the original PEP, so of course you can withdraw it (or reject it), but...
Are you sure? See this discussion, I don't think it's as simple as all that.
From a library maintainers point of view, I personally want to get away from using __version__ strings at all. They’re kind of a pain to remember to bump on every new release. PEP 396 has been dormant for a long time and I have no interest in pushing it forward any more. If someone else wants to take up the cause, I recommend creating a new PEP and referring back to 396. -Barry
On 4/15/2021 12:38 PM, Barry Warsaw wrote: that.
From a library maintainers point of view, I personally want to get away from using __version__ strings at all. They’re kind of a pain to remember to bump on every new release.
At least some in the stdlib have not been. For re.__version__, not since Nov 2001. -- Terry Jan Reedy
On Thu, Apr 15, 2021 at 12:27 PM Terry Reedy <tjreedy@udel.edu> wrote:
At least some in the stdlib have not been. For re.__version__, not since Nov 2001.
From Serhiy over on python-ideas, he's started an effort to clean that up. https://mail.python.org/archives/list/python-dev@python.org/thread/KBU4EU2JU... ). I believe there is an unfinished PR -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 9:38 AM Barry Warsaw <barry@python.org> wrote:
PEP 396 has been dormant for a long time and I have no interest in
That’s a tooling issue, and I don’t think that Python itself should take any official stance on the tooling. But yes, anything that Python takes an official stance on should be able to be easily supported with not-too-complex tools.and I think a __version__ string fits this need. Personally, I use the __version__ string in my packages as the canonical source when I bold my packages. And there are fancier tools for more complex needs. pushing it forward any more. If someone else wants to take up the cause, I recommend creating a new PEP and referring back to 396. Fair enough -- I'm still gathering data to see if I should do just that. I am leaning toward at least SOME PEP about this -- having __version__ around as a semi-convention with no official recommendation as to if or how it should be used is kind of a mess. -CHB
On Thu, 15 Apr 2021 at 21:03, Christopher Barker <pythonchb@gmail.com> wrote:
I am leaning toward at least SOME PEP about this -- having __version__ around as a semi-convention with no official recommendation as to if or how it should be used is kind of a mess.
On the other hand, having a PEP that says you "may" have a version attribute, but mandates nothing, is pretty useless. People who conform to that convention don't need telling, people who don't have little reason to change, and people wanting to use the information still have to cater for it being missing, so they gain nothing. But a PEP that makes __version__ a requirement fails instantly, because there's too many packages in the wild that violate it. Welcome to the world of packaging PEPs :-)
I'm not sure what to make of all this, though I'm leaning toward better suporting the distiction by asking for __version__ strings in top-level packages -- and maybe making importlib.metadata.version a bit smarter about looking for packages, and not just distributions.
Personally, I'm leaning more and more towards the view that it's *distributions* that are versioned, and that's an established practice. Versioning packages as well (except in the very limited sense that package X installed by distribution Y can expose Y's version via its API) only adds confusion and makes the situation less, not more, useful for users. So I'm now -1 on any PEP to try to formalise the use of __version__. If people want to discuss the use of a __version__ attribute, they should consider contributing to the discussions on versions in https://packaging.python.org/guides/ (where the necessary context is already present). Paul
Hi, sorry for being late to the party, but I may not be the only one wondering… Le 14/04/2021 à 20:56, Barry Warsaw a écrit :
I’d forgotten that this PEP was in Deferred state. I think it should be rejected and I plan on making that change. importlib.metadata is a much better approach to programmatically determining package versions.
https://docs.python.org/3/library/importlib.metadata.html#distribution-versi...
This is indeed the correct approach, thanks for letting me learn this. However, I unsuccessfully searched for the canonical way to look up the distribution name based on either a module name or an imported module object. Is there one? Looks like it could be computed based on information in "*.egg-info/installed-files.txt", but it's far from trivial. Is "installed-files.txt" even guaranteed to exist for all distributions? Cheers, Baptiste
On Mon, Apr 26, 2021 at 9:37 AM Baptiste Carvello < devel2021@baptiste-carvello.net> wrote:
If you mean how to tie a module back to its name on PyPI, you should be able to look up the "Name" in the project's metadata: https://docs.python.org/3/library/importlib.metadata.html#distribution-metad... . -Brett
On Mon, Apr 26, 2021 at 10:31 PM Brett Cannon <brett@python.org> wrote:
The missing bit here, for me, is how do you map a module back to it's project (distribution)? For example: ```
This is because the distribution calls itself 'beautifulsoup4' instead.
The same goes for another package: `umap`, for which the distribution is
called `umap-learn`
This is the best I could come up with from reading the docs:
import bs4 #<- This is the module we want the version of
import importlib
import sys
from itertools import chain
from pathlib import Path
loaders = sys.meta_path
target_path = Path(bs4.__file__)
distros = list(chain(*(finder.find_distributions() for finder in
loaders if hasattr(finder, 'find_distributions'))))
distros_files = chain(*(f for f in (d.files for d in distros)))
distro_files = [(d, d.locate_file(f)) for d in distros if d.files for f
in d.files]
matching = [d for d, f in distro_files if f == target_path]
for match in matching:
print("Found Version:", match.version)
Steve
On Mon, Apr 26, 2021 at 3:25 PM Stestagg <stestagg@gmail.com> wrote:
Unfortunately I thought importlib.metadata would have used the module name instead of the metadata details, but in hindsight am guessing that the .dist-info is what it's using to do the lookup and that's based on the package name instead of the project name. This is a long-standing issue with projects that use project names which differ from their module name, but there's no good way without checking what files a project installed (which is what you're doing below). -Brett
On Tue, 27 Apr 2021 at 18:01, Brett Cannon <brett@python.org> wrote:
Unfortunately I thought importlib.metadata would have used the module name instead of the metadata details, but in hindsight am guessing that the .dist-info is what it's using to do the lookup and that's based on the package name instead of the project name.
This is a long-standing issue with projects that use project names which differ from their module name, but there's no good way without checking what files a project installed (which is what you're doing below).
That's correct. There is no link from a given import name back to the PyPI project name. And the metadata for installed packages (defined here: https://packaging.python.org/specifications/recording-installed-packages/) is keyed on the PyPI project name, as the OP noted ("beautifulsoup4" rather than "bs4", "setuptools" rather than "pkg_resources", etc).
The following is a bit simpler. The file.locate() method of a distribution is undocumented - but I tried to use resolve() on the path object I got from dist.files, and it appears not to be implemented (at least in Python 3.9) - PackagePath objects are a subclass of *Pure* Path objects, not concrete paths :-( TBH, I suspect the fact that it's undocumented is an oversight, it's clearly deliberately added. import importlib.metadata from pathlib import Path import pkg_resources target = Path(pkg_resources.__file__) for dist in importlib.metadata.distributions(): for file in dist.files: path = file.locate() if path == target: print(f"{dist.metadata['name']}: {dist.version}") break To be honest, if anyone were interested in making a PEP from any of this, having Python modules contain a __distribution_name__ attribute that links the module back to the PyPI distribution, would probably be more useful than standardising __version__. But I'm not sufficiently interested to do anything more than mention that as a possibility. Paul
Replying to the original email because my comment is more a thought after reading the whole thread rather than a response to any particular message. What I love about __version__ is that it is simple. It doesn't come with the complexity of solutions like `importlib.metadata.version`. The maintainers of the project arranged for it to be set**. It doesn't depend on how the module was packaged or distributed. I also don't think it's really a language update or requires any changes in the stdlib -- it's just a standard for the community to follow where none currently exists. It's also probably far from the end of the world*** if it doesn't happen, but I still like the idea. ** It might have been updated somewhat automatically -- my point here is that distutils, setuptools, packaging, pip, wheels, PyPI were all not involved. *** Although that is likely true of most PEPs, even much more complicated ones. :) Schiavo, Simon
Replying to myself to update my own thoughts: I think __version__ and packaging tools answer separate questions. __version__ answers the question of "this module I imported, what version is it". The packaging tools answer the question "what dependency does this distribution satisfy". This mismatch is why importlib.metadata.version("bs4") doesn't return a useful answer. A single distribution might even install multiple top-level modules with different versions. Schiavo, Simon
On Wed, Apr 14, 2021 at 7:48 AM Christopher Barker <pythonchb@gmail.com> wrote:
So what do you'all think? After thirteen years, it would be nice to put this to bed.
There are two main use cases for versions: * Display them to the user * Compare versions to check if one is newer, older or the same I dislike using strings for comparison. You need to use packaging.version for that: https://packaging.pypa.io/en/latest/version.html Many C libraries provide the version as a number of as 3 numbers (major, minor, micro). In its C API, Python provides all of them: * PY_VERSION_HEX: single number * (PY_MAJOR_VERSION, PY_MINOR_VERSION, PY_MICRO_VERSION, PY_RELEASE_LEVEL, PY_RELEASE_SERIAL): as 5 numbers * PY_VERSION: string In my Python projects, I like to provide the version as a tuple which can be used directly for comparison: version_a <= version_b. Example: VERSION = (2, 2, 1) __version__ = '.'.join(map(str, VERSION)) The tuple might contain strings like "beta" or "rc", as soon as comparison makes sense ;-) Sadly, such tuple is no standardized. Which part is the major version? How to format it as a string? Good luck with trying to standardize that ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, 14 Apr 2021 at 13:28, Victor Stinner <vstinner@python.org> wrote:
Agreed, programmatic use should really conform to PEP 440 versioning, as implemented in packaging.version, if it's to align with existing packaging standards (which currently focus on *project* versions rather than package versions, but the distinction is not something we should be making a big deal of).
Agreed. We should stick to PEP 440 format strings, and not try to dictate other formats like (named) tuples. But does the PEP want to cover programmatic use *at all*? It would be possible to define __version__ as "a human-readable string in PEP 440 format" and note that any use other than for display to a person is outside the scope of the PEP. That's closer in spirit to the 13-year old version we currently have, but it does miss an opportunity as we now do have a standard version class ("standard" in the sense of "a packaging standard", but not "in the stdlib"). Paul
In my Python projects, I like to provide the version as a tuple which can be used directly for comparison
To add to this, comparing tuples doesn't always work well for projects where multiple release lines are maintained simultaneously, e.g. user-facing changes introduced in minor/point releases across several major versions. People use version numbers in wildly different ways. Barney On Wed, 14 Apr 2021 at 13:26, Victor Stinner <vstinner@python.org> wrote:
I recently encountered this, which is very useful, but only for a human-readable perspective.
On Wed, Apr 14, 2021 at 2:01 PM Barney Gale <barney.gale@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, Apr 14, 2021 at 7:48 AM David Mertz <mertz@gnosis.cx> wrote:
Well, THAT is a great argument for some official standardization! There is sometimes a need for that sort of thing, but I think it's best handled by either putting __version__ in each sub_package, or having a different attribute altogether. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Agreed! I was trying to figure out why an API from Vaex 3.x was no longer working, and to my human eye, this quickly pointed me at the issue. However as a way to automatically check for versions, this is a mess. I have no idea what additional keys the next version might add or remove, for example. I'm not even sure if this is dynamically determined based on optional components being installed or not. Providing this kind of information *somehow* feels like a useful thing to do. But .__version__ is probably not the right way to do it. On Wed, Apr 14, 2021 at 5:08 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, Apr 14, 2021 at 5:24 AM Victor Stinner <vstinner@python.org> wrote:
Yes, though it looks a little too heavyweight to be required, nifty features like ``is_prerelease` is going pretty far. On the other hand, maybe no harm in all that, as long as: - It's easy to do the easy stuff (which is looks like it is) - It would be added to the stdlib However, there is a lot of code out there that's already using strings for __version__, so it would be a real shame if we had to ask everyone to upgrade. And Version isn't going to be in older versions of the Python stdlib, so hard to make code compatible. But: If we make a Version object that can be compared to (confoming) strings then there might be a path forward.
In my Python projects, I like to provide the version as a tuple which can be used directly for comparison: version_a <= version_b. Example:
Yes, and tuples are already built in to Python, so an easier lift than adding the Version object. Another possible issue: using Version would require an extra import in many module initializations -- is that a performance issue that would matter? In any case, I think the PEP should specify a standard for what is in __version__ And ideally it would be backward compatible with strings, which are the most commonly used currently. And conform to PEP 440 Sadly, such tuple is no standardized. Which
part is the major version? How to format it as a string?
Good luck with trying to standardize that ;-)
well, PEP 440 gets us close -- and I *think* compliant strings could be unambiguously converted to-from tuples. Anyway, the question now for me is whether this is worth any more of my time. So: - Is there someone willing to sponsor? - Do folks generally think there's a chance of it being accepted without a huge debate and expansion of scope. If the answers are not yes, I can better spend my Python time of other things. BTW: one source of hope: __version__ strings are pretty common, but, as evidenced already by this brief discussion, it's not by any means a universal standard -- so we really can't break anything that already works in any universal way. What i mean by that is that anything done here might break someone's special use case of __version__ (Like the one David Mertz identified), but that scheme only works with that one package's code anyway. There are no general tools that expect that scheme. And it could be preserved by making a subclass of Version (Or str) that was also a mapping :-) - CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Apr 14, 2021 at 5:44 PM Christopher Barker <pythonchb@gmail.com> wrote:
I like having a `Version` object that is easily importable in the standard library. I think it should relatively polymorphic. I.e. it should accept a module or package as an argument, but also should accept a string or a tuple. Maybe other objects from which one could reasonably extract a version. In particular, I think initializing this object with a module object should at least look for a .__version__ attribute, and appropriately case either a string (that looks sufficiently version-like) or a tuple. I think that if it doesn't succeed, it should become some sort of "cannot determine" object that is neither less than nor greater than any other Version object. In particular, the Vaex example with a dictionary of versions of each component should probably just become this "cannot determine" value (but as an instance of Version). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, 14 Apr 2021 at 18:01, David Mertz <mertz@gnosis.cx> wrote:
If it's not basically equivalent to packaging.version.Version (and based on PEP 440) then we'll be creating a nightmare of confusion, because PEP 440 versions are fundamental to packaging.
In particular, I think initializing this object with a module object should at least look for a .__version__ attribute, and appropriately case either a string (that looks sufficiently version-like) or a tuple. I think that if it doesn't succeed, it should become some sort of "cannot determine" object that is neither less than nor greater than any other Version object. In particular, the Vaex example with a dictionary of versions of each component should probably just become this "cannot determine" value (but as an instance of Version).
What's wrong with Version(module.__version__)? And if the __version__ attribute isn't a valid version, raise an exception? That's what packaging.version does, and it's worked fine for the packaging ecosystem. Is there a benefit that justifies being different here? Can I remind people that the packaging community have done a *huge* amount of work designing and standardising concepts like version identifiers, version comparisons and constraints, etc. (PEPs 440 and 508). Understanding those standards should really be the basic starting point for any discussion like this. Paul
On Wed, Apr 14, 2021 at 9:12 PM Paul Moore <p.f.moore@gmail.com> wrote:
Are you suggesting that users should have to install an external module to tell what version of packages they are using?! What's wrong with Version(module.__version__)? And if the __version__
Doesn't that seem really, really painful to use in an interactive shell? This is honestly (a simplified version of) what I tried in trying to learn packaging.version.Version. It was not a good experience: % conda create -n test python=3.9 pandas [...] % conda activate test % python
I cannot find a case where the more verbose spelling is ever nicer than just referencing `mod.__version__` directly for such exploration. Yes, I get that formal packaging has different needs. But here it just seems like a lot more work to get a lot less information back. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Wed, 14 Apr 2021 at 21:59, David Mertz <mertz@gnosis.cx> wrote:
No. To tell what version of a package they are using, a string is sufficient. They only need a version object if they want to do additional processing (like comparing versions, or checking whether a version meets a constraint). Given that the packaging ecosystem already has a canonical version object (provided by the packaging library), which has been used and tested extensively in many environments, inventing a different API seems at best ill-advised. Whether the stdlib needs a version object. rather than leaving that functionality to a 3rd party library, is the same question that comes up for *any* functionality that's proposed for the stdlib, and I have no particular opinion in this case.
It's designed for programmatic use, not interactive use, yes. But that's sort of my point, why do you want anything more than the bare string in the REPL? What are you planning on doing with it? Paul
On Wed, 14 Apr 2021 22:23:44 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
Tangentially, until now projects could use distutils's LooseVersion if they wanted to compare version numbers reliably. With distutils being deprecated, they'll have to either depending on packaging (which is a large dependency just for comparison version numbers) or vendor packaging's Version class (which is doable but still some bothersome additional work). Regards Antoine.
Paul Bryan:
Seems like this is something that should make its way into stdlib?
In the last 10 years, the trend is more to remove anything related to packaging *outside* the stdlib :-) Since it's evolving way faster than Python releases and the stdlib cannot be updated once installed, I don't think that it's a good idea. On Thu, Apr 15, 2021 at 12:22 AM Antoine Pitrou <antoine@python.org> wrote:
If packaging is too big and if packaging maintainters like the idea, maybe packaging.version could be moved into a dedicated package? I didn't look if it makes sense from a technical point of view. $ wc -l packaging/version.py packaging/_structures.py 556 packaging/version.py 86 packaging/_structures.py 642 total version.py uses _structures.py (InfinityType, NegativeInfinityType). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 15 Apr 2021 11:37:28 +0200 Victor Stinner <vstinner@python.org> wrote:
What is "evolving way faster"? packaging.version implements PEP 440, which has been approved in 2014. Recent changes have been really minor: https://github.com/pypa/packaging/commits/main/packaging/version.py
On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner <vstinner@python.org> wrote:
Well, maybe all of us don't think that's a good idea ;-) But anyway, I would say removing anything *related* to packaging outside the stdlib is a bad idea -- requiring an external tool to build a package is OK, but requireing an external use packages, not so much. Presumably that's why importlib.metadata exists in the stdlib. Version is arguably useful from the package user side. As I believe Victor mentioned, there are two uses for version information: display to the user -- for which version strings are fine, or programmatic comparison -- for which something like the Version object is very helpful. Do we only need to use version information programmatically when we are creating (or installing) packages? I don't think so -- I know I have code that (poorly) does version checking programmatically. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 4:55 PM Christopher Barker <pythonchb@gmail.com> wrote:
Presumably that's why importlib.metadata exists in the stdlib.
I was so hopeful about this, but in the end... not really. I have not used this capability before. Here are a few different situations I know of: pandas 1.2.4 vaex {'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0', 'vaex-hdf5': '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0', 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.11.1'} bs4 4.9.3 pandas 1.2.4 vaex 4.1.0 bs4 PackageNotFoundError('bs4') It seems like (somehow or another) importlib.metadata is doing something perhaps more useful for vaex. But it is doing distinctly worse for re, statistics, and bs4. Version is arguably useful from the package user side. As I believe Victor
I think the key thing that Chris and I are pointing to is that there is a definite use for versions that is NOT for package maintainers/creators. Interactive use is definitely one such case, and eyeballing something, even if it is the oddball use that Vaex has, can help with figuring out an issue. But in actual code, I relatively often do something like this. Or rather, the below is what I would find really nice to be able to do. ver = robust_version(module) if ver >= (5, 2, 1): doit_modern_style() elif ver < (5, 2, 1): doit_old_style else: doit_unversioned_style() I'd like code like that to ALWAYS succeed. No exceptions raised. And it would *usually* use reasonable heuristics to figure out some sort of structured version info as well as is possible. I.e. probably look in several places and return some custom object that represents a "best effort." That object might be like NaN in being neither less than nor more than other things, in the fallback case. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Apr 15, 2021 at 9:36 AM David Mertz <mertz@gnosis.cx> wrote:
I don't see how importlib.metadata plays into your code sample since you didn't import it to use it?
And is installing a dependency that much of a burden for that code?
All doable with importlib.metadata and 'packaging'. -Brett
On 4/15/2021 12:35 PM, David Mertz wrote:
re 2.2.1
re.__version__ was last modified 2001-10-07 by F. Lundh in bec95b9d8825b39cff46a8c645fa0eeb8409854e. What does '2.2.1' mean? Only Fredrik knows. The commit message says that he rewrote the pattern.sub and pattern.subn methods in C. That should have been user invisible except for speed. I believe that it is dead data left only for back compatibility and because of an inability to give a deprecation warning.
re PackageNotFoundError('re')
It seems like (somehow or another) importlib.metadata is doing something perhaps more useful for vaex. But it is doing distinctly worse for re,
Is an effectively meaningless and misleading value better than none? IDLE once had a version last set, I believe, in 2.5 or 2.6, by K. Kaiser, with whatever meaning it had to him. I removed it in 3.6 (allowed by PEP 434 + RM decision). No one complained. The real version of any stdlib module is the Python version it is released with, plus and external app versions it is associated with (sqlite and ssl, for instance). It has been suggested, if not decided, that other versions should go. -- Terry Jan Reedy
On Thu, Apr 15, 2021 at 9:36 AM David Mertz <mertz@gnosis.cx> wrote:
I was so hopeful about this, but in the end... not really. I have not used this capability before. Here are a few different situations I know of:
... ... re PackageNotFoundError('re') statistics PackageNotFoundError('statistics') pandas 1.2.4 vaex 4.1.0 bs4 PackageNotFoundError('bs4')
funny you should try bs4, which I also used as an example in another post. But what you have found is that there is a difference between a pacakge (which you can import) and a "distribution" which is something you install. importlib.metadata is looking for distributions. re and statistics are not distributions, but rather, built in pacakges (modules). bs4 is not a distributiuion, it is a package that is provided by the "beautifulsoup4" distribution. In [3]: importlib.metadata.version("beautifulsoup4") Out[3]: '4.9.3' Frankly, I'm still a bit confused about the distiction, but I do know for sure that a single distribution (maybe somethign that you can install via PyPi) can install more than one top-level package -- and certainly the packages/modules installed can have a different name than the distribution name. And if a distribution installs more than one package, they may have different version numbers. I'm not sure what to make of all this, though I'm leaning toward better suporting the distiction by asking for __version__ strings in top-level packages -- and maybe making importlib.metadata.version a bit smarter about looking for packages, and not just distributions. If you look at the docstring of metadata.version: """ In [2]: importlib.metadata.version? Signature: importlib.metadata.version(distribution_name) Docstring: Get the version string for the named package. :param distribution_name: The name of the distribution package to query. :return: The version string for the package as defined in the package's "Version" metadata key. File: ~/miniconda3/envs/py3/lib/python3.9/importlib/metadata.py Type: function """ It's a bit inconsistent with the use of the term "distribution" vs "package". That should get cleaned up if nothing else. Also, the Exception raised is "PackageNotFoundError" -- which should maybe be "DistributionNotFoundError"? Version is arguably useful from the package user side. As I believe Victor mentioned, there are two uses for version information: display to the user -- for which version strings are fine, or programmatic comparison -- for which something like the Version object is very helpful. Do we only need to use version information programmatically when we are creating (or installing) packages? I don't think so -- I know I have code that (poorly) does version checking programmatically.
Exactly -- and I htink we are close, if pacakges(modules) had compliant __version__ strings, thenyou could do (with the Version object from packaging -- maybe why it should be in the stdlib) ver = Version(module.__version__) if ver >= Version("5.2.1"): doit_modern_style() elif ver < Version("5.2.1"): doit_old_style else: doit_unversioned_style() And if my PR is accepted (which looks unlikley) to allow camparison between Version objects and strings: ver = Version(module.__version__) if ver >= "5.2.1": doit_modern_style() elif ver < "5.2.1": doit_old_style else: doit_unversioned_style() A key distiction here from the importlib.metadata approach is the level of indirection -- I like being able to ask a module itself what version it is, rather than asking some other part of teh system to go look it up for me. So I could do: import numpy as np print("using np version:", np.__version) And this is pretty consitent with the rest of Python, where many objects (functions, classes, modules) have a __name__ attribute if things "know" their name, shouldn't they "know" their version as well? -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 9:54 PM Christopher Barker <pythonchb@gmail.com> wrote:
This wouldn't be bad. A little more verbose, but fine. The thing is, I almost never care about the version of a *distribution* in my programs. I care about a version of the *package*. Since, as Chris mentions, a distribution could contain multiple packages, the same package could be vendorized (in different versions) by different distributions. The question I want to ask above is always because "version such-and-such changed an API," and that's about a package. When I care about distribution versions, it is because there are dependencies stated in requirements.txt or environment.yaml or whatever.deb. Asking for the `.__version__` attributes feels like needless redundancy in Chris' variation. The constructor should be able to ask "Is this a module? If so, does it have that attribute". But whatever, I can learn to do that easily enough. Someone else mentioned that the versions of standard library modules are an odd collection of relics. I do not disagree, of course. But sometimes the stdlib modules *do* change. I thought of the `re` example because I was doing a tutorial on it, and something or another depended on a change between 3.8 and 3.9 (I think, I forget the detail). For training material, this can just be approached as describing the Python version... but for running code, a version check could be relevant. Therefore, I think this `Version` constructor (or `robust_version()` function, or whatever) should have access to that. I.e. if not an automated update to `.__version__` inside each stdlib module during builds, then a fallback to "If this is a stdlib module, report sys.version_info (within the right kind of "Version" object). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Apr 15, 2021 at 8:50 AM Christopher Barker <pythonchb@gmail.com> wrote:
As an active participant in the PyPA side of the packaging ecosystem and one of the maintainers of 'packaging', I am definitely NOT in favour of moving any of 'packaging' into the stdlib, nor breaking it up. We are taking distutils out of the stdlib for a reason. Backsliding on plans that have been in place for years is not something I would want to see happen as the motivations have not changed.
You can totally use a package already as things exist today and will in the future; you're just wanting a specific way to interpret a package's metadata which I don't view as the same thing.
Presumably that's why importlib.metadata exists in the stdlib.
It's there because of how tightly it binds to the import system; same goes for importlib.resources.
No one said that's the *only* use-case, but just because there's more than that does not mean we should move any packaging code into the stdlib to support a zero-install use-case. A key point I want to make here is the version metadata is in every project's .dist-info/METADATA file. That's what importlib.metadata knows how to read and hence why people have been suggesting to use it. The version information is fundamental to the wheel format and in fact you can't have a wheel without it. So asking every Python project to set a __version__ isn't going to change accessibility of version numbers when it comes to installed projects.
On Thu, 15 Apr 2021 10:28:53 -0700 Brett Cannon <brett@python.org> wrote:
This seems gratuitously dogmatic. Version numbers are a simple feature that is stably encoded in PEP 440. It's purely computational, does not involve reading or saving any persistent state in the filesystem, making any network access, etc. Why wouldn't it have its place in the stdlib?
That specific way is a Python standard (PEP 440). Having the functionality in the stdlib would encourage people to use it. Not having it in the stdlib encourages people to use adhoc version parsing, or worse, naive string comparison.
That doesn't have much to do with the suggestion of shipping a Version class in the stdlib, though. Many projects already provide a __version__, and that field is often inspected programmatically in dependent packages (to work around behaviour changes, for example). Regards Antoine.
On Thu, Apr 15, 2021 at 10:53 AM Antoine Pitrou <antoine@python.org> wrote:
Technically PyPA specs might start as PEPs but their official home is packaging.python.org and updates can occur there outside of the PEP process. In the case of version specifiers ( https://packaging.python.org/specifications/version-specifiers/), they point back to the original PEP, but that is not universally true nor guaranteed to hold in the future. In other words I personally don't view the fact that it's a PEP as a reason to have it in the stdlib (else pretty much every single packaging PEP is missing). We also have other things standardized in PEPs and not covered in the stdlib, e.g. @ as an operator. So I don't think version comparison occurs often enough to be in the stdlib, and the fact that an external project exists which isn't interested in being put in the stdlib suggests to me it isn't worth doing. But that's a separate topic to discuss at the language summit. :)
I don't know if I draw the same line since the packaging community is relying on 'packaging' without issue for this exact thing.
But this thread is about standardizing on __version__ which then led to wanting to bring in some Version object into the stdlib to make that more useful. So to me they are tied together, else a separate thread should probably be started if a new module *just *for version parsing support is desired.
On Thu, 15 Apr 2021 16:12:57 -0700 Brett Cannon <brett@python.org> wrote:
Ah, I should submit a question then.
Right, but the functionality is useful generally, not only for the packaging generally.
You're right, this thread was hijacked somewhat for the Version object discussion. Regards Antoine.
On Wed, Apr 14, 2021 at 2:23 PM Paul Moore <p.f.moore@gmail.com> wrote:
exactly -- if we do anything at all here, it should probably be at least. if you provide a __version__ attribute, it should be a PEP 440 compliant string. That would at least make the use __version__ a touch more consistent. The next, more controversial, step would be to suggest that people SHOULD provide a __version__string on all top-level packages. I would like that -- the current situation where many, but not all packages have __version__ is really annoying. They only need a version object if they want to do additional
processing (like comparing versions, or checking whether a version meets a constraint).
And indeed, we could add the ability for packaging.version.Version objects to be able to compare themselves with compatible strings -- I think that would be pretty handy. Given that the packaging ecosystem already has a canonical version
absolutely.
I don't think it's the same as any functionality -- if we do want to better standardize package versioning in Python, and I think we should, then the Version object may become something useful to, and maybe even needed by, virtually every third party package. Which makes it a pretty prime candidate for the stdlib. Alternatively, the packaging package is pretty small, but if it grows, it might be good to split out the run-time vs build-time pieces. It's designed for programmatic use, not interactive use, yes. But
that's sort of my point, why do you want anything more than the bare string in the REPL? What are you planning on doing with it?
there is something in between the REPL and full on system development -- something simple for quickly scripts is nice too. But a simple standardised version string is fine for that. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Paul Moore wrote:
What's wrong with Version(module.__version__)? And if the __version__ attribute isn't a valid version, raise an exception?
I don't have a deep answer, but I do think __version__ should be specified (or at least mentioned) at https://docs.python.org/3/reference/datamodel.html At the moment, I can't even find a listing of possible __dunder__ attributes, though I'm almost sure I've seen one in the past. -jJ
https://docs.python.org/3/reference/datamodel.html On Wed, 2021-04-14 at 22:56 +0000, Jim J. Jewett wrote:
On Wed, Apr 14, 2021 at 7:04 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:
Given the intent to reject PEP 394, I can't imagine any good would come of that. Drop a __version__ in your modules if you find it valuable, but otherwise, there's nothing to do here. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
On 4/14/2021 12:38 PM, Christopher Barker wrote:
I think that before these can be answered, you need to decide why this needs to be standardized. I don't see any reason to standardize it unless there's some programmatic use for these version strings. If it's just the user poking around on the REPL, then I think the status quo is fine. I've read the PEP, and the User Stories section talks about user convenience, not programmatic access. I also think the distribution version is more useful than any __version__ information in each module/package. I realize this information might not be available, depending on how the code was installed (say, just by copying some files into the right place). Eric
On Wed, 14 Apr 2021 at 18:04, Eric V. Smith <eric@trueblade.com> wrote:
Agreed. The original PEP was very limited, just proposing a string value for user convenience, with no programmatic interface. Doing anything more than that is opening up a whole load of complexity and compatibility (with packaging) that no-one has expressed any need for. The distribution version (exposed via importlib.metadata) is the correct thing to use for anything programmatic. Paul PS I see Barry plans on rejecting the PEP, which I think is probably the right decision, given the way this thread has developed.
On Wed, Apr 14, 2021 at 4:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
PS I see Barry plans on rejecting the PEP, which I think is probably the right decision, given the way this thread has developed.
Barry makes good plans. Putting the version into the sources is a bit of an anti-pattern. IMO. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "There is nothing more uncommon than common sense." --Frank Lloyd Wright
On Apr 13, 2021, at 22:40, Christopher Barker <pythonchb@gmail.com> wrote:
Turns out this was suggested in PEP 396 -- and deferred almost 13 years ago!
I’d forgotten that this PEP was in Deferred state. I think it should be rejected and I plan on making that change. importlib.metadata is a much better approach to programmatically determining package versions. https://docs.python.org/3/library/importlib.metadata.html#distribution-versi... -Barry
On Wed, Apr 14, 2021 at 12:10 PM Barry Warsaw <barry@python.org> wrote: I’d forgotten that this PEP was in Deferred state. I think it should be
Barry, You wrote the original PEP, so of course you can withdraw it (or reject it), but... Are you sure? See this discussion, I don't think it's as simple as all that. Honestly, I'm still kind of confused about a "distribution" vs a package or module, but in general, it seems to me that the packaging community has done a great job of building a system that can accommodate a lot of complexity, but as they say: The easy things should be easy, and the hard things should be possible. And the full process of using packaging and distributions, and a Version object and all that is not so easy anymore --at least compared to putting: __version__ = "1.2.3" in a package __init__.py. Anyway, a couple points: 1) it seems self-evident that it would be easier for the whole community if "how to declare", and "how to find" the version of a package was standardized: "there's only one way to do it" 1b) It seems like a really good idea that the (or one of the) official recommended way to do it is supported in the standard library itself, and using a string is pretty easy to support :-) -- do we really want people to have to install a third party package to set or get the version of a simple module? 2) __version__ strings are already pretty darn common -- maybe more common than proper distribution metadata? -- are we saying that everyone should stop doing that? 3) I don't know that having __version__ strings is incompatible with packging.version.Version or importlib.metadata.version 4) Yes, there are real advantages to the more complex system for programmatically working with versions. But there are real advantages to something simple for human readability / interactive use -- and, frankly, simple scripting. See David Mertz's recent post -- I went through a similar process in iPython. Consider this analogy: if I want to know what version of git I have installed, I can run: $ git --version And that's a convention adhered to by many *nix command line tools. Would we all really be happier if we had to go use an rpm or deb or homebrew, or ?? command to see what package i have installed? I think this is similar -- while a good distribution management system is a good thing, it's also good if I can tell something about the version of a package/module from the package itself. With regard to having both __version__ and importlib.metadata.version, see beautiful soup for example: In [10]: import beautifulsoup4 ... ModuleNotFoundError: No module named 'beautifulsoup4' oops, that's not what the importable module is called. So: In [11]: import bs4 That worked -- now, what version is it? In [12]: importlib.metadata.version('bs4') --------------------------------------------------------------------------- PackageNotFoundError Traceback (most recent call last) ... PackageNotFoundError: bs4 [*] That didn't work -- I guess I need to use the distribution name: In [13]: importlib.metadata.version('beautifulsoup4') Out[13]: '4.9.3' OK, that worked. But I imported "bs4", so what if I do this? In [14]: bs4.__version__ Out[14]: '4.9.3' Ah yes, that worked too. So I guess what I'm suggesting is that we officially recommend that folks do what the BeautifulSoup folks, and many others, are already doing. In many (most) cases, the distribution name and the importable package name are the same, but this does accommodate both. [*] One other annoyance to importlib.metadata.version is that in addition to having to import importlib to use it, I also need to use importlib.metadata.PackageNotFoundError if I want to trap the exception. BTW: I'm not making that last point up -- I recently updated some in-house code (that someone else had written) in a web app that presented the version of various packages core to the system. We had just refactor and one of these packages what now optional, but the whole system was crashing on that darn PackageNotFoundError. I ended up jsut commenting out that package, rather than dealing with how to catch the exception and deal with it properly. I would have so much rathered a simple: try: import the_package the_pacakge_version = the_package.__version__ except the_package_version = None Granted, it's not that hard to "do it right", but I still like things simple :-) Anyway, as the original PEP author is no longer supportive, this is dead in the water unless another core dev is interested in sponsoring this (or a new) PEP. If not, I can stop wasting my and everyone else's time. -Chris -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Apr 14, 2021, at 23:11, Christopher Barker <pythonchb@gmail.com> wrote:
You wrote the original PEP, so of course you can withdraw it (or reject it), but...
Are you sure? See this discussion, I don't think it's as simple as all that.
From a library maintainers point of view, I personally want to get away from using __version__ strings at all. They’re kind of a pain to remember to bump on every new release. PEP 396 has been dormant for a long time and I have no interest in pushing it forward any more. If someone else wants to take up the cause, I recommend creating a new PEP and referring back to 396. -Barry
On 4/15/2021 12:38 PM, Barry Warsaw wrote: that.
From a library maintainers point of view, I personally want to get away from using __version__ strings at all. They’re kind of a pain to remember to bump on every new release.
At least some in the stdlib have not been. For re.__version__, not since Nov 2001. -- Terry Jan Reedy
On Thu, Apr 15, 2021 at 12:27 PM Terry Reedy <tjreedy@udel.edu> wrote:
At least some in the stdlib have not been. For re.__version__, not since Nov 2001.
From Serhiy over on python-ideas, he's started an effort to clean that up. https://mail.python.org/archives/list/python-dev@python.org/thread/KBU4EU2JU... ). I believe there is an unfinished PR -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Apr 15, 2021 at 9:38 AM Barry Warsaw <barry@python.org> wrote:
PEP 396 has been dormant for a long time and I have no interest in
That’s a tooling issue, and I don’t think that Python itself should take any official stance on the tooling. But yes, anything that Python takes an official stance on should be able to be easily supported with not-too-complex tools.and I think a __version__ string fits this need. Personally, I use the __version__ string in my packages as the canonical source when I bold my packages. And there are fancier tools for more complex needs. pushing it forward any more. If someone else wants to take up the cause, I recommend creating a new PEP and referring back to 396. Fair enough -- I'm still gathering data to see if I should do just that. I am leaning toward at least SOME PEP about this -- having __version__ around as a semi-convention with no official recommendation as to if or how it should be used is kind of a mess. -CHB
On Thu, 15 Apr 2021 at 21:03, Christopher Barker <pythonchb@gmail.com> wrote:
I am leaning toward at least SOME PEP about this -- having __version__ around as a semi-convention with no official recommendation as to if or how it should be used is kind of a mess.
On the other hand, having a PEP that says you "may" have a version attribute, but mandates nothing, is pretty useless. People who conform to that convention don't need telling, people who don't have little reason to change, and people wanting to use the information still have to cater for it being missing, so they gain nothing. But a PEP that makes __version__ a requirement fails instantly, because there's too many packages in the wild that violate it. Welcome to the world of packaging PEPs :-)
I'm not sure what to make of all this, though I'm leaning toward better suporting the distiction by asking for __version__ strings in top-level packages -- and maybe making importlib.metadata.version a bit smarter about looking for packages, and not just distributions.
Personally, I'm leaning more and more towards the view that it's *distributions* that are versioned, and that's an established practice. Versioning packages as well (except in the very limited sense that package X installed by distribution Y can expose Y's version via its API) only adds confusion and makes the situation less, not more, useful for users. So I'm now -1 on any PEP to try to formalise the use of __version__. If people want to discuss the use of a __version__ attribute, they should consider contributing to the discussions on versions in https://packaging.python.org/guides/ (where the necessary context is already present). Paul
Hi, sorry for being late to the party, but I may not be the only one wondering… Le 14/04/2021 à 20:56, Barry Warsaw a écrit :
I’d forgotten that this PEP was in Deferred state. I think it should be rejected and I plan on making that change. importlib.metadata is a much better approach to programmatically determining package versions.
https://docs.python.org/3/library/importlib.metadata.html#distribution-versi...
This is indeed the correct approach, thanks for letting me learn this. However, I unsuccessfully searched for the canonical way to look up the distribution name based on either a module name or an imported module object. Is there one? Looks like it could be computed based on information in "*.egg-info/installed-files.txt", but it's far from trivial. Is "installed-files.txt" even guaranteed to exist for all distributions? Cheers, Baptiste
On Mon, Apr 26, 2021 at 9:37 AM Baptiste Carvello < devel2021@baptiste-carvello.net> wrote:
If you mean how to tie a module back to its name on PyPI, you should be able to look up the "Name" in the project's metadata: https://docs.python.org/3/library/importlib.metadata.html#distribution-metad... . -Brett
On Mon, Apr 26, 2021 at 10:31 PM Brett Cannon <brett@python.org> wrote:
The missing bit here, for me, is how do you map a module back to it's project (distribution)? For example: ```
This is because the distribution calls itself 'beautifulsoup4' instead.
The same goes for another package: `umap`, for which the distribution is
called `umap-learn`
This is the best I could come up with from reading the docs:
import bs4 #<- This is the module we want the version of
import importlib
import sys
from itertools import chain
from pathlib import Path
loaders = sys.meta_path
target_path = Path(bs4.__file__)
distros = list(chain(*(finder.find_distributions() for finder in
loaders if hasattr(finder, 'find_distributions'))))
distros_files = chain(*(f for f in (d.files for d in distros)))
distro_files = [(d, d.locate_file(f)) for d in distros if d.files for f
in d.files]
matching = [d for d, f in distro_files if f == target_path]
for match in matching:
print("Found Version:", match.version)
Steve
On Mon, Apr 26, 2021 at 3:25 PM Stestagg <stestagg@gmail.com> wrote:
Unfortunately I thought importlib.metadata would have used the module name instead of the metadata details, but in hindsight am guessing that the .dist-info is what it's using to do the lookup and that's based on the package name instead of the project name. This is a long-standing issue with projects that use project names which differ from their module name, but there's no good way without checking what files a project installed (which is what you're doing below). -Brett
On Tue, 27 Apr 2021 at 18:01, Brett Cannon <brett@python.org> wrote:
Unfortunately I thought importlib.metadata would have used the module name instead of the metadata details, but in hindsight am guessing that the .dist-info is what it's using to do the lookup and that's based on the package name instead of the project name.
This is a long-standing issue with projects that use project names which differ from their module name, but there's no good way without checking what files a project installed (which is what you're doing below).
That's correct. There is no link from a given import name back to the PyPI project name. And the metadata for installed packages (defined here: https://packaging.python.org/specifications/recording-installed-packages/) is keyed on the PyPI project name, as the OP noted ("beautifulsoup4" rather than "bs4", "setuptools" rather than "pkg_resources", etc).
The following is a bit simpler. The file.locate() method of a distribution is undocumented - but I tried to use resolve() on the path object I got from dist.files, and it appears not to be implemented (at least in Python 3.9) - PackagePath objects are a subclass of *Pure* Path objects, not concrete paths :-( TBH, I suspect the fact that it's undocumented is an oversight, it's clearly deliberately added. import importlib.metadata from pathlib import Path import pkg_resources target = Path(pkg_resources.__file__) for dist in importlib.metadata.distributions(): for file in dist.files: path = file.locate() if path == target: print(f"{dist.metadata['name']}: {dist.version}") break To be honest, if anyone were interested in making a PEP from any of this, having Python modules contain a __distribution_name__ attribute that links the module back to the PyPI distribution, would probably be more useful than standardising __version__. But I'm not sufficiently interested to do anything more than mention that as a possibility. Paul
Replying to the original email because my comment is more a thought after reading the whole thread rather than a response to any particular message. What I love about __version__ is that it is simple. It doesn't come with the complexity of solutions like `importlib.metadata.version`. The maintainers of the project arranged for it to be set**. It doesn't depend on how the module was packaged or distributed. I also don't think it's really a language update or requires any changes in the stdlib -- it's just a standard for the community to follow where none currently exists. It's also probably far from the end of the world*** if it doesn't happen, but I still like the idea. ** It might have been updated somewhat automatically -- my point here is that distutils, setuptools, packaging, pip, wheels, PyPI were all not involved. *** Although that is likely true of most PEPs, even much more complicated ones. :) Schiavo, Simon
Replying to myself to update my own thoughts: I think __version__ and packaging tools answer separate questions. __version__ answers the question of "this module I imported, what version is it". The packaging tools answer the question "what dependency does this distribution satisfy". This mismatch is why importlib.metadata.version("bs4") doesn't return a useful answer. A single distribution might even install multiple top-level modules with different versions. Schiavo, Simon
participants (16)
-
Antoine Pitrou -
Baptiste Carvello -
Barney Gale -
Barry Warsaw -
Brett Cannon -
Christopher Barker -
David Mertz -
Eric V. Smith -
Fred Drake -
Jim J. Jewett -
Paul Bryan -
Paul Moore -
Simon Cross -
Stestagg -
Terry Reedy -
Victor Stinner