Donald Stufft schreef op 17-12-14 01:05:

...

On Dec 16, 2014, at 6:53 PM, Maurits van Rees wrote:

<words>

Yea, it’s unfortunate that local versions don’t work prior to setuptools 8.0, but the older versions more or less escape everything that’s not alpha numeric and . and - to the - character so there’s only so much we can do with that.

It’s correct that existing versions of pip do not support any new syntax with PEP 440, that will be part of pip 6.0 which I’m hoping to release this week. We were waiting on setuptools 8 to release with PEP 440 support and now we’re just making sure that setuptools is calmed down before releasing a pip 6.0.

I agree with Paul, please file a bug for the local versions not working, they _should_ work and it should be a release blocker for 6.0.

Done: https://github.com/pypa/pip/issues/2203 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl

Donald Stufft schreef op 17-12-14 01:54:

...

On Dec 16, 2014, at 7:46 PM, Maurits van Rees wrote:

Maurits van Rees schreef op 17-12-14 00:53:

...

I have created a very basic python project called 'myproject'. It does nothing. I have released a few versions here: http://pypi.zestsoftware.nl/public/packagingtest/

I have now also distributed myproject version 1.1. (This has a base.jinja2 file and requires Jinja2[i18n], which I need for checking a corner case; I may report that later).

Installing 1.1 unexpectedly gives problems, both with pip and zc.buildout.

- zc.buildout 2.2.5 and setuptools 7.0: all is well.

- zc.buildout 2.3.1 and setuptools 8.0.4:

* When I specify 'myproject = 1.1' in the buildout config, myproject does not get updated. It sticks at version 1.1+maurits.3.

This isn’t a bug, assuming that buildout is translating ``my project = 1.1`` to something like ==1.1.

That is indeed what buildout does. Note that buildout config files are basically parsed by ConfigParser, so '==' is not allowed there. That is why buildout uses a different notation here.

Local versions are used to indicate something that is compatible with 1.1 and they sort as newer than the same version without the local version. The reason they match for a ``==1.1`` is because you want downstream distributors like Debian to be able to set their versions to 1.0+debian.1 without breaking things for people who do ``==1.1``.

That is most definitely not what I want. If I tell setuptools or pip or buildout that I want version X, then I don't want roughly version X, but I want exactly version X. When I request 1.0rc1 and I get 1.0, that is wrong. When I request 1.0 and I get 1.0rc1, that is wrong. To me, the same is true for local version identifiers. To me, this is the basis for repeatable deployments, knowing that given a buildout file or a pip requirements file you get the same versions and the same code that you got when you tried it a month earlier. It avoids "it works on my machine" when I get an error that a colleague does not get. For clarity, how I use internal releases, so how I would want to use local version identifiers, is: 1. Use largetrout 1.0 in a project. 2. Notice a bug. 3. See that the bug is already fixed in development, but there is no release yet. 4. Checkout largetrout dev, setup.py version is 1.1.dev0. 5. Set version to 1.1+internal1 (an internal release of a snapshot of the code that is expected to end up in 1.1). 6. Make an internal release and use it. 7. Five more bugs get fixed in largetrout. 8. largetrout 1.1 is released. Now when I tell pip or buildout to use largetrout 1.1 I definitely want those five bug fixes from step 7 included.

...

- pip 1.5.6, setuptools 7.0:

* pip warns about three different versions: $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1 Downloading/unpacking myproject==1.1 Downloading myproject-1.1+maurits.3.zip Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject Requested myproject==1.1, but installing version 1.1-maurits.3 * The warning is correct: not 1.1, but 1.1-maurits.3 is installed.

That warning might actually be from a stale build directory laying around and not related to the local version at all.

Let me check. Nope, the same happens with a fresh virtualenv: mauritsvanrees@procyon:tmp $ virtualenv-2.7 venv-older Using real prefix '/Users/mauritsvanrees/buildout/python2.7/parts/opt' New python executable in venv-older/bin/python2.7 Also creating executable in venv-older/bin/python Installing setuptools, pip...done. mauritsvanrees@procyon:tmp $ cd venv-older mauritsvanrees@procyon:venv-older $ . bin/activate (venv-older)mauritsvanrees@procyon:venv-older $ pip install -U setuptools==7.0 # this is optional Downloading/unpacking setuptools==7.0 Downloading setuptools-7.0-py2.py3-none-any.whl (534kB): 534kB downloaded Installing collected packages: setuptools Found existing installation: setuptools 3.6 Uninstalling setuptools: Successfully uninstalled setuptools Successfully installed setuptools Cleaning up... (venv-older)mauritsvanrees@procyon:venv-older $ pip list pip (1.5.6) setuptools (7.0) wsgiref (0.1.2) (venv-older)mauritsvanrees@procyon:venv-older $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1 Downloading/unpacking myproject==1.1 http://pypi.zestsoftware.nl/public/packagingtest/ uses an insecure transport scheme (http). Consider using https if pypi.zestsoftware.nl has it available Downloading myproject-1.1+maurits.3.zip Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject warning: no files found matching '*rst' warning: no previously-included files matching '*.pyc' found anywhere in distribution Requested myproject==1.1, but installing version 1.1-maurits.3 Installing collected packages: myproject Running setup.py install for myproject warning: no files found matching '*rst' warning: no previously-included files matching '*.pyc' found anywhere in distribution Successfully installed myproject Cleaning up... (venv-older)mauritsvanrees@procyon:venv-older $ ls -1 lib/python2.7/site-packages/ _markerlib easy_install.py easy_install.pyc myproject myproject-1.1_maurits.3-py2.7.egg-info pip pip-1.5.6.dist-info pkg_resources.py pkg_resources.pyc setuptools setuptools-7.0.dist-info -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl

On 12/16/2014 05:49 PM, Donald Stufft wrote:

So the *primary* use case that motivated local versions is things like when Debian patches a copy of a project they can indicate that they’ve done so by changing the version to 1.0+dfsg1 or so instead of 1.0. A related use case is the one you’re talking about. However the primary use case is what influenced the fact that ==1.1 matches ==1.0+foobar.

Important to note, is that ==1.0+foobar will only install that patched version, not any 1.0 version. You can also approximate the kind of pinning you want with the === (which is really the arbitrary equality indicator, which is generally used for people who want to install a version like “dog” which we can’t parse). It’s possible that we could add some sort of a “None” indicator for local versions that says “1.0 and exactly 1.0” though I’m not sure how off the top of my head (Maybe ==1.0+).

Let me see if I understand correctly: I release my dbf package as 1.3. Somebody at Debian fixes a bug, and rather than wait for me to release the next version just slap a local tag on it -- so now there is: - 1.3 (mine) - 1.3+debian1 At this point, a non-debian user asking for 1.3 will get mine, whereas a debian user would get the 1.3+debian1 version. Correct? Plus, a debian user asking for 1.4 would also get 1.3+debian1? Now, I fix that bug plus a few more, and make a 1.4 release. A non-debian user would get my 1.4 release. Which version will a debian user get? - 1.3+debian1 ? - 1.4 ? If my 1.4 does not sort higher than a 1.3+blahblah anything, that's not good. -- ~Ethan~

On 12/16/2014 06:48 PM, Donald Stufft wrote:

Now if you have 1.3+debian1 installed via apt-get (or any means really), and you’ll get the following behaviors (show with pip):

- pip install yourthing -> 1.3+debian1 satisfies the constraint of anything, so it stays installed. - pip install yourthing==1.3 -> 1.3+debian1 satisfies the constraint of ==1.3, so it stays installed. - pip install yourthing>=1.3 -> 1.3+debian1 satisfies the constraint of >= 1.3, so it stays installed. - pip install yourthing>=1.4 -> 1.3+debian1 does not satisfy the constraint of >=1.4, so pip will upgrade it to 1.4. - pip install —upgrade youthing -> You’ve requested an upgrade, so pip will see 1.4 exists and will install it. - pip install —upgrade yourthing==1.3 -> You’ve requested an upgrade, but there’s nothing newer than 1.3+debian1 that matches the constraint so it stays installed - pip install -upgrade youthing>=1.3 -> You’ve requested an upgrade, pip will see 1.4 exists and satisfies the constraint and will install it. - pip install —upgrade yourthing>1.4 -> you’ve requested an upgrade and 1.3+debian1 is not >= 1.4, so pip will see 1.4 exists and install it.

Does that make sense?

Yes. Thank you for that _very_ thorough explanation. :) To continue with Maurits' use-case, in order to get /exactly/ 1.3, '===' is the operator to use? Or are we still discussing that? Personally, I think pip install yourthing is 1.3 ;) -- ~Ethan~

On 17 December 2014 at 13:31, Ethan Furman wrote:

Oh, to be clear: There are no guarantees that 1.4 actually includes the bug-fixes in +debian1, correct? It's just a big hope?

Correct. The local versions were mostly added such that Linux distros could indicate to Python specific tools when patches had been applied using the distro packaging system. This is actually a pretty normal practice (especially in a long lived distro like RHEL/CentOS - our version numbers indicate the *baseline* version of a package), but it's currently completely invisible to the Python level tooling (a patched distro provided package will currently report the same Python level version metadata as a pristine upstream package). The promise we make is that any such patches shouldn't break compatibility at all (that's the whole point of the baseline+backports long term support model), which is why the comparison semantics are defined to ignore them by default. That lets us incorporate security fixes (for example) without breaking anything, while still providing an indication that something has changed in the software. Whether those patches make their way upstream will depend on the patch and the upstream project. Some distro integration patches (like pip dependency unbundling) will never go upstream. Others will be backports of bug fixes from later upstream feature releases to earlier versions that are integrated into the distro. And some will be patches that distros apply because we need them quickly for some reason, and then subsequently work through the process of getting the change (or a variant thereof) accepted upstream (because carrying downstream patches long term is expensive, we prefer not to do it if we don't have to). So while the usage model Maurits describes here is valid (and should work with setuptools 8 + pip 6), it isn't really the primary intended use case - it's aimed at when you're installing Python packages but using something other than the Python specific tooling to do it (e.g. apt-get, yum, conda, etc). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Donald Stufft schreef op 17-12-14 04:59:

...

On Dec 16, 2014, at 10:25 PM, Ethan Furman wrote: To continue with Maurits' use-case, in order to get /exactly/ 1.3, '===' is the operator to use? Or are we still discussing that? Personally, I think

pip install yourthing is 1.3

Currently the === operator will give you that, but it’s not really designed for that. It’s designed as an escape hatch for versions we can’t parse. An example of where this matters is that ==1.3 will match version 1.3.0, but ===1.3 will not. So if we want something better than that for “give me exactly 1.3 without any patches” then that’s under discussion.

Current pip 1.5.6 does not recognize the === operator and gives a ValueError. That is expected. pip dev does recognize it, but I do not see the behaviour you say. Let me try from scratch: $ virtualenv-2.7 venv-newest $ cd venv-newest $ . bin/activate $ pip install -U setuptools # gets 8.0.4 $ pip install https://github.com/pypa/pip/tarball/develop#egg=pip-dev $ pip install -U --trusted-host pypi.zestsoftware.nl -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject===1.1 Collecting myproject===1.1 Downloading myproject-1.1+maurits.3.zip Requested myproject===1.1, but installing version 1.1+maurits.3 Installing collected packages: myproject Running setup.py install for myproject Successfully installed myproject $ ls -1 lib/python2.7/site-packages _markerlib easy_install.py easy_install.pyc myproject myproject-1.1+maurits.3-py2.7.egg-info pip pip-1.5.6.dist-info pip-6.0.dev1-py2.7.egg-info pkg_resources.py pkg_resources.pyc setuptools setuptools-8.0.4.dist-info $ pip list myproject (1.1+maurits.3) pip (6.0.dev1) setuptools (8.0.4) -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl

Maurits van Rees schreef op 17-12-14 01:46:

Maurits van Rees schreef op 17-12-14 00:53:

...

I have created a very basic python project called 'myproject'. It does nothing. I have released a few versions here: http://pypi.zestsoftware.nl/public/packagingtest/

I have now also distributed myproject version 1.1. (This has a base.jinja2 file and requires Jinja2[i18n], which I need for checking a corner case; I may report that later).

I will report that now. I ran into some problems earlier with an internal release of Babel. Problem is that I use Jinja2 and this has a requirement on 'Babel>=0.8'. This gives extra problems in some cases, maybe more because entry points are used. In some cases installing works fine, but actually using the package gives an error. For clarity: I don't think the notes below indicate a bug in setuptools or pip or buildout. It just shows some possibly unexpected behaviour. Latest official Babel release is 1.3. I have created a few development releases and added them to http://pypi.zestsoftware.nl/public/ Some versions there are PEP 440 compliant, others not. Some may make sense, other not. In order of creation: - setup.py version 2.0a0.zest.20141021: Babel-2.0a0.zest.20141021.zip - setup.py version 2.0a1.zest.20141021: Babel-2.0a1.zest.20141021.zip - setup.py version 2.0a1+zest.20141216: Babel-2.0a1-zest.20141216.zip - setup.py version 2.0a1+zest.201412162: Babel-2.0a1+zest.201412162.zip Let's try them out. Babel and buildout ------------------ With the earlier given buildout, I added an extra directory to the find-links: find-links = http://pypi.zestsoftware.nl/public/packagingtest/ http://pypi.zestsoftware.nl/public/ Then I updated the Babel version in the [versions] section. - Babel = 2.0a1.zest.20141021 * Works in setuptools 7. * Does not work setuptools 8: Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8 - Babel = 2.0a1-zest.20141216 * Works in setuptools 7. * Does not work setuptools 8: Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8 - Babel = 2.0a1+zest.201412162 * Does not work in setuptools 8: ValueError: ("Expected ',' or end-of-list in", 'Babel[]==2.0a1+zest.201412162', 'at', '+zest.201412162') * Works in setuptools 8 Babel and pip ------------- - Babel==2.0a1.zest.20141021 * pip 1.5.6 works fine * pip dev fails when using it: $ ../venv-newest/bin/python setup.py extract_messages --mapping-file=extract.ini --output-file=foo.pot running extract_messages extracting messages from myproject/__init__.py extracting messages from myproject/base.jinja2 Traceback (most recent call last): File "setup.py", line 11, in <module> 'Jinja2[i18n]', File "/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/frontend.py", line 305, in run for filename, lineno, message, comments, context in extracted: File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", line 163, in extract_from_dir strip_comment_tags): File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", line 190, in extract_from_file strip_comment_tags)) File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", line 249, in extract func = entry_point.load(require=True) File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", line 2242, in load self.require(env, installer) File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", line 2256, in require items = working_set.resolve(reqs, env, installer) File "/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", line 751, in resolve raise VersionConflict(tmpl % args) pkg_resources.VersionConflict: Babel 2.0a1.zest.20141021 is installed but Babel>=0.8 is required by [] * DANGER: a version that previously worked fine and still installs fine, is broken when using it. This may be troublesome. Mostly you should just be aware of it. Basically, this wrongly named internal version number of my Babel distribution does not work nicely with setuptools 8 when you also use jinja2. * I only wonder if something needs fixing in Babel's usage of the entry point. - Babel==2.0a1-zest.20141216 * pip 1.5.6 works fine * pip dev works fine (with version warning while installing) * Note that it works because the setup.py version is a valid 440 version: 2.0a1+zest.20141216 - Babel==2.0a1+zest.20141216 * pip 1.5.6 fails to install: plus is not accepted. * pip dev fails to install. Error message is: "No distributions matching the version for Babel==2.0a1+zest.20141216". This is logical, because this distribution was created with setuptools 7 or earlier, which results in a file name that does not exactly match the version. - Babel==2.0a1+zest.201412162 * pip 1.5.6 fails to install: plus is not accepted. * pip dev fails to install. * pip dev with pull request #2205 (Expand our regex to support the new PEP 440 version characters) works fine. - pip install http://pypi.zestsoftware.nl/public/Babel-2.0a1+zest.201412162.zip * pip 1.5.6 works fine * pip dev works fine -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl