On Dec 15, 2014, at 1:41 PM, Robert Collins wrote:

+1 - I think thats a great idea. We've had to pin < setuptools 8 in the OpenStack ecosystem too.

As far as I’m aware, Openstack is just about ready to unpin setuptools. I believe the only thing holding it back is a pbr change.

It would have been nice to offer some period of time where the cases which were going to change would have warned. I realise thats non-trivial, but this is systemic infrastructure at the core of our dependency graph - everything that expresses requirements is affected.

-Rob

On 16 December 2014 at 07:27, Jim Fulton wrote:

...

I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot.

For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools <8 to give more time to respond to these changes.

Jim

-- Jim Fulton http://www.linkedin.com/in/jimfulton _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

-- Robert Collins Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Jim Fulton schreef op 15-12-14 19:51:

On Mon, Dec 15, 2014 at 1:45 PM, Donald Stufft wrote:

...

...

On Dec 15, 2014, at 1:27 PM, Jim Fulton wrote:

I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot.

For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools <8 to give more time to respond to these changes.

Possibly. Or maybe a zc.buildout 2.2.x that requires setuptools<8 or a bootstrap.py that defaults to zc.buildout 2.2.5 and setuptools 7.0. (Just some options, I haven't thought them through.)

...

Is there something I’m not aware that is broken currently? I thought the transition was going pretty smoothly overall considering that a core piece of code inside of setuptools was touched.

The problem is that version numbers and requirements aren't interpreted the same way. I think that's going to take some time to sort out. In the mean time, I'd like to try to stop the pain.

An example of some pain, although it is only a repeated warning: $ curl -O https://bootstrap.pypa.io/bootstrap-buildout.py $ python2.7 bootstrap-buildout.py Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.4.zip Extracting in TMP1 Now working in TMP1/setuptools-8.0.4 Building a Setuptools egg in TMP2 TMP2/setuptools-8.0.4-py2.7.egg TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 'zc.buildout-1.0.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. ... (dozens more warnings, two for every zc.buildout version on PyPI) ... TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. I'm not sure if that is setuptools doing funky stuff or zc.buildout doing funky stuff. Probably both. And probably they both *need* to do funky stuff. ;-)

I think you've been very responsive and that's much appreciated.

+1 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl

On Dec 15, 2014, at 6:46 PM, Maurits van Rees wrote:

Maurits van Rees schreef op 15-12-14 23:50:

...

TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.

Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it. This prints the warnings, because it sees for example a file '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip' This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.

Ah, a demo with plain setuptools 0.8.4 is easy to setup:

$ mkdir bar $ touch bar/project-1.0.zip $ . bin/activate (venv) $ python Python 2.7.8 (default, Jul 28 2014, 10:41:45) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

...

from setuptools import package_index pi = package_index.PackageIndex() pi.add_find_links(['bar']) /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. RuntimeWarning,

Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Dec 16, 2014, at 1:42 AM, anatoly techtonik wrote:

On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft wrote:

...

...

On Dec 15, 2014, at 6:46 PM, Maurits van Rees wrote:

Maurits van Rees schreef op 15-12-14 23:50:

...

TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.

Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it. This prints the warnings, because it sees for example a file '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip' This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.

Ah, a demo with plain setuptools 0.8.4 is easy to setup:

$ mkdir bar $ touch bar/project-1.0.zip $ . bin/activate (venv) $ python Python 2.7.8 (default, Jul 28 2014, 10:41:45) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

...

from setuptools import package_index pi = package_index.PackageIndex() pi.add_find_links(['bar']) /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. RuntimeWarning,

Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.

I'd like to opt-out from PEP440 and use semantic versioning (that is incompatible with PEP440) for my Python packages. I think pip/setuptools/whatever should not stay in the way and give authors ability to choose versioning strategy that is most suitable for their workflow.

To achieve this: 1. pip/setuptools needs to be aware of different versioning strategies 2. package needs to convey which versioning strategy it uses 3. pip/setuptools needs a documented requirement for versions semantic (what it understands about versions and how it uses them)

For example of the above - pip could document that it doesn't install development and alpha versions unless explicitly requested with command line switch, and instead of taking users to read gory details of PEP440 just mention how package may tell pip that it is alpha or development.

Previous vague attempt to request the same thing https://github.com/pypa/pip/issues/1557 (just for the history) -- anatoly t.

It’s not going to happen, it can’t happen reasonably. Luckily though it’s quite trivial for an author to adapt semver to work with PEP 440 syntax. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Dec 16, 2014, at 2:24 AM, anatoly techtonik wrote:

On Tue, Dec 16, 2014 at 9:45 AM, Donald Stufft wrote:

...

...

On Dec 16, 2014, at 1:42 AM, anatoly techtonik wrote:

On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft wrote:

...

...

On Dec 15, 2014, at 6:46 PM, Maurits van Rees wrote:

Maurits van Rees schreef op 15-12-14 23:50:

...

TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.

Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it. This prints the warnings, because it sees for example a file '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip' This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.

Ah, a demo with plain setuptools 0.8.4 is easy to setup:

$ mkdir bar $ touch bar/project-1.0.zip $ . bin/activate (venv) $ python Python 2.7.8 (default, Jul 28 2014, 10:41:45) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

>> from setuptools import package_index >> pi = package_index.PackageIndex() >> pi.add_find_links(['bar']) /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. RuntimeWarning, >>

Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.

I'd like to opt-out from PEP440 and use semantic versioning (that is incompatible with PEP440) for my Python packages. I think pip/setuptools/whatever should not stay in the way and give authors ability to choose versioning strategy that is most suitable for their workflow.

To achieve this: 1. pip/setuptools needs to be aware of different versioning strategies 2. package needs to convey which versioning strategy it uses 3. pip/setuptools needs a documented requirement for versions semantic (what it understands about versions and how it uses them)

For example of the above - pip could document that it doesn't install development and alpha versions unless explicitly requested with command line switch, and instead of taking users to read gory details of PEP440 just mention how package may tell pip that it is alpha or development.

Previous vague attempt to request the same thing https://github.com/pypa/pip/issues/1557 (just for the history) -- anatoly t.

It’s not going to happen, it can’t happen reasonably. Luckily though it’s quite trivial for an author to adapt semver to work with PEP 440 syntax.

Or move to other distribution method than PyPI FWIW. If OpenStack ecosystem could be able to provide an alternative, I'd gladly switch. There are already too many packaging bicycles out there for dynamic languages - Ruby, npm, PyPI - but in fact the only difference that people need for server side of this stuff is distinctive design and URL/subdomain.

Openstack won’t be providing one, they are adapting server to PEP 440 syntax. If you want to go elsewhere be my guest. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Tue, Dec 16, 2014 at 5:08 PM, Jeremy Stanley wrote:

On 2014-12-16 02:25:33 -0500 (-0500), Donald Stufft wrote:

...

Openstack won’t be providing one, they are adapting server to PEP 440 syntax. [...]

Confirmed, we're very close now and will likely be interoperating with Setuptools 8 in our test infrastructure for all supported branches later this week (depending on what direction we end up going for embedded Git SHAs in unreleased dev sdist/wheel version strings).

It would be more useful if the powers behind OpenStack could sponsor development of proper PEP scheme to untangle its own workflow instead of relying on imperfect PEP solutions that are being done by volunteers in their free time. I don't know where Donald finds his time for all his contributions, but in my world it is unreal even without full time job. Judging from activity in OpenStack community, maybe it can teach PSF a thing or two. -- anatoly t.

On Dec 16, 2014, at 10:40 AM, Jeremy Stanley wrote:

The way I see it, we're all volunteers. We voluntarily work on free software and associated support infrastructure. Some of us are simply lucky enough to find sponsors willing to pay us to do it full time (and while I won't speak for Donald I get the impression he feels the same, whether he's working on PSF or OpenStack projects).

Yea, I’m incredibly lucky to have found someone willing to pay me for my hobby. Rackspace pays me to work on Python packaging as well as pays me to work on Openstack. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 17 December 2014 at 02:48, Donald Stufft wrote:

...

On Dec 16, 2014, at 10:40 AM, Jeremy Stanley wrote:

The way I see it, we're all volunteers. We voluntarily work on free software and associated support infrastructure. Some of us are simply lucky enough to find sponsors willing to pay us to do it full time (and while I won't speak for Donald I get the impression he feels the same, whether he's working on PSF or OpenStack projects).

Yea, I’m incredibly lucky to have found someone willing to pay me for my hobby. Rackspace pays me to work on Python packaging as well as pays me to work on Openstack.

Similarly, these days I get to spend (Red Hat) work time on helping to get Fedora to play nice with upstream Python packaging (as well as helping to wrangle other Python integration challenges, including providing feedback and advice on the Python 3 migration work, and playing a significant role in maintaining Red Hat's relationship with the PSF). While most of my work day still goes to Red Hat internal infrastructure work (including learning to take advantage of new technologies like Docker and OpenShift), having that ability to divert time into upstream when necessary makes a huge difference. Untangling the issues in the Python packaging ecosystem is just an incredibly complex change management problem, as anything we do to tighten up the interoperability standards or the underlying security model is almost certainly going to break things for someone, somewhere. Those ecosystem specific constraints are thus far more heavily weighted as a design consideration than interoperability with third party versioning conventions (although we do aim to accommodate those where practical). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Dec 15, 2014 at 01:45:28PM -0500, Donald Stufft wrote:

...

On Dec 15, 2014, at 1:27 PM, Jim Fulton wrote: I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot.

For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools <8 to give more time to respond to these changes.

Is there something I’m not aware that is broken currently? I thought the transition was going pretty smoothly overall considering that a core piece of code inside of setuptools was touched.

~80 zope.* test builds are currently failing, for mysterious reasons due to setuptools 8.0.x not interacting well with zc.buildout: https://mail.zope.org/pipermail/zope-dev/2014-December/046509.html Here's a summary of the various errors: https://mail.zope.org/pipermail/zope-dev/2014-December/046508.html Newer builds also added a bunch of warnings of the form /var/lib/jenkins/jobs/zopetoolkit_trunk/workspace/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'zc.recipe.testrunner-1.0.5 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. Recent threads on distutils-sig@ help explain some of what's happening. E.g. the 'zope.app.wsgi<3.11,<4.0dev,>=3.12' probably used to be interpreted as an "<3.11 or >=3.12" and just needs to be hunted down and replaced with "!=3.11.*", or something like that. (It's painful when you get requirement conflict errors with no indication about the source of those requirements.) Marius Gedminas -- The difference between Microsoft and 'Jurassic Parc': In one, a mad businessman makes a lot of money with beasts that should be extinct. The other is a film.

On Wed, Dec 17, 2014 at 10:09:56AM +0000, Chris Withers wrote:

FWIW, I've also been seeing failures in all of my buildout-based library testing Jenkins jobs:

This one only on Windows and Python 2.7 only, NOT Python 2.6:

C:\Jenkins\workspace\checker-buildout\aeb5917b>C:\Python27\python.exe bootstrap.py ... ImportError: No module named email

This sounds like https://github.com/buildout/buildout/issues/217

...which I see on a couple of jobs, also this, on Python 3, Linux:

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip Extracting in /tmp/tmp71bne9 Now working in /tmp/tmp71bne9/setuptools-8.0.1 Building a Setuptools egg in /tmp/tmpdgs_c0 /tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg Creating directory '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin'. Creating directory '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/parts'. Creating directory '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/develop-eggs'. Generated script '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin/buildout'. /tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130: RuntimeWarning: You have iterated over the result of pkg_resources.parse_version. This is a legacy behavior which is inconsistent with the new version class introduced in setuptools 8.0. That class should be used directly instead of attempting to iterate over the result. /var/lib/jenkins/.buildout/eggs/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130: RuntimeWarning: You have iterated over the result of pkg_resources.parse_version. This is a legacy behavior which is inconsistent with the new version class introduced in setuptools 8.0. That class should be used directly instead of attempting to iterate over the result. While: Installing. Checking for upgrades. Getting distribution for 'zc.buildout>=2.2.5'.

An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", line 1946, in main getattr(buildout, command)(args) File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", line 475, in install self._maybe_upgrade() File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", line 910, in _maybe_upgrade allow_hosts = self._allow_hosts File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", line 844, in install return installer.install(specs, working_set) File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", line 631, in install for_buildout_run=for_buildout_run): File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", line 473, in _get_dist dist, avail = self._satisfied(requirement) File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", line 268, in _satisfied best_available = self._obtain(req, source) File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", line 427, in _obtain if distv > bestv: TypeError: unorderable types: SetuptoolsVersion() > tuple()

Haven't seen that one before.

...and this one on Windows Python 3:

C:\Jenkins\workspace\mush-buildout\a1017537>C:\Python33\python.exe bootstrap.py Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip Extracting in c:\users\jenkins\appdata\local\temp\tmpkij7_u Now working in c:\users\jenkins\appdata\local\temp\tmpkij7_u\setuptools-8.0.1 Building a Setuptools egg in c:\users\jenkins\appdata\local\temp\tmpgy_a7q c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg Traceback (most recent call last): File "<string>", line 138, in use_setuptools ImportError: No module named 'pkg_resources'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "bootstrap.py", line 92, in <module> ez['use_setuptools'](**setup_args) File "<string>", line 140, in use_setuptools File "<string>", line 128, in _do_download File "c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg\setuptools\__init__.py", line 5, in <module> File "C:\Python33\lib\distutils\core.py", line 17, in <module> from distutils.dist import Distribution File "C:\Python33\lib\distutils\dist.py", line 15, in <module> from distutils.fancy_getopt import FancyGetopt, translate_longopt File "C:\Python33\lib\distutils\fancy_getopt.py", line 12, in <module> import getopt ImportError: No module named 'getopt'

Might be https://github.com/buildout/buildout/issues/217 again?

Links to these builds are here, if you're familiar with Jenkins:

http://jenkins.simplistix.co.uk/job/mush-buildout/ http://jenkins.simplistix.co.uk/job/checker-buildout/

Marius Gedminas -- To express oneself In seventeen syllables Is very diffic -- John Cooper Clark.