[Distutils] change in setuptools 8.0

[Jim Fulton]

On Mon, Dec 15, 2014 at 12:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 15, 2014, at 9:05 AM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> Donald Stufft schreef op 15-12-14 13:20:
>>>
>>>> On Dec 15, 2014, at 6:03 AM, Robin Becker <robin at reportlab.com> wrote:
>>>>
>>>> A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency
>>>>
>>>> pillow==2.0.0,>=2.4.0
>>>>
>>>> which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied.
>>>>
>>>> In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi.
>>>>
>>>> Unfortunately we had this as part of the install_requires as
>>>>
>>>> install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']
>>>>
>>>> so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :(
>>>
>>> Historically the meaning of a comma inside of a version specifier is… well complicated. Honestly I have a hard time even putting into words what a comma means at all in a historical context. Sometimes it acts as an OR, sometimes it acts as and AND, and sometimes it acts as something else that I can’t quite explain.
>>>
>>> This was part of how setuptools was designed, it valued giving an answer, any answer, over saying “Sorry this doesn’t make sense”. You can see this most clearly in the version parsing code which would allow versions such as “dog” or “this isn’t a version but setuptools will parse it as one”. In PEP 440 we attempted to standardize what a version and what a specifier means, and as part of that we made the decision that we are going to be stricter in what we accept. This means that some things that used to be valid versions are no longer valid versions and in your case, relying on the old, complicated behavior, of a comma that sometimes means different things.
>>>
>>> So yea, in a PEP 440 world the comma is AND.
>>
>> Sounds sane.
>>
>> But I now run into unexpected behaviour when two packages have a constraint on the same third package.  For example one has 'zest.releaser==3.50' and another has 'zest.releaser>=3.40'. Wanted and expected behaviour is to get 3.50, as that satisfies both constraints.
>>
>> You can test this in a virtualenv with setuptools 8.0.2:
>>
>> $ pip install 'zest.releaser==3.50,>=3.40'
>> Downloading/unpacking zest.releaser>=3.40,==3.50
>>  Downloading zest.releaser-3.53.2.zip
>> ...
>>
>> So expected is 3.50, but you get the latest version, currently 3.53.2.
>> Sound like a bug?
>
> Try with the develop version of pip. pip bundles setuptools internally in order to prevent issues from setuptools accidentally getting uninstalled breaking pip and such,
>
>>
>>
>> Where I am seeing this error in practice is in a buildout.  I have not managed to reproduce my error in a small enough buildout that is sane to share.  But for the idea, it goes like this. Latest buildout-bootstrap.py gives me zc.buildout 2.3.0 and setuptools 0.8.2. The buildout config has pinned zc.buildout to version 2.2.5 and setuptools to 7.0 and allow-picked-versions to false.  Then I run bin/buildout.   It fails with:
>>
>> While:
>>  Installing.
>>  Getting section _mr.developer.
>>  Initializing section _mr.developer.
>>  Installing recipe zc.recipe.egg.
>>  Getting distribution for 'zc.buildout==2.2.5,>=1.5.0'.
>> Error: Picked: zc.buildout = 2.2.5
>>
>> So one package (zc.recipe egg 1.3.2, but similar with latest 2.0.1) has a dependency on zc.buildout>=1.5.0 and the buildout config pins zc.buildout to 2.2.5 and this somehow fails.
>>
>> Oddly enough, it goed alright when I set allow-picked-versions to true...
>>
>>
>> For the record, it then goes wrong later with an error that indicates a casualty of the more strict version checking:
>>
>> The constraint, 2.0.5, is not consistent with the requirement, 'five.localsitemanager>2.0dev'.
>> While:
>>  Installing zeoclient.
>> Error: Bad constraint 2.0.5 five.localsitemanager>2.0dev
>>
>> The bad constraint '>2.0dev' is in the five.grok package.  I guess it should have been '>2.0.dev0' (or by now simply '>=2.0').  I'll pick it up for that package.
>>
>
> I don’t know much (or anything really) about buildout or what it does with that.

Buildout doesn't try to merge dependency requirements from different packages.
It just installs packages and their dependencies and builds up a working set.

It does merge constraints from the versions section (if any) with
whatever requirement
it's working on at any point in time.

I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0
is coming from a setup spec or from some buildout section.  It was a
valid spec once
but isn't any more.

Buildout 2 before 2.3 did a lot of strange gymnastics to try to merge
version constraints
with requirement specs given the strange old rules.

Now, with the new sane rules, it just adds the constraint to the
whatever existing spec is
in the requirement.

There error "Bad constraint 2.0.5 five.localsitemanager>2.0dev" looks
to me like a bug. :)
When buildout gets a single-version constraint, like 2.0.5, it tests
whether that version
is "in" (__contains__) a requirement.

>>> import pkg_resources
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>> '2.0.5' in r
False
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>> '2.0.5' in r
True

I have a feeling the definition of "dev" has changed too.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton