buildout 2.2.3: not completely right?
Hi, Since buildout 2.2.3 is out, I've been getting mails out of quite some jenkins jobs. Those jobs always run "python bootstrap.py" followed by "bin/buildout". We always pin buildout, so most of them are at version 2.2.1 $ python bootstrap.py Setting socket time out to 1 seconds. Generated script '<http://jenkins.3di.lizard.net/job/python-subgrid/ws/bin/buildout'.> $ bin/buildout Setting socket time out to 1 seconds. While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 2.2.3 "python bootstrap.py" generates a bin/buildout with a 2.2.3 version in it. And running bin/buildout doesn't want to downgrade to the specified version 2.2.1. The bootstrap.py I'm using is http://downloads.buildout.org/2/bootstrap.py I've seen some other problems with versions when there's a globally installed version already available at a higher version. A colleague installed ansible globally on the jenkins machine (...) so there's a new jinja2 in usr/local/lib/pythonsomething. The jinja2 in buildout is pinned to a lower version, but buildout refuses to downgrade it and fails with an "I already have that new version". So... I haven't totally tracked it down yet, but I'm throwing it out on the list here in case others have seen it too. I'll continue debugging after lunch :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 31-10-14 12:38, Reinout van Rees wrote:
$ python bootstrap.py Setting socket time out to 1 seconds. Generated script '<http://jenkins.3di.lizard.net/job/python-subgrid/ws/bin/buildout'.> $ bin/buildout Setting socket time out to 1 seconds. While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 2.2.3
Ok, I tracked it down: https://github.com/buildout/buildout/issues/198#issuecomment-61258074 The short summary: the main problem is line 652: https://github.com/buildout/buildout/pull/194/files#diff-784f9bb654bbcee96dc... The working set is queried for just the package name. It grabs whatever it can (a system package? a zc.buildout 2.2.3 installed by the bootstrap.py) and uses that. It doesn't look at the actual requirement, just at the plain package name (req.key). Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 31-10-14 14:21, Reinout van Rees wrote:
Ok, I tracked it down: https://github.com/buildout/buildout/issues/198#issuecomment-61258074
And I have a pull request that fixes it. https://github.com/buildout/buildout/pull/199 All the buildouts that failed this morning work again with this fix. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 31-10-14 18:16, Reinout van Rees wrote:
On 31-10-14 14:21, Reinout van Rees wrote:
Ok, I tracked it down: https://github.com/buildout/buildout/issues/198#issuecomment-61258074
And I have a pull request that fixes it. https://github.com/buildout/buildout/pull/199
All the buildouts that failed this morning work again with this fix.
Ok, 2.2.4 is out with the fix in. A colleague came to me this morning, however, with a problem. Ouch, it isn't completely fixed yet! Turns out the `env.best_match(req, ws)` fails on anything that is installed globally with a version that doesn't match the one we want. In my colleague's case, he has nose 1.3.1 installed globally. Buildout has a pin on 1.3.4. env.best_match() returns a VersionConflict. So it seems we should ignore that too. **But**, this means buildout just gained the ability to use globally installed packages! Hurray, I don't need https://pypi.python.org/pypi/syseggrecipe anymore! I can imagine that was not quite the effect we wanted! I'll dive in a bit deeper. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 03-11-14 10:22, Reinout van Rees wrote:
Turns out the `env.best_match(req, ws)` fails on anything that is installed globally with a version that doesn't match the one we want.
In my colleague's case, he has nose 1.3.1 installed globally. Buildout has a pin on 1.3.4. env.best_match() returns a VersionConflict.
So it seems we should ignore that too.
I'm on to something. The `env.best_match()` works based on the working set. So: why is the working set filled with all the system stuff? Normally the working set starts out empty! I did a bit of extra logging and found out that buildout extensions and buildout recipes get installed with a system-wide working set. Regular requirements are installed based on a n empty working set. The problem I'm seeing is with pbp.recipe.noserunner (ancient, but handy). This has a dependency on "nose" which it finds globally with a wrong version. The other problem is with collective.recipe.sphinxbuilder, which depends on jinja2 which is installed globally in some cases ("I need ansible installed globally"). => Why does the buildout install and the extension/recipe install start out with a full-blown working set? It might be a left-over from the time when we used the globally installed setuptools. The newest bootstraps make this unnecessary, right? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 03-11-14 11:28, Reinout van Rees wrote:
I'm on to something.
https://github.com/buildout/buildout/pull/201 Multiple complications at the same time. The new buildout was stricter than the old one with constraining dependencies to the right version. For instance when installing dependencies for extensions. mr.developer depends on buildout and setuptools. Constraining them to the right version fails if the buildout was bootstrapped with a different buildout/setuptools version... Extra complication was that installing extensions happens BEFORE checking if setuptools/buildout needs upgrading/downgrading :-) The pull request also solves a problem with recipes that themselves have dependencies. pdp.recipe.noserunner depends on nose, which can be installed globally. If that's the wrong version, you get an unclear version conflict. With the pull request, buildout will ignore that conflict and instead will forcefully install the right version. This only happens when installing buildout/setuptools/extensions/recipes, though, not with the regular install/dependency handling. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
participants (1)
-
Reinout van Rees