[buildout] weird behaviour with system-installed packages and cross-buildout contamination
Hi Jim, This has had me tearing my hair out somewhat. If you want to play along at home, here's the code: https://github.com/Simplistix/testfixtures/tree/py3k Background: I'm on Mac OS X, Python2.6 is compiled from source, Python2.7 is an EPD install, so has *lot* packages in it (this is a good thing, and means I don't have to compile numpy and friends). So, porting testfixtures to Python 3, I got all the tests passing on 3.3, but now Jenkins tells me there are problems on 2.7. Okay, bootstrap there: buzzkill:testfixtures chris$ python2.7 bootstrap.py Creating directory '/Users/chris/LocalGIT/testfixtures/bin'. Creating directory '/Users/chris/LocalGIT/testfixtures/parts'. Creating directory '/Users/chris/LocalGIT/testfixtures/develop-eggs'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. buzzkill:testfixtures chris$ bin/buildout Develop: '/Users/chris/LocalGIT/testfixtures/.' Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/tox'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv-2.7'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/py'. Installing docs. ... But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in guessing this is to do with Buildout 2's desire not to try and protect you form system packages? If so, not problem with it not installing a new nose, but surely it's a bug that I don't end up with bin/nosetests? Oh well, my 2.6 will show the same problem and is clean, so lets bootstrap with that: buzzkill:testfixtures chris$ python2.6 bootstrap.py Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.34.tar.gz Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpWmSNdS Now working in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpWmSNdS/distribute-0.6.34 Building a Distribute egg in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpG09Ey9 /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpG09Ey9/distribute-0.6.34-py2.6.egg Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. buzzkill:testfixtures chris$ bin/buildout Upgraded: distribute version 0.6.24; restarting. Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. Develop: '/Users/chris/LocalGIT/testfixtures/.' Uninstalling docs. Uninstalling py. Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/tox'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv-2.6'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/py'. Installing docs. Generated script '/Users/chris/LocalGIT/testfixtures/bin/rst2'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/pkginfo'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/docpy'. buzzkill:testfixtures chris$ bin/nos -bash: bin/nos: No such file or directory Wah? Still no nose. Now things get really bizarre. The only way I can get nose back, with *any* python install now, is: buzzkill:testfixtures chris$ git clean -fxd Removing .installed.cfg Removing bin/ Removing develop-eggs/ Removing parts/ Removing testfixtures.egg-info/ Removing testfixtures/__init__.pyc Removing testfixtures/comparison.pyc Removing testfixtures/compat.pyc Now if I buildout with a non-nosed base python, I get nose back: buzzkill:testfixtures chris$ python2.6 bootstrap.py ... buzzkill:testfixtures chris$ bin/buildout Develop: '/Users/chris/LocalGIT/testfixtures/.' Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/nosetests-2.6'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/nosetests'. ... Which elements of the above should I file bugs for? Suffice to say, buildout 2 has not been a very pleasant experience this evening :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 13-02-13 00:45, Chris Withers wrote:
But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in guessing this is to do with Buildout 2's desire not to try and protect you form system packages? If so, not problem with it not installing a new nose, but surely it's a bug that I don't end up with bin/nosetests?
I looked at your code and I cannot find a dependency on nose. Not in the buildout.cfg and not in the setup.py. So buildout rightfully doesn't give you a bin/nosetests. I *do* see nose in "tox.ini", but that's not something that buildout reads. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham"
On 13/02/2013 08:59, Reinout van Rees wrote:
On 13-02-13 00:45, Chris Withers wrote:
But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in guessing this is to do with Buildout 2's desire not to try and protect you form system packages? If so, not problem with it not installing a new nose, but surely it's a bug that I don't end up with bin/nosetests?
I looked at your code and I cannot find a dependency on nose. Not in the buildout.cfg and not in the setup.py. So buildout rightfully doesn't give you a bin/nosetests.
I *do* see nose in "tox.ini", but that's not something that buildout reads.
You're ignoring these lines in setup.py: https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16 I don't want to maintain a list of test dependencies in two files, testfixtures[test] will give you the same list of dependencies as the tox testenv. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 13-02-13 10:15, Chris Withers wrote:
You're ignoring these lines in setup.py:
https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16
You're right, I missed that part. I tried your buildout with a 2.0 bootstrap and I did get a bin/nosetests. I also made a virtualenv and pip-installed nose into that and then ran the buildout: buildout downloaded its own copy. Buildout doesn't isolate you from the system python, but that does not mean it searches the system packages for eggs to use. (You need something like http://pypi.python.org/pypi/syseggrecipe to explicitly grab a system egg). My guess is that the .installed.cfg somehow interferes. I've never really trusted that one and I never looked into how it worked. If something unexplainable happens, I often blow this file away and start again. Chris, does it work for you if you start from scratch in an empty dir? Your jenkins output seems to suggest that you let jenkins create a fresh empty dir for you, so it *should* work... (Unrelated: you're using nose-cov; I can now get a similar coverage output out of nose itself, so the nose-cov functionality is probably included in recent nose releases. Great to get that coverage feedback right in your face after running the tests :-) ) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham"
On 13/02/2013 09:40, Reinout van Rees wrote:
On 13-02-13 10:15, Chris Withers wrote:
You're ignoring these lines in setup.py:
https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16
You're right, I missed that part.
I tried your buildout with a 2.0 bootstrap and I did get a bin/nosetests.
There was a reason my original posting was so long...
My guess is that the .installed.cfg somehow interferes. I've never really trusted that one and I never looked into how it worked.
Indeed, but we don't have any choice, do we?
Chris, does it work for you if you start from scratch in an empty dir? Your jenkins output seems to suggest that you let jenkins create a fresh empty dir for you, so it *should* work...
The problem I was talking about in this thread is not jenkins-related. And yes, as I said in the original posting, a git clean -fxd followed by re-bootstrapping solves the problem.
(Unrelated: you're using nose-cov; I can now get a similar coverage output out of nose itself, so the nose-cov functionality is probably included in recent nose releases. Great to get that coverage feedback right in your face after running the tests :-) )
I don't think the built-in support works well with newest coverage releases and doesn't spit out xml compatible with the cobertura plugin for Jenkins. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (3)
-
Chris Withers -
Chris Withers -
Reinout van Rees