buildout 2 craziness: "upgrading" to distribute 0.6.26 and using site-packages from another python install
Hi All, This is driving me nuts, I hope someone can help. So, pre-amble, I'm on a Mac, I have a few python installations, the ones in play here are: buzzkill:xlutils chris$ which python /Library/Frameworks/EPD64.framework/Versions/Current/bin/python buzzkill:xlutils chris$ which python2.6 /usr/local/bin/python2.6 buzzkill:xlutils chris$ which python2.7 /Library/Frameworks/EPD64.framework/Versions/Current/bin/python2.7 The buildout I'm working with is here: https://github.com/python-excel/xlutils/blob/master/buildout.cfg My python2.7 has a site-packages xlrd (it's EPD, which includes a bunch of packages), so I'm using python2.6 on this machine, since it doesn't (it's a plain source install on python). Okay, so, as you'll see from github above, I have the latest bootstrap.py, and the initial bootstrap looks fine: buzzkill:xlutils chris$ python2.6 bootstrap.py Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpxnVlCD Now working in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpxnVlCD/distribute-0.6.35 Building a Distribute egg in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpg4hxVn /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpg4hxVn/distribute-0.6.35-py2.6.egg buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/distribute-0.6.35-py2.6.egg', '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Okay, so now try running that buildout: buzzkill:xlutils chris$ bin/buildout Upgraded: distribute version 0.6.26; restarting. Generated script '/Users/chris/LocalGIT/xlutils/bin/buildout'. Develop: '/Users/chris/LocalGIT/xlutils/.' Develop: '/Users/chris/LocalGIT/xlutils/../xlrd' Traceback (most recent call last): File "/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpwnpB08", line 13, in <module> exec(compile(open('/Users/chris/LocalGIT/xlutils/../xlrd/setup.py').read(), '/Users/chris/LocalGIT/xlutils/../xlrd/setup.py', 'exec')) File "/Users/chris/LocalGIT/xlutils/../xlrd/setup.py", line 18, in <module> from xlrd.info import __VERSION__ ImportError: No module named info While: Installing. Processing develop directory '/Users/chris/LocalGIT/xlutils/../xlrd'. An internal error occured due to a bug in either zc.buildout or in a recipe being used: <snip> File "/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg/zc/buildout/easy_install.py", line 129, in call_subprocess % repr(args)[1:-1]) Exception: Failed to run command: '/usr/local/bin/python2.6', '/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpwnpB08', '-q', 'develop', '-mxN', '-d', '/Users/chris/LocalGIT/xlutils/develop-eggs/tmpUg8rV8build' Okay, so, a few comments: - "upgrading" to distribute version 0.6.26 from 0.6.35. Whu?! - The xlrd in /Users/chris/LocalGIT/xlrd certainly does have an info module. Now the really crazy bit: buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Come again?! Why is site-packages from a *different python installation* ending up in bin/buildout? Any help here would be very gratefully received... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 05/04/2013 11:52, Chris Withers wrote:
Hi All,
This is driving me nuts, I hope someone can help.
Okay, so using my cunning buildout knowledge (huh!) I tried this: [buildout] develop = . ../xlrd ../xlwt parts = test py docs versions = versions [versions] distribute=0.6.35 <snip> buzzkill:xlutils chris$ bin/buildout Develop: '/Users/chris/LocalGIT/xlutils/.' Develop: '/Users/chris/LocalGIT/xlutils/../xlrd' Develop: '/Users/chris/LocalGIT/xlutils/../xlwt' Unused options for buildout: 'unzip'. Updating test. Updating py. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'. Updating docs. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'. YAY! (or so I thought) buzzkill:xlutils chris$ bin/test <snip> Ran 327 tests with 2 failures and 27 errors in 0.478 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. buzzkill:xlutils chris$ cat bin/test #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/LocalGIT/xlutils', '/Users/chris/buildout-eggs/zope.testrunner-4.3.3-py2.6.egg', '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', '/Users/chris/buildout-eggs/zope.exceptions-4.0.6-py2.6.egg', <snip> Oh FFS :-( Why is the python2.6 buildout using xlrd from site-packages in a python2.7 installation?! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, Apr 05, 2013 at 12:11:18PM -0300, Chris Withers wrote:
On 05/04/2013 11:52, Chris Withers wrote:
Hi All,
This is driving me nuts, I hope someone can help.
Okay, so using my cunning buildout knowledge (huh!) I tried this:
[buildout] develop = . ../xlrd ../xlwt parts = test py docs versions = versions
[versions] distribute=0.6.35 <snip>
buzzkill:xlutils chris$ bin/buildout Develop: '/Users/chris/LocalGIT/xlutils/.' Develop: '/Users/chris/LocalGIT/xlutils/../xlrd' Develop: '/Users/chris/LocalGIT/xlutils/../xlwt' Unused options for buildout: 'unzip'. Updating test. Updating py. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'. Updating docs. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'.
YAY! (or so I thought)
buzzkill:xlutils chris$ bin/test <snip> Ran 327 tests with 2 failures and 27 errors in 0.478 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
buzzkill:xlutils chris$ cat bin/test #!/usr/local/bin/python2.6
import sys sys.path[0:0] = [ '/Users/chris/LocalGIT/xlutils', '/Users/chris/buildout-eggs/zope.testrunner-4.3.3-py2.6.egg',
'/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', '/Users/chris/buildout-eggs/zope.exceptions-4.0.6-py2.6.egg', <snip>
Oh FFS :-( Why is the python2.6 buildout using xlrd from site-packages in a python2.7 installation?!
Check develop-eggs/, there may be a distribute.egg-link file that puts /Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages in your sys.path. When in doubt, rm -rf develop-eggs before retrying. Marius Gedminas -- 1 + 1 = 3 -- from a Microsoft advertisement
On 05/04/2013 12:15, Marius Gedminas wrote:
Check develop-eggs/, there may be a distribute.egg-link file that puts /Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages in your sys.path.
When in doubt, rm -rf develop-eggs before retrying.
Well, this worked. My guess as to what happened: I was using python2.7 (which is EPD, containing a tonne of packages including an older version of xlrd). Since that xlrd was getting picked instead of the one in the develop line (wasn't that bug supposed to be fixed in buildout 2?!) I switched to python2.6, which I knew was clean. It sounds like I should have had the prescience (!) to do: rm -rf develop-eggs/ .installed.cfg ...before I did: python2.6 bootstrap.py *sigh* At times like this (when I've just wasted 1.5hrs chasing my tail) I really truly hate buildout :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, Apr 5, 2013 at 11:36 AM, Chris Withers <chris@python.org> wrote:
On 05/04/2013 12:15, Marius Gedminas wrote:
Check develop-eggs/, there may be a distribute.egg-link file that puts
/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages in your sys.path.
When in doubt, rm -rf develop-eggs before retrying.
Well, this worked.
My guess as to what happened: I was using python2.7 (which is EPD, containing a tonne of packages including an older version of xlrd).
BTW, I have no idea what EPD means.
Since that xlrd was getting picked instead of the one in the develop line (wasn't that bug supposed to be fixed in buildout 2?!)
There was quite a bit of discussion about this recently and there was no consensus that develop eggs should be preferred to over non-develop eggs with higher versions.
I switched to python2.6, which I knew was clean.
It sounds like I should have had the prescience (!) to do:
rm -rf develop-eggs/ .installed.cfg
...before I did:
python2.6 bootstrap.py
*sigh*
At times like this (when I've just wasted 1.5hrs chasing my tail) I really truly hate buildout :-(
It's times like this that I really really hate it when people insist on using dirty Pythons when they should know better. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
On 05/04/2013 12:48, Jim Fulton wrote:
My guess as to what happened: I was using python2.7 (which is EPD, containing a tonne of packages including an older version of xlrd).
BTW, I have no idea what EPD means.
Enthought Python Distribution, I should emphasise *was* using above...
Since that xlrd was getting picked instead of the one in the develop line (wasn't that bug supposed to be fixed in buildout 2?!)
There was quite a bit of discussion about this recently and there was no consensus that develop eggs should be preferred to over non-develop eggs with higher versions.
The non-develop eggs here have significantly *lower* version numbers. (and really, if I say I'm developing a package, why would a non-development version be picked?!)
At times like this (when I've just wasted 1.5hrs chasing my tail) I really truly hate buildout :-(
It's times like this that I really really hate it when people insist on using dirty Pythons when they should know better.
Two comments, the first an order of magnitude more important than the second: 1. I *was using a clean python*. Buildout decided to use some crap *it* left lying around from a totally different Python installation when I ran python2.6<clean, build form source> bootstrap.py 2. There is some practicality here. EPD provides suitable versions of a tonne of packages I can't and often don't even want to try compiling from source. Unfortunately, it provides some packages of which I want newer versions. I know this is a hard problem, one that Gary Poster tried to solve, and I appreciate that implementation didn't turn out well... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, Apr 5, 2013 at 12:01 PM, Chris Withers <chris@python.org> wrote:
On 05/04/2013 12:48, Jim Fulton wrote:
My guess as to what happened: I was using python2.7 (which is EPD, containing a tonne of packages including an older version of xlrd).
BTW, I have no idea what EPD means.
Enthought Python Distribution, I should emphasise *was* using above...
Since that xlrd was getting picked instead of the one in the develop line (wasn't that bug supposed to be fixed in buildout 2?!)
There was quite a bit of discussion about this recently and there was no consensus that develop eggs should be preferred to over non-develop eggs with higher versions.
The non-develop eggs here have significantly *lower* version numbers. (and really, if I say I'm developing a package, why would a non-development version be picked?!)
There are people who agree with you and people who don't. <shrug>
At times like this (when I've just wasted 1.5hrs chasing my tail) I really truly hate buildout :-(
It's times like this that I really really hate it when people insist on using dirty Pythons when they should know better.
Two comments, the first an order of magnitude more important than the second:
1. I *was using a clean python*. Buildout decided to use some crap *it* left lying around from a totally different Python installation when I ran python2.6<clean, build form source> bootstrap.py
But you were using a dirty Python and that left your buildout in a bad state. There's not a lot I can (or am willing) to do about that.
2. There is some practicality here. EPD provides suitable versions of a tonne of packages I can't and often don't even want to try compiling from source.
I understand. I sure hope wheels help this situation. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
On 05/04/2013 13:16, Jim Fulton wrote:
The non-develop eggs here have significantly *lower* version numbers. (and really, if I say I'm developing a package, why would a non-development version be picked?!)
There are people who agree with you and people who don't.<shrug>
What's your view on it? My take: how can I ensure that when I put in a develop line, that package gets used, regardless of the version number, but *especially* when the version number is greater than other packages found, even if that version number is a .dev version (which is usually the case..)
Two comments, the first an order of magnitude more important than the second:
1. I *was using a clean python*. Buildout decided to use some crap *it* left lying around from a totally different Python installation when I ran python2.6<clean, build form source> bootstrap.py
But you were using a dirty Python and that left your buildout in a bad state.
No, I was using a Python that left my buildout in the state any python, clean or dirty, would have done...
There's not a lot I can (or am willing) to do about that.
Having bootstrap.py really make sure things are started again from scratch would likely solve all these problems. (The .installed.cfg stuff I remember hitting before, when only using clean pythons) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 06-04-13 02:55, Chris Withers wrote:
Having bootstrap.py really make sure things are started again from scratch would likely solve all these problems. (The .installed.cfg stuff I remember hitting before, when only using clean pythons)
Removing .installed.cfg: not sure. Some recipes fail when installing themselves when there's still something left over from the last run. And 'install' versus 'update' is used when the .installed.cfg is gone. OTOH, that's somethint that needs fixing in the recipes. Removing the develop eggs: that actually sounds like a very good idea. It would have solved quite some issues with system packages here at the office. (And yes, we use system packages too. Big geographical stack with numpy, mapnik, matplotlib, geos, scipy, netcdf... Impractical to compile from source. Though I'm sometimes tempted to make a buildout that compiles all that stuff for me :-) ) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham"
On Fri, Apr 05, 2013 at 11:52:17AM -0300, Chris Withers wrote:
This is driving me nuts, I hope someone can help.
So, pre-amble, I'm on a Mac, I have a few python installations, the ones in play here are:
<snip>
buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6
import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/distribute-0.6.35-py2.6.egg', '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', ] ... buzzkill:xlutils chris$ bin/buildout Upgraded: distribute version 0.6.26; restarting. Generated script '/Users/chris/LocalGIT/xlutils/bin/buildout'. ... buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6
import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', ]
I've seen this on Ubuntu. IIRC a Python 3-bootstrapped buildout ended up with /usr/lib/python2.7/dist-packages in sys.path, in my case.
Come again?! Why is site-packages from a *different python installation* ending up in bin/buildout?
Any help here would be very gratefully received...
Workaround #1: run 'bin/buildout -N' so it won't try to upgrade distribute. Workaround #2: always always always wrap your buildouts in a virtualenv. I started using a 'virtual-bootstrap' script that does #!/bin/sh rm -rf develop-eggs .installed.cfg python virtualenv --distribute python python/bin/python bootstrap.py bin/buildout "$@" and haven't encountered any issues yet. Marius Gedminas -- When in trouble or in doubt, run in circles, scream and shout.
On 05/04/2013 12:14, Marius Gedminas wrote:
Any help here would be very gratefully received...
Workaround #1: run 'bin/buildout -N' so it won't try to upgrade distribute.
I just pinned distribute in the buildout.cfg, as I showed, saves forgetting to pass -N later.
Workaround #2: always always always wrap your buildouts in a virtualenv.
How would this have helped here? The python2.6 I ran bootstrap with is totally clean, compiled by me...
I started using a 'virtual-bootstrap' script that does
#!/bin/sh rm -rf develop-eggs .installed.cfg python virtualenv --distribute python python/bin/python bootstrap.py bin/buildout "$@"
and haven't encountered any issues yet.
My problem is that I *want* to use that python2.7 as a base for a lot of my projects. I absolutely don't want to be trying to compile numpy/etc on a Mac when I'm already paying Enthought to do that for me ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (5)
-
Chris Withers -
Chris Withers -
Jim Fulton -
Marius Gedminas -
Reinout van Rees