Buildout not updating setuptools
Hi, I'm having a problem with buildout. My system has setuptools 0.6c5 installed. The buildout bootstrap.py script doesn't install the latest version, and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ... The problem seems to be that bootstrap.py creates a link for setuptools in develop-eggs. Why would that be? Thanks, Graham
On Jul 2, 2007, at 8:08 AM, Graham Stratton wrote: Sorry to take so long to reply. I needed to find time to try to reproduce this.
Hi, I'm having a problem with buildout.
My system has setuptools 0.6c5 installed. The buildout bootstrap.py script doesn't install the latest version,
No, It will use the existing setuptools installation if there is one.
and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ...
I tried to reproduce this, but was unable to.
The problem seems to be that bootstrap.py creates a link for setuptools in develop-eggs. Why would that be?
Is your existing installation of setuptools a develop egg? Buildout always prefers develop eggs that satisfies a requirement, so if you started with a develop egg of setuptools, it won't be upgraded to a later version unless you specifially request one. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Hi Jim, thanks for your time.
and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ...
I tried to reproduce this, but was unable to.
Here's what I did on a clean ubuntu server from the VMWare appliance site: #Install the system setuptools (0.6c5): $ sudo apt-get install subversion python-setuptools $ svn co svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/ . $ vi buildout.cfg ***** [buildout] parts = foxtrot [foxtrot] recipe = zc.recipe.egg interpreter = python2.5 eggs = routes ****** $ python2.5 bootstrap.py $ ls develop-eggs/ setuptools.egg-link $ cat develop-eggs/setuptools.egg-link /usr/lib/python2.5/site-packages $ ls eggs/ zc.buildout-1.0.0b28-py2.5.egg $./bin/buildout -v Installing 'zc.buildout', 'setuptools'. We have the best distribution that satisfies 'zc.buildout'. Picked: zc.buildout = 1.0.0b28 We have a develop egg: setuptools 0.6c5 Installing 'zc.recipe.egg'. We have the best distribution that satisfies 'zc.recipe.egg'. Picked: zc.recipe.egg = 1.0.0b6 Uninstalling foxtrot. Installing foxtrot. Installing 'routes'. We have no distributions for routes that satisfies 'routes'. Couldn't find index page for 'routes' (maybe misspelled?) Getting distribution for 'routes'. We have a develop egg: setuptools 0.6c5 Running easy_install: /usr/bin/python2.5 "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd" "/home/notroot/eggs/tmp_IqDV_" "-q" "/tmp/ tmpDtCTlqget_dist/Routes-1.7.tar.gz" path=/usr/lib/python2.5/site-packages The required version of setuptools (>=0.6c6) is not available, and can't be installed while this script is running. Please install a more recent version first. (Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) error: Setup script exited with 2 An error occured when trying to install Routes 1.7.Look above this message for any errors thatwere output by easy_install. While: Installing foxtrot. Getting distribution for 'routes'. Error: Couldn't install: Routes 1.7 I hope that's useful output/input, Graham
Oh system Pythons. I can only reproduce this using the ubuntu python-setuptools package. Installing setuptools myself using ez_setup doesn't give this behavior. The Unbuntu python-setuptools package installation is rather odd. It doesn't install setuptools as a normal egg. Rather it installs it in such a way that it shows up as a develop egg. jim@starbase14:~/tmp$ python Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import pkg_resources pkg_resources.working_set.find(pkg_resources.Requirement.parse ('setuptools')) setuptools 0.6c5 (/usr/lib/python2.5/site-packages)
Note that the location of the setuptools egg is site packages.
d = pkg_resources.working_set.find (pkg_resources.Requirement.parse('setuptools')) d.precedence == pkg_resources.DEVELOP_DIST True
So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location. IMO, this is just wrong and broken. Does anyone disagree? I could probably add logic to the bootstrap script to get around this, but I'm not very motivated to do so. I'm inclined to say that if you choose to use this version of setuptools, by using your system Python and a system package of setuptools, then you'll have to live with its limitations. Of course, your system easy_install won't install routs either: jim@starbase14:~/tmp$ sudo easy_install routes Password: Searching for routes Reading http://cheeseshop.python.org/pypi/routes/ Couldn't find index page for 'routes' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://cheeseshop.python.org/pypi/ Reading http://cheeseshop.python.org/pypi/Routes/1.7 Reading http://routes.groovie.org/ Best match: Routes 1.7 Downloading http://cheeseshop.python.org/packages/source/R/Routes/ Routes- 1.7.tar.gz#md5=6b8892b8aef7495228376f4e8b6c747b Processing Routes-1.7.tar.gz Running Routes-1.7/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-goT506/Routes-1.7/egg-dist-tmp-fT_EqK The required version of setuptools (>=0.6c6) is not available, and can't be installed while this script is running. Please install a more recent version first. (Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) error: Setup script exited with 2 The good news is that setuptools can repair itself: sudo easy_install -U setuptools Doing this causes setuptools to upgrade itself to a regular egg-based install. At this point (at least on my system), I can install routes and, when I run buildout, it doesn't create an egg link in the develo- eggs directory. Of course, at that point, you've probably messed up your system packaging I will probably do something about this in the long run though just to avoid the question coming up. Jim On Jul 7, 2007, at 2:07 PM, Graham Stratton wrote:
Hi Jim, thanks for your time.
and nor does running bin/buildout. The result is that the Routes install fails with The required version of setuptools (>=0.6c6) is not available, ...
I tried to reproduce this, but was unable to.
Here's what I did on a clean ubuntu server from the VMWare appliance site:
#Install the system setuptools (0.6c5): $ sudo apt-get install subversion python-setuptools $ svn co svn://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/ .
$ vi buildout.cfg ***** [buildout] parts = foxtrot
[foxtrot] recipe = zc.recipe.egg interpreter = python2.5 eggs = routes ******
$ python2.5 bootstrap.py
$ ls develop-eggs/ setuptools.egg-link $ cat develop-eggs/setuptools.egg-link /usr/lib/python2.5/site-packages $ ls eggs/ zc.buildout-1.0.0b28-py2.5.egg
$./bin/buildout -v Installing 'zc.buildout', 'setuptools'. We have the best distribution that satisfies 'zc.buildout'. Picked: zc.buildout = 1.0.0b28 We have a develop egg: setuptools 0.6c5 Installing 'zc.recipe.egg'. We have the best distribution that satisfies 'zc.recipe.egg'. Picked: zc.recipe.egg = 1.0.0b6 Uninstalling foxtrot. Installing foxtrot. Installing 'routes'. We have no distributions for routes that satisfies 'routes'. Couldn't find index page for 'routes' (maybe misspelled?) Getting distribution for 'routes'. We have a develop egg: setuptools 0.6c5 Running easy_install: /usr/bin/python2.5 "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd" "/home/notroot/eggs/tmp_IqDV_" "-q" "/tmp/ tmpDtCTlqget_dist/Routes-1.7.tar.gz" path=/usr/lib/python2.5/site-packages
The required version of setuptools (>=0.6c6) is not available, and can't be installed while this script is running. Please install a more recent version first.
(Currently using setuptools 0.6c5 (/usr/lib/python2.5/site-packages)) error: Setup script exited with 2 An error occured when trying to install Routes 1.7.Look above this message for any errors thatwere output by easy_install. While: Installing foxtrot. Getting distribution for 'routes'. Error: Couldn't install: Routes 1.7
I hope that's useful output/input,
Graham _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote:
So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location.
IMO, this is just wrong and broken. Does anyone disagree?
Yes, because it's fine for them to install it as a develop egg; that's how system packages are *supposed* to be installed. In other words, it's not wonky, unless they also changed something besides just installing it .egg-info style. Thus, if buildout is assuming that "DEVELOP_DIST" egg paths contain only modules or packages that are part of that egg, it is broken. Note, for example, that as of Python 2.5, the distutils install *all* packages with an .egg-info file, which is detected by pkg_resources as a DEVELOP_DIST. A plain old Python 2.5 install with stock system packages will be chock full of develop-style eggs, in other words, even if the packages in question didn't use setuptools at all. This is by design. If you want to be able to figure out if something is really a "develop" installation, you need to look for an .egg-link file, not the mere existence of a DEVELOP_DIST egg. Only the "develop" command writes .egg-link files.
The good news is that setuptools can repair itself:
sudo easy_install -U setuptools
Doing this causes setuptools to upgrade itself to a regular egg-based install. At this point (at least on my system), I can install routes and, when I run buildout, it doesn't create an egg link in the develo- eggs directory. Of course, at that point, you've probably messed up your system packaging
Installing setuptools to a $PYTHONPATH directory should work fine... but might not, depending on whether the system setuptools has a 'site.py' included, or whether it was removed by some busy-body. The next 0.6 version will have a workaround for this so that you can still do $PYTHONPATH installation even if the system setuptools was hacked on by an over-eager packager thinking that a 'site.py' shouldn't be in site-packages. :)
On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote:
At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote:
So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location.
IMO, this is just wrong and broken. Does anyone disagree?
Yes, because it's fine for them to install it as a develop egg; that's how system packages are *supposed* to be installed.
Really? Why? If this is true, then "develop" seems to be a misnomer.
In other words, it's not wonky, unless they also changed something besides just installing it .egg-info style.
Thus, if buildout is assuming that "DEVELOP_DIST" egg paths contain only modules or packages that are part of that egg, it is broken.
Note, for example, that as of Python 2.5, the distutils install *all* packages with an .egg-info file, which is detected by pkg_resources as a DEVELOP_DIST. A plain old Python 2.5 install with stock system packages will be chock full of develop-style eggs, in other words, even if the packages in question didn't use setuptools at all.
This is by design.
If you want to be able to figure out if something is really a "develop" installation, you need to look for an .egg-link file, not the mere existence of a DEVELOP_DIST egg. Only the "develop" command writes .egg-link files.
But you can't look for an egg-link file if all you have is a distribution. The dist location points to the contents of the egg link file. How is one to determine, given a DEVELOP_DIS egg in an environment whether it was created by the develop command? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 07:33 AM 7/9/2007 -0400, Jim Fulton wrote:
On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote:
At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote:
So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location.
IMO, this is just wrong and broken. Does anyone disagree?
Yes, because it's fine for them to install it as a develop egg; that's how system packages are *supposed* to be installed.
Really? Why? If this is true, then "develop" seems to be a misnomer.
Yep - which is why I've been referring to them as "egg-info" or "single version, externally managed" eggs. :) The code still uses one symbol for DEVELOP_DIST because the selection precedence of such eggs is the same, regardless of whether they're generated by "install" or "develop".
If you want to be able to figure out if something is really a "develop" installation, you need to look for an .egg-link file, not the mere existence of a DEVELOP_DIST egg. Only the "develop" command writes .egg-link files.
But you can't look for an egg-link file if all you have is a distribution. The dist location points to the contents of the egg link file. How is one to determine, given a DEVELOP_DIS egg in an environment whether it was created by the develop command?
I didn't say it was easy. ;) You'd have to check for .egg-link files along the environment path, and see if there's one that points to the distribution in question. One possible distinction is that distutils generates .egg-info *files* with version information in the file name, and setuptools "install" and bdist packages include the version information in the directory name by default. So you can rule out any egg for which a versioned filename exists as not being "develop" targets. Unfortunately, you can't assume the reverse: just because it's an .egg-info directory and *doesn't* have any version information, you don't know for sure that it was generated by develop. It could be a system package produced by one of those people who likes to tinker, and decided to remove the version info from the directory name for some crazy reason. OTOH, you could just say the heck with 'em. :) You can easily check whether the .egg-info is a file: if it's a DEVELOP_DIST, and ``dist.list_metadata('')`` contains 1 or 0 entries, it's a metadata file, and definitely not a develop. In order to check the directory version info, you'll have to steal a peak at the ``dist._provider.egg_info`` path and see if it's got a '-' in it before the '.'. If yes, it's not a "develop"-generated egg.
On Jul 9, 2007, at 11:42 AM, Phillip J. Eby wrote:
At 07:33 AM 7/9/2007 -0400, Jim Fulton wrote:
On Jul 8, 2007, at 12:56 PM, Phillip J. Eby wrote:
At 07:17 AM 7/8/2007 -0400, Jim Fulton wrote:
So, the Ubuntu packager for setuptools decided not to install setuptools as an actual egg, but as a wonky sort-of develop egg with a less than ideal location.
IMO, this is just wrong and broken. Does anyone disagree?
Yes, because it's fine for them to install it as a develop egg; that's how system packages are *supposed* to be installed.
Really? Why? If this is true, then "develop" seems to be a misnomer.
Yep - which is why I've been referring to them as "egg-info" or "single version, externally managed" eggs. :)
The code still uses one symbol for DEVELOP_DIST because the selection precedence of such eggs is the same, regardless of whether they're generated by "install" or "develop".
IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should.
If you want to be able to figure out if something is really a "develop" installation, you need to look for an .egg-link file, not the mere existence of a DEVELOP_DIST egg. Only the "develop" command writes .egg-link files.
But you can't look for an egg-link file if all you have is a distribution. The dist location points to the contents of the egg link file. How is one to determine, given a DEVELOP_DIS egg in an environment whether it was created by the develop command?
I didn't say it was easy. ;) You'd have to check for .egg-link files along the environment path, and see if there's one that points to the distribution in question.
One possible distinction is that distutils generates .egg-info *files* with version information in the file name, and setuptools "install" and bdist packages include the version information in the directory name by default. So you can rule out any egg for which a versioned filename exists as not being "develop" targets.
Unfortunately, you can't assume the reverse: just because it's an .egg-info directory and *doesn't* have any version information, you don't know for sure that it was generated by develop. It could be a system package produced by one of those people who likes to tinker, and decided to remove the version info from the directory name for some crazy reason. OTOH, you could just say the heck with 'em. :)
You can easily check whether the .egg-info is a file: if it's a DEVELOP_DIST, and ``dist.list_metadata('')`` contains 1 or 0 entries, it's a metadata file, and definitely not a develop.
In order to check the directory version info, you'll have to steal a peak at the ``dist._provider.egg_info`` path and see if it's got a '-' in it before the '.'. If yes, it's not a "develop"-generated egg.
So I guess there's nothing actually stored in the egg info do indicate that it was built with develop. I'll have to think about what this means for buildout. I suppose I should simply treat anything in the buildout's develop-eggs directory to be deserving of special treatment and ignore the precedence. Things will only get put there when buildout invokes the develop command or by the custom egg builder, whos output should get special treatment that it''s not getting now. Thanks for, um, clarifying this for me. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote:
IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should.
I could have .egg-link files use a different precedence, but I wouldn't make it the highest. Develop eggs have the lowest precedence for two reasons: one, easy_install prefers to use eggs it can copy, and two, develop eggs have lower precedence to prevent you thinking that your version requirements are correct when they aren't. That is, if you depend on 1.9.dev-r1201, and that picks up your development version in place of an .egg file with the same version, you won't be able to tell that your requirement is wrong. This can occur if you are working on an SVN checkout - your develop egg has the same version as the built one, but you could have changed it. This was actually more of a happy coincidence than something I planned ahead of time, but it does occur in practice. :) Note that in any event, if we make platform-specific eggs have higher precedence, then platform-specific built eggs will have to take precedence over develop eggs, because "true" develop eggs are never viewed as "platform-specific" (because they have no platform name in their .egg-info directory name).
On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote:
At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote:
IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should.
I could have .egg-link files use a different precedence, but I wouldn't make it the highest. Develop eggs have the lowest precedence for two reasons: one, easy_install prefers to use eggs it can copy,
Why does this matter to users?
and two, develop eggs have lower precedence to prevent you thinking that your version requirements are correct when they aren't.
That is, if you depend on 1.9.dev-r1201, and that picks up your development version in place of an .egg file with the same version, you won't be able to tell that your requirement is wrong. This can occur if you are working on an SVN checkout - your develop egg has the same version as the built one, but you could have changed it.
That's a fair argument. I think it is just as bad to be making changes to an svn checkout and having those changes have no effect. I suppose that any time we select among two distributions with the same version based on precedence, we should issue a warning, at least when one of them is a develop dist.
This was actually more of a happy coincidence than something I planned ahead of time, but it does occur in practice. :)
It made me very unhappy when I first started working with setuptools. I use develop eggs a lot and I definately expect them to be used.
Note that in any event, if we make platform-specific eggs have higher precedence, then platform-specific built eggs will have to take precedence over develop eggs, because "true" develop eggs are never viewed as "platform-specific" (because they have no platform name in their .egg-info directory name).
This argument sounds overly pedantic to me. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 02:09 PM 7/9/2007 -0400, Jim Fulton wrote:
On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote:
At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote:
IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should.
I could have .egg-link files use a different precedence, but I wouldn't make it the highest. Develop eggs have the lowest precedence for two reasons: one, easy_install prefers to use eggs it can copy,
Why does this matter to users?
It doesn't! -- that's precisely why easy_install is free to pick the one that's easier for it. :)
I suppose that any time we select among two distributions with the same version based on precedence, we should issue a warning, at least when one of them is a develop dist.
That might indeed be doable... at least for tools like buildout and easy_install. I'm not sure it's worth doing so in pkg_resources.
This was actually more of a happy coincidence than something I planned ahead of time, but it does occur in practice. :)
It made me very unhappy when I first started working with setuptools. I use develop eggs a lot and I definately expect them to be used.
If you're not installing them with -m, they will be used, because they'll be the one in easy-install.pth.
Note that in any event, if we make platform-specific eggs have higher precedence, then platform-specific built eggs will have to take precedence over develop eggs, because "true" develop eggs are never viewed as "platform-specific" (because they have no platform name in their .egg-info directory name).
This argument sounds overly pedantic to me.
It's actually a comment about the *implementation*. Develop eggs *can't* include version or platform information in their .egg-info directory names, or it makes them useless.
On Jul 9, 2007, at 2:41 PM, Phillip J. Eby wrote:
At 02:09 PM 7/9/2007 -0400, Jim Fulton wrote:
On Jul 9, 2007, at 1:45 PM, Phillip J. Eby wrote:
At 01:10 PM 7/9/2007 -0400, Jim Fulton wrote:
IMO, they shouldn't have the same precedence. IMO installations made with the develop command should have the highest precedence. After all, why would someone use the develop command if they didn't want to see the effects of changes they're making in development. Could we can change the develop command to generate a different symbol, like REALLY_DEVELOP_DIST and give it the highest precedence? (Tempted to wink, but not really kidding...) It appears that it doesn't actually generate anything atm, but I think it should.
I could have .egg-link files use a different precedence, but I wouldn't make it the highest. Develop eggs have the lowest precedence for two reasons: one, easy_install prefers to use eggs it can copy,
Why does this matter to users?
It doesn't! -- that's precisely why easy_install is free to pick the one that's easier for it. :)
The this I was refering to was: "easy_install prefers to use eggs it can copy". Users don't care about this. I think users do care how develop distributions are handled. ...
This was actually more of a happy coincidence than something I planned ahead of time, but it does occur in practice. :)
It made me very unhappy when I first started working with setuptools. I use develop eggs a lot and I definately expect them to be used.
If you're not installing them with -m, they will be used, because they'll be the one in easy-install.pth.
I install *everything* with -m. When creating the working set for a particular application, I need to decide what should go in the working set. setuptools, left to it's own devices, will choose a non- develop egg over a develop egg. This made me sad.
Note that in any event, if we make platform-specific eggs have higher precedence, then platform-specific built eggs will have to take precedence over develop eggs, because "true" develop eggs are never viewed as "platform-specific" (because they have no platform name in their .egg-info directory name).
This argument sounds overly pedantic to me.
It's actually a comment about the *implementation*. Develop eggs *can't* include version or platform information in their .egg-info directory names, or it makes them useless.
Yes, but I don't see why that should effect their precedence. BTW, at this point, this discussion is pedantic, so we can stop. I can see that, for buildout, I'm not going to be able to use precedence. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
participants (4)
-
Graham Stratton -
Graham Stratton -
Jim Fulton -
Phillip J. Eby