patches: ez_setup.py: don't import setuptools into your current address space in order to learn its version number
Folks: We're gradually converting the allmydata.org tahoe project [1] and its spin-off packages to use setuptools. One problem that comes up is that ez_setup.py might require a newer version of setuptools than the version of setuptools already installed. For example, although Tahoe currently uses setuptools to manage its dependencies on zfec, foolscap, and nevow [2], but if we execute simplejson's ez_setup.py then it requires setuptools v0.6c7 and refuses to proceed if an earlier version is installed. One solution for this problem could be for the packager of simplejson (Bob Ippolito) to use the "min_version" patch that we contributed to ez_setup.py [3]. This is assuming that simplejson doesn't actually *require* the latest version of setuptools, and we could successfully install with a slightly older version, but what if a package actually does require a newer version of setuptools than the one that is already installed? The following patch was created by Tahoe contributor Arno Washck and then modified by me in order to get around such a problem. The idea is simple enough -- reload setuptools. There may be some subtleties that we still need to work out. This patch hasn't been tested in its current form. --- ez_setup.py~ 2007-09-10 12:36:30.000000000 -0600 +++ ez_setup.py 2007-09-18 15:15:23.000000000 -0600 @@ -95,13 +95,9 @@ pkg_resources.require("setuptools>="+version) except pkg_resources.VersionConflict, e: - # XXX could we install in a subprocess here? - print >>sys.stderr, ( - "The required version of setuptools (>=%s) is not available, and\n" - "can't be installed while this script is running. Please install\n" - " a more recent version first.\n\n(Currently using %r)" - ) % (version, e.args[0]) - sys.exit(2) + egg = download_setuptools(version, download_base, to_dir, download_delay) + sys.path.insert(0, egg) + reload(setuptools); setuptools.bootstrap_install_from = egg def download_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, ------- An alternative idea figure out the version of the current install of setuptools without importing the package. The following patch might accomplish that. It is also not tested. --- ez_setup.py~ 2007-09-10 12:36:30.000000000 -0600 +++ ez_setup.py 2007-09-18 15:26:50.000000000 -0600 @@ -77,31 +77,13 @@ this routine will print a message to ``sys.stderr`` and raise SystemExit in an attempt to abort the calling script. """ - try: - import setuptools - if setuptools.__version__ == '0.0.1': - print >>sys.stderr, ( - "You have an obsolete version of setuptools installed. Please\n" - "remove it from your system entirely before rerunning this script." - ) - sys.exit(2) - except ImportError: - egg = download_setuptools(version, download_base, to_dir, download_delay) - sys.path.insert(0, egg) - import setuptools; setuptools.bootstrap_install_from = egg - + verstr = os.system("python -c \"import setuptools;print setuptools.__version__\"") import pkg_resources - try: - pkg_resources.require("setuptools>="+version) - except pkg_resources.VersionConflict, e: - # XXX could we install in a subprocess here? - print >>sys.stderr, ( - "The required version of setuptools (>=%s) is not available, and\n" - "can't be installed while this script is running. Please install\n" - " a more recent version first.\n\n(Currently using %r)" - ) % (version, e.args[0]) - sys.exit(2) + if pkg_resources.parse_version(verstr) < pkg_resources.parse_version(version): + egg = download_setuptools(version, download_base, to_dir) + sys.path.insert(0, egg) + import setuptools; setuptools.bootstrap_install_from = egg def download_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, ------- Regards, Zooko [1] http://allmydata.org [2] http://allmydata.org/trac/tahoe/browser/calcdeps.py [3] http://mail.python.org/pipermail/distutils-sig/2007-September/ 008257.html
At 03:29 PM 9/18/2007 -0600, zooko wrote:
Folks:
We're gradually converting the allmydata.org tahoe project [1] and its spin-off packages to use setuptools. One problem that comes up is that ez_setup.py might require a newer version of setuptools than the version of setuptools already installed.
For example, although Tahoe currently uses setuptools to manage its dependencies on zfec, foolscap, and nevow [2], but if we execute simplejson's ez_setup.py then it requires setuptools v0.6c7 and refuses to proceed if an earlier version is installed.
One solution for this problem could be for the packager of simplejson (Bob Ippolito) to use the "min_version" patch that we contributed to ez_setup.py [3]. This is assuming that simplejson doesn't actually *require* the latest version of setuptools, and we could successfully install with a slightly older version, but what if a package actually does require a newer version of setuptools than the one that is already installed?
The following patch was created by Tahoe contributor Arno Washck and then modified by me in order to get around such a problem. The idea is simple enough -- reload setuptools. There may be some subtleties that we still need to work out. This patch hasn't been tested in its current form.
It won't work correctly. First, pkg_resources is part of setuptools, so if you install a new version of setuptools, you have to reload() it too. Second, it's not *safe* to reload it, if it or setuptools were already imported at the time the function is called. That's because easy_install runs the setup.py of a package it's building from source. So if you use easy_install to install a package that needs a newer version, reloading pkg_resources or setuptools (and note that setuptools is a package with lots of submodules!) will break the host easy_install process. Basically, that's why ez_setup.py is written the way it is. The best case scenario here would be to check if setuptools or pkg_resources are already imported, and if not, then the pkg_resources version check + reload of pkg_resources would work. But if they are already imported, there is no way to reload them in a way that won't hose something that's already in progress in the caller's context. In other words, we could maybe fix this for "setup.py install", but not for easy_install. I'm not sure how useful that is, though, since if you have setuptools installed, why download "foo", unpack it, and "setup.py install", when you can just "easy_install foo" in one go? About the only thing that *might* work would be to have easy_install detect the error when trying to build the package, and then download a new setuptools and run the setup.py in a subprocess with that setuptools on the path, and finish up the current install by switching to the new setuptools. But that's just insane, because what if the user is running e.g., easy_install -zmaxd to build a collection of eggs for distribution? It makes no sense to install the new setuptools in the *distribution* directory. So, unfortunately, the end user is the only one who can safely upgrade their setuptools version (especially if they're doing it via .rpm or .deb or some such!).
In other words, we could maybe fix this for "setup.py install", but not for easy_install. I'm not sure how useful that is, though, since if you have setuptools installed, why download "foo", unpack it, and "setup.py install", when you can just "easy_install foo" in one go?
Oh, perhaps this also explains why you didn't understand the use case for the previous patch. I love easy_install, in some cases we and our users need to execute "setup.py install" instead. There are a few different reasons for this that I would be happy to go over with you, but hopefully as far as patching ez_setup.py, if I can write patches which improve this use case without harming other use cases, you will accept that some people prefer to execute "./setup.py install" and not "easy_install"? Here is the current README for our decentralized file storage project: http://allmydata.org/trac/tahoe/browser/README Here is my personal notes on how I like to use setuptools-packaged software under GNU stow: https://zooko.com/log-2007.html#d2007-06-02 Regards, Zooko
At 01:29 PM 9/27/2007 -0600, zooko wrote:
In other words, we could maybe fix this for "setup.py install", but not for easy_install. I'm not sure how useful that is, though, since if you have setuptools installed, why download "foo", unpack it, and "setup.py install", when you can just "easy_install foo" in one go?
Oh, perhaps this also explains why you didn't understand the use case for the previous patch. I love easy_install, in some cases we and our users need to execute "setup.py install" instead.
Unless you are using --root or --single-version-externally-managed, there is actually no difference: "setup.py install" actually invokes the rough equivalent of "easy_install .".
There are a few different reasons for this that I would be happy to go over with you, but hopefully as far as patching ez_setup.py, if I can write patches which improve this use case without harming other use cases, you will accept that some people prefer to execute "./setup.py install" and not "easy_install"?
The only reason (besides habit) to do that is to do a single-version installation -- in which case trying to make ez_setup do the right thing is a red herring. To do a single-version install you have to either know precisely what you're doing, or else you're building a system package -- in which case you'd darn well better have a compatible setuptools installed. To put it another way, I'm not interested in making it easier for people to shoot themselves in the foot. The error message is there for good reasons, and I've thought long and hard about ways to get rid of it. I'm willing to listen to new ideas, I just don't think a safe solution for upgrading setuptools in-place is possible without either multiple processes or multiple interpreters in a single process.
On Sep 27, 2007, at 10:51 AM, Phillip J. Eby wrote:
It won't work correctly. First, pkg_resources is part of setuptools, so if you install a new version of setuptools, you have to reload() it too.
Second, it's not *safe* to reload it, if it or setuptools were already imported at the time the function is called. That's because easy_install runs the setup.py of a package it's building from source. So if you use easy_install to install a package that needs a newer version, reloading pkg_resources or setuptools (and note that setuptools is a package with lots of submodules!) will break the host easy_install process.
Okay. Attached is a patch that doesn't import pkg_resources and doesn't reload anything. I haven't yet tested it -- I am submitting it only to inform the discussion and get any early feedback from you on whether the very idea is sound. Regards, Zooko diff -rN -u old-up/setuptools-0.6c7/ez_setup.py new-up/ setuptools-0.6c7/ez_setup.py --- old-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:24.000000000 -0600 +++ new-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:25.000000000 -0600 @@ -1,4 +1,4 @@ -#!python +#!/usr/bin/env python """Bootstrap setuptools installation If you want to use setuptools in your package's setup.py, just include this @@ -13,7 +13,7 @@ This file can also be run as a script to install or upgrade setuptools. """ -import sys +import os, re, subprocess, sys DEFAULT_VERSION = "0.6c7" DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3] @@ -44,8 +44,6 @@ 'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53', } -import sys, os - def _validate_md5(egg_name, data): if egg_name in md5_data: from md5 import md5 @@ -58,6 +56,42 @@ sys.exit(2) return data +# The following code to parse versions is copied from pkg_resources.py so that +# we can parse versions without importing that module. +component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE) +replace = {'pre':'c', 'preview':'c','-':'final-','rc':'c','dev':'@'}.get + +def _parse_version_parts(s): + for part in component_re.split(s): + part = replace(part,part) + if not part or part=='.': + continue + if part[:1] in '0123456789': + yield part.zfill(8) # pad for numeric comparison + else: + yield '*'+part + + yield '*final' # ensure that alpha/beta/candidate are before final + +def parse_version(s): + parts = [] + for part in _parse_version_parts(s.lower()): + if part.startswith('*'): + if part<'*final': # remove '-' before a prerelease tag + while parts and parts[-1]=='*final-': parts.pop() + # remove trailing zeros from each series of numeric parts + while parts and parts[-1]=='00000000': + parts.pop() + parts.append(part) + return tuple(parts) + +def setuptools_is_new_enough(required_version): + """Return True if setuptools is already installed and has a version + number >= required_version.""" + sub = subprocess.Popen([sys.executable, "-c", "import setuptools;print setuptools.__version__"], stdout=subprocess.PIPE) + verstr = sub.stdout.read().strip() + ver = parse_version(verstr) + return ver and ver >= parse_version(required_version) def use_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, @@ -74,32 +108,11 @@ this routine will print a message to ``sys.stderr`` and raise SystemExit in an attempt to abort the calling script. """ - try: - import setuptools - if setuptools.__version__ == '0.0.1': - print >>sys.stderr, ( - "You have an obsolete version of setuptools installed. Please\n" - "remove it from your system entirely before rerunning this script." - ) - sys.exit(2) - except ImportError: + if not setuptools_is_new_enough(version): egg = download_setuptools(version, download_base, to_dir, download_delay) sys.path.insert(0, egg) import setuptools; setuptools.bootstrap_install_from = egg - import pkg_resources - try: - pkg_resources.require("setuptools>="+version) - - except pkg_resources.VersionConflict, e: - # XXX could we install in a subprocess here? - print >>sys.stderr, ( - "The required version of setuptools (>=%s) is not available, and\n" - "can't be installed while this script is running. Please install\n" - " a more recent version first.\n\n(Currently using %r)" - ) % (version, e.args[0]) - sys.exit(2) - def download_setuptools( version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, delay = 15 @@ -150,9 +163,14 @@ def main(argv, version=DEFAULT_VERSION): """Install or upgrade setuptools and EasyInstall""" - try: - import setuptools - except ImportError: + if setuptools_is_new_enough(version): + if argv: + from setuptools.command.easy_install import main + main(argv) + else: + print "Setuptools version",version,"or greater has been installed." + print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)' + else: egg = None try: egg = download_setuptools(version, delay=0) @@ -162,31 +180,6 @@ finally: if egg and os.path.exists(egg): os.unlink(egg) - else: - if setuptools.__version__ == '0.0.1': - # tell the user to uninstall obsolete version - use_setuptools(version) - - req = "setuptools>="+version - import pkg_resources - try: - pkg_resources.require(req) - except pkg_resources.VersionConflict: - try: - from setuptools.command.easy_install import main - except ImportError: - from easy_install import main - main(list(argv)+[download_setuptools(delay=0)]) - sys.exit(0) # try to force an exit - else: - if argv: - from setuptools.command.easy_install import main - main(argv) - else: - print "Setuptools version",version,"or greater has been installed." - print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)' - - def update_md5(filenames): """Update our built-in md5 registry"""
Okay. Attached is a patch that doesn't import pkg_resources and doesn't reload anything. I haven't yet tested it -- I am submitting it only to inform the discussion and get any early feedback from you on whether the very idea is sound.
Okay, I tested this patch on Mac OS X and it worked. First I installed setuptools v0.6c3, then I tried to execute allmydata.org 's "setup.py install". I got the error message saying that it couldn't install a newer version of setuptools (v0.6c7) when an older version was already loaded. Then I applied my patch to ez_setup.py and tried again. This time it installed v0.6c7. Regards, Zooko
At 04:36 PM 10/2/2007 -0600, zooko wrote:
Okay. Attached is a patch that doesn't import pkg_resources and doesn't reload anything. I haven't yet tested it -- I am submitting it only to inform the discussion and get any early feedback from you on whether the very idea is sound.
Okay, I tested this patch on Mac OS X and it worked. First I installed setuptools v0.6c3, then I tried to execute allmydata.org 's "setup.py install". I got the error message saying that it couldn't install a newer version of setuptools (v0.6c7) when an older version was already loaded. Then I applied my patch to ez_setup.py and tried again. This time it installed v0.6c7.
Interesting. But the patch doesn't actually address the real issue with importing; i.e., upgrading setuptools in-place *while setuptools is already running*. In other words, the test I care about is what happens when you run easy_install on the package, not "setup.py install". Or, to put it more succinctly: if 'pkg_resources' in sys.modules or 'setuptools' in sys.modules: # setuptools is installed, but can't be upgraded, so # just version check (using pkg_resources) and exit if # it's not a good enough version. else: # okay to import pkg_resources and check for version, # as long as afterwards we del sys.modules['pkg_resources'] # and re-import it. So the subprocess and the duplication of code from pkg_resources is unnecessary, since it does not fix the problem when the code is run via easy_install.
So the subprocess and the duplication of code from pkg_resources is unnecessary, since it does not fix the problem when the code is run via easy_install.
Okay, how's this:
Regards,
Zooko
<ez_setup.py> _
Oh, I'm sorry -- that one still has subprocess. Please stay tuned for something closer to what you suggested. Regards, Zooko
Okay, this still uses subprocess but doesn't duplicate parse_version () from pkg_resources. Unlike the previous one that I posted, this one works because it does "import subprocess" at the top... Regards, Zooko
At 01:51 PM 10/3/2007 -0600, zooko wrote:
Okay, this still uses subprocess but doesn't duplicate parse_version () from pkg_resources. Unlike the previous one that I posted, this one works because it does "import subprocess" at the top...
...which is why it won't work with Python 2.3. The subprocess module wasn't added until 2.4. Luckily, the subprocess is unnecessary; it's sufficient to: del sys.modules['pkg_resources'] after upgrading setuptools, as long as it wasn't there to start with.
On Oct 3, 2007, at 5:20 PM, Phillip J. Eby wrote:
del sys.modules['pkg_resources']
after upgrading setuptools, as long as it wasn't there to start with.
There's something about this that I don't understand. How is deleting the name from sys.modules different from reloading it, which as you already explained [1] is not going to work? I can think of two approaches that will work for python >= 2.3: 1. Import setuptools to get a copy of its __version__ attribute, import pkg_resources to use its parse_version() function, then del them from sys.modules. (As above, I don't understand how deleting them from sys.modules and then importing them is different from reloading them.) 2. Use os.system() instead of subprocess() to get the version of the installed setuptools, include a copy of the parse_version() function from pkg_resources. The second approach seems clean and robust to me, and I would be happy to submit a patch that does it if you are interested in seeing one. Regards, Zooko [1] http://mail.python.org/pipermail/distutils-sig/2007-September/ 008309.html
At 03:00 PM 10/4/2007 -0600, zooko wrote:
On Oct 3, 2007, at 5:20 PM, Phillip J. Eby wrote:
del sys.modules['pkg_resources']
after upgrading setuptools, as long as it wasn't there to start with.
There's something about this that I don't understand. How is deleting the name from sys.modules different from reloading it, which as you already explained [1] is not going to work?
This won't work for that either, so that's not why I'm suggesting deleting it. The reason I'm suggesting it for this scenario is to ensure that pkg_resources initializes with clean module contents, which won't happen in a reload().
I can think of two approaches that will work for python >= 2.3:
1. Import setuptools to get a copy of its __version__ attribute,
It's not necessary to import setuptools. Using pkg_resources.require() is sufficient to do the job, and will get the actual version (including dev-r### tags). The only reason ez_setup uses __version__ is to find a *legacy* setuptools (i.e. version 0.0.1) -- and that is the only thing __version__ should be used for.
import pkg_resources to use its parse_version() function, then del them from sys.modules. (As above, I don't understand how deleting them from sys.modules and then importing them is different from reloading them.)
With regard to whether it's safe to do it, there is no difference. If they were in sys.modules to start with, it's not safe to reload() or delete them. If they *weren't* there, then it's okay to reload() or delete them.
2. Use os.system() instead of subprocess() to get the version of the installed setuptools, include a copy of the parse_version() function from pkg_resources.
It's not necessary to do that, because it doesn't help to have another process. If setuptools or pkg_resources are already in sys.modules when a setup.py runs, it's already too late to upgrade safely.
participants (2)
-
Phillip J. Eby -
zooko