EasyInstall: installing non-distutil packages

Looking at some of the packages I'm interested in, there's a couple ones that don't use distutils. Of course, fixing this is best, but I'm wondering if those can be installed as well. Typically you can install these packages just by copying them into place. So... thinking about the best way to do this. I think it should be possible to simply create a kind of generic set of attributes... maybe (very untested)... def setup_args(package_dir): attrs = {} attrs['name'] = os.path.basename(package_dir) dirs = [] os.path.walk(package_dir, (lambda arg, dirpath, names: pkgs.append(dirpath)), None) attrs['packages'] = [ attrs['name'] + '.' + d.replace('/', '.') for d in dirs if os.path.exists(os.path.join(package_dir, d, '__init__.py'))] return attrs Then somehow call distutils.core.setup(**attrs). And somehow package_dir has to get passed in, but I haven't figured out how that happens yet. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

At 05:17 PM 5/29/2005 -0500, Ian Bicking wrote:
Hm. They're going to get a 0.0.0 version number. :( I suppose there could be an option to '--make-setup=version' that would let you automatically create a setup.py using that version number. But that seems rather hacky; I'd rather spend the effort writing a setup.py for the package in a local directory, then submitting it to the author as a patch. Or, if the package is orphaned, building a .egg and distributing it for others' benefit.

Phillip J. Eby wrote:
Eh, crappy metadata is inevitable, and it's an incentive to create a proper distutil script. Having something that works, even if sub-optimal, is good enough for me. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

On May 29, 2005, at 3:17 PM, Ian Bicking wrote:
You should name them, so we can publicly humiliate them into doing the right thing :) The only non-distutils-using package I have used lately is PyKQueue <http://ox.eicat.ca/~dgilbert/files/PyKQueue.html>. I submitted a working setup.py to the maintainer, but he has not posted a new version that uses it. -bob

Bob Ippolito wrote:
I wrote one of them ;) -- obviously I could fix that one, but once I do I lose the opportunity to make sure the tool worked with The World We Have Now. And then there's still two other packages -- the flup servers, and PySourceColor.py, which is distributed as a single .py file (which is one of the more common cases where distutils isn't used). -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

Well, I gave this a first go. I've attached a diff, but I'm not actually advocating that it go in. In this, you have to explicitly give the package name, because it's often not clear from the download location. So here's two examples: python easy_install.py --force-package=flup -d app-packages/ http://svn.saddi.com/flup/trunk/flup python easy_install.py --force-package=Component -d app-packages/ http://svn.w4py.org/Component/trunk Hrm... but I don't really like it. First, force_package is passed into Installer.__init__, and it should be part of the spec. Also, I don't think you always need to explicitly give the package name, sometimes it is self-evident. Also it doesn't work with plain Python files, which can't even be properly downloaded at this point, and for which eggs seem a little excessive. (To give an example of a single file I'd like to be able to install: http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py) I'm thinking maybe a better way to do this would be some sort of patch process, so that I could distribute a setup.py seperately, or some other custom build process, that easy_install could detect and run. And, of course, obviously I *could* fix these upstream, but I think this is a useful problem to solve through easy_install, even if these particular packages get fixed later. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org --- orig/setuptools-0.3a2/easy_install.py 2005-05-29 18:01:42.000000000 -0500 +++ easy_install.py 2005-05-29 21:28:03.357133760 -0500 @@ -151,6 +151,10 @@ directory is on ``sys.path`` at runtime, and to use ``pkg_resources.require()`` to enable the installed package(s) that you need. + +``--force-package=PACKAGE_NAME`` + If the downloaded package doesn't include a ``setup.py`` file, then it + will be installed as a simple package. """ import sys, os.path, pkg_resources, re, zipimport, zipfile, tarfile, shutil @@ -167,7 +171,8 @@ pth_file = None - def __init__(self, instdir=None, zip_ok=False, multi=None, tmpdir=None): + def __init__(self, instdir=None, zip_ok=False, multi=None, tmpdir=None, + force_package=None): if tmpdir is None: from tempfile import mkdtemp @@ -193,9 +198,11 @@ self.instdir = instdir self.zip_ok = zip_ok self.multi = multi + self.force_package = force_package def close(self): if os.path.isdir(self.tmpdir): + return rmtree(self.tmpdir,True) def __del__(self): @@ -258,6 +265,9 @@ setup_script = os.path.join(self.tmpdir, 'setup.py') if not os.path.exists(setup_script): setups = glob(os.path.join(self.tmpdir, '*', 'setup.py')) + if not setups and self.force_package: + self._create_forced_setup(setup_script) + setups = [setup_script] if not setups: raise RuntimeError( "Couldn't find a setup script in %s" % dist_filename @@ -326,6 +336,32 @@ finally: tarobj.close() + def _create_forced_setup(self, filename): + from glob import glob + package_name = self.force_package + f = open(filename, 'w') + f.write('from distutils.core import setup\n\n') + attrs = {} + attrs['name'] = package_name + dirs = [] + package_dirs = glob(os.path.join(self.tmpdir, '*', '__init__.py')) + if not package_dirs: + raise RuntimeError( + "No directories found with __init__.py files") + packages = [os.path.dirname(d[len(self.tmpdir):].lstrip(os.sep)) for d in package_dirs] + pkg_dir_name = None + for package in packages: + pkg_name = package.split('.')[0] + if pkg_dir_name and pkg_name != pkg_dir_name: + raise RuntimeError( + "More than one potential package directory (%r and %r)" + % (pkg_dir_name, pkg_name)) + pkg_dir_name = pkg_name + attrs['packages'] = ['.'.join([package_name] + d.split(os.sep)[1:]) for d in packages] + attrs['package_dir'] = {package_name: pkg_dir_name} + f.write('setup(**%r)' % attrs) + f.close() + def _run_setup(self, setup_script): from setuptools.command import bdist_egg sys.modules.setdefault('distutils.command.bdist_egg', bdist_egg) @@ -547,6 +583,9 @@ parser.add_option("-m", "--multi-version", action="store_true", dest="multi", default=None, help="make apps have to require() a version") + parser.add_option("--force-package", + dest="force_package", default=None, metavar="PACKAGE_NAME", + help="install as a package (with name PACKAGE_NAME) even if setup.py is missing") (options, args) = parser.parse_args() @@ -555,7 +594,8 @@ parser.error("No urls, filenames, or requirements specified") for spec in args: - inst = factory(options.instdir, options.zip_ok, options.multi) + inst = factory(options.instdir, options.zip_ok, options.multi, + force_package=options.force_package) try: print "Downloading", spec downloaded = inst.download(spec)

Ian Bicking wrote:
Maybe this kind of thing shouldn't be installed similarly at all. These files don't have versions and there's no real benefit to turning them into egg files. The benefits are all fake when we make up the metadata. Maybe a simpler way of doing this, besides patching, would just be a kind of shortcut installation. Python source files and package directories just get installed directly (into site-packages or --install-dir). Crude but transitional. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

At 12:20 AM 5/30/2005 -0500, Ian Bicking wrote:
Yes, I've kind of thought about doing that (patches, or special modes/options/whatever), but it seemed just a bit Microsofty, if you know what I mean. :) On the other hand, Microsoft's commitment to backward compatibility has been legendary, and it's not necessarily a bad thing to emulate. It should help us achieve Total World Domination more quickly, which is why I was willing to put in the research time to figure out how to do sandboxing. So, I guess there are two ends of the "backward compatibility" spectrum here: * Low-end: Single .py files and other setup-free distributions * High-end: Distributions that muck about with their installation paths in one way or another I've so far found that there are almost as many ways to hack package data paths as there are packages; through considerable tweaking I've managed to get bdist_egg now to handle more packages without virtualization, but it's a regression-testing nightmare, because all the stuff is order sensitive, and stuff that looks unnecessary is in fact necessary to support different packages' ways of getting at the same information. I think, however, that all of those kludges will have to stick around until Python 2.6, using the Python 2.5 timeframe to warn that 'data_files' should not be used to install data into site-packages, and that 'package_data' is the correct approach. As for me, at the moment I've successfully gotten aquarium, IPython, and numarray to install their data only into their eggs, without any virtualization. So I'll be checking those kludges in momentarily. I have to say that in hindsight, the absence of adequate support for package-data installation in previous versions of the distutils was perhaps its single largest flaw. It appears very rare that anybody ever creates custom distutils commands, unless they needed to in order to work around that problem. As a consequence, it makes it much harder to get bdist_egg to trap such installs and route them to the right place.

At 05:17 PM 5/29/2005 -0500, Ian Bicking wrote:
Hm. They're going to get a 0.0.0 version number. :( I suppose there could be an option to '--make-setup=version' that would let you automatically create a setup.py using that version number. But that seems rather hacky; I'd rather spend the effort writing a setup.py for the package in a local directory, then submitting it to the author as a patch. Or, if the package is orphaned, building a .egg and distributing it for others' benefit.

Phillip J. Eby wrote:
Eh, crappy metadata is inevitable, and it's an incentive to create a proper distutil script. Having something that works, even if sub-optimal, is good enough for me. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

On May 29, 2005, at 3:17 PM, Ian Bicking wrote:
You should name them, so we can publicly humiliate them into doing the right thing :) The only non-distutils-using package I have used lately is PyKQueue <http://ox.eicat.ca/~dgilbert/files/PyKQueue.html>. I submitted a working setup.py to the maintainer, but he has not posted a new version that uses it. -bob

Bob Ippolito wrote:
I wrote one of them ;) -- obviously I could fix that one, but once I do I lose the opportunity to make sure the tool worked with The World We Have Now. And then there's still two other packages -- the flup servers, and PySourceColor.py, which is distributed as a single .py file (which is one of the more common cases where distutils isn't used). -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

Well, I gave this a first go. I've attached a diff, but I'm not actually advocating that it go in. In this, you have to explicitly give the package name, because it's often not clear from the download location. So here's two examples: python easy_install.py --force-package=flup -d app-packages/ http://svn.saddi.com/flup/trunk/flup python easy_install.py --force-package=Component -d app-packages/ http://svn.w4py.org/Component/trunk Hrm... but I don't really like it. First, force_package is passed into Installer.__init__, and it should be part of the spec. Also, I don't think you always need to explicitly give the package name, sometimes it is self-evident. Also it doesn't work with plain Python files, which can't even be properly downloaded at this point, and for which eggs seem a little excessive. (To give an example of a single file I'd like to be able to install: http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py) I'm thinking maybe a better way to do this would be some sort of patch process, so that I could distribute a setup.py seperately, or some other custom build process, that easy_install could detect and run. And, of course, obviously I *could* fix these upstream, but I think this is a useful problem to solve through easy_install, even if these particular packages get fixed later. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org --- orig/setuptools-0.3a2/easy_install.py 2005-05-29 18:01:42.000000000 -0500 +++ easy_install.py 2005-05-29 21:28:03.357133760 -0500 @@ -151,6 +151,10 @@ directory is on ``sys.path`` at runtime, and to use ``pkg_resources.require()`` to enable the installed package(s) that you need. + +``--force-package=PACKAGE_NAME`` + If the downloaded package doesn't include a ``setup.py`` file, then it + will be installed as a simple package. """ import sys, os.path, pkg_resources, re, zipimport, zipfile, tarfile, shutil @@ -167,7 +171,8 @@ pth_file = None - def __init__(self, instdir=None, zip_ok=False, multi=None, tmpdir=None): + def __init__(self, instdir=None, zip_ok=False, multi=None, tmpdir=None, + force_package=None): if tmpdir is None: from tempfile import mkdtemp @@ -193,9 +198,11 @@ self.instdir = instdir self.zip_ok = zip_ok self.multi = multi + self.force_package = force_package def close(self): if os.path.isdir(self.tmpdir): + return rmtree(self.tmpdir,True) def __del__(self): @@ -258,6 +265,9 @@ setup_script = os.path.join(self.tmpdir, 'setup.py') if not os.path.exists(setup_script): setups = glob(os.path.join(self.tmpdir, '*', 'setup.py')) + if not setups and self.force_package: + self._create_forced_setup(setup_script) + setups = [setup_script] if not setups: raise RuntimeError( "Couldn't find a setup script in %s" % dist_filename @@ -326,6 +336,32 @@ finally: tarobj.close() + def _create_forced_setup(self, filename): + from glob import glob + package_name = self.force_package + f = open(filename, 'w') + f.write('from distutils.core import setup\n\n') + attrs = {} + attrs['name'] = package_name + dirs = [] + package_dirs = glob(os.path.join(self.tmpdir, '*', '__init__.py')) + if not package_dirs: + raise RuntimeError( + "No directories found with __init__.py files") + packages = [os.path.dirname(d[len(self.tmpdir):].lstrip(os.sep)) for d in package_dirs] + pkg_dir_name = None + for package in packages: + pkg_name = package.split('.')[0] + if pkg_dir_name and pkg_name != pkg_dir_name: + raise RuntimeError( + "More than one potential package directory (%r and %r)" + % (pkg_dir_name, pkg_name)) + pkg_dir_name = pkg_name + attrs['packages'] = ['.'.join([package_name] + d.split(os.sep)[1:]) for d in packages] + attrs['package_dir'] = {package_name: pkg_dir_name} + f.write('setup(**%r)' % attrs) + f.close() + def _run_setup(self, setup_script): from setuptools.command import bdist_egg sys.modules.setdefault('distutils.command.bdist_egg', bdist_egg) @@ -547,6 +583,9 @@ parser.add_option("-m", "--multi-version", action="store_true", dest="multi", default=None, help="make apps have to require() a version") + parser.add_option("--force-package", + dest="force_package", default=None, metavar="PACKAGE_NAME", + help="install as a package (with name PACKAGE_NAME) even if setup.py is missing") (options, args) = parser.parse_args() @@ -555,7 +594,8 @@ parser.error("No urls, filenames, or requirements specified") for spec in args: - inst = factory(options.instdir, options.zip_ok, options.multi) + inst = factory(options.instdir, options.zip_ok, options.multi, + force_package=options.force_package) try: print "Downloading", spec downloaded = inst.download(spec)

Ian Bicking wrote:
Maybe this kind of thing shouldn't be installed similarly at all. These files don't have versions and there's no real benefit to turning them into egg files. The benefits are all fake when we make up the metadata. Maybe a simpler way of doing this, besides patching, would just be a kind of shortcut installation. Python source files and package directories just get installed directly (into site-packages or --install-dir). Crude but transitional. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

At 12:20 AM 5/30/2005 -0500, Ian Bicking wrote:
Yes, I've kind of thought about doing that (patches, or special modes/options/whatever), but it seemed just a bit Microsofty, if you know what I mean. :) On the other hand, Microsoft's commitment to backward compatibility has been legendary, and it's not necessarily a bad thing to emulate. It should help us achieve Total World Domination more quickly, which is why I was willing to put in the research time to figure out how to do sandboxing. So, I guess there are two ends of the "backward compatibility" spectrum here: * Low-end: Single .py files and other setup-free distributions * High-end: Distributions that muck about with their installation paths in one way or another I've so far found that there are almost as many ways to hack package data paths as there are packages; through considerable tweaking I've managed to get bdist_egg now to handle more packages without virtualization, but it's a regression-testing nightmare, because all the stuff is order sensitive, and stuff that looks unnecessary is in fact necessary to support different packages' ways of getting at the same information. I think, however, that all of those kludges will have to stick around until Python 2.6, using the Python 2.5 timeframe to warn that 'data_files' should not be used to install data into site-packages, and that 'package_data' is the correct approach. As for me, at the moment I've successfully gotten aquarium, IPython, and numarray to install their data only into their eggs, without any virtualization. So I'll be checking those kludges in momentarily. I have to say that in hindsight, the absence of adequate support for package-data installation in previous versions of the distutils was perhaps its single largest flaw. It appears very rare that anybody ever creates custom distutils commands, unless they needed to in order to work around that problem. As a consequence, it makes it much harder to get bdist_egg to trap such installs and route them to the right place.
participants (3)
-
Bob Ippolito
-
Ian Bicking
-
Phillip J. Eby