Mailman 3 December 2006 - Distutils-SIG

ORed requirements Proof of Concept
by Matt Good Dec. 16, 2006

Dec. 16, 2006

Based on the recent discussions regarding setuptools requirements where more than one package may satisfy a requirement. Setuptools doesn't have a way to say "my package requires either A or B". I've created an inital patch to the setuptools trunk for specifying this as: install_requires = [ "A | B", ] Requirements with multiple candidates like "A | B" will be scanned first for an already installed version, and if one isn't found it will try to install each candidate in order until one is successful. This patch changes the values returned by pkg_resources.parse_requirements. The items yielded by the method will still be Requirement objects for standard requirements, but it will yield a tuple of Requirement objects if there are multiple candidates. It also does not yet handle this scenario ideally: install_requires = [ 'A | B', ], extras_require = { 'foo': 'B', } If the user wants to install the entry point "foo", setuptools should detect that 'B' will be installed which will satisfy the installation requirement 'A | B'. Right now it will still install 'A' to satisfy the installation requirement, then also install 'B' to satisfy the 'foo' extra. -- Matt Good Index: setuptools/package_index.py =================================================================== --- setuptools/package_index.py (revision 53031) +++ setuptools/package_index.py (working copy) @@ -468,6 +468,13 @@ set, development and system eggs (i.e., those using the ``.egg-info`` format) will be ignored. """ + if isinstance(requirement, tuple): + for req in requirement: + dist = self.fetch_distribution(req, tmpdir, force_scan, source, + develop_ok) + if dist is not None: + return dist + return None # process a Requirement self.info("Searching for %s", requirement) Index: setuptools/command/easy_install.py =================================================================== --- setuptools/command/easy_install.py (revision 53031) +++ setuptools/command/easy_install.py (working copy) @@ -414,7 +414,7 @@ if not self.editable: self.install_site_py() try: - if not isinstance(spec,Requirement): + if not isinstance(spec,(Requirement,tuple)): if URL_SCHEME(spec): # It's a url, download it to tmpdir and process self.not_editable(spec) Index: pkg_resources.py =================================================================== --- pkg_resources.py (revision 53031) +++ pkg_resources.py (working copy) @@ -14,6 +14,7 @@ """ import sys, os, zipimport, time, re, imp, new, pkgutil # XXX +from distutils.errors import DistutilsError from sets import ImmutableSet from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open @@ -465,20 +466,20 @@ processed = {} # set of processed requirements best = {} # key -> dist to_activate = [] + env = [env] - while requirements: - req = requirements.pop(0) # process dependencies breadth-first + def process(req, installer): if req in processed: # Ignore cyclic or redundant dependencies - continue + return dist = best.get(req.key) if dist is None: # Find the best distribution and add it to the map dist = self.by_key.get(req.key) if dist is None: - if env is None: - env = Environment(self.entries) - dist = best[req.key] = env.best_match(req, self, installer) + if env[0] is None: + env[0] = Environment(self.entries) + dist = best[req.key] = env[0].best_match(req, self, installer) if dist is None: raise DistributionNotFound(req) # XXX put more info here to_activate.append(dist) @@ -488,6 +489,27 @@ requirements.extend(dist.requires(req.extras)[::-1]) processed[req] = True + while requirements: + req = requirements.pop(0) # process dependencies breadth-first + if isinstance(req, tuple): + found_match = False + # first try without installer to check pre-installed packages + for inst in (None, installer): + for r in req: + try: + process(r, inst) + found_match = True + break + except (DistributionNotFound, VersionConflict, + DistutilsError), e: + pass + if found_match: + break + if not found_match: + raise e + else: + process(req, installer) + return to_activate # return list of distros to activate def find_plugins(self, @@ -1645,6 +1667,8 @@ s = s.strip() if s and not s.startswith('#'): # skip blank lines/comments yield s + elif isinstance(strs,tuple): + yield ' | '.join(strs) else: for ss in strs: for s in yield_lines(ss): @@ -2212,7 +2236,7 @@ if match: p = match.end() # skip the terminator, if any return line, p, items - for line in lines: + def parse_one(line): match = DISTRO(line) if not match: raise ValueError("Missing distribution spec", line) @@ -2229,9 +2253,16 @@ line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec") specs = [(op,safe_version(val)) for op,val in specs] - yield Requirement(project_name, specs, extras) + return Requirement(project_name, specs, extras) + for line in lines: + req = [parse_one(i) for i in line.split('|')] + if len(req)==1: + yield req[0] + else: + yield tuple(req) + def _sort_dists(dists): tmp = [(dist.hashcmp,dist) for dist in dists] tmp.sort()

1 0

Error when trying to create an rpm
by Thomas Dybdahl Ahle Dec. 16, 2006

Dec. 16, 2006

Hi, When I try to create an rpm using distutils, I get an error like this: error: Installed (but unpackaged) file(s) found: /usr/share/games/pychess/sidepanel/bookPanel.pyc /usr/share/games/pychess/sidepanel/bookPanel.pyo ... I found this solution http://www.mail-archive.com/distutils-sig@python.org/msg02655.html which removed most of the lines, but besides my modules, I do also have some .py files in a more plug-in like way, which are not stored in modules. For these files the error still appears. I was wondering if there was a way to not have the files created, but simply not have them checked? I don't understand why it does that.

1 0

warnings while merging eggs with namespace packages
by Bryce Hendrix Dec. 12, 2006

Dec. 12, 2006

Please forgive me if this has been asked before, I thought it had been, but I can't find it on the list now... We've been trying to switch our python distribution over to eggs, but when importing multiple eggs that share the same base namespace, we get tons of warnings "Module %s was already imported from %, but %s is being added to sys.path." I can see where this might be a valid message, so I don't want to filter it out always, but bombarding the user with 40 duplicate warnings when they just start up the interpreter is not acceptable either. Ideally, we'd get this message one time per namespace. Are there any suggestions as how we could implement this behavior? Is this grounds for a feature request? Thanks, Bryce

1 0

Re: [Distutils] setuptools extras_require
by Phillip J. Eby Dec. 12, 2006

Dec. 12, 2006

At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: >Is there any way to exclude packages in extras_require as opposed to >just include them. In Turbogears we have Sqlobject in by default. but >if the extra "future" is used it should drop SO and install sqlalchemy >instead. Not currently, no. Do you have any suggestions as to how this might be spelled in the requirements API?

5 6

SETUPTOOLS - Loading Eggs which do not have an egg_info
by Ilias Lazaridis Dec. 12, 2006

Dec. 12, 2006

I have the following use-case: within an application, i like to load several plugins from an folder, without any user-interaction app/plugins/pluginone app/plugins/plugintwo the plugins (standard sources prepared for setuptools, coming from svn) are arranged like this: pluginone/one pluginone/setup.py #standard setup.py file Is there any documented way to load such a "raw egg" (egg sources without a generated egg-info) into the application, without the need to generate the egg-info? . -- http://lazaridis.com

4 26

trouble with data_files + RPM
by Torsten Bronger Dec. 12, 2006

Dec. 12, 2006

Hallöchen! I try to distribute a Python application with i18n files (.mo). My setup.py says: ... data_files = [('share/locale/de/LC_MESSAGES', ['po/de/felo.mo'])], packages = ['felo'], package_dir = {'felo': 'src'}, The directory structure seen from setup.py is: ./setup.py po/ src/ This works perfectly with the bdist_dumb command. However, when I try to generate an RPM, it says: ... running install_data creating /var/tmp/Felo-1.0-1-buildroot/usr/share creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale/de creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale/de/LC_MESSAGES error: can't copy 'po/de/felo.mo': doesn't exist or not a regular file error: Bad exit status from /var/tmp/rpm-tmp.6388 (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.6388 (%install) error: command 'rpmbuild' failed with exit status 1 What can I do about it? Thank you! Tschö, Torsten. -- Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646 (See http://ime.webhop.org for Jabber, MSN, etc.)

1 0

Re: [Distutils] SetupTools: What Files Were Installed?
by Phillip J. Eby Dec. 11, 2006

Dec. 11, 2006

At 11:34 AM 12/11/2006 -0500, Alexander Michael wrote: >It sounds like it is safe to assume 1, but that 2 is indeterminate (as >easy_install does not leave it's own trace about the user overrides at >the time of installation). It's indeterminate, but verifiable. That is, you can simply look at PATH or other such directories and tell whether a given script came from the egg. > The result being that there is no generic >mechanism to find the files that need to be deleted in order to >uninstall a package without resorting to requiring that the package >was installed with --record=FILENAME. Right; the "nest" command framework will have a mechanism to deal with this; nests will know what files they've installed, and nest will provide complete package management facilities for the files involved.

1 0

Re: [Distutils] SetupTools: What Files Were Installed?
by Phillip J. Eby Dec. 11, 2006

Dec. 11, 2006

At 09:43 AM 12/11/2006 -0500, Alexander Michael wrote: >Currently, I am easy_install'ing with --record=FILENAME set to save >the list of installed filenames in a known location so that I can >later visit it to determine what needs to be removed in order to >uninstall the package. It would be nice to reduce the footprint on >the filesystem by not leaving this trace, but is it possible or even a >good idea? > >Is there a re-usable bit of the SetupTools internals that would allow >me to recreate the contents of this file dynamically on-demand without >reinstalling? Sort of. There's no official API, but the data is more-or-less there. See below. >If not, then in the simple packages that I have been working with, >there are two things that need to be found a deleted: > >1) The egg file or directory in site-packages. >2) Any console or GUI scripts in the Scripts/bin directory. > >I can find the egg file or directory by looking at >pkg_resources.Distribution.location, but how to find the scripts? What >about a non-default script location setting? The scripts' names can be determined from the entry points and the 'scripts' metadata directory. Their link to the specific egg(s) can be confirmed by inspecting the comment lines at the beginning of the script, as easy_install always writes a comment describing the type of script and the egg it came from. So, in principle you can check script directories for scripts of the right name(s) and verify their contents to ensure they are actually associated with the egg you plan to remove, and not just another script with the same name.

2 1

SetupTools: What Files Were Installed?
by Alexander Michael Dec. 11, 2006

Dec. 11, 2006

Currently, I am easy_install'ing with --record=FILENAME set to save the list of installed filenames in a known location so that I can later visit it to determine what needs to be removed in order to uninstall the package. It would be nice to reduce the footprint on the filesystem by not leaving this trace, but is it possible or even a good idea? Is there a re-usable bit of the SetupTools internals that would allow me to recreate the contents of this file dynamically on-demand without reinstalling? If not, then in the simple packages that I have been working with, there are two things that need to be found a deleted: 1) The egg file or directory in site-packages. 2) Any console or GUI scripts in the Scripts/bin directory. I can find the egg file or directory by looking at pkg_resources.Distribution.location, but how to find the scripts? What about a non-default script location setting? Again, maybe it is not even a good idea to attempt to avoid --record=FILENAME. Your thoughts are appreciated. Thanks, Alex

1 0

setuptools extras_require
by Elvelind Grandin Dec. 10, 2006

Dec. 10, 2006

Is there any way to exclude packages in extras_require as opposed to just include them. In Turbogears we have Sqlobject in by default. but if the extra "future" is used it should drop SO and install sqlalchemy instead. -- cheers elvelind grandin

1 0