Mailman 3 February 2006 - Distutils-SIG

setuptools 0.6a10/a9 conflict
by Ian Bicking 14 Feb '06

14 Feb '06

I just installed setuptools 0.6a10 on a host, and that host had a copy of 0.6a9 in site-packages. 0.6a10 got installed into a different directory that had been set up like the Administrator Install instructions (that is, add the directory to the path in sitecustomize). The site-packages (0.6a9) version had a setuptools.pth file pointing to it (in site-packages itself). The new 0.6a10 also had a setuptools.pth file, but in a later path. After that, when I ran easy_install it imported from the 0.6a9 pkg_resources module, not the 0.6a10 version. Once I deleted the old versions it worked again. Presumably it should have given a version error (or just magically worked correctly); it shouldn't have imported from the wrong version. Not sure if setuptools should have caught that, or what. Anyway, data point. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

setuptools 0.6a10 activation
by Ian Bicking 14 Feb '06

14 Feb '06

I had a setup where I had a package "SampleApp". It was installed once normally (as an egg), version 0.0.0dev_r3284. Then I made a checkout, and ran "python setup.py develop" on the checkout. easy-install.pth has /usr/iscape/web/aedemo/src/SampleApp in it, and SampleApp.egg-link points to the same place (with an egg-info directory and whatnot). It was working fine. Then we upgraded to 0.6a10 and the old version became the active version. Confused me a bit; I ran python setup.py develop and it worked again. Actually, now that I think about it, I don't know what easy-install.pth looked like before I re-ran develop. Hrm... and I lost the output of the develop command (if it had any useful information). Anyway, I know it was working before we installed 0.6a10 (whether it should have or not), and broken after. Damn installation problems... hard to reproduce. Anyway, data point at least. If I see it happen again I'll look more closely. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

1 0

setuptools 0.6a10 released
by Phillip J. Eby 14 Feb '06

14 Feb '06

This version is almost entirely devoted to fit-and-finish issues such as improved error messages, better PYTHONPATH, --prefix, and "site" directory support, better SourceForge handling, etc. There are also a few minor new features such as: * The 'find_plugins()' API I described here previously * A few new methods on Environment and Distribution * You can package single-module distributions for EasyInstall by using just a URL: pointing a link to a .py file with a "#egg=project-version" tag on the end will tell EasyInstall to auto-create a setup.py to go with it. Or, if you simply list a .py file as the "Download URL" of a PyPI listing, EasyInstall will automatically figure out the project name and version number, without needing the tag. I'm going to have limited time and net access from now until the weekend, so my responses may not be as quick as they usually are, but please continue to inform me about any issues you may encounter.

1 0

Re: [Distutils] API for finding plugins
by Phillip J. Eby 14 Feb '06

14 Feb '06

At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote: >On 2/7/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > My question at this point is, is this algorithm sane? For example, is it > > reasonable to fall back to older versions of plugins in the case of missing > > dependencies or version conflicts, or would it be better to just disable > > those plugins altogether? I can also imagine that some environments might > > want to have the option of "failing safe" by switching back to a known-good > > prior configuration, or else failing altogether, rather than have arbitrary > > drops or version fallbacks. > >To date, I've just been hunting among the installed packages for >things that satisfy particular entry points. I'm not certain I >understand the intersection between plugins here and entry points. (If >this is already documented in the setuptools documentation, feel free >to say that...) The difference is that plugins of this kind are installed into an application instance-specific directory or directories, rather than to the default sys.path. Chandler plugins, for example, might be installed in a user's profile directory, much like Firefox extensions. The key here is that an application wants to scan its plugin directory or directories and figure out what eggs can safely be added to sys.path, thus making their entry points available. In theory, if the application uses easy_install to set up and configure its plugins, everything would be fine of course, but an application's plugin directory isn't normally a "site" directory; i.e., it's not going to support .pth files and so can't have a "default" version set for each package. Thus, a more dynamic approach is usually needed. Currently, the Zope "basket" product and the Trac plugin system scan a plugin directory in this fashion, looking for suitable eggs and adding them to sys.path. It's possible of course to do this selection by looking for application-specific entry points, and in fact some of these programs do that. Indeed, I started out with that idea in my first draft code for Chandler. What I soon realized, however, is that there is going to be other metadata besides entry points. For example, we're expecting to supply i18n metadata listing message catalogs and other localized resources, so that you can translate one plugin using data in another. A plugin that consists solely of translations isn't going to contain any entry points, so using entry points as a filter isn't a great idea, and neither is scanning eggs for an ever-increasing list of possible metadata that might identify a Chandler plugin. Thus, it became clear to me that I needed to have some way to just activate everything in the plugin directory, and look for metadata and entry points later by scanning the working_set. But "everything" needs to be a consistently-ordered and "guaranteed" usable set of distributions. Thinking over the plugin API some more, at this point I'm leaning towards just adding a 'fallback=True' flag, so that if version fallback is undesirable it can be turned off. Systems wanting to implement an even stricter fallback mechanism (i.e., always fall back to a known working set of plugins if a new set can't be initialized) could then just check the error_info dictionary and resort to their fallback plan if there's anything in it. (The reason why different fallback plans may be desirable is that some applications may have user data that depends on a particular version of a plugin; arbitrary upgrade or downgrade of a plugin could conceivably cause data corruption in such scenarios.)

5 17

More DWIMmy goodness for easy_install
by Phillip J. Eby 13 Feb '06

13 Feb '06

FYI, I just checked in some additional target directory checks for easy_install, based on an idea inspired by a comment from Exarkun on my blog. It seems that a lot of the most recent "new" problems reported with setuptools have to do with site-packages directories under secondary prefixes, like Debian's /usr/local or various 64-bit distros' /usr/lib64. EasyInstall hasn't previously been able to detect that these are valid "site" directories, leading to unnecessary pain for users of these platforms. So, the latest SVN version now contains logic to verify the "site"-ishness of the installation directory, via the crude (but effective) expedient of writing a .pth file to the directory and spawning another Python process to see if the .pth file actually gets processed. This should almost completely eliminate the need for --site-dirs, and along with it, the need to actually read the installation instructions. :) In fact, the only time now that somebody should care about the "custom installation location" docs is if they want to set up a custom "site" directory and *not* use PYTHONPATH to do it. In addition to this new empirical test for site-ness, I've also added writability and existence checks, with hopefully-useful error messages. So we should now be a lot closer to a place where package developers can feel safe using ez_setup as part of their installation process, even if they're very particular about the user experience on a wide variety of platforms. However, testing is appreciated, as always. I'm particularly interested in whether this version can correctly deal with installation targets like /usr/lib64 and Debian's /usr/local/python2.x/site-packages directories, *without* needing to specify site-dirs on the command line or in a config file.

1 0

API for finding plugins
by Phillip J. Eby 11 Feb '06

11 Feb '06

I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: * It should not cause anything to be imported before it actually needs to be used by the application * It should be able to deal gracefully with multiple installed versions of a project in designated plugin directories, falling back to older versions if a newer version's requirements can't be met * It should not actually add anything to sys.path until all plugins have been analyzed. The proposed API would be a 'find_plugins()' method added to the WorkingSet class, as follows: ========== find_plugins(plugin_env, full_env=None) Scan `plugin_env` and identify which distributions could be added to this working set without version conflicts or missing requirements. Example usage:: distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) map(working_set.add, distributions) # add plugins+libs to sys.path print "Couldn't load", errors # display errors The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's "plugin directory" or directories. The `full_env`, if supplied, should be an ``Environment`` instance that contains all usable distributions, *including* those listed in `plugin_env`. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned. This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance. Most applications will use this method mainly on the master ``working_set`` instance in ``pkg_resources``, and then immediately add the returned distributions to the working set so that they are available on sys.path. This will make it possible to find any entry points, and allow any other metadata tracking and hooks to be activated. The resolution algorithm used by ``find_plugins()`` is as follows. First, the project names of the distributions present in `plugin_env` are sorted. Then, each project's eggs are tried in descending version order (i.e., newest version first). An attempt is made to resolve that egg's dependencies. If the attempt is successful, the egg and its dependencies are added to the output list and to a temporary copy of the working set. The process continues with the next project, and no older eggs for that project are tried. If the resolution attempt fails, however, the error is added to the error dictionary and the next older version of the plugin is tried, until a working version is found. Note that this algorithm gives precedence to satisfying the dependencies of alphabetically prior project names in case of version conflicts. If two projects named "AaronsPlugin" and "ZekesPlugin" both need different versions of "TomsLibrary", then "AaronsPlugin" will win and "ZekesPlugin" will be disabled due to version conflict. ========== My question at this point is, is this algorithm sane? For example, is it reasonable to fall back to older versions of plugins in the case of missing dependencies or version conflicts, or would it be better to just disable those plugins altogether? I can also imagine that some environments might want to have the option of "failing safe" by switching back to a known-good prior configuration, or else failing altogether, rather than have arbitrary drops or version fallbacks. Ergo, this API is only a preliminary proposal. I'd like to hear from folks who've tried to implement their own plugin finders, or who would like to have one, as to their thoughts on what kind of fallback approaches seem best. That way, I can refine the API better, and perhaps get the benefit of some of your field experience with previously-tried approaches.

3 5

Script-specific .pth files?
by Jim Fulton 11 Feb '06

11 Feb '06

Here's a nutty idea I thought of when thinking about some of the recent back and forth. I realized, when thinking about "plugins" versus "scripts" is that a key use case is getting a script installed so that it will run. A script comes from an egg, which depends on other eggs, and so on. (Of course, this gets more exciting for complex script, like, say, Zope. :) It occurs to me that, when a script is installed, easy_install could generate a .pth file that named exactly those eggs the script needed. A wrapper script could be generated that used this .pth file. There wouldn't be any need for run-time analysis. The script wouldn't need to manipulate PYTHONPATH to find the needed eggs, as all of the needed eggs would be captured in the script-specific .pth file. Rather than determining the working set at run time, the working set would be determined at install time. I see a number of advantages to this approach: - No complicated run-time path gymnastics - Faster start-up, as there is no run-time searching for eggs or analysis of which eggs to use - More determinstic behavior. An installed script will always use the same eggs. The human who installed the script can inspect the .pth file to see exactly what's being used. They can even modify it if the feel so bold. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 2

DWIM installation with setuptools
by Phillip J. Eby 11 Feb '06

11 Feb '06

Okay, so I've been thinking about all the recent complaints/requests regarding the perils of PYTHONPATH, site-packages, --prefix, and all that. And I'm thinking, is there some way I can work around all this stuff so that installation is totally DWIMmish -- that is, that the install can just "do what I mean", while still allowing a default version of a package to be set. This already works perfectly with root access or a virtual Python, so we can pretty much ignore those setups for this discussion (except to make sure we don't break them). So, mostly what's left to worry about is being able to select an arbitrary installation location, and still have it work. If we assume you're smart enough to pick a location that will already be on your PYTHONPATH (and we could verify that by checking that it already is), then all we have to worry about is making sure that the right version of a package gets added to sys.path at runtime. For scripts this isn't that big of a deal. I could create a specialized bootstrap loader version of pkg_resources.py and dump it alongside any scripts installed by easy_install. It'd be a pain to implement, but I could probably make it work such that sys.path was searched for a setuptools egg, and then have that get used to bootstrap the rest. What this doesn't work for is the interactive interpreter, or code that doesn't use require(). Even if the relevant directory is on sys.path, Python doesn't normally read .pth files in PYTHONPATH directories. I've also learned that many Linux distros appear to hack sys.path in odd ways, and I'm not positive that all of them process .pth files in the directories they add. Certainly they don't support .pth for PYTHONPATH directories. Since it's not just the interactive interpreter, hacking .pythonrc isn't going to help much. That leaves the 'site' and 'sitecustomize' modules. The current PYTHONPATH install support hijacks the 'site' module with a special .pth-honoring feature, so as long as you have the setuptools egg listed directly in PYTHONPATH, you're golden. But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH *before* you can reasonably do anything. Part of the essence of what we desire is that any required steps be done for you *during* the installation rather than requiring an extra step before or afterwards. Unfortunately, this means that hacking 'site' can't work unless we actually install a site.py. Will that work? Well, yes and no. It appears Python will recognize an added site.py that appears in a PYTHONPATH, but many platforms will not recognize a site.py that's in the script directory but not PYTHONPATH. Windows seems to be the only platform where putting a site.py in the script directory has any effect. This is really good news, however. It appears to mean that slapping a copy of my hacked 'site.py' (i.e., the one that honors .pth files along PYTHONPATH) into the easy_install target directory would make PYTHONPATH-based installs essentially DWIMmish. To complete the illusion, I'd need to add some code to script wrappers to process the setuptools.pth file in the script directory if pkg_resources can't be otherwise imported. We could then lobby for the Python 2.5 site.py to behave the same way as my hacked one does: i.e., process .pth files found in PYTHONPATH directories, inserting them ahead of site-packages and its ilk, so that it wouldn't be necessary to install the hacked site.py in such cases. (It's also not necessary to install it in known "site" directories, such as site-packages and any directories listed in --site-dirs. But even if it's installed unnecessarily, it shouldn't hurt anything.) Since 'site' is a stdlib module and isn't normally intended for customization, it shouldn't conflict with any user-installed modules. However, easy_install could detect an existing site.py and warn if its content is different from the one easy_install wants to use. I'm trying to remember why I didn't do this around the time I implemented the original PYTHONPATH support (which first required a hacked 'site' module). I think I was thrown by the lack of cross-platform script directory support for importing 'site', and it didn't occur to me that I could easily work around that from within scripts. Also, it's easy to mentally confuse the script directory and the install directory while thinking about some of this stuff. But as long as the install directory (irrespective of the script directory) is on PYTHONPATH and has a hacked site.py, what's in the script directory doesn't matter. The only case where the script directory matters is if: 1. it's the same directory the libraries were installed to, 2. it's not on PYTHONPATH, and 3. the user puts a script of their own in there and expects it to work anywhere but Windows This edge case could be handled, however, by refusing to install libraries to non-PYTHONPATH directories unless you're on Windows or you use --multi-version. I'm somewhat uncertain about this, though, because it doesn't seem to let you create a self-sufficient application directory. Right now, you can use "easy_install -ad somedir ..." to copy all needed eggs and scripts to the target directory, making a self-contained application directory that has everything it needs to run. OTOH, I suppose if you just made that -mad instead of -ad, it would work just fine for that purpose under the proposed new regime. So, to sum up, here are the changes that would take place: * easy_install would treat all PYTHONPATH-specified directories as though they were listed in --site-dirs, but it would also copy and compile a hacked site.py into them during installation, if they aren't *actually* listed in --site-dirs or in setuptools' default computation of "site" directories. * Currently, easy_install defaults to -m if you install to a non-"site" directory; it would now instead stop and refuse to install to such directories unless you explicitly specify -m. * easy_install could now meaningfully grow a --prefix option, since it will now DWIM as long as the target dir is recognizable as (or convertible to) a "site" directory. And, in the cases where DWIM can't be guaranteed, it will stop and inform the user that they need to fix PYTHONPATH, --site-dirs, or use -m/--multi-version mode. What do y'all think? Do we have a winner yet?

3 6

Re: [Distutils] pkg_resources.find_distributions("./")
by Phillip J. Eby 11 Feb '06

11 Feb '06

At 10:33 AM 2/11/2006 +0100, Elvelind Grandin wrote: >On 2/11/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > At 11:14 PM 2/10/2006 +0100, Elvelind Grandin wrote: > > >On 2/10/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > > > There is no reliable way to tell what distribution you are in, if > all you > > > > have is the directory name. If you have a *module* name, however, > you can > > > > use get_provider(modulename).get_metadata('PKG-INFO') to retrieve the > > > > package's PKG-INFO text, which you can then use to find the > distribution > > > > name and version. > > > > > > > > > >The problem is that the package might not be installed. And > > >get_provides needs it to be right? > > > > The module just needs to be importable. > > > > > >When I run get_provider("xx") it just seems to return an emtpy >DefaultProvider instance. there is no egg_name and no metadata. but I >can import the package. Argh. I forgot about .egg-info distributions; they don't include this data, and if there's more than one of them installed in the same directory (e.g. site-packages) there is currently no way to tell *which* distribution the module is actually from. I'd have to start including some kind of installation manifest in the egg-info and search all of them. This is another fallout from adding system package support that needs to get fixed. For now, I'd do something like: if '.' in modulename: top = modulename.split('.')[0] else: top = modulename candidates = [] for dist in working_set: if not module.__file__.startswith(dist.location): continue if top not in list(dist.get_metadata_lines('top_level.txt')): continue candidates.append(dist) This should give you a list of candidate distributions.

1 0

A prefix option?
by Ben Bangert 11 Feb '06

11 Feb '06

I'm wondering if there's any plans for a prefix option, which functions like the common makefile prefix option for where the lib/ bin dir will then be, etc. My main reason for asking is this blog entry regarding setuptools installs: http://bitworking.org/news/ Please_stop_using_setuptools__at_least_exclusively__for_now____ Cheers, Ben

4 9