From owner-freebsd-isp at freebsd.org Wed Feb 1 23:11:45 2006 From: owner-freebsd-isp at freebsd.org (owner-freebsd-isp at freebsd.org) Date: Wed, 01 Feb 2006 14:11:45 -0800 Subject: [Distutils] Your message to freebsd-isp awaits moderator approval Message-ID: Your mail to 'freebsd-isp' with the subject Spam Is being held until the list moderator can review it for approval. The reason it is being held: SpamAssassin identified this message as possible spam Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: http://lists.freebsd.org/mailman/confirm/freebsd-isp/bb6585809095b7ce2f6ebb1d82573318abeb0438 PLEASE NOTE! If you would like to post freely to the list, please subscribe first. If you post from multiple addresses, you can subscribe each address and go into the options page and select 'no mail' for all but one address. This will allow you to post without delay in the future. Sorry for the hassle, but certain immature people made this necessary. From bob at redivi.com Fri Feb 3 23:50:39 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 3 Feb 2006 14:50:39 -0800 Subject: [Distutils] pkg_resources and Mac OS X universal builds Message-ID: <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> Modern Mac OS X applications need to support both the i386 and ppc architectures in what's called a universal build (AKA fat in NeXT- ese). Ronald Oussoren and I are very far along in making python24- maint capable of doing this with patches all over the build procedure and a few smaller patches to distutils itself. One such patch basically expands upon the macosx platform string from pkg_resources, with the additional clause that a fat build of Python (one configured to build multi-architecture extensions) will report its architecture as "fat" rather than "i386" or "ppc". In order for setuptools' pkg_resources to inherit this behavior, I need the attached patch to be accepted. Basically it just skips over the custom behavior if get_platform already returns a sanitized value. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools-r42230-fat.patch Type: application/octet-stream Size: 1308 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060203/b8d185ab/attachment.obj -------------- next part -------------- From pje at telecommunity.com Sat Feb 4 01:03:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Feb 2006 19:03:47 -0500 Subject: [Distutils] pkg_resources and Mac OS X universal builds In-Reply-To: <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> Message-ID: <5.1.1.6.0.20060203190245.042fbd90@mail.telecommunity.com> At 02:50 PM 2/3/2006 -0800, Bob Ippolito wrote: >Modern Mac OS X applications need to support both the i386 and ppc >architectures in what's called a universal build (AKA fat in NeXT- >ese). Ronald Oussoren and I are very far along in making python24- maint >capable of doing this with patches all over the build procedure >and a few smaller patches to distutils itself. One such patch >basically expands upon the macosx platform string from pkg_resources, >with the additional clause that a fat build of Python (one configured >to build multi-architecture extensions) will report its architecture >as "fat" rather than "i386" or "ppc". > >In order for setuptools' pkg_resources to inherit this behavior, I >need the attached patch to be accepted. Basically it just skips over >the custom behavior if get_platform already returns a sanitized value. Implemented in SVN rev 42231. From parlar at gmail.com Sun Feb 5 16:05:03 2006 From: parlar at gmail.com (Jay Parlar) Date: Sun, 5 Feb 2006 07:05:03 -0800 Subject: [Distutils] Twisted and easy_install Message-ID: I'm having trouble doing an install of the newest Twisted from the Cheeseshop, using easy_install. I took a quick look through the archives of distutils-sig, and couldn't find anything. I'm running on OS X 10.3.9, latest version of setuptools (did an 'easy_install -U setuptools' right before I tried installing Twisted). I've pasted the result below. Any thoughts? Searching for twisted Reading http://www.python.org/pypi/twisted/ Couldn't find index page for 'twisted' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://www.python.org/pypi/ Reading http://www.python.org/pypi/Twisted/2.1.0 Reading http://www.twistedmatrix.com Reading http://twistedmatrix.com/projects/core/ Best match: Twisted 2.1.0 Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/2.1/Twisted-2.1.0.tar.bz2 Processing Twisted-2.1.0.tar.bz2 Running Twisted-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ssfbd7/Twisted-2.1.0/egg-dist-tmp--DvGj- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/bin/easy_install", line 7, in ? sys.exit( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 1138, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 230, in run self.easy_install(spec, not self.no_deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 317, in easy_install return self.install_item(spec, download, tmpdir, deps) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 338, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 532, in install_eggs return self.build_and_install(setup_script, setup_base) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 807, in build_and_install self.run_setup(setup_script, setup_base, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 796, in run_setup run_setup(setup_script, args) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 26, in run_setup DirectorySandbox(setup_dir).run( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 63, in run return func() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 29, in {'__file__':setup_script, '__name__':'__main__'} File "setup.py", line 113, in ? File "./twisted/python/dist.py", line 69, in setup File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/bdist_egg.py", line 167, in run self.run_command("egg_info") File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/egg_info.py", line 129, in run writer(self, ep.name, os.path.join(self.egg_info,ep.name)) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/egg_info.py", line 321, in write_toplevel_names pkgs = dict.fromkeys( File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/dist.py", line 630, in iter_distribution_names yield ext.name AttributeError: 'bool' object has no attribute 'name' Jay P. From jim at zope.com Sun Feb 5 16:52:23 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 05 Feb 2006 10:52:23 -0500 Subject: [Distutils] Easy install, making "Traditional" PYTHONPATH-based Installation work better Message-ID: <43E61F37.9040706@zope.com> I'be been learning about setuptools and eggs and am very pleased. There is one area where I'd like to see a small tweak. In: http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations four options are presented for installing outside of Python installation. Naturally, I prefer the the last option, which the documentation recommends against. :) Here's why. An important use case to me is the ability for a developer to work in a development sandbox without affecting anything outside the sandbox. The solution needs to be cross platform. The "Administrator Installation" option doesn't work for me because it requires modifying the Python setup. Neither the "Mac OS X "User" Installation" nor the 'Creating a "Virtual" Python' options work for me because they are not cross platform. I also find the "Virtual Python" approach a bit heavy handed. In playing a bit, I find the "Traditional" PYTHONPATH-based Installation works pretty well except for the problem of having to set the path manually when invoking scripts. This problem could be addressed by having the generated scripts include the custom path settings. I recommend adding two new options to the easy_install section of the configuration file: script_path A list of paths, separated by commas to be added to sys.path by generated scripts. script_pth_files A list of pth files, separated by commas to be added be processed by script files. Generated scripts would include code to manipulate sys.path based on this information. So, for example, in a workspace with a 'lib' directory containing eggs, I'd have: [easy_install] site_dirs = '/here/lib' script_path = '/here/lib' script_pth_files = '/here/lib/easy-install.pth' The script_pth_files option is needed, I think, to avoid hardcoding the setuptools egg path in the generated scripts. Note that it could be argued that the site_dirs option should be enough and that generated scripts should simply reflect this option. Thoughts? For now, I can create a wrapper around easy_install that does custom script generation, which could serve as a prototype for the proposed change. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sun Feb 5 19:22:53 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Feb 2006 13:22:53 -0500 Subject: [Distutils] Twisted and easy_install In-Reply-To: Message-ID: <5.1.1.6.0.20060205132027.0419f970@mail.telecommunity.com> At 07:05 AM 2/5/2006 -0800, Jay Parlar wrote: >I'm having trouble doing an install of the newest Twisted from the >Cheeseshop, using easy_install. I took a quick look through the >archives of distutils-sig, and couldn't find anything. I'm running on >OS X 10.3.9, latest version of setuptools (did an 'easy_install -U >setuptools' right before I tried installing Twisted). I've pasted the >result below. Any thoughts? [...] >AttributeError: 'bool' object has no attribute 'name' Twisted does some interesting mangling of the build_ext process in order to support optional extensions, that are only determined at build time. I've been meaning to add a facility to do this in setuptools, but it's not done yet, and in any case Twisted would have to *use* that facility in order for it to work. I'm afraid Twisted isn't supportable by easy_install at the present time. When I do add support for optional extensions, I plan to look carefully at what Twisted is doing in order to make sure that the facility I add will be able to meet their needs. From pje at telecommunity.com Sun Feb 5 19:55:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Feb 2006 13:55:39 -0500 Subject: [Distutils] Easy install, making "Traditional" PYTHONPATH-based Installation work better In-Reply-To: <43E61F37.9040706@zope.com> Message-ID: <5.1.1.6.0.20060205134200.01b1f5a8@mail.telecommunity.com> At 10:52 AM 2/5/2006 -0500, Jim Fulton wrote: >I'be been learning about setuptools and eggs and am very pleased. >There is one area where I'd like to see a small tweak. > >In: > > >http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations > >four options are presented for installing outside of Python installation. > >Naturally, I prefer the the last option, which the documentation recommends >against. :) Here's why. An important use case to me is the ability for >a developer to work in a development sandbox without affecting anything >outside the sandbox. The solution needs to be cross platform. The >"Administrator Installation" option doesn't work for me because it requires >modifying the Python setup. Neither the "Mac OS X "User" Installation" >nor the 'Creating a "Virtual" Python' options work for me because they >are not cross platform. I also find the "Virtual Python" approach a bit >heavy handed. > >In playing a bit, I find the "Traditional" PYTHONPATH-based Installation >works pretty well except for the problem of having to set the path manually >when invoking scripts. This problem could be addressed by having >the generated scripts include the custom path settings. Unfortunately, that's probably harder than you think. The reason I advise against the PYTHONPATH-based approach is that it uses a sneaky hack to get site directory processing to work "properly" (i.e., as desired). When you use the PYTHONPATH-based approach, it only works if you put the setuptools egg directly on the PYTHONPATH at *Python startup*, because setuptools includes its own version of the 'site' module that bypasses the standard Python 'site' module, so that it can scan .pth files that are *already on* PYTHONPATH. This means that no Python code embedded in the scripts is going to affect this. If you want to use PYTHONPATH, you have to set PYTHONPATH in the environment. By the time the Python script itself is running, sys.path has already been manipulated such that it's no longer possible to process .pth files in a way that gives them precedence over system-installed packages. Now, if you don't care about the .pth files and just want to use whatever the script's require() picks up, then it's not a problem. However, since your post talked about .pth and site-dirs, I assumed that means you want to process that stuff. >For now, I can create a wrapper around easy_install that does >custom script generation, which could serve as a prototype for the >proposed change. Try it out, and see if you can get a privately installed package to *override* a system-installed one, without setting PYTHONPATH in the environment prior to Python startup. I'll be surprised and intrigued if you can find a way. (*Adding* packages this way is of course easy; it's overriding system-installed packages that's the problem, because you have to get the .pth stuff in *front* of site-packages in sys.path, and that normally doesn't happen. It only happens with setuptools because of its hacked 'site.py'.) If you don't care about .pth files, then you can simply install the desired eggs or egg-links to the same directory as the scripts, and PYTHONPATH only needs to include the setuptools egg. It's either that, or perhaps put some bootstrap code in every script to locate setuptools if it's not already on sys.path. From bob at redivi.com Sun Feb 5 20:08:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Sun, 5 Feb 2006 11:08:06 -0800 Subject: [Distutils] Twisted and easy_install In-Reply-To: <5.1.1.6.0.20060205132027.0419f970@mail.telecommunity.com> References: <5.1.1.6.0.20060205132027.0419f970@mail.telecommunity.com> Message-ID: <2FF481C5-F42D-46EB-92F9-827EBECE0F29@redivi.com> On Feb 5, 2006, at 10:22 AM, Phillip J. Eby wrote: > At 07:05 AM 2/5/2006 -0800, Jay Parlar wrote: >> I'm having trouble doing an install of the newest Twisted from the >> Cheeseshop, using easy_install. I took a quick look through the >> archives of distutils-sig, and couldn't find anything. I'm running on >> OS X 10.3.9, latest version of setuptools (did an 'easy_install -U >> setuptools' right before I tried installing Twisted). I've pasted the >> result below. Any thoughts? > [...] >> AttributeError: 'bool' object has no attribute 'name' > > Twisted does some interesting mangling of the build_ext process in > order to > support optional extensions, that are only determined at build > time. I've > been meaning to add a facility to do this in setuptools, but it's > not done > yet, and in any case Twisted would have to *use* that facility in > order for > it to work. I'm afraid Twisted isn't supportable by easy_install > at the > present time. Not only that, but the top-level script for Twisted isn't distutils based. It is a script that invokes several setup.py files for each of the subprojects if given "build" or "install" as options, and otherwise takes totally different arguments. Even if it did install with easy_install it would be a little misleading since Twisted requires Zope.Interface and has a bunch of extras (like SSL or serial support). Twisted really could use a good refactor to support setuptools, but the core team isn't interested in doing that. I believe Zope.Interface is not compatible with setuptools also. I'm not sure why... I think it's the distribution name or something. -bob From parlar at gmail.com Sun Feb 5 20:13:18 2006 From: parlar at gmail.com (Jay Parlar) Date: Sun, 5 Feb 2006 11:13:18 -0800 Subject: [Distutils] Twisted and easy_install In-Reply-To: <2FF481C5-F42D-46EB-92F9-827EBECE0F29@redivi.com> References: <5.1.1.6.0.20060205132027.0419f970@mail.telecommunity.com> <2FF481C5-F42D-46EB-92F9-827EBECE0F29@redivi.com> Message-ID: On 2/5/06, Bob Ippolito wrote: > > On Feb 5, 2006, at 10:22 AM, Phillip J. Eby wrote: > > [snip] > > Twisted does some interesting mangling of the build_ext process in > > order to > > support optional extensions, that are only determined at build > > time. I've > > been meaning to add a facility to do this in setuptools, but it's > > not done > > yet, and in any case Twisted would have to *use* that facility in > > order for > > it to work. I'm afraid Twisted isn't supportable by easy_install > > at the > > present time. > > Not only that, but the top-level script for Twisted isn't distutils > based. It is a script that invokes several setup.py files for each > of the subprojects if given "build" or "install" as options, and > otherwise takes totally different arguments. > > Even if it did install with easy_install it would be a little > misleading since Twisted requires Zope.Interface and has a bunch of > extras (like SSL or serial support). Twisted really could use a good > refactor to support setuptools, but the core team isn't interested in > doing that. > > I believe Zope.Interface is not compatible with setuptools also. I'm > not sure why... I think it's the distribution name or something. > That's more or less what I figured. I mostly wanted to post the error message because it's non-obvious what went wrong, and wanted to make sure someone in the know saw it. I think I still remember how to install packages the old fashioned way, something or other about 'setup.py install' :) Jay P. From jim at zope.com Sun Feb 5 20:16:11 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 05 Feb 2006 14:16:11 -0500 Subject: [Distutils] Easy install, making "Traditional" PYTHONPATH-based Installation work better In-Reply-To: <5.1.1.6.0.20060205134200.01b1f5a8@mail.telecommunity.com> References: <5.1.1.6.0.20060205134200.01b1f5a8@mail.telecommunity.com> Message-ID: <43E64EFB.6080202@zope.com> Phillip J. Eby wrote: > At 10:52 AM 2/5/2006 -0500, Jim Fulton wrote: > >> I'be been learning about setuptools and eggs and am very pleased. >> There is one area where I'd like to see a small tweak. >> >> In: >> >> >> http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations >> >> >> four options are presented for installing outside of Python installation. >> >> Naturally, I prefer the the last option, which the documentation >> recommends >> against. :) Here's why. An important use case to me is the ability for >> a developer to work in a development sandbox without affecting anything >> outside the sandbox. The solution needs to be cross platform. The >> "Administrator Installation" option doesn't work for me because it >> requires >> modifying the Python setup. Neither the "Mac OS X "User" Installation" >> nor the 'Creating a "Virtual" Python' options work for me because they >> are not cross platform. I also find the "Virtual Python" approach a bit >> heavy handed. >> >> In playing a bit, I find the "Traditional" PYTHONPATH-based Installation >> works pretty well except for the problem of having to set the path >> manually >> when invoking scripts. This problem could be addressed by having >> the generated scripts include the custom path settings. > > > Unfortunately, that's probably harder than you think. I doubt it. :) > The reason I > advise against the PYTHONPATH-based approach is that it uses a sneaky > hack to get site directory processing to work "properly" (i.e., as > desired). > > When you use the PYTHONPATH-based approach, it only works if you put the > setuptools egg directly on the PYTHONPATH at *Python startup*, because > setuptools includes its own version of the 'site' module that bypasses > the standard Python 'site' module, so that it can scan .pth files that > are *already on* PYTHONPATH. I realize this. > This means that no Python code embedded in the scripts is going to > affect this. Unless it invokes the setuptools site,py code itself. This is all about getting the path hacked. > If you want to use PYTHONPATH, you have to set PYTHONPATH > in the environment. By the time the Python script itself is running, > sys.path has already been manipulated such that it's no longer possible > to process .pth files in a way that gives them precedence over > system-installed packages. Overriding system-installed packages wasn't a primary goal, although that would be nice. > Now, if you don't care about the .pth files and just want to use > whatever the script's require() picks up, then it's not a problem. Actually, this happens to be the case for me. I plan to always use -m with easy_install. > However, since your post talked about .pth and site-dirs, I assumed that > means you want to process that stuff. The only reason that *I* need to do so is to get at the setuptools installation. Scripts need to get at setuptools and I don't want the scripts to hard code the name if the setuptools egg, which has a version number in it. Of course, if this was to be a general solution, then I think we'd need to deal with the .pth files. > >> For now, I can create a wrapper around easy_install that does >> custom script generation, which could serve as a prototype for the >> proposed change. > > > Try it out, and see if you can get a privately installed package to > *override* a system-installed one, without setting PYTHONPATH in the > environment prior to Python startup. I'll be surprised and intrigued if > you can find a way. (*Adding* packages this way is of course easy; it's > overriding system-installed packages that's the problem, because you > have to get the .pth stuff in *front* of site-packages in sys.path, and > that normally doesn't happen. It only happens with setuptools because > of its hacked 'site.py'.) Well, I could have the generated script code call the hacked site.py from setuptools, so it should work as long as the system packages in question haven't been imported yet, which is good enough for me. > If you don't care about .pth files, then you can simply install the > desired eggs or egg-links to the same directory as the scripts, I don't want to do that, although I can live with it I suppose. > and > PYTHONPATH only needs to include the setuptools egg. It's either that, > or perhaps put some bootstrap code in every script to locate setuptools > if it's not already on sys.path. Again, my plan was to put some bootstrap code in every script that aranged to have the working library and the setuptools egg in the system path. I'll try it and let you know what comes out of it. It's too bad setuptools isn't in the standard library. If it was, I'd probably just stick all my eggs and scripts in the same place and be done ith it. :) It's still tempting. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sun Feb 5 20:42:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Feb 2006 14:42:03 -0500 Subject: [Distutils] Easy install, making "Traditional" PYTHONPATH-based Installation work better In-Reply-To: <43E64EFB.6080202@zope.com> References: <5.1.1.6.0.20060205134200.01b1f5a8@mail.telecommunity.com> <5.1.1.6.0.20060205134200.01b1f5a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060205143717.036bb7f8@mail.telecommunity.com> At 02:16 PM 2/5/2006 -0500, Jim Fulton wrote: >It's too bad setuptools isn't in the standard library. If it was, I'd >probably just stick all my eggs and scripts in the same place and be >done ith it. :) It's still tempting. Note that it's only actually necessary for *pkg_resources* to be importable; you don't have to have all of setuptools. If you installed pkg_resources.py to your target directory, your whole plan would work just fine. By the way, note that if people are developing from source, you probably want to use 'setup.py develop' rather than 'easy_install', since it allows you to edit the code and continue to run. The 'develop' command should work the same with all this, just make sure there's a pkg_resources.py in the target directory when you actually run your scripts. Indeed, given your requirements as I understand them, that should really be all you need. You don't need a custom site-dirs, you need only configure the installation directory (--install-dir) for easy_install and develop. Develop inherits configuration from easy_install, so a ~/.pydistutils.cfg like this: [easy_install] install_dir = wherever should do the trick as long as wherever/pkg_resources.py exists. From bob at redivi.com Sun Feb 5 23:00:20 2006 From: bob at redivi.com (Bob Ippolito) Date: Sun, 5 Feb 2006 14:00:20 -0800 Subject: [Distutils] pkg_resources and Mac OS X compatibility In-Reply-To: <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> References: <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> Message-ID: <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> With the way that we're returning the distutils platform on the universal branch of Mac OS X, we need another patch to pkg_resources. The reason for this is that distutils.util.get_platform() returns the platform that it is trying to produce binaries for, which is often not the exact current platform. More specifically, our current strategy is to produce a version of Python that will build extensions that are compatible with Mac OS X 10.3, but the actual building of extensions requires Mac OS X 10.4 or later (because the toolchain doesn't exist on Mac OS X 10.3). In some rare cases, people will want to produce packages that explicitly require Mac OS X 10.4 or later, which they can do by setting the MACOSX_DEPLOYMENT_TARGET=10.4 environment variable when running setup.py. This will influence the value returned by disutils.util.get_platform(), and will influence the compiler and linker if extensions are built. This patch adds an internal _get_max_platform(plat) function that returns the actual runtime version of Mac OS X, for use in compatible_platforms. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools-r42250-fat.patch Type: application/octet-stream Size: 1239 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060205/ea316c35/attachment.obj From bob at redivi.com Sun Feb 5 23:07:33 2006 From: bob at redivi.com (Bob Ippolito) Date: Sun, 5 Feb 2006 14:07:33 -0800 Subject: [Distutils] [Pythonmac-SIG] pkg_resources and Mac OS X compatibility In-Reply-To: <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> References: <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> Message-ID: <9FF4B86E-D69B-4749-8347-D76A4708E1E2@redivi.com> On Feb 5, 2006, at 2:00 PM, Bob Ippolito wrote: > With the way that we're returning the distutils platform on the > universal branch of Mac OS X, we need another patch to > pkg_resources. The reason for this is that > distutils.util.get_platform() returns the platform that it is > trying to produce binaries for, which is often not the exact > current platform. More specifically, our current strategy is to > produce a version of Python that will build extensions that are > compatible with Mac OS X 10.3, but the actual building of > extensions requires Mac OS X 10.4 or later (because the toolchain > doesn't exist on Mac OS X 10.3). > > In some rare cases, people will want to produce packages that > explicitly require Mac OS X 10.4 or later, which they can do by > setting the MACOSX_DEPLOYMENT_TARGET=10.4 environment variable when > running setup.py. This will influence the value returned by > disutils.util.get_platform(), and will influence the compiler and > linker if extensions are built. > > This patch adds an internal _get_max_platform(plat) function that > returns the actual runtime version of Mac OS X, for use in > compatible_platforms. Oops, wrong patch.. here's the correct one. Sorry about that: -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools-r42250-fat-1.patch Type: application/octet-stream Size: 1250 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060205/1fb218d0/attachment.obj From tlau at tlau.org Tue Feb 7 05:39:21 2006 From: tlau at tlau.org (Tessa Lau) Date: Mon, 06 Feb 2006 20:39:21 -0800 Subject: [Distutils] Swig support Message-ID: <43E82479.3080804@tlau.org> Hi, I'm using distutils with python 2.4.2 on Linux and trying to distribute a complex package including an extension module. I'm using SWIG to wrap the extension module. If I invoke swig normally from the command line with "swig -python notes.i", then swig generates a "notes.py" wrapper as well as a "notes_wrap.c" source code file. It expects this source file to be compiled into a module named "_notesmodule.so" (on Linux). I tried recreating the same functionality with distutils, using the following setup.py code: ext_modules = [Extension('mypkg.notes', ['wrapper/notes.i'], include_dirs=['include'], swig_opts=['-Iinclude'])] What happens is that distutils converts "notes.i" into "notes.so" (which cannot be imported as a module) and fails to install the generated "notes.py" wrapper script, which is also required for the module to work. The "notes.so" built by distutils cannot be imported by python because the swig-generated C source code declares the module name to be "_notes", and this doesn't correspond to the name of the created shared library. This is similar to the shadow class behavior found in earlier releases of swig, but now it's the default behavior. Is this supposed to work? Am I doing something wrong, or is SWIG support just broken in this release? Thanks, --Tessa From pje at telecommunity.com Tue Feb 7 07:09:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 01:09:14 -0500 Subject: [Distutils] API for finding plugins Message-ID: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: * It should not cause anything to be imported before it actually needs to be used by the application * It should be able to deal gracefully with multiple installed versions of a project in designated plugin directories, falling back to older versions if a newer version's requirements can't be met * It should not actually add anything to sys.path until all plugins have been analyzed. The proposed API would be a 'find_plugins()' method added to the WorkingSet class, as follows: ========== find_plugins(plugin_env, full_env=None) Scan `plugin_env` and identify which distributions could be added to this working set without version conflicts or missing requirements. Example usage:: distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) map(working_set.add, distributions) # add plugins+libs to sys.path print "Couldn't load", errors # display errors The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's "plugin directory" or directories. The `full_env`, if supplied, should be an ``Environment`` instance that contains all usable distributions, *including* those listed in `plugin_env`. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned. This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance. Most applications will use this method mainly on the master ``working_set`` instance in ``pkg_resources``, and then immediately add the returned distributions to the working set so that they are available on sys.path. This will make it possible to find any entry points, and allow any other metadata tracking and hooks to be activated. The resolution algorithm used by ``find_plugins()`` is as follows. First, the project names of the distributions present in `plugin_env` are sorted. Then, each project's eggs are tried in descending version order (i.e., newest version first). An attempt is made to resolve that egg's dependencies. If the attempt is successful, the egg and its dependencies are added to the output list and to a temporary copy of the working set. The process continues with the next project, and no older eggs for that project are tried. If the resolution attempt fails, however, the error is added to the error dictionary and the next older version of the plugin is tried, until a working version is found. Note that this algorithm gives precedence to satisfying the dependencies of alphabetically prior project names in case of version conflicts. If two projects named "AaronsPlugin" and "ZekesPlugin" both need different versions of "TomsLibrary", then "AaronsPlugin" will win and "ZekesPlugin" will be disabled due to version conflict. ========== My question at this point is, is this algorithm sane? For example, is it reasonable to fall back to older versions of plugins in the case of missing dependencies or version conflicts, or would it be better to just disable those plugins altogether? I can also imagine that some environments might want to have the option of "failing safe" by switching back to a known-good prior configuration, or else failing altogether, rather than have arbitrary drops or version fallbacks. Ergo, this API is only a preliminary proposal. I'd like to hear from folks who've tried to implement their own plugin finders, or who would like to have one, as to their thoughts on what kind of fallback approaches seem best. That way, I can refine the API better, and perhaps get the benefit of some of your field experience with previously-tried approaches. From bob at redivi.com Tue Feb 7 08:47:04 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 6 Feb 2006 23:47:04 -0800 Subject: [Distutils] Supporting arch specific eggs with a fat Python and setuptools Message-ID: <3B1DD8F2-BE3D-4186-BCED-C8D8D23D96F1@redivi.com> With the recent introduction of i386 architecture Macs, it's becoming a necessity to support two architectures for the platform: PPC and i386. Fortunately this is done somewhat easily using Apple's GCC 4 compiler and their linker toolchain. Ronald and I have already basically made all of the necessary changes to Python (in a branch on another svn server) and distutils in order to make Python compile with multiple architectures, and to make distutils also do the right thing. This solution works great (in theory and limited testing) for everyone except Mac OS X 10.3 users. We have gone out of our way to ensure that this build of Python is compatible with Mac OS X 10.3.9 so that those users aren't totally left in the dark, but there is a problem: the GCC 4 toolchain required to build extensions is not available to Mac OS X 10.3 users. They simply can not build binaries compatible with i386.. and if they could, they couldn't link it together as a universal binary. Bummer. The workaround for this issue is to have a way to swap out the Makefile with a PPC-only Makefile for those users. The problem with that is setuptools. Currently setuptools doesn't include a lot of intelligence for choosing eggs given multiple choices. Here's a scenario of various Python builds and the naming of eggs they produce: Mac OS X 10.4 with current means: pysqlite-2.0.5-py2.4-macosx-10.4-ppc.egg Mac OS X 10.3 with current means, or with the universal build and a swapped Makefile: pysqlite-2.0.5-py2.4-macosx-10.3-ppc.egg Mac OS X 10.4 with the universal build: pysqlite-2.0.5-py2.4-macosx-10.3-fat.egg The decision of which egg to choose is subjective, but the greatest benefit overall is achieved by choosing the maximally compatible version: pysqlite-2.0.5-py2.4-macosx-10.3-fat.egg For users of Mac OS X, here is the egg-sorting algorithm I'm proposing. This sort must be used only with eggs that are compatible with the current architecture -- which means either "fat" or _macosx_arch(os.uname()[4].replace(' ', '_')). Also allowing _macosx_arch and universally allowing fat are going to require another patch to setuptools (that I have not yet written, because it's not the right thing to do until egg-sorting is right). # a and b must only differ in platform specifics! def cmp_platform_eggs(a, b): if a.arch == "fat" and b.arch == "fat": pass elif a.arch == "fat": return 1 elif b.arch == "fat": return -1 # lower version sorts higher return -cmp(a.macosx_version, b.macosx_version) This sort means that fat will always be preferred, and lower requirements sort higher. This makes it easier for people who wish to redistribute self-contained applications to other users (a la py2app, py2exe, cx_Freeze), which is a very important use case for this platform. -bob From dangoor at gmail.com Tue Feb 7 12:59:15 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 7 Feb 2006 06:59:15 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> Message-ID: <3f085ecd0602070359l36b01ac7r1fcff6c3c7a16b7c@mail.gmail.com> On 2/7/06, Phillip J. Eby wrote: > My question at this point is, is this algorithm sane? For example, is it > reasonable to fall back to older versions of plugins in the case of missing > dependencies or version conflicts, or would it be better to just disable > those plugins altogether? I can also imagine that some environments might > want to have the option of "failing safe" by switching back to a known-good > prior configuration, or else failing altogether, rather than have arbitrary > drops or version fallbacks. To date, I've just been hunting among the installed packages for things that satisfy particular entry points. I'm not certain I understand the intersection between plugins here and entry points. (If this is already documented in the setuptools documentation, feel free to say that...) Kevin From pje at telecommunity.com Tue Feb 7 14:11:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 08:11:00 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <3f085ecd0602070359l36b01ac7r1fcff6c3c7a16b7c@mail.gmail.co m> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote: >On 2/7/06, Phillip J. Eby wrote: > > My question at this point is, is this algorithm sane? For example, is it > > reasonable to fall back to older versions of plugins in the case of missing > > dependencies or version conflicts, or would it be better to just disable > > those plugins altogether? I can also imagine that some environments might > > want to have the option of "failing safe" by switching back to a known-good > > prior configuration, or else failing altogether, rather than have arbitrary > > drops or version fallbacks. > >To date, I've just been hunting among the installed packages for >things that satisfy particular entry points. I'm not certain I >understand the intersection between plugins here and entry points. (If >this is already documented in the setuptools documentation, feel free >to say that...) The difference is that plugins of this kind are installed into an application instance-specific directory or directories, rather than to the default sys.path. Chandler plugins, for example, might be installed in a user's profile directory, much like Firefox extensions. The key here is that an application wants to scan its plugin directory or directories and figure out what eggs can safely be added to sys.path, thus making their entry points available. In theory, if the application uses easy_install to set up and configure its plugins, everything would be fine of course, but an application's plugin directory isn't normally a "site" directory; i.e., it's not going to support .pth files and so can't have a "default" version set for each package. Thus, a more dynamic approach is usually needed. Currently, the Zope "basket" product and the Trac plugin system scan a plugin directory in this fashion, looking for suitable eggs and adding them to sys.path. It's possible of course to do this selection by looking for application-specific entry points, and in fact some of these programs do that. Indeed, I started out with that idea in my first draft code for Chandler. What I soon realized, however, is that there is going to be other metadata besides entry points. For example, we're expecting to supply i18n metadata listing message catalogs and other localized resources, so that you can translate one plugin using data in another. A plugin that consists solely of translations isn't going to contain any entry points, so using entry points as a filter isn't a great idea, and neither is scanning eggs for an ever-increasing list of possible metadata that might identify a Chandler plugin. Thus, it became clear to me that I needed to have some way to just activate everything in the plugin directory, and look for metadata and entry points later by scanning the working_set. But "everything" needs to be a consistently-ordered and "guaranteed" usable set of distributions. Thinking over the plugin API some more, at this point I'm leaning towards just adding a 'fallback=True' flag, so that if version fallback is undesirable it can be turned off. Systems wanting to implement an even stricter fallback mechanism (i.e., always fall back to a known working set of plugins if a new set can't be initialized) could then just check the error_info dictionary and resort to their fallback plan if there's anything in it. (The reason why different fallback plans may be desirable is that some applications may have user data that depends on a particular version of a plugin; arbitrary upgrade or downgrade of a plugin could conceivably cause data corruption in such scenarios.) From pje at telecommunity.com Tue Feb 7 14:37:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 08:37:11 -0500 Subject: [Distutils] [Pythonmac-SIG] pkg_resources and Mac OS X compatibility In-Reply-To: <9FF4B86E-D69B-4749-8347-D76A4708E1E2@redivi.com> References: <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> Message-ID: <5.1.1.6.0.20060207083254.038944b8@mail.telecommunity.com> At 02:07 PM 2/5/2006 -0800, Bob Ippolito wrote: >On Feb 5, 2006, at 2:00 PM, Bob Ippolito wrote: >>With the way that we're returning the distutils platform on the >>universal branch of Mac OS X, we need another patch to >>pkg_resources. The reason for this is that >>distutils.util.get_platform() returns the platform that it is >>trying to produce binaries for, which is often not the exact >>current platform. More specifically, our current strategy is to >>produce a version of Python that will build extensions that are >>compatible with Mac OS X 10.3, but the actual building of >>extensions requires Mac OS X 10.4 or later (because the toolchain >>doesn't exist on Mac OS X 10.3). >> >>In some rare cases, people will want to produce packages that >>explicitly require Mac OS X 10.4 or later, which they can do by >>setting the MACOSX_DEPLOYMENT_TARGET=10.4 environment variable when >>running setup.py. This will influence the value returned by >>disutils.util.get_platform(), and will influence the compiler and >>linker if extensions are built. >> >>This patch adds an internal _get_max_platform(plat) function that >>returns the actual runtime version of Mac OS X, for use in >>compatible_platforms. > >Oops, wrong patch.. here's the correct one. Sorry about that: I've implemented a similar - but different - patch. Yours causes setuptools' tests to fail on non-Mac platforms, including non-Mac darwin. From pje at telecommunity.com Tue Feb 7 14:43:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 08:43:05 -0500 Subject: [Distutils] Supporting arch specific eggs with a fat Python and setuptools In-Reply-To: <3B1DD8F2-BE3D-4186-BCED-C8D8D23D96F1@redivi.com> Message-ID: <5.1.1.6.0.20060207083812.0389aa60@mail.telecommunity.com> At 11:47 PM 2/6/2006 -0800, Bob Ippolito wrote: >This sort must be used only with eggs that are compatible >with the current architecture -- which means either "fat" or >_macosx_arch(os.uname()[4].replace(' ', '_')). Also allowing >_macosx_arch and universally allowing fat are going to require >another patch to setuptools (that I have not yet written, because >it's not the right thing to do until egg-sorting is right). > > # a and b must only differ in platform specifics! > def cmp_platform_eggs(a, b): > if a.arch == "fat" and b.arch == "fat": > pass > elif a.arch == "fat": > return 1 > elif b.arch == "fat": > return -1 > # lower version sorts higher > return -cmp(a.macosx_version, b.macosx_version) > >This sort means that fat will always be preferred, and lower >requirements sort higher. This makes it easier for people who wish >to redistribute self-contained applications to other users (a la >py2app, py2exe, cx_Freeze), which is a very important use case for >this platform. The way I'd recommend implementing this is via a _platform_key() function that takes a platform string and converts it to something that sorts the way you want it to; higher values are preferred to lower values. Note that pkg_resources sorts distributions by version and other precedence information first; platform is the very *last* thing considered in a sort. It also can't be guaranteed that you'll be comparing your custom object with a common platform. So please try to write the patch with these constraints in mind. On the line that's currently line 1758 of pkg_resources.py, you'll see a reference to 'self.platform'; change that to '_platform_key(self.platform)' once you've got your precedence algorithm figured out. Note also that self.platform can be None, so _platform_key should not assume its parameter is a string. From dangoor at gmail.com Tue Feb 7 15:41:48 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 7 Feb 2006 09:41:48 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> Message-ID: <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> On 2/7/06, Phillip J. Eby wrote: > At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote: > >To date, I've just been hunting among the installed packages for > >things that satisfy particular entry points. I'm not certain I > >understand the intersection between plugins here and entry points. (If > >this is already documented in the setuptools documentation, feel free > >to say that...) > > The difference is that plugins of this kind are installed into an > application instance-specific directory or directories, rather than to the > default sys.path. Chandler plugins, for example, might be installed in a > user's profile directory, much like Firefox extensions. > > The key here is that an application wants to scan its plugin directory or > directories and figure out what eggs can safely be added to sys.path, thus > making their entry points available. In theory, if the application uses > easy_install to set up and configure its plugins, everything would be fine > of course, but an application's plugin directory isn't normally a "site" > directory; i.e., it's not going to support .pth files and so can't have a > "default" version set for each package. Ahh, OK. Your explanation makes the difference clear. I'm pretty sure that TurboGears doesn't need plugins (in this terminology) yet, but I could imagine that some TurboGears apps may like to use plugins (much like Trac does). Kevin From ianb at colorstudy.com Tue Feb 7 18:31:25 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Feb 2006 11:31:25 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> Message-ID: <43E8D96D.4030603@colorstudy.com> Kevin Dangoor wrote: > On 2/7/06, Phillip J. Eby wrote: > >>At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote: >> >>>To date, I've just been hunting among the installed packages for >>>things that satisfy particular entry points. I'm not certain I >>>understand the intersection between plugins here and entry points. (If >>>this is already documented in the setuptools documentation, feel free >>>to say that...) >> >>The difference is that plugins of this kind are installed into an >>application instance-specific directory or directories, rather than to the >>default sys.path. Chandler plugins, for example, might be installed in a >>user's profile directory, much like Firefox extensions. >> >>The key here is that an application wants to scan its plugin directory or >>directories and figure out what eggs can safely be added to sys.path, thus >>making their entry points available. In theory, if the application uses >>easy_install to set up and configure its plugins, everything would be fine >>of course, but an application's plugin directory isn't normally a "site" >>directory; i.e., it's not going to support .pth files and so can't have a >>"default" version set for each package. > > > Ahh, OK. Your explanation makes the difference clear. I'm pretty sure > that TurboGears doesn't need plugins (in this terminology) yet, but I > could imagine that some TurboGears apps may like to use plugins (much > like Trac does). I actually think there's some intersection between what you and I are doing, and what Trac and Chandler are doing. In that, I'm a little worried that what we're doing is going to have scaling problems once a rich set of plugins is available. The opt-in nature of a plugin directory solves some of these problems; and much of what Phillip is describing (in terms of being able to scan for lots of kinds of content and other things provided by plugins) does not seem so far out of what I am doing with eggs and entry points. *Except*, that with what we're doing there's not an ownership relationship. If Paste and TG provide things to each other, one of them doesn't own the other -- TG wouldn't go in Paste's plugin dir, nor vice versa. I think even if the packages were refactored to be more granular, it still wouldn't work out that way. At the same time, I feel like opt-in plugins -- explicitly enumerated in some way -- are going to be necessary for us. Right now if an egg is not activated by default, you won't see its entry points when scanning by entry point group. This has confused me before, which I figure is a good predictor that other people are going to be confused by this too. Right now in Paste there's a couple ways you opt-in. One is with middleware, which doesn't get picked up implicitly, but instead gets specifically activated and configured. The other is with PasteScript plugins, where a list of applicable plugins is put in your .egg-info directory for the project. Those generally work pretty well. But I'm not sure how to apply that in other areas, like PasteScript commands that aren't project-local. I also worry -- but have not actually profiled -- that the scanning is causing Paste Script to be slow on startup (Paste might be too, but I wouldn't notice as much). It would be good to look into this more closely; but it would also be really good to catch any problems now -- and fix that -- instead of later when it might lead to people bitching about it and talking down setuptools as a result. I also want content plugins, like the internationalization plugins (but it might be Javascript or templates or whatever). And certainly that applies to TG as well. I've been doing this at work, but using entry points, where the entry points points to an object that is a string that points to a resource that is then fetched with pkg_resources. Something more direct would be appreciated ;) I like entry point groups, and applying the same idea to resources (but without the indirection) would be great. Hmm... and reading over this, I realize that a "plugin directory" is not that different from the virtual python setups, and I think that kind of setup is really the only good way to deploy web applications. "Working set" -- which is already an idea in setuptools -- is probably the concept that encompasses both uses. As a developer, I'd like it to be easier and safer to change my working set -- right now I have hacks in sitecustomize and distutils to make that easy. This switching of working sets is basically what Jim wants in a Zope Instance as well. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Feb 7 19:17:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 13:17:10 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43E8D96D.4030603@colorstudy.com> References: <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> Message-ID: <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> At 11:31 AM 2/7/2006 -0600, Ian Bicking wrote: >At the same time, I feel like opt-in plugins -- explicitly enumerated in >some way -- are going to be necessary for us. Right now if an egg is not >activated by default, you won't see its entry points when scanning by >entry point group. This has confused me before, which I figure is a good >predictor that other people are going to be confused by this too. Others have been confused too, but it's also quite easy to understand once you've bumped into it once or twice. Especially since you really don't want the entry points for *every* version of a project to show up in the list. Explicit activation on sys.path is the poor man's global "plugin directory", so now I'm finally following up on having a clean way to have a *real* plugin directory. >Right now in Paste there's a couple ways you opt-in. One is with >middleware, which doesn't get picked up implicitly, but instead gets >specifically activated and configured. The other is with PasteScript >plugins, where a list of applicable plugins is put in your .egg-info >directory for the project. Those generally work pretty well. But I'm not >sure how to apply that in other areas, like PasteScript commands that >aren't project-local. Configuration files, perhaps? >I also worry -- but have not actually profiled -- that the scanning is >causing Paste Script to be slow on startup (Paste might be too, but I >wouldn't notice as much). It would be good to look into this more >closely; but it would also be really good to catch any problems now -- and >fix that -- instead of later when it might lead to people bitching about >it and talking down setuptools as a result. FYI, the best way to minimize scan overhead is to avoid repeatedly creating Environment instances; this is the slowest thing you can possibly do. Note that most pkg_resources APIs were designed to be called once at startup and that's it, so they usually are able to create a default Environment if one isn't passed in. However, if you have an API that gets called "at runtime" and it invokes pkg_resources APIs that take Environment instances, you may want to create one early and stash it away for future use. This prevents the most expensive scanning from taking place repeatedly. Note that Distribution objects cache their entry point information, so looking for entry points repeatedly on the same WorkingSet doesn't do any disk I/O. >I also want content plugins, like the internationalization plugins (but it >might be Javascript or templates or whatever). And certainly that applies >to TG as well. I've been doing this at work, but using entry points, >where the entry points points to an object that is a string that points to >a resource that is then fetched with pkg_resources. Something more direct >would be appreciated ;) Um, you do realize that this is *exactly* what I was campaigning for on the Web-SIG when I said, "we need a resource deployment standard", don't you? :) This is relevant for me for Chandler too; I'd like for us to be able to release our coming egg translations and resources library as an implementation of a standard, rather than as a Chandler-only thing. I suppose it technically goes beyond the Web-SIG at that point and should maybe go to i18n-SIG, or maybe here if distutils/setuptools is part of it. OTOH, web framework and template system authors are among the most important stakeholders for such an effort, since they are the most readily identified cases of existing systems that would need to implement such a standard. However, it may be that it's a big enough discussion with a diverse enough set of stakeholders that it deserves its own (hopefully time limited) SIG. Anyway, I have more cycles that I can devote to a resource deployment standard than I can to a templating standard; the former has direct impact on my "day job" work, and the latter does not. >I like entry point groups, and applying the same idea to resources (but >without the indirection) would be great. Again, this is precisely what I wanted to do with a resource deployment standard. (And note also that such a standard would make a more standardized approach to the find_template issue much more possible/practical -- another reason I thought we should tackle deployment before rendering.) >Hmm... and reading over this, I realize that a "plugin directory" is not >that different from the virtual python setups, and I think that kind of >setup is really the only good way to deploy web applications. "Working >set" -- which is already an idea in setuptools -- is probably the concept >that encompasses both uses. As a developer, I'd like it to be easier and >safer to change my working set -- right now I have hacks in sitecustomize >and distutils to make that easy. This switching of working sets is >basically what Jim wants in a Zope Instance as well. Ideas welcome, especially ones with working implementations. ;) From ianb at colorstudy.com Tue Feb 7 19:43:04 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Feb 2006 12:43:04 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> References: <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> Message-ID: <43E8EA38.5020500@colorstudy.com> Phillip J. Eby wrote: > At 11:31 AM 2/7/2006 -0600, Ian Bicking wrote: > >> At the same time, I feel like opt-in plugins -- explicitly enumerated >> in some way -- are going to be necessary for us. Right now if an egg >> is not activated by default, you won't see its entry points when >> scanning by entry point group. This has confused me before, which I >> figure is a good predictor that other people are going to be confused >> by this too. > > > Others have been confused too, but it's also quite easy to understand > once you've bumped into it once or twice. Especially since you really > don't want the entry points for *every* version of a project to show up > in the list. Explicit activation on sys.path is the poor man's global > "plugin directory", so now I'm finally following up on having a clean > way to have a *real* plugin directory. Indeed -- and I'm thinking we all need to go that way, because the poor man's plugin directory is already showing some problems. >> Right now in Paste there's a couple ways you opt-in. One is with >> middleware, which doesn't get picked up implicitly, but instead gets >> specifically activated and configured. The other is with PasteScript >> plugins, where a list of applicable plugins is put in your .egg-info >> directory for the project. Those generally work pretty well. But I'm >> not sure how to apply that in other areas, like PasteScript commands >> that aren't project-local. > > > Configuration files, perhaps? Yes, but I've avoiding those so far. I'd have to locate them (in /etc/paster/*?), and then provide options to specify them, and so forth. It will probably be necessary... but I feel like there's something useful in defering that too. But maybe that is irrational. >> I also worry -- but have not actually profiled -- that the scanning is >> causing Paste Script to be slow on startup (Paste might be too, but I >> wouldn't notice as much). It would be good to look into this more >> closely; but it would also be really good to catch any problems now -- >> and fix that -- instead of later when it might lead to people bitching >> about it and talking down setuptools as a result. > > > FYI, the best way to minimize scan overhead is to avoid repeatedly > creating Environment instances; this is the slowest thing you can > possibly do. Note that most pkg_resources APIs were designed to be > called once at startup and that's it, so they usually are able to create > a default Environment if one isn't passed in. However, if you have an > API that gets called "at runtime" and it invokes pkg_resources APIs that > take Environment instances, you may want to create one early and stash > it away for future use. This prevents the most expensive scanning from > taking place repeatedly. I guess the problem is that for Paste Script, the startup overhead and the runtime overhead are the same thing. Like, 8 seconds to complete "paster help" (which gives a listing of all available commands). After the first run it warms up and is much faster (disk caching?), but that still doesn't feel good to me. Now that it's warmed up I can't tell how slow a normal command would be -- possible faster, since "help" necessarily looks at all packages. > Note that Distribution objects cache their entry point information, so > looking for entry points repeatedly on the same WorkingSet doesn't do > any disk I/O. > > >> I also want content plugins, like the internationalization plugins >> (but it might be Javascript or templates or whatever). And certainly >> that applies to TG as well. I've been doing this at work, but using >> entry points, where the entry points points to an object that is a >> string that points to a resource that is then fetched with >> pkg_resources. Something more direct would be appreciated ;) > > > Um, you do realize that this is *exactly* what I was campaigning for on > the Web-SIG when I said, "we need a resource deployment standard", don't > you? :) Yes, and I completely agreed with you! Though I think it's a little larger of a problem than can be solved with eggs alone, unless everyone creates eggs for all their content, even instance-specific content. The developer overhead for that seems too large, since much of the content won't be produced by developers. It's also something that isn't even Python specific (kind of by definition -- this is what distinguishes content from code). > This is relevant for me for Chandler too; I'd like for us to > be able to release our coming egg translations and resources library as > an implementation of a standard, rather than as a Chandler-only thing. > I suppose it technically goes beyond the Web-SIG at that point and > should maybe go to i18n-SIG, or maybe here if distutils/setuptools is > part of it. OTOH, web framework and template system authors are among > the most important stakeholders for such an effort, since they are the > most readily identified cases of existing systems that would need to > implement such a standard. > > However, it may be that it's a big enough discussion with a diverse > enough set of stakeholders that it deserves its own (hopefully time > limited) SIG. Anyway, I have more cycles that I can devote to a > resource deployment standard than I can to a templating standard; the > former has direct impact on my "day job" work, and the latter does not. Right now I suspect many possible stakeholders won't see how it applies to their cases, or feel ready to take that discussion on. It's still too abstract, unless perhaps it can be phrased in terms of some existing standard (which may not be a Python-specific standard at all). Kind of like how WSGI was phrased in terms of CGI. I am not familiar enough with this stuff to have any idea what such standards exist. Seems like internationalization is the most common such case, but even there people aren't utilizing that functionality to the degree they could (e.g., internationalization should provide hooks for customizing the text of an application, but it isn't used for that very often). >> I like entry point groups, and applying the same idea to resources >> (but without the indirection) would be great. > > > Again, this is precisely what I wanted to do with a resource deployment > standard. (And note also that such a standard would make a more > standardized approach to the find_template issue much more > possible/practical -- another reason I thought we should tackle > deployment before rendering.) My hope is that find_template can be implemented with such a resource deployment standard, even if the template stuff happens first. And if that is possible, that would give the resource deployment standard a kind of instance applicability to a real problem, which is a bonus. >> Hmm... and reading over this, I realize that a "plugin directory" is >> not that different from the virtual python setups, and I think that >> kind of setup is really the only good way to deploy web applications. >> "Working set" -- which is already an idea in setuptools -- is probably >> the concept that encompasses both uses. As a developer, I'd like it >> to be easier and safer to change my working set -- right now I have >> hacks in sitecustomize and distutils to make that easy. This >> switching of working sets is basically what Jim wants in a Zope >> Instance as well. > > > Ideas welcome, especially ones with working implementations. ;) Yeah, I'm still trying to figure out what I want ;) Heck, I'm only half way into articulating what I see as the current outstanding problems. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Feb 7 20:11:49 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 14:11:49 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43E8EA38.5020500@colorstudy.com> References: <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> At 12:43 PM 2/7/2006 -0600, Ian Bicking wrote: >I guess the problem is that for Paste Script, the startup overhead and the >runtime overhead are the same thing. Like, 8 seconds to complete "paster >help" (which gives a listing of all available commands). After the first >run it warms up and is much faster (disk caching?), but that still doesn't >feel good to me. Now that it's warmed up I can't tell how slow a normal >command would be -- possible faster, since "help" necessarily looks at all >packages. I'm assuming here that the problem is needing to import each command to get its description and display it? >>>I also want content plugins, like the internationalization plugins (but >>>it might be Javascript or templates or whatever). And certainly that >>>applies to TG as well. I've been doing this at work, but using entry >>>points, where the entry points points to an object that is a string that >>>points to a resource that is then fetched with pkg_resources. Something >>>more direct would be appreciated ;) >> >>Um, you do realize that this is *exactly* what I was campaigning for on >>the Web-SIG when I said, "we need a resource deployment standard", don't >>you? :) > >Yes, and I completely agreed with you! Um, suuuure you did, Mr. "Objections only on the Spec and not the Scope". ;-) > Though I think it's a little larger of a problem than can be solved > with eggs alone, I don't need the spec to be egg-specific, only egg-*compatible*. >>However, it may be that it's a big enough discussion with a diverse >>enough set of stakeholders that it deserves its own (hopefully time >>limited) SIG. Anyway, I have more cycles that I can devote to a resource >>deployment standard than I can to a templating standard; the former has >>direct impact on my "day job" work, and the latter does not. > >Right now I suspect many possible stakeholders won't see how it applies to >their cases, or feel ready to take that discussion on. It's still too >abstract, unless perhaps it can be phrased in terms of some existing >standard (which may not be a Python-specific standard at all). Kind of >like how WSGI was phrased in terms of CGI. > >I am not familiar enough with this stuff to have any idea what such >standards exist. Seems like internationalization is the most common such >case, but even there people aren't utilizing that functionality to the >degree they could (e.g., internationalization should provide hooks for >customizing the text of an application, but it isn't used for that very often). I'm not aware of any existing standards, either, but I'm thinking that what's needed is more of an API for resource retrieval keyed on some set of simple values or wildcards, with some way to aggregate search results from multiple sources (so that e.g. databases, eggs, and directory trees) can all "offer" resources on demand. I'm thinking that you would call this API (at the low level) via something like: my_page_source = resource_set.find_resource( resource=['my_page'], for_project=['MyProject'], locale=['en','de'], layer=['some_layer', 'other_layer'], ).as_string() Of course, this would in most cases be wrapped by some higher-level API that eliminates most of the parameters from needing to be specified (e.g. by a framework that knows what locales and skin layers are in effect and what project the requesting code is calling from). For performance, you could extract subset resource sets and use them instead of querying a top-level resource set. There are a lot of things that would still have to be worked out, of course, like precedence among parameters, wildcards, and speed/space efficiencies. But the basic idea is that you just do queries on a relatively simple dataset. Since providers determine where the resources are and how they're retrieved, this wouldn't actually require everybody to agree on where things are or how they're organized for input, but would instead only require agreeing on an abstract addressing scheme, not unlike an LDAP directory schema. (Representing such a directory with good speed/space tradeoffs is of course one of the biggest hurdles, implementation-wise.) With respect to eggs specifically, I expect that setup() would grow an argument to accept an input data structure of some kind, with various utility routines to generate the data directly from common directory structure layouts. This is basically what we'd planned to do for Chandler message catalogs and localizable resources, but with a more simplistic data structure. >My hope is that find_template can be implemented with such a resource >deployment standard, even if the template stuff happens first. And if >that is possible, that would give the resource deployment standard a kind >of instance applicability to a real problem, which is a bonus. Well, I've already got a real use case in Chandler, so it's not as though the real-world needs don't exist. :) From bob at redivi.com Tue Feb 7 20:48:15 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 7 Feb 2006 11:48:15 -0800 Subject: [Distutils] [Pythonmac-SIG] pkg_resources and Mac OS X compatibility In-Reply-To: <5.1.1.6.0.20060207083254.038944b8@mail.telecommunity.com> References: <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> <5.1.1.6.0.20060207083254.038944b8@mail.telecommunity.com> Message-ID: <3CD0B049-6130-49C2-9CAD-ED230DD24A8E@redivi.com> On Feb 7, 2006, at 5:37 AM, Phillip J. Eby wrote: > At 02:07 PM 2/5/2006 -0800, Bob Ippolito wrote: >> On Feb 5, 2006, at 2:00 PM, Bob Ippolito wrote: >>> With the way that we're returning the distutils platform on the >>> universal branch of Mac OS X, we need another patch to >>> pkg_resources. The reason for this is that >>> distutils.util.get_platform() returns the platform that it is >>> trying to produce binaries for, which is often not the exact >>> current platform. More specifically, our current strategy is to >>> produce a version of Python that will build extensions that are >>> compatible with Mac OS X 10.3, but the actual building of >>> extensions requires Mac OS X 10.4 or later (because the toolchain >>> doesn't exist on Mac OS X 10.3). >>> >>> In some rare cases, people will want to produce packages that >>> explicitly require Mac OS X 10.4 or later, which they can do by >>> setting the MACOSX_DEPLOYMENT_TARGET=10.4 environment variable when >>> running setup.py. This will influence the value returned by >>> disutils.util.get_platform(), and will influence the compiler and >>> linker if extensions are built. >>> >>> This patch adds an internal _get_max_platform(plat) function that >>> returns the actual runtime version of Mac OS X, for use in >>> compatible_platforms. >> >> Oops, wrong patch.. here's the correct one. Sorry about that: > > I've implemented a similar - but different - patch. Yours causes > setuptools' tests to fail on non-Mac platforms, including non-Mac > darwin. Could you explain how? I don't see it. The condition is:: if m is not None and sys.platform == "darwin": ... How does that fail on pure Darwin? The condition should be False because m has to be None at that point and the branch shouldn't happen. -bob From pje at telecommunity.com Tue Feb 7 21:42:33 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Feb 2006 15:42:33 -0500 Subject: [Distutils] [Pythonmac-SIG] pkg_resources and Mac OS X compatibility In-Reply-To: <3CD0B049-6130-49C2-9CAD-ED230DD24A8E@redivi.com> References: <5.1.1.6.0.20060207083254.038944b8@mail.telecommunity.com> <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> <6D4EE44B-ECA5-4014-B339-16BB6DA4CE3A@redivi.com> <6B099161-BDFF-42A3-BF33-5D3748878B66@redivi.com> <5.1.1.6.0.20060207083254.038944b8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060207153653.0215d718@mail.telecommunity.com> At 11:48 AM 2/7/2006 -0800, Bob Ippolito wrote: >On Feb 7, 2006, at 5:37 AM, Phillip J. Eby wrote: >>I've implemented a similar - but different - patch. Yours causes >>setuptools' tests to fail on non-Mac platforms, including non-Mac >>darwin. > >Could you explain how? I don't see it. The condition is:: > > if m is not None and sys.platform == "darwin": > ... Note that that's because I added the "darwin" check; it wasn't in your original patch. >How does that fail on pure Darwin? The condition should be False >because m has to be None at that point and the branch shouldn't happen. You're making the assumption that platform compatibility checks between Mac OS X version strings only happen when you're *running* on Mac OS X, but the test suite contains tests for the platform compatibility checking code that runs on all platforms. Thus, the "m is not None" condition can apply regardless of platform, at least for test purposes. From ianb at colorstudy.com Wed Feb 8 00:21:17 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Feb 2006 17:21:17 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <43E8D96D.4030603@colorstudy.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <43E8D96D.4030603@colorstudy.com> Message-ID: <43E92B6D.1080700@colorstudy.com> Ian Bicking wrote: > Hmm... and reading over this, I realize that a "plugin directory" is not > that different from the virtual python setups, and I think that kind of > setup is really the only good way to deploy web applications. "Working > set" -- which is already an idea in setuptools -- is probably the > concept that encompasses both uses. As a developer, I'd like it to be > easier and safer to change my working set -- right now I have hacks in > sitecustomize and distutils to make that easy. This switching of > working sets is basically what Jim wants in a Zope Instance as well. One of the open questions I have here, is how the working set is determined. Here's some current practices: * Scripts include all the information about their working set. Jim brought this up recently, where in addition to the version a script will also contain the PYTHONPATH in some form. In this, you explicitly run some particular script, implicitly activating a working set. * The virtual Python installations basically work like this too. "Administrator Install" basically keys it of $HOME, which is great for per-user installations but doesn't get any more fine grained. "Virtual" Python uses the Python executable itself, and then all scripts will give an absolute path to the executable they use, so this same script-based dispatch continues to work with other commands. "Traditional" PYTHONPATH seems a little annoying, but maybe is kind of the right direction. * At work we're using an environmental variable ("ACTIVE_SITE") which changes sys.path (in sitecustomize) and the distutils installation options. This is like basing it on $HOME, but a bit more flexible (since $HOME has a lot of meanings). Another option would be some other environmental variable with a $HOME fallback. But this requires some distutils monkeypatching to make work. Are there other ways we can identify the user's intended working set? I don't think environmental variables are easy to work with on Windows, and it's a little opaque from a GUI user's perspective. $HOME isn't granular enough. Multiple scripts can work, but I've found it challenging to manage when the binaries in $PATH work, but potentially with bad side-effects (the discipline of keeping working sets separated isn't enforced, or even suggested by making it easier to do the right thing). Multiple scripts seems the most workable and transparent from the point of view of a GUI user, and not particularly bad from the perspective of a Posix command-line user either. [The more I think about it, the more I think site-packages in its entirety is pointless, distracting, and dangerous. But I don't think a PEP proposing its removal would be well received at this time ;) ] -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed Feb 8 04:40:13 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Feb 2006 21:40:13 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> References: <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> Message-ID: <43E9681D.6030900@colorstudy.com> Phillip J. Eby wrote: > At 12:43 PM 2/7/2006 -0600, Ian Bicking wrote: > >> I guess the problem is that for Paste Script, the startup overhead and >> the runtime overhead are the same thing. Like, 8 seconds to complete >> "paster help" (which gives a listing of all available commands). >> After the first run it warms up and is much faster (disk caching?), >> but that still doesn't feel good to me. Now that it's warmed up I >> can't tell how slow a normal command would be -- possible faster, >> since "help" necessarily looks at all packages. > > > I'm assuming here that the problem is needing to import each command to > get its description and display it? Oh, yes, that too ;) That probably is the bigger problem, and inevitable. That doesn't happen except with help. So maybe I am worried about nothing. >>>> I also want content plugins, like the internationalization plugins >>>> (but it might be Javascript or templates or whatever). And >>>> certainly that applies to TG as well. I've been doing this at work, >>>> but using entry points, where the entry points points to an object >>>> that is a string that points to a resource that is then fetched with >>>> pkg_resources. Something more direct would be appreciated ;) >>> >>> >>> Um, you do realize that this is *exactly* what I was campaigning for >>> on the Web-SIG when I said, "we need a resource deployment standard", >>> don't you? :) >> >> >> Yes, and I completely agreed with you! > > > Um, suuuure you did, Mr. "Objections only on the Spec and not the > Scope". ;-) Yes, and now we can start a separate spec. Mmm... yummy specs. OK, that's not actually my reaction at all to discussing specs, but I still think it is easier in smaller chunks. > I'm not aware of any existing standards, either, but I'm thinking that > what's needed is more of an API for resource retrieval keyed on some set > of simple values or wildcards, with some way to aggregate search results > from multiple sources (so that e.g. databases, eggs, and directory > trees) can all "offer" resources on demand. > > I'm thinking that you would call this API (at the low level) via > something like: > > my_page_source = resource_set.find_resource( > resource=['my_page'], for_project=['MyProject'], > locale=['en','de'], layer=['some_layer', 'other_layer'], > ).as_string() Are the arguments arbitrary? I.e., can I add my own? Like domain=['blog.ianbicking.org', 'ianbicking.org'], or user=['ianb', None]? Are resources typed in any way? Similar to entry point groups...? > Of course, this would in most cases be wrapped by some higher-level API > that eliminates most of the parameters from needing to be specified > (e.g. by a framework that knows what locales and skin layers are in > effect and what project the requesting code is calling from). For > performance, you could extract subset resource sets and use them instead > of querying a top-level resource set. I often find myself wanting to just override one little bit. Subset resources would potentially break that, unless they are a subset that is resolved all at once instead of a subset that has to provide all the necessary resources. At least if you are describing what I think you are describing. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Wed Feb 8 06:36:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 00:36:25 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43E9681D.6030900@colorstudy.com> References: <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> At 09:40 PM 2/7/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>I'm assuming here that the problem is needing to import each command to >>get its description and display it? > >Oh, yes, that too ;) That probably is the bigger problem, and >inevitable. That doesn't happen except with help. So maybe I am worried >about nothing. Probably. ;) A fix, however, would be to change your entry point names to include a description. Currently, entry point names can contain any characters you like besides '='. (And leading/trailing spaces are skipped.) This means that you can define entry points like this, as long as you do the name parsing in your own tools: commit (ci,checkin) - Commit the current version = some.module:commit So, this entry point would have a name of "commit (ci,checkin) - Commit the current version", and you can then parse it to get what you need. I plan to use this format for "nest" command entry points in 0.7, with the parenthesized names being used to identify aliases that refer to the same command. This will let a help command operate without needing to import anything, and in particular it means that any extras required for the command won't have to be downloaded and installed unless you try to do something with it more directly. The downside to this approach is that you can't just look up a command name directly, you have to iterate over all entry points in the command group until you find a match. However, this search would take place anyway, it's just hidden under the API, so the performance is the same apart from the parsing overhead. >>I'm not aware of any existing standards, either, but I'm thinking that >>what's needed is more of an API for resource retrieval keyed on some set >>of simple values or wildcards, with some way to aggregate search results >>from multiple sources (so that e.g. databases, eggs, and directory trees) >>can all "offer" resources on demand. >>I'm thinking that you would call this API (at the low level) via >>something like: >> my_page_source = resource_set.find_resource( >> resource=['my_page'], for_project=['MyProject'], >> locale=['en','de'], layer=['some_layer', 'other_layer'], >> ).as_string() > >Are the arguments arbitrary? I.e., can I add my own? Like >domain=['blog.ianbicking.org', 'ianbicking.org'], or user=['ianb', None]? Yeah, that's the idea. >Are resources typed in any way? Similar to entry point groups...? I think resources should only provide access to string/stream/filename (ala pkg_resources resources) and their metadata attributes (like locale, layer, etc.) If you want to have more elaborate typing, you can simply use another attribute to define it. For example, a content_type attribute or an attribute that says what entry point to use to adapt the resource to some interface. >>Of course, this would in most cases be wrapped by some higher-level API >>that eliminates most of the parameters from needing to be specified (e.g. >>by a framework that knows what locales and skin layers are in effect and >>what project the requesting code is calling from). For performance, you >>could extract subset resource sets and use them instead of querying a >>top-level resource set. > >I often find myself wanting to just override one little bit. Subset >resources would potentially break that, unless they are a subset that is >resolved all at once instead of a subset that has to provide all the >necessary resources. At least if you are describing what I think you are >describing. I'm not sure I follow you. If it's something you "override", then you'd have to leave it out of your subset criteria. What I'm describing is the ability to have a subset snapshot for performance reasons, not simply a restricted view over a larger set. (Although that also sounds like a useful thing to have.) Mainly, my concerns about this approach are that, without tuning or hinting for a particular access profile, it's going to be tough to have a fast data structure that's also memory-efficient. Creating indexes on all attributes means a space consumption of roughly one dictionary or set per unique attribute value. That is, every relatively-unique key consumes a dictionary of its own, consuming hundreds of bytes. Every resource will have at least 1 relatively-unique attribute value, namely its ID. On the plus side, even as the total number of resources grows due to variants, the raw overhead for the dictionaries should remain the same, since each new language or layer will only add one new unique key (the language or layer). So, it's probably not as bad to just index everything as I'm worrying it would be. Efficiently handling search precedence across multiple resource providers is also an interesting problem. You really want the result precedence to be based on stuff like the locale and layers, *not* on which provider found the data. This means that searches like the example I gave have to either be broken down into a variety of single-value searches done in sequence, each one executed in "parallel" against all backends. Either that, or there has to be a kind of sort-merge done against results yielded by the backends to ensure that the "best" results are yielded first. Probably the best thing to do is going to be to require searches to be prioritized on input, e.g.: find_resource( ('resource', ['my_page']), ('for_project', ['MyProject']), ('layer', ['some_layer','other_layer']), ('locale', ['en','de']), ... The above is saying, "first look for resources named my_page that are for MyProject, and of those you find, give precedence to 'some_layer' over 'other_layer' ones. And within those, give precedence to locales of 'en' then 'de'. This approach has the benefit of allowing entire backends to be excluded early from the search, since it doesn't matter what layers or locales an egg has resources for if it doesn't have 'my_page' for 'MyProject'. From pje at telecommunity.com Wed Feb 8 06:56:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 00:56:47 -0500 Subject: [Distutils] Changes to --find-links and --upgrade in SVN Message-ID: <5.1.1.6.0.20060208004729.02164e60@mail.telecommunity.com> Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file. From ianb at colorstudy.com Wed Feb 8 08:10:54 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Feb 2006 01:10:54 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> References: <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> Message-ID: <43E9997E.5000300@colorstudy.com> Phillip J. Eby wrote: > At 09:40 PM 2/7/2006 -0600, Ian Bicking wrote: > >> Phillip J. Eby wrote: >> >>> I'm assuming here that the problem is needing to import each command >>> to get its description and display it? >> >> >> Oh, yes, that too ;) That probably is the bigger problem, and >> inevitable. That doesn't happen except with help. So maybe I am >> worried about nothing. > > > Probably. ;) A fix, however, would be to change your entry point names > to include a description. Currently, entry point names can contain any > characters you like besides '='. (And leading/trailing spaces are > skipped.) > > This means that you can define entry points like this, as long as you do > the name parsing in your own tools: > > commit (ci,checkin) - Commit the current version = some.module:commit Somehow this bothers me. In practice I guess this gives me all the data I want... and yet I don't really feel confident it gives me that data. And it also feels like the entry point becomes unstable at a certain point. Of course, if the help was a resource instead of a Python object... >> Are resources typed in any way? Similar to entry point groups...? > > > I think resources should only provide access to string/stream/filename > (ala pkg_resources resources) and their metadata attributes (like > locale, layer, etc.) If you want to have more elaborate typing, you can > simply use another attribute to define it. For example, a content_type > attribute or an attribute that says what entry point to use to adapt the > resource to some interface. I'd very much like to get the actual location of the resource as well. This is what should show up in debugging output. Also, logging of some sort should go in early, I think, as it should be expected that resource resolution will have to be debugged, and the only way I imagine it being debuggable is through log messages. >>> Of course, this would in most cases be wrapped by some higher-level >>> API that eliminates most of the parameters from needing to be >>> specified (e.g. by a framework that knows what locales and skin >>> layers are in effect and what project the requesting code is calling >>> from). For performance, you could extract subset resource sets and >>> use them instead of querying a top-level resource set. >> >> >> I often find myself wanting to just override one little bit. Subset >> resources would potentially break that, unless they are a subset that >> is resolved all at once instead of a subset that has to provide all >> the necessary resources. At least if you are describing what I think >> you are describing. > > > I'm not sure I follow you. If it's something you "override", then you'd > have to leave it out of your subset criteria. What I'm describing is > the ability to have a subset snapshot for performance reasons, not > simply a restricted view over a larger set. (Although that also sounds > like a useful thing to have.) OK, that's probably all I mean. I just don't want someone to get "/images/", and then look inside their for "plus.gif", and get a not found error because it looks inside the directory named /images/ that was found which only contains a subset of files. > Mainly, my concerns about this approach are that, without tuning or > hinting for a particular access profile, it's going to be tough to have > a fast data structure that's also memory-efficient. Creating indexes on > all attributes means a space consumption of roughly one dictionary or > set per unique attribute value. That is, every relatively-unique key > consumes a dictionary of its own, consuming hundreds of bytes. Every > resource will have at least 1 relatively-unique attribute value, namely > its ID. > > On the plus side, even as the total number of resources grows due to > variants, the raw overhead for the dictionaries should remain the same, > since each new language or layer will only add one new unique key (the > language or layer). So, it's probably not as bad to just index > everything as I'm worrying it would be. OK, I will trust you on efficiency ;) > Efficiently handling search precedence across multiple resource > providers is also an interesting problem. You really want the result > precedence to be based on stuff like the locale and layers, *not* on > which provider found the data. This means that searches like the > example I gave have to either be broken down into a variety of > single-value searches done in sequence, each one executed in "parallel" > against all backends. Either that, or there has to be a kind of > sort-merge done against results yielded by the backends to ensure that > the "best" results are yielded first. > > Probably the best thing to do is going to be to require searches to be > prioritized on input, e.g.: > > find_resource( > ('resource', ['my_page']), > ('for_project', ['MyProject']), > ('layer', ['some_layer','other_layer']), > ('locale', ['en','de']), > ... > > The above is saying, "first look for resources named my_page that are > for MyProject, and of those you find, give precedence to 'some_layer' > over 'other_layer' ones. And within those, give precedence to locales > of 'en' then 'de'. > > This approach has the benefit of allowing entire backends to be excluded > early from the search, since it doesn't matter what layers or locales an > egg has resources for if it doesn't have 'my_page' for 'MyProject'. You wouldn't have to order the criteria to deal with that, since it seems like a fairly easy query optimization to see that since there was only one option given for each of resource and for_project, it must be satisfied regardless of precedence. Unless there was some kind of wildcard that a resource could provide. Like providing "my_page" for any project. I don't expect there to be such a wildcard...? Though I can almost imagine more complex search rules. But I'm having a hard time coming up with one. I thought I had one, then it slipped away... -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From rasky at develer.com Wed Feb 8 14:49:40 2006 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 8 Feb 2006 14:49:40 +0100 Subject: [Distutils] Decompressing .egg files Message-ID: <023201c62cb6$81ff4ca0$bf03030a@trilan> Hello, is there an easy_install option to decompress existing .egg files? I could probably run "easy_install -Z name" but that could potentially download and install another version. I'd need an option which pretty much this semantic: "if extension `name` exists and its current version is packaged as egg file, decompress it into a directory". Does it exist already? -- Giovanni Bajo From cwmoad at gmail.com Wed Feb 8 14:57:23 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Wed, 8 Feb 2006 08:57:23 -0500 Subject: [Distutils] Decompressing .egg files In-Reply-To: <023201c62cb6$81ff4ca0$bf03030a@trilan> References: <023201c62cb6$81ff4ca0$bf03030a@trilan> Message-ID: <6382066a0602080557m779ee818n2de531011ad08e79@mail.gmail.com> I think you can just unzip them, since they are zip files. Some programs might complain about the different extension though. On 2/8/06, Giovanni Bajo wrote: > Hello, > > is there an easy_install option to decompress existing .egg files? I could > probably run "easy_install -Z name" but that could potentially download and > install another version. I'd need an option which pretty much this semantic: > "if extension `name` exists and its current version is packaged as egg file, > decompress it into a directory". Does it exist already? > -- > Giovanni Bajo > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From rasky at develer.com Wed Feb 8 15:05:22 2006 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 8 Feb 2006 15:05:22 +0100 Subject: [Distutils] Decompressing .egg files References: <023201c62cb6$81ff4ca0$bf03030a@trilan> <6382066a0602080557m779ee818n2de531011ad08e79@mail.gmail.com> Message-ID: <027201c62cb8$b3d3bac0$bf03030a@trilan> Charlie Moad wrote: > I think you can just unzip them, since they are zip files. Some > programs might complain about the different extension though. True, but you'll have to locate which .egg file is the current one for a given module (there is no 1:1 match between python module names and egg files, and you can have multiple versions installed). Also you can't just decompress it: you need to create a wrapper directory with the same name (including extension) of the .egg file. This also means that you need to do it in two steps since you can't have both a file and a directory with the same name at the same time. I was wondering if there was already something ready-made so that I could just do "easy_install --decompress pysqlite" and forget about it. If not, I hope the future "nest" command will handle this. -- Giovanni Bajo From pje at telecommunity.com Wed Feb 8 16:13:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 10:13:32 -0500 Subject: [Distutils] Decompressing .egg files In-Reply-To: <023201c62cb6$81ff4ca0$bf03030a@trilan> Message-ID: <5.1.1.6.0.20060208100739.021a35a0@mail.telecommunity.com> At 02:49 PM 2/8/2006 +0100, Giovanni Bajo wrote: >Hello, > >is there an easy_install option to decompress existing .egg files? I could >probably run "easy_install -Z name" but that could potentially download and >install another version. No, it couldn't. A download would only occur if you use --find-links (prior to the current SVN version) or --upgrade, or if 'name' can't be found locally. > I'd need an option which pretty much this semantic: >"if extension `name` exists and its current version is packaged as egg file, >decompress it into a directory". Does it exist already? Use --install-dir (-d) and --always-copy (-a) with -Z; this will always result in an unpacked egg in the install directory, whether the source was packed or not: easy_install -Zad tgtdir name If you don't want dependencies, you should also use --no-deps (-N): easy_install -ZaNd tgtdir name In either case, this will install the egg(s) for 'name' into 'tgtdir', and will copy/unpack them there even if they're already on sys.path. No online search will be done unless you have 'find_links' and/or 'upgrade' set in one of your distutils config files (setup.cfg, ~/.pydistutils.cfg, etc.). And you could prevent it even then using --allow-hosts (-H) to disable any remote access. From pje at telecommunity.com Wed Feb 8 16:24:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 10:24:59 -0500 Subject: [Distutils] Decompressing .egg files In-Reply-To: <5.1.1.6.0.20060208100739.021a35a0@mail.telecommunity.com> References: <023201c62cb6$81ff4ca0$bf03030a@trilan> Message-ID: <5.1.1.6.0.20060208102118.04350288@mail.telecommunity.com> At 10:13 AM 2/8/2006 -0500, Phillip J. Eby wrote: >Use --install-dir (-d) and --always-copy (-a) with -Z; this will always >result in an unpacked egg in the install directory, whether the source was >packed or not: > > easy_install -Zad tgtdir name > >If you don't want dependencies, you should also use --no-deps (-N): > > easy_install -ZaNd tgtdir name > >In either case, this will install the egg(s) for 'name' into 'tgtdir', and >will copy/unpack them there even if they're already on sys.path. No online >search will be done unless you have 'find_links' and/or 'upgrade' set in >one of your distutils config files (setup.cfg, ~/.pydistutils.cfg, >etc.). And you could prevent it even then using --allow-hosts (-H) to >disable any remote access. I forgot to mention, by the way, that 'tgtdir' cannot be the same directory where the original egg is. If you want to update an egg in place, use a temporary directory like this: easy_install -aNd some_temp_dir name easy_install -aNZ some_temp_dir/name*.egg which will first copy the egg to the temporary directory, then unpack it back to its original location (more or less). From rasky at develer.com Wed Feb 8 16:26:27 2006 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 8 Feb 2006 16:26:27 +0100 Subject: [Distutils] Decompressing .egg files References: <5.1.1.6.0.20060208100739.021a35a0@mail.telecommunity.com> Message-ID: <03ef01c62cc4$0782c2a0$bf03030a@trilan> Phillip J. Eby wrote: >> I'd need an option which pretty much this semantic: >> "if extension `name` exists and its current version is packaged as egg >> file, decompress it into a directory". Does it exist already? > > Use --install-dir (-d) and --always-copy (-a) with -Z; this will always > result in an unpacked egg in the install directory, whether the source was > packed or not: > > easy_install -Zad tgtdir name This works, but specifying the target directory is something I would like to avoid. My scenario is this: I run "easy_install foobar", and easy_install downloads and installs foobar.egg (zip file). After that, I decide that I need it to be a directory, *not* a zip file. I'd expect something like "easy_install -Z foobar" to be sufficient. I tried with: easy_install -Zad c:\python24\lib\site-packages foobar and it worked, but it feels wrong to be forced to specify the site-packages directory. I just want to obtain the same result *as if* I had specified -Z when it first downloaded the file. > In either case, this will install the egg(s) for 'name' into 'tgtdir', and > will copy/unpack them there even if they're already on sys.path. No online > search will be done unless you have 'find_links' and/or 'upgrade' set in > one of your distutils config files (setup.cfg, ~/.pydistutils.cfg, > etc.). And you could prevent it even then using --allow-hosts (-H) to > disable any remote access. Thanks for this explanation. -- Giovanni Bajo From ianb at colorstudy.com Wed Feb 8 17:40:07 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Feb 2006 10:40:07 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> References: <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> Message-ID: <43EA1EE7.5060300@colorstudy.com> Phillip J. Eby wrote: > Probably the best thing to do is going to be to require searches to be > prioritized on input, e.g.: > > find_resource( > ('resource', ['my_page']), > ('for_project', ['MyProject']), > ('layer', ['some_layer','other_layer']), > ('locale', ['en','de']), > ... Thoughts about the return value: I think there should be a dictionary returned, that contains various metadata. In this case it should contain at least {'resource': 'my_page', 'for_project': 'MyProject', 'layer': 'other_layer', 'locale': 'en'}, to represent exactly what was found. Other metadata can be useful, like the resource_location (the actual filename or other user-readable name). Content-type is useful in many contexts -- for instance, you might want some kind of image, but then you need to know exactly what kind you got back. If it gets a stream, knowing Content-length is useful as well. Also, either encoding should be given (for text resources), or unicode returns should be allowed (I prefer unicode return values). I think the resource container usually knows the encoding, not the entity requesting the resource. That dictionary could also be a container for callables that produce things like filenames and streams. Or they could be returned in a different way. Or it could be an actual object, with methods for those things. Somehow I have become fond of dictionaries. I'm starting to get a better feel for how this overlaps with templates -- coming at it from this direction is easier than from the WSGI direction. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Feb 8 18:19:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 12:19:41 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43E9997E.5000300@colorstudy.com> References: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060208120039.0218bc60@mail.telecommunity.com> At 01:10 AM 2/8/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>Probably. ;) A fix, however, would be to change your entry point names >>to include a description. Currently, entry point names can contain any >>characters you like besides '='. (And leading/trailing spaces are skipped.) >>This means that you can define entry points like this, as long as you do >>the name parsing in your own tools: >> commit (ci,checkin) - Commit the current version = some.module:commit > >Somehow this bothers me. In practice I guess this gives me all the data I >want... and yet I don't really feel confident it gives me that data. And >it also feels like the entry point becomes unstable at a certain point. Um, okay. Those sound like personal issues that I can't help you with. ;) Entry point names were designed with this kind of flexible processing in mind, however. You can embed as much data as you want in the name, as long as there are no line breaks or '=', and the name doesn't start with a '#'. >I'd very much like to get the actual location of the resource as well. >This is what should show up in debugging output. That should be done in the resource object's __repr__, then. But I can see having an attribute that's effectively some str() of the object's location. (By which I mean it's human readable, but not machine-usable, since the resource provider could be a file, a database, or who knows what.) > Also, logging of some sort should go in early, I think, as it should be > expected that resource resolution will have to be debugged, and the only > way I imagine it being debuggable is through log messages. You lost me on that. I'm having trouble seeing what you'll be able to (or need to) debug that way. The most likely problems are that you misspell something (in which case there'll be no matches at all) or you're missing a provider (in which case studying the list of the providers will give you the answer). In each case, inspecting the current state of the system seems sufficient. OTOH, as long as the interfaces are well-defined, nothing stops you from creating logging "middleware" for debugging purposes, I suppose. I just don't want to embed that stuff in core code, if for no other reason than that the logging module is a PITA. >OK, that's probably all I mean. I just don't want someone to get >"/images/", and then look inside their for "plus.gif", and get a not found >error because it looks inside the directory named /images/ that was found >which only contains a subset of files. Oh, no... I'm assuming that the attribute namespaces are entirely flat. If you want directories or some other kind of hierarchy you would need to simulate them using 'directory' attributes or something like that. I'm assuming also that an individual resource can have more than one value for an attribute, so that a single resource could be e.g. registered for both 'en' and 'en-US' locales. >>Efficiently handling search precedence across multiple resource providers >>is also an interesting problem. You really want the result precedence to >>be based on stuff like the locale and layers, *not* on which provider >>found the data. This means that searches like the example I gave have to >>either be broken down into a variety of single-value searches done in >>sequence, each one executed in "parallel" against all backends. Either >>that, or there has to be a kind of sort-merge done against results >>yielded by the backends to ensure that the "best" results are yielded first. >>Probably the best thing to do is going to be to require searches to be >>prioritized on input, e.g.: >> find_resource( >> ('resource', ['my_page']), >> ('for_project', ['MyProject']), >> ('layer', ['some_layer','other_layer']), >> ('locale', ['en','de']), >> ... >>The above is saying, "first look for resources named my_page that are for >>MyProject, and of those you find, give precedence to 'some_layer' over >>'other_layer' ones. And within those, give precedence to locales of 'en' >>then 'de'. >>This approach has the benefit of allowing entire backends to be excluded >>early from the search, since it doesn't matter what layers or locales an >>egg has resources for if it doesn't have 'my_page' for 'MyProject'. > >You wouldn't have to order the criteria to deal with that, since it seems >like a fairly easy query optimization "Easy query optimization" is usually an oxymoron. :) > to see that since there was only one option given for each of resource > and for_project, it must be satisfied regardless of precedence. True. On the other hand, for the use cases where this is true, you'll usually statically know that at the call point. And, for the ones that have multiple values, following the order given is critical. But that's probably a matter for tuning of an *actual* implementation, rather than at the design level. :) > Unless there was some kind of wildcard that a resource could > provide. Like providing "my_page" for any project. I don't expect there > to be such a wildcard...? I don't either. Multiple values for an attribute, yes. Wildcard attributes, not really, at least not as part of the system itself. Wildcards can be simulated by either omitting a value from a search, or having an explicit wildcard value that's always searched last in a list. From pje at telecommunity.com Wed Feb 8 19:07:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 13:07:47 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43EA1EE7.5060300@colorstudy.com> References: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060208122010.02260dd8@mail.telecommunity.com> At 10:40 AM 2/8/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>Probably the best thing to do is going to be to require searches to be >>prioritized on input, e.g.: >> find_resource( >> ('resource', ['my_page']), >> ('for_project', ['MyProject']), >> ('layer', ['some_layer','other_layer']), >> ('locale', ['en','de']), >> ... > >Thoughts about the return value: > >I think there should be a dictionary returned, that contains various metadata. Actually, there should be a find_resources() that yields resources according to the precedence given, and find_resource() will simply return the first resource. > In this case it should contain at least {'resource': 'my_page', > 'for_project': 'MyProject', 'layer': 'other_layer', 'locale': 'en'}, to > represent exactly what was found. There would actually be a Resource object, with either a mapping or attribute interface to these things, as well as methods. The attributes are going to be tuples of strings, however, since they can be multi-valued. > Other metadata can be useful, like the resource_location (the actual > filename or other user-readable name). Content-type is useful in many > contexts -- for instance, you might want some kind of image, but then you > need to know exactly what kind you got back. If it gets a stream, > knowing Content-length is useful as well. Also, either encoding should > be given (for text resources), or unicode returns should be allowed (I > prefer unicode return values). I think the resource container usually > knows the encoding, not the entity requesting the resource. Content type would probably need to be mime_major+mime_minor attributes, so you could request just 'image', without implementing more complex matches for searching. Content length I don't see as a searchable attribute, but I can see having a method of some sort to query that, and the same is true for encoding. >That dictionary could also be a container for callables that produce >things like filenames and streams. Or they could be returned in a >different way. Or it could be an actual object, with methods for those >things. Somehow I have become fond of dictionaries. Heh. I think objects are a reasonable way to go here; there isn't much call for wrapping resources themselves with middleware, and they don't do very much to begin with. >I'm starting to get a better feel for how this overlaps with templates -- >coming at it from this direction is easier than from the WSGI direction. So far the biggest architectural flaw (efficiency-wise) that I see in all this is that if your resources are all in eggs, and you have a *lot* of them, you have to read *all* of the eggs' resource indexes before you can return a single match. While it's true that you could have a shorter list for each egg that indicates only what attribute/value combinations are offered by that egg, you still have to read *that* list for all of them for the first search, and for eggs with a small number of resources it'll be almost as fast to just read the full index to avoid multiple I/O operations. Anyway, conceptually I think this is something that's useful for pretty much any extensible, localizable Python application, especially ones that are web-based. I can see many potential implementations for how you get the resource data *in* to the system, too. For example, peak.web already has an .ini file format that lets you set content type rules, using section headings that give filenames or wildcards, and then the entries list properties to be assigned. I'm thinking that on the egg side, I'd use a new setuptools entry point for "resource finder" plugins. Their job will be to scope out the distribution source for resources and add them to an index. The index would then be written to the egg's metadata directory. A 'resource_finders' keyword to setup would list the names of the entry points to use, so that you don't have every possible resource finder chugging away and adding false positives. It might be that the keyword would be a dictionary, e.g.: setup( ... publish_resources = { 'peak.web': ['somepkg/foo', ...], 'chandler.translations': {'someparcel':'foobar/LC_MESSAGES'}, ... } ) That is, 'publish_resources' would be a dictionary mapping entry point names to arguments that define how those plugins should locate and index the resources. The above would fire the peak.web and chandler.translations plugins from the 'setuptools.resource_finders' entry point group in order to index the available resources for publication. (Notice that each resource finder can have a different parameter format, if it likes.) Hm. After all this talk about the thing, I kind of want to just go implement it so *I* can use it, standards be damned. :) OTOH, I think it would be really good to get more feedback on the concept before doing that. I'd be especially curious to hear from the framework developers, esp. of Zope, TurboGears, and Myghty. Zope of course already has similar resource-finding abilities, but they don't tie to eggs. They should have good feedback about both what you need to be able to search on, and what kind of performance issues are likely to arise. From ianb at colorstudy.com Wed Feb 8 19:10:28 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Feb 2006 12:10:28 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060208120039.0218bc60@mail.telecommunity.com> References: <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060208120039.0218bc60@mail.telecommunity.com> Message-ID: <43EA3414.8050807@colorstudy.com> Phillip J. Eby wrote: >> Also, logging of some sort should go in early, I think, as it should >> be expected that resource resolution will have to be debugged, and the >> only way I imagine it being debuggable is through log messages. > > > You lost me on that. I'm having trouble seeing what you'll be able to > (or need to) debug that way. The most likely problems are that you > misspell something (in which case there'll be no matches at all) or > you're missing a provider (in which case studying the list of the > providers will give you the answer). In each case, inspecting the > current state of the system seems sufficient. I disagree strongly here. There are two cases I see that need to be debugged: you get the wrong resource, and you get no resource at all. Both of these can be very confusing. Certainly I regularly find 404s to be confounding. I don't want to have to put print statements in other people's resource-finding code in order to figure out what's going wrong. I'm regularly putting print statements in pkg_resources to figure out what's going wrong. Also, "inspecting the current state of the system" isn't really feasible. The current state is in memory, often in opaque objects. And people who are debugging this are often going to be non-programmers, who don't have any means or understanding of the internal state of the system. > OTOH, as long as the interfaces are well-defined, nothing stops you from > creating logging "middleware" for debugging purposes, I suppose. I just > don't want to embed that stuff in core code, if for no other reason than > that the logging module is a PITA. I don't really care about the logging module. If you could just pass in a callable where things are written to (defaut to None which does no writing), that would be good enoguh -- no log levels or anything fancy like that. >> OK, that's probably all I mean. I just don't want someone to get >> "/images/", and then look inside their for "plus.gif", and get a not >> found error because it looks inside the directory named /images/ that >> was found which only contains a subset of files. > > > Oh, no... I'm assuming that the attribute namespaces are entirely > flat. If you want directories or some other kind of hierarchy you would > need to simulate them using 'directory' attributes or something like > that. I'm assuming also that an individual resource can have more than > one value for an attribute, so that a single resource could be e.g. > registered for both 'en' and 'en-US' locales. OK, that's fine then; I want opaque names (not hierarchical names) anyway. >> Unless there was some kind of wildcard that a resource could >> provide. Like providing "my_page" for any project. I don't expect >> there to be such a wildcard...? > > > I don't either. Multiple values for an attribute, yes. Wildcard > attributes, not really, at least not as part of the system itself. > Wildcards can be simulated by either omitting a value from a search, or > having an explicit wildcard value that's always searched last in a list. Yeah, I figured ad hoc conventions for wildcards would be sufficient. A provider could, potentially, say that every resource it provides also satisfies '*' for some attribute, and you could query with '*' at the end of the list. I assume when a criteria is omited entirely then any resource will match that criteria. I suppose there is possibly a use case for "satisfies some related criteria", even if you don't have any specific criteria in mind. E.g., "satisfies some template_language=anything", even if you don't care template language in particular. But you don't want to match CSS files if you are looking for a template. OTOH, that could be done with "resource_type=['template']" or something like that. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Feb 8 19:46:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 13:46:38 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43EA3414.8050807@colorstudy.com> References: <5.1.1.6.0.20060208120039.0218bc60@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <5.1.1.6.0.20060207130037.038978e0@mail.telecommunity.com> <5.1.1.6.0.20060207134602.041a1ea8@mail.telecommunity.com> <5.1.1.6.0.20060207234831.03757510@mail.telecommunity.com> <5.1.1.6.0.20060208120039.0218bc60@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060208131910.02263688@mail.telecommunity.com> At 12:10 PM 2/8/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>> Also, logging of some sort should go in early, I think, as it should >>> be expected that resource resolution will have to be debugged, and the >>> only way I imagine it being debuggable is through log messages. >> >>You lost me on that. I'm having trouble seeing what you'll be able to >>(or need to) debug that way. The most likely problems are that you >>misspell something (in which case there'll be no matches at all) or >>you're missing a provider (in which case studying the list of the >>providers will give you the answer). In each case, inspecting the >>current state of the system seems sufficient. > >I disagree strongly here. There are two cases I see that need to be >debugged: you get the wrong resource, and you get no resource at all. Both >of these can be very confusing. Certainly I regularly find 404s to be >confounding. I don't want to have to put print statements in other >people's resource-finding code in order to figure out what's going >wrong. I'm regularly putting print statements in pkg_resources to figure >out what's going wrong. > >Also, "inspecting the current state of the system" isn't really >feasible. The current state is in memory, often in opaque objects. And >people who are debugging this are often going to be non-programmers, who >don't have any means or understanding of the internal state of the system. Actually, I'm assuming that the system would provide sufficient state info to allow you to create a viewer or query tool that would let you see and search through all available resources, or to dump them to a tab-delimited file that you can massage in Excel or Access or some such thing to see what's going on. It is, after all, just a database, so a few filters and sorts should suffice to isolate what is or isn't there. Mostly, I think debugging effectively needs to take place on the *input* end. That is, on the files that are going into your egg to begin with. Resource finders should display warnings about the resources they didn't or couldn't index. On the other hand, this highlights the fact that rerunning egg_info every time you add templates to a project would probably be a PITA in itself, meaning that what you probably really want is for resource finders to be something that runs at lookup time, not egg building time. What sucks about that is, you'll have to import all the finders at run time, unless you have some kind of project filter. Bleah. By project filter, I mean some way for a project to indicate what other projects it offers resources for. That way, at runtime you could skip loading finders for eggs that don't relate to the project you're looking up. This is about the only kind of hint I can think of that can reasonably be hardcoded to prevent unnecessary dynamic lookups at runtime. The whole thing seems a little messy, since you could have both runtime resource finding and build-time resource finding, which sounds like it's just asking for trouble. It might be better to just use runtime searching only... in which case we end up back at needing logging, unless there's a prescan process involved. Ah well. In any case, we have to support runtime searches for development uses. >I assume when a criteria is omited entirely then any resource will match >that criteria. Yeah. > I suppose there is possibly a use case for "satisfies some related > criteria", even if you don't have any specific criteria in mind. E.g., > "satisfies some template_language=anything", even if you don't care > template language in particular. But you don't want to match CSS files > if you are looking for a template. OTOH, that could be done with > "resource_type=['template']" or something like that. Yeah, I really want to keep the data model ridiculously simple; you shouldn't have to implement a "real" database for this stuff. From ben at groovie.org Wed Feb 8 23:27:59 2006 From: ben at groovie.org (Ben Bangert) Date: Wed, 8 Feb 2006 14:27:59 -0800 Subject: [Distutils] setuptools requiring itself to run... thus not running Message-ID: A user of one of my webapps reported that after running upgrade one night, setuptools was no longer functioning properly. Apparently it somehow got installed in multi-version mode, requiring pkg_resources to be imported *before* you could use setuptools. Which of course leaves it in the somewhat amusing state that it needs to import pkg_resources before it can import pkg_resources.... Here's the messages he received during the install: molokai:/usr/lib64/python2.4/site-packages # python /in/hellahella/ ez_setup.py -U setuptools Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/ setuptools-0.6a9-py2.4.egg Searching for setuptools Reading http://www.python.org/pypi/setuptools/ Reading http://peak.telecommunity.com/DevCenter/setuptools Best match: setuptools 0.6a9 Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/ setuptools-0.6a9-py2.4.egg#md5=8f6e01fc12fb1cd006dc0d6c04327ec1 Processing setuptools-0.6a9-py2.4.egg creating /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg Extracting setuptools-0.6a9-py2.4.egg to /usr/lib64/python2.4/site- packages Installing easy_install script to /usr/bin Installed /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg Because this distribution was installed --multi-version or --install- dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("setuptools") # latest installed version pkg_resources.require("setuptools==0.6a9") # this exact version pkg_resources.require("setuptools>=0.6a9") # this version or higher Processing dependencies for setuptools Processing setuptools-0.6a9-py2.4.egg removing '/usr/lib64/python2.4/site-packages/setuptools-0.6a9- py2.4.egg' (and everything under it) creating /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg Extracting setuptools-0.6a9-py2.4.egg to /usr/lib64/python2.4/site- packages Installing easy_install script to /usr/bin Installed /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg Because this distribution was installed --multi-version or --install- dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("setuptools") # latest installed version pkg_resources.require("setuptools==0.6a9") # this exact version pkg_resources.require("setuptools>=0.6a9") # this version or higher Processing dependencies for setuptools==0.6a9 Searching for setuptools Reading http://www.python.org/pypi/setuptools/ Reading http://peak.telecommunity.com/DevCenter/setuptools Best match: setuptools 0.6a9 Processing setuptools-0.6a9-py2.4.egg Installing easy_install script to /usr/bin Using /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg Because this distribution was installed --multi-version or --install- dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("setuptools") # latest installed version pkg_resources.require("setuptools==0.6a9") # this exact version pkg_resources.require("setuptools>=0.6a9") # this version or higher Processing dependencies for setuptools Thanks, Ben From ben at groovie.org Wed Feb 8 23:42:32 2006 From: ben at groovie.org (Ben Bangert) Date: Wed, 8 Feb 2006 14:42:32 -0800 Subject: [Distutils] A prefix option? Message-ID: <84BE8F25-1433-493F-9493-4C57A67A267B@groovie.org> I'm wondering if there's any plans for a prefix option, which functions like the common makefile prefix option for where the lib/ bin dir will then be, etc. My main reason for asking is this blog entry regarding setuptools installs: http://bitworking.org/news/ Please_stop_using_setuptools__at_least_exclusively__for_now____ Cheers, Ben From pje at telecommunity.com Thu Feb 9 00:22:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 18:22:44 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <84BE8F25-1433-493F-9493-4C57A67A267B@groovie.org> Message-ID: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> At 02:42 PM 2/8/2006 -0800, Ben Bangert wrote: >I'm wondering if there's any plans for a prefix option, which >functions like the common makefile prefix option for where the lib/ >bin dir will then be, etc. My main reason for asking is this blog >entry regarding setuptools installs: >http://bitworking.org/news/ >Please_stop_using_setuptools__at_least_exclusively__for_now____ Rather than stop using setuptools, package authors should consult: http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know which has a (hopefully) comprehensive list of things that you may need or want to inform your users about. A --prefix option wouldn't have removed the need for *any* of them, I'm afraid. Note also that among the installation options offered at: http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations is the option to create a "virtual" Python installation, and the supplied virtual-python.py script *does* accept a --prefix option, and would have allowed him to do exactly what he wanted to do by simply using the ~/bin/python created automatically by virtual-python.py. I've written the author a letter privately apologizing for the inconvenience and giving him the links above. I'll probably also post a copy of the letter on my blog, to help raise awareness that setuptools using packages should prominently refer to at least the Custom Installation Locations link. From pje at telecommunity.com Thu Feb 9 00:33:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Feb 2006 18:33:03 -0500 Subject: [Distutils] setuptools requiring itself to run... thus not running In-Reply-To: Message-ID: <5.1.1.6.0.20060208182430.040c8a10@mail.telecommunity.com> At 02:27 PM 2/8/2006 -0800, Ben Bangert wrote: >A user of one of my webapps reported that after running upgrade one >night, setuptools was no longer functioning properly. Apparently it >somehow got installed in multi-version mode, requiring pkg_resources >to be imported *before* you could use setuptools. Which of course >leaves it in the somewhat amusing state that it needs to import >pkg_resources before it can import pkg_resources.... That's *very* odd. EasyInstall has special code in there to write an extra setuptools.pth file, such that you can never *really* install setuptools in multi-version mode. Even if it's not in easy-install.pth, it should always be in setuptools.pth. The only way it should be able to do this is if you are installing to a directory that easy_install doesn't consider a "site" directory. Judging from the output, my guess here would be that the standard algorithm for determining a site-packages directory location isn't working on this platform; it's probably coming up with /usr/lib/python2.4/site-packages instead. They need to add: [easy_install] site_dirs = /usr/lib64/python2.4/site-packages to /usr/lib64/python2.4/distutils/distutils.cfg. This file probably doesn't exist; just create it. I'm really curious how this particular Python got built, such that it thinks /usr/lib64 is the place where this stuff goes. There might be other problems, but from the output you provided I'm guessing that this is the only actual issue currently. Certainly it's the most likely source of the issue. From micktwomey at gmail.com Thu Feb 9 08:59:56 2006 From: micktwomey at gmail.com (Michael Twomey) Date: Thu, 9 Feb 2006 07:59:56 +0000 Subject: [Distutils] Some negative press for easy_install Message-ID: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> Hi, Joe Gregorio has a fairly negative experience with easy_install here: http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ I think his points boil roughly down to these: 1. (not directly related to his first comment, but I think this might be a source of problems) easy_install isn't always called easy_install. In my case I've wound up with easy_install.py on some systems at some point in the past. Would a possible solution be to allow "python -m easy_install"? Then again I think it's pretty consistent these days. 2. No --prefix option. I think this is a case of RTFM, since it's not normally as simple as specifying a prefix. Maybe a help message which explains why --prefix isn't supported might be useful? 3. In the error message he gets it shows bog standard distutils help, so he tries --help-commands and gets an error. I think this is a definite source of frustration. 4. He tries the --install-dir option which installs the package but of course this leads to a multi-install so he can't do a plain import foo. It might be good to link to http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations for an explanation here. 5. At this point he gives up and copies the source from svn. I think in all the above there is an element of RTFM but he does have a legitimate point when it comes to the command line help, based off that alone you aren't likely to figure out the best way to use easy_install if you are doing a non-system python install. I think a link to http://peak.telecommunity.com/DevCenter/EasyInstall in the help would work wonders. cheers, Michael From micktwomey at gmail.com Thu Feb 9 09:02:25 2006 From: micktwomey at gmail.com (Michael Twomey) Date: Thu, 9 Feb 2006 08:02:25 +0000 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> Message-ID: <50a522ca0602090002m1947f91fo@mail.gmail.com> Whoops, I just noticed previous messages on this, ignore me. Michael On 09/02/06, Michael Twomey wrote: > Hi, > > Joe Gregorio has a fairly negative experience with easy_install here: > http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ > > I think his points boil roughly down to these: > > 1. (not directly related to his first comment, but I think this might > be a source of problems) easy_install isn't always called > easy_install. In my case I've wound up with easy_install.py on some > systems at some point in the past. Would a possible solution be to > allow "python -m easy_install"? Then again I think it's pretty > consistent these days. > > 2. No --prefix option. I think this is a case of RTFM, since it's not > normally as simple as specifying a prefix. Maybe a help message which > explains why --prefix isn't supported might be useful? > > 3. In the error message he gets it shows bog standard distutils help, > so he tries --help-commands and gets an error. I think this is a > definite source of frustration. > > 4. He tries the --install-dir option which installs the package but of > course this leads to a multi-install so he can't do a plain import > foo. It might be good to link to > http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations > for an explanation here. > > 5. At this point he gives up and copies the source from svn. > > I think in all the above there is an element of RTFM but he does have > a legitimate point when it comes to the command line help, based off > that alone you aren't likely to figure out the best way to use > easy_install if you are doing a non-system python install. I think a > link to http://peak.telecommunity.com/DevCenter/EasyInstall in the > help would work wonders. > > cheers, > Michael > From p.f.moore at gmail.com Thu Feb 9 12:55:35 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Feb 2006 11:55:35 +0000 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> Message-ID: <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> On 2/9/06, Michael Twomey wrote: > Hi, > > Joe Gregorio has a fairly negative experience with easy_install here: > http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ > > I think his points boil roughly down to these: [...] > 5. At this point he gives up and copies the source from svn. This is the one that struck me as the biggest issue. Joe is right that setuptools is still in alpha - so authors offering setuptools-based installs should be fully aware that they risk breakage, frustration, and similar from their users. That's to be expected. If you don't like the pain of alpha software, then avoid it. If routes *needed* setuptools functionality, then fine - but explain this prominently somewhere: "This package uses setuptools, which is currently in alpha status - there may be issues installing or using the software. If you hit any problems, please report them to the distils-sg, and thank you for helping to test setuptools". But clearly routes does not need setuptolols functionality (or the Routes tests aren't complete - as Joe said that all the tests run). So why not provide a non-setuptools build, for users who don't want to fight with the bleeding edge? Just my 2p worth... Paul. From jim at zope.com Thu Feb 9 13:15:35 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 09 Feb 2006 07:15:35 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> Message-ID: <43EB3267.9020803@zope.com> Phillip J. Eby wrote: > I recently started work on adding egg support to Chandler ( > http://chandler.osafoundation.org/ ), and ran into some interesting issues > with respect to plugin discovery. Specifically, it's not easy to do it > well with the APIs that pkg_resources currently offers. I suspect that > others who've worked on plugin loading for application environments like > Zope and Trac have probably run into similar issues. > > I'm proposing, therefore, to add a new API to pkg_resources to make > plugin-finding easier. Among the requirements: I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set. Does that capture what you want to do? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Thu Feb 9 13:31:29 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 09 Feb 2006 07:31:29 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> References: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> Message-ID: <43EB3621.9010008@zope.com> Phillip J. Eby wrote: > At 02:42 PM 2/8/2006 -0800, Ben Bangert wrote: > >>I'm wondering if there's any plans for a prefix option, which >>functions like the common makefile prefix option for where the lib/ >>bin dir will then be, etc. My main reason for asking is this blog >>entry regarding setuptools installs: >>http://bitworking.org/news/ >>Please_stop_using_setuptools__at_least_exclusively__for_now____ > > > Rather than stop using setuptools, package authors should consult: > > http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know > > which has a (hopefully) comprehensive list of things that you may need or > want to inform your users about. A --prefix option wouldn't have removed > the need for *any* of them, I'm afraid. Note also that among the > installation options offered at: > > http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations > > is the option to create a "virtual" Python installation, and the supplied > virtual-python.py script *does* accept a --prefix option, and would have > allowed him to do exactly what he wanted to do by simply using the > ~/bin/python created automatically by virtual-python.py. I really don't think the virtual python approach is viable, at a minimum because it doesn't work on windows. It is also unacceptably heavy IMO. I think we need a simple way to handle custom directories that is: - cross platform - doesn't require modifying the Python install - allows chained/multile custom directories, which means the trick of putting eggs and scripts in the same directory doesn't work, - is simple. :) It appears that this has something to do with your plugin proposal. I'll try to participate in that discussion to tease this out a bit more. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ben at groovie.org Thu Feb 9 17:02:17 2006 From: ben at groovie.org (Ben Bangert) Date: Thu, 9 Feb 2006 08:02:17 -0800 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> Message-ID: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> On Feb 9, 2006, at 3:55 AM, Paul Moore wrote: > If routes *needed* setuptools functionality, then fine - but explain > this prominently somewhere: "This package uses setuptools, which is > currently in alpha status - there may be issues installing or using > the software. If you hit any problems, please report them to the > distils-sg, and thank you for helping to test setuptools". But clearly > routes does not need setuptolols functionality (or the Routes tests > aren't complete - as Joe said that all the tests run). So why not > provide a non-setuptools build, for users who don't want to fight with > the bleeding edge? It's mainly because Routes is relied on by quite a few other setuptools-enabled packages, so being able to easy install it was necessary. I didn't have a non-setuptools build mainly because I couldn't see how to setup a setup.py file in such a way that I could make both versions at once. I'm assuming I'd need two setup.py's and to swap them in the build depending on if it was a setuptools build or not. Not a lot of fun, but I can see the utility of it for those not wishing to grapple with setuptools. Does this mean there should perhaps be some criteria or a recommendation to developers using setuptools on when they should still supply a non-setuptools build of their project? - Ben From dangoor at gmail.com Thu Feb 9 17:10:29 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 9 Feb 2006 11:10:29 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> Message-ID: <3f085ecd0602090810ufc7a31fie7207cfe420eb420@mail.gmail.com> On 2/9/06, Ben Bangert wrote: > It's mainly because Routes is relied on by quite a few other > setuptools-enabled packages, so being able to easy install it was > necessary. I didn't have a non-setuptools build mainly because I > couldn't see how to setup a setup.py file in such a way that I could > make both versions at once. I'm assuming I'd need two setup.py's and > to swap them in the build depending on if it was a setuptools build > or not. Could you do something like this: try: from setuptools import setup except ImportError: from distutils.core import setup On your system, you'd then be able to build eggs at will. Other people who download an sdist but don't have setuptools will just get normal sdist-like behavior. Kevin From gh at ghaering.de Thu Feb 9 17:35:18 2006 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Thu, 09 Feb 2006 17:35:18 +0100 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> Message-ID: <43EB6F46.8090508@ghaering.de> Ben Bangert wrote: > I didn't have a non-setuptools build mainly because I > couldn't see how to setup a setup.py file in such a way that I could > make both versions at once. I'm assuming I'd need two setup.py's and > to swap them in the build depending on if it was a setuptools build > or not. [...] See http://initd.org/tracker/pysqlite/changeset/227 I just changed pyqlite's build process so that you can build without setuptools again. setup.py is the traditional distutils script. extended_setup.py is the one that imports everything from setup.py and adds additional setuptools features. HTH, -- Gerhard From pje at telecommunity.com Thu Feb 9 18:04:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:04:06 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> Message-ID: <5.1.1.6.0.20060209115551.01f4d350@mail.telecommunity.com> At 07:59 AM 2/9/2006 +0000, Michael Twomey wrote: >Hi, > >Joe Gregorio has a fairly negative experience with easy_install here: >http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ > >I think his points boil roughly down to these: > >1. (not directly related to his first comment, but I think this might >be a source of problems) easy_install isn't always called >easy_install. In my case I've wound up with easy_install.py on some >systems at some point in the past. Would a possible solution be to >allow "python -m easy_install"? Then again I think it's pretty >consistent these days. It has been consistent throughout 0.6, and I think it was even at some point in 0.5. There is still an "easy_install" module whose sole purpose is to tell you that "python -m easy_install" doesn't work any more, and that will go away in 0.7. >2. No --prefix option. I think this is a case of RTFM, since it's not >normally as simple as specifying a prefix. Maybe a help message which >explains why --prefix isn't supported might be useful? After thinking this over, I've decided that by 0.6b1 (yes, I'm going beta) easy_install should be able to detect that it's self-installing and should verify that your system is adequately prepared for installation, and if not, display helpful hints about what's wrong and links to the docs. That should fix the RTFM issue here, and bypass the concern about whether package authors are informing people about TFM. :) In the interim, I think I should probably create a user-targeted quickstart page, along the lines of "If you've never used EasyInstall, here's what you should know", and ask package authors to link to it. >3. In the error message he gets it shows bog standard distutils help, >so he tries --help-commands and gets an error. I think this is a >definite source of frustration. Yeah, I've been putting off fixing that because customizing that message is basically impossible without copying a crapload of code. But it's definitely a requirement for 0.6b1. >4. He tries the --install-dir option which installs the package but of >course this leads to a multi-install so he can't do a plain import >foo. It might be good to link to >http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations >for an explanation here. Yeah, and a shorter URL would help, too. :) But mainly, checking the configuration on an initial install to make sure the system's configured to begin with would be helpful. Maybe eventually an interactive configuration mode that edits your config files for you would be even better. >5. At this point he gives up and copies the source from svn. > >I think in all the above there is an element of RTFM but he does have >a legitimate point when it comes to the command line help, based off >that alone you aren't likely to figure out the best way to use >easy_install if you are doing a non-system python install. I think a >link to http://peak.telecommunity.com/DevCenter/EasyInstall in the >help would work wonders. From pje at telecommunity.com Thu Feb 9 18:14:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:14:01 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <3f085ecd0602090810ufc7a31fie7207cfe420eb420@mail.gmail.com > References: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> Message-ID: <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> At 11:10 AM 2/9/2006 -0500, Kevin Dangoor wrote: >On 2/9/06, Ben Bangert wrote: > > It's mainly because Routes is relied on by quite a few other > > setuptools-enabled packages, so being able to easy install it was > > necessary. I didn't have a non-setuptools build mainly because I > > couldn't see how to setup a setup.py file in such a way that I could > > make both versions at once. I'm assuming I'd need two setup.py's and > > to swap them in the build depending on if it was a setuptools build > > or not. > >Could you do something like this: > >try: > from setuptools import setup >except ImportError: > from distutils.core import setup > >On your system, you'd then be able to build eggs at will. Other people >who download an sdist but don't have setuptools will just get normal >sdist-like behavior. Almost. You'll have to use MANIFEST.in or they won't be able to do bdist_rpm or certain other commands from your sdist. That is, you can't use setuptools' revision control features, since they won't have them available. Aside from that minor point, this is the approach I'd recommend to authors who are only using setuptools in order to build eggs and to upload stuff to PyPI, and aren't pointing their users to the docs. You will have to forego all of the normal conveniences of setuptools, however. Personally, however, I think that the short statement somebody else posted about "hey, I use setuptools, and if you're installing anywhere besides your system's site-packages directory you should check out this link" (to the Custom Installation Locations doc) is more than sufficient. I've been keeping setuptools in "alpha" mainly for API-change reasons, but it seems to me that the massive exposure to eggs given by TurboGears has shaken out all but the "new user hasn't read the doc and needs a custom install" problems. What impressed me most about Joe's blog post is not his installation failed, but just how close he was to actual success without even reading the documentation. So, as far as I'm concerned, as soon as it's at the point where he'd have succeeded in what he did, setuptools should qualify as beta quality. And that's just a bit of spit-and-polish away. So even though I'm not thrilled about the blog post, it's actually a pretty positive milestone in two ways: 1. It highlights how little there is left to go to get to a quality installation experience 2. It highlights how popular setuptools has already become. From pje at telecommunity.com Thu Feb 9 18:21:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:21:11 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <43EB3621.9010008@zope.com> References: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote: >I really don't think the virtual python approach is viable, at a minimum >because it doesn't work on windows. It is also unacceptably heavy IMO. What do you mean by "heavy"? >I think we need a simple way to handle custom directories that is: > >- cross platform > >- doesn't require modifying the Python install > >- allows chained/multile custom directories, which means the trick > of putting eggs and scripts in the same directory doesn't work, PYTHONPATH-based installs allow all of this, as long as you add the setuptools .egg to PYTHONPATH. >- is simple. :) PYTHONPATH is simple everywhere but Windows, and it's complex there only because you have to right click a bunch of icons to get to the editing facility. ISTM that what's missing from your requirements statement is how you would like to activate or deactivate these various custom directories. An alternate Python executable? Script wrappers? Commands you run to set what directories are active? >It appears that this has something to do with your plugin proposal. Not that I'm aware of, actually. The plugin facility is intended for an "application instance" path, distinct from a general/global installation location. >I'll try to participate in that discussion to tease this out a bit more. Design and implementation assistance greatly appreciated, on this thread as well as that one. From pje at telecommunity.com Thu Feb 9 18:23:42 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:23:42 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.co m> References: <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> Message-ID: <5.1.1.6.0.20060209122159.01f74978@mail.telecommunity.com> At 11:55 AM 2/9/2006 +0000, Paul Moore wrote: >explain this prominently somewhere: "This package uses setuptools, which >is currently in alpha status - there may be issues installing or using >the software. If you hit any problems, please report them to the >distils-sg, and thank you for helping to test setuptools". +1. This is perfect. That's why I wrote the "what your users should know" section, even though it wasn't available until recently. From pje at telecommunity.com Thu Feb 9 18:26:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:26:03 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43EB3267.9020803@zope.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209122347.01f6fb30@mail.telecommunity.com> At 07:15 AM 2/9/2006 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >>I recently started work on adding egg support to Chandler ( >>http://chandler.osafoundation.org/ ), and ran into some interesting >>issues with respect to plugin discovery. Specifically, it's not easy to >>do it well with the APIs that pkg_resources currently offers. I suspect >>that others who've worked on plugin loading for application environments >>like Zope and Trac have probably run into similar issues. >>I'm proposing, therefore, to add a new API to pkg_resources to make >>plugin-finding easier. Among the requirements: > >I don't fully understand the goal here. From later discussion, I think >you envision a model where people drop eggs into some directory and an >application should be able to analyze this directory to determine >which ones to use, meaning which to include in some working set. >In addition to automatically determining the working >set, the application should be able to find the entry points in that >working set. > >Does that capture what you want to do? Yes, similar to the Zope 2 Basket product or Trac's plugin facility. It also relates to non-entrypoint metadata, like the resource system that Ian and I have been discussing. The goal there is to allow the located plugin eggs to offer translations, localizations, skins, etc. for other eggs to use. I expect this will be relevant to Zope also, so I'd appreciate your input there as well. I've been away from Zope 3 a little too long to know whether the ideas we're discussing can be made to work for it as well. From pje at telecommunity.com Thu Feb 9 18:30:46 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 12:30:46 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> References: <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> Message-ID: <5.1.1.6.0.20060209122628.03779b90@mail.telecommunity.com> At 08:02 AM 2/9/2006 -0800, Ben Bangert wrote: >On Feb 9, 2006, at 3:55 AM, Paul Moore wrote: > > > If routes *needed* setuptools functionality, then fine - but explain > > this prominently somewhere: "This package uses setuptools, which is > > currently in alpha status - there may be issues installing or using > > the software. If you hit any problems, please report them to the > > distils-sg, and thank you for helping to test setuptools". But clearly > > routes does not need setuptolols functionality (or the Routes tests > > aren't complete - as Joe said that all the tests run). So why not > > provide a non-setuptools build, for users who don't want to fight with > > the bleeding edge? > >It's mainly because Routes is relied on by quite a few other >setuptools-enabled packages, so being able to easy install it was >necessary. FYI, easy_install works with distutils packages just fine, as long as they don't get too fancy in their customization. >I didn't have a non-setuptools build mainly because I >couldn't see how to setup a setup.py file in such a way that I could >make both versions at once. I'm assuming I'd need two setup.py's and >to swap them in the build depending on if it was a setuptools build >or not. > >Not a lot of fun, but I can see the utility of it for those not >wishing to grapple with setuptools. Does this mean there should >perhaps be some criteria or a recommendation to developers using >setuptools on when they should still supply a non-setuptools build of >their project? I think the criterion is simply this: if you don't want to have to inform your users about the things listed in the "What Your Users Should Know" section of the manual, then you should make setuptools optional for the time being. When that section of the manual is effectively built in to setuptools, then you won't need to supply that extra information. However, in most cases the notification to users can be as simple as Paul Moore's suggested wording and a link to the Custom Installation docs. From strawman at astraw.com Thu Feb 9 19:38:31 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 09 Feb 2006 10:38:31 -0800 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> References: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> Message-ID: <43EB8C27.8050104@astraw.com> Phillip J. Eby wrote: >At 11:10 AM 2/9/2006 -0500, Kevin Dangoor wrote: > > >>Could you do something like this: >> >>try: >> from setuptools import setup >>except ImportError: >> from distutils.core import setup >> >>On your system, you'd then be able to build eggs at will. Other people >>who download an sdist but don't have setuptools will just get normal >>sdist-like behavior. >> >> > >Almost. You'll have to use MANIFEST.in or they won't be able to do >bdist_rpm or certain other commands from your sdist. That is, you can't >use setuptools' revision control features, since they won't have them >available. > > Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered. Matplotlib now has setupegg.py: from setuptools import setup execfile('setup.py', {'additional_params' : {'namespace_packages' : ['matplotlib.toolkits']}}) I guess the point is that just because someone has setuptools installed on their system doesn't mean they want to use it. Another option, for a single setup.py, is something like the following: if 'setuptools' in sys.modules: use_setuptools_options() This way, setup.py can be setuptools-aware without doing 'import setuptools', but the user would have to do: python -c "import setuptools; execfile('setup.py')" From robert.kern at gmail.com Thu Feb 9 20:31:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 09 Feb 2006 13:31:35 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43EB8C27.8050104@astraw.com> References: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> Message-ID: Andrew Straw wrote: > Note that matplotlib tried essentially this for a while, but apparently > some folks really didn't like it. I'm not sure what exactly broke on > their systems (they didn't complain to the mailing list), but when > setup.py reverted to a plain distutils script, they cheered. That's right, they complained to me, instead. :-) The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg. > Matplotlib > now has setupegg.py: > > from setuptools import setup > execfile('setup.py', > {'additional_params' : > {'namespace_packages' : ['matplotlib.toolkits']}}) > > I guess the point is that just because someone has setuptools installed > on their system doesn't mean they want to use it. > > Another option, for a single setup.py, is something like the following: > > if 'setuptools' in sys.modules: > use_setuptools_options() > > This way, setup.py can be setuptools-aware without doing 'import > setuptools', but the user would have to do: > python -c "import setuptools; execfile('setup.py')" And this is my preferred method for those packages that want to provide the appropriate information for egg builds but not (yet) *require* setuptools or egg builds. It means that "easy_install somepackage==dev" works (a failing of the setupegg.py approach) and allows "python setup.py install" to work as standard distutils does. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pje at telecommunity.com Thu Feb 9 20:34:40 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 14:34:40 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43EB8C27.8050104@astraw.com> References: <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209143104.01f37c88@mail.telecommunity.com> At 10:38 AM 2/9/2006 -0800, Andrew Straw wrote: >Note that matplotlib tried essentially this for a while, but apparently >some folks really didn't like it. I'm not sure what exactly broke on >their systems (they didn't complain to the mailing list) I wish more people would complain in useful ways; at least the blog post that started this thread was *specific*, even if it was directed to the wrong forum. From pje at telecommunity.com Thu Feb 9 20:45:53 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 14:45:53 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: References: <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> Message-ID: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> At 01:31 PM 2/9/2006 -0600, Robert Kern wrote: >Andrew Straw wrote: > > > Note that matplotlib tried essentially this for a while, but apparently > > some folks really didn't like it. I'm not sure what exactly broke on > > their systems (they didn't complain to the mailing list), but when > > setup.py reverted to a plain distutils script, they cheered. > >That's right, they complained to me, instead. :-) > >The issue was that they had setuptools on their system because they were >building eggs for another project, but specifically *didn't* want to install >matplotlib as an egg. setuptools-based packages can be forced to install the old-fashioned way using: setup.py install --single-version-externally-managed as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record. (This is covered in the "What Your Users Should Know" section of the setuptools manual, btw, under the subheading "Creating System Packages".) If somebody could write a standalone version of the What Your Users Should Know section that everybody can link to, that would likely help resolve these problems sooner. (I'll get to it eventually if nobody else does.) Setuptools now has most of the features that are needed to get used in these situations, but the information that end users need isn't quickly accessible from the current docs. This is a really good place to be, relatively speaking, as it means a lot of progress has been made: the stuff works and it's actually documented, too! It's just buried in reference docs and developer guides for people who actively *want* to use the tools, and not pulled out in a quick reference form good enough for busy folks who just want to *use* a package that just happens to be based on setuptools. From bob at redivi.com Thu Feb 9 20:55:21 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 9 Feb 2006 11:55:21 -0800 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43EB8C27.8050104@astraw.com> References: <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> Message-ID: <229010A6-B5D0-4AAC-9F31-266D8B4A01E5@redivi.com> On Feb 9, 2006, at 10:38 AM, Andrew Straw wrote: > This way, setup.py can be setuptools-aware without doing 'import > setuptools', but the user would have to do: > python -c "import setuptools; execfile('setup.py')" Maybe we should get an easy_setup that does this? easy_install is fine if you want to install, but not if you want to bdist_egg or use any of the other setuptools-specific features. -bob From robert.kern at gmail.com Thu Feb 9 21:10:20 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 09 Feb 2006 14:10:20 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> References: <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> Message-ID: <43EBA1AC.3060801@gmail.com> Phillip J. Eby wrote: > At 01:31 PM 2/9/2006 -0600, Robert Kern wrote: > >> Andrew Straw wrote: >> >> > Note that matplotlib tried essentially this for a while, but apparently >> > some folks really didn't like it. I'm not sure what exactly broke on >> > their systems (they didn't complain to the mailing list), but when >> > setup.py reverted to a plain distutils script, they cheered. >> >> That's right, they complained to me, instead. :-) >> >> The issue was that they had setuptools on their system because they were >> building eggs for another project, but specifically *didn't* want to >> install >> matplotlib as an egg. > > setuptools-based packages can be forced to install the old-fashioned way > using: > > setup.py install --single-version-externally-managed > > as long as you also specify a --root directory or a --record file. This > is of course not upgradeable or uninstallable without help from a > packaging tool that can utilize the results of --root or --record. And this particular user did not want that, either. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From jjl at pobox.com Thu Feb 9 21:07:08 2006 From: jjl at pobox.com (John J Lee) Date: Thu, 9 Feb 2006 20:07:08 +0000 (GMT Standard Time) Subject: [Distutils] Changes to --find-links and --upgrade in SVN In-Reply-To: <5.1.1.6.0.20060208004729.02164e60@mail.telecommunity.com> References: <5.1.1.6.0.20060208004729.02164e60@mail.telecommunity.com> Message-ID: On Wed, 8 Feb 2006, Phillip J. Eby wrote: [...] > happen. EasyInstall will now *only* go online if a dependency can't be > resolved locally, if -U or --upgrade is used, or if you provided suitable > direct URLs via an argument or --find-links, or via a link in a local .html > file. Great, that annoyed me too. John From pje at telecommunity.com Thu Feb 9 21:36:28 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 15:36:28 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43EBA1AC.3060801@gmail.com> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> At 02:10 PM 2/9/2006 -0600, Robert Kern wrote: >Phillip J. Eby wrote: > > At 01:31 PM 2/9/2006 -0600, Robert Kern wrote: > > > >> Andrew Straw wrote: > >> > >> > Note that matplotlib tried essentially this for a while, but apparently > >> > some folks really didn't like it. I'm not sure what exactly broke on > >> > their systems (they didn't complain to the mailing list), but when > >> > setup.py reverted to a plain distutils script, they cheered. > >> > >> That's right, they complained to me, instead. :-) > >> > >> The issue was that they had setuptools on their system because they were > >> building eggs for another project, but specifically *didn't* want to > >> install > >> matplotlib as an egg. > > > > setuptools-based packages can be forced to install the old-fashioned way > > using: > > > > setup.py install --single-version-externally-managed > > > > as long as you also specify a --root directory or a --record file. This > > is of course not upgradeable or uninstallable without help from a > > packaging tool that can utilize the results of --root or --record. > >And this particular user did not want that, either. I'm confused. They didn't want to be able to uninstall? Didn't want to point --record to /dev/null, or --root to /? Or something else? From robert.kern at gmail.com Thu Feb 9 21:55:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 09 Feb 2006 14:55:36 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 02:10 PM 2/9/2006 -0600, Robert Kern wrote: > >>Phillip J. Eby wrote: >>>setuptools-based packages can be forced to install the old-fashioned way >>>using: >>> >>> setup.py install --single-version-externally-managed >>> >>>as long as you also specify a --root directory or a --record file. This >>>is of course not upgradeable or uninstallable without help from a >>>packaging tool that can utilize the results of --root or --record. >> >>And this particular user did not want that, either. > > I'm confused. They didn't want to be able to uninstall? Didn't want to > point --record to /dev/null, or --root to /? Or something else? Didn't want setuptools involved at all. He's wasted more hours on it than he ever really wanted to in the days before non-root installs were reasonably documented. He just doesn't trust it. He only has setuptools installed at all because I've convinced him to distribute eggs of his project (and that I will help him troubleshoot setuptools issues). -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From ianb at colorstudy.com Thu Feb 9 22:45:08 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Feb 2006 15:45:08 -0600 Subject: [Distutils] A prefix option? In-Reply-To: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> References: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> Message-ID: <43EBB7E4.5060702@colorstudy.com> Phillip J. Eby wrote: >>I think we need a simple way to handle custom directories that is: >> >>- cross platform >> >>- doesn't require modifying the Python install >> >>- allows chained/multile custom directories, which means the trick >> of putting eggs and scripts in the same directory doesn't work, > > > PYTHONPATH-based installs allow all of this, as long as you add the > setuptools .egg to PYTHONPATH. Would putting sys.path manipulations directly into the generated scripts handle all these issues, then? Including the Windows problem of environmental variables being hard to access. >>It appears that this has something to do with your plugin proposal. > > > Not that I'm aware of, actually. The plugin facility is intended for an > "application instance" path, distinct from a general/global installation > location. Aren't we talking about setting up application instance paths here, except where the application is Python itself? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Feb 9 23:03:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 17:03:57 -0500 Subject: [Distutils] DWIM installation with setuptools Message-ID: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> Okay, so I've been thinking about all the recent complaints/requests regarding the perils of PYTHONPATH, site-packages, --prefix, and all that. And I'm thinking, is there some way I can work around all this stuff so that installation is totally DWIMmish -- that is, that the install can just "do what I mean", while still allowing a default version of a package to be set. This already works perfectly with root access or a virtual Python, so we can pretty much ignore those setups for this discussion (except to make sure we don't break them). So, mostly what's left to worry about is being able to select an arbitrary installation location, and still have it work. If we assume you're smart enough to pick a location that will already be on your PYTHONPATH (and we could verify that by checking that it already is), then all we have to worry about is making sure that the right version of a package gets added to sys.path at runtime. For scripts this isn't that big of a deal. I could create a specialized bootstrap loader version of pkg_resources.py and dump it alongside any scripts installed by easy_install. It'd be a pain to implement, but I could probably make it work such that sys.path was searched for a setuptools egg, and then have that get used to bootstrap the rest. What this doesn't work for is the interactive interpreter, or code that doesn't use require(). Even if the relevant directory is on sys.path, Python doesn't normally read .pth files in PYTHONPATH directories. I've also learned that many Linux distros appear to hack sys.path in odd ways, and I'm not positive that all of them process .pth files in the directories they add. Certainly they don't support .pth for PYTHONPATH directories. Since it's not just the interactive interpreter, hacking .pythonrc isn't going to help much. That leaves the 'site' and 'sitecustomize' modules. The current PYTHONPATH install support hijacks the 'site' module with a special .pth-honoring feature, so as long as you have the setuptools egg listed directly in PYTHONPATH, you're golden. But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH *before* you can reasonably do anything. Part of the essence of what we desire is that any required steps be done for you *during* the installation rather than requiring an extra step before or afterwards. Unfortunately, this means that hacking 'site' can't work unless we actually install a site.py. Will that work? Well, yes and no. It appears Python will recognize an added site.py that appears in a PYTHONPATH, but many platforms will not recognize a site.py that's in the script directory but not PYTHONPATH. Windows seems to be the only platform where putting a site.py in the script directory has any effect. This is really good news, however. It appears to mean that slapping a copy of my hacked 'site.py' (i.e., the one that honors .pth files along PYTHONPATH) into the easy_install target directory would make PYTHONPATH-based installs essentially DWIMmish. To complete the illusion, I'd need to add some code to script wrappers to process the setuptools.pth file in the script directory if pkg_resources can't be otherwise imported. We could then lobby for the Python 2.5 site.py to behave the same way as my hacked one does: i.e., process .pth files found in PYTHONPATH directories, inserting them ahead of site-packages and its ilk, so that it wouldn't be necessary to install the hacked site.py in such cases. (It's also not necessary to install it in known "site" directories, such as site-packages and any directories listed in --site-dirs. But even if it's installed unnecessarily, it shouldn't hurt anything.) Since 'site' is a stdlib module and isn't normally intended for customization, it shouldn't conflict with any user-installed modules. However, easy_install could detect an existing site.py and warn if its content is different from the one easy_install wants to use. I'm trying to remember why I didn't do this around the time I implemented the original PYTHONPATH support (which first required a hacked 'site' module). I think I was thrown by the lack of cross-platform script directory support for importing 'site', and it didn't occur to me that I could easily work around that from within scripts. Also, it's easy to mentally confuse the script directory and the install directory while thinking about some of this stuff. But as long as the install directory (irrespective of the script directory) is on PYTHONPATH and has a hacked site.py, what's in the script directory doesn't matter. The only case where the script directory matters is if: 1. it's the same directory the libraries were installed to, 2. it's not on PYTHONPATH, and 3. the user puts a script of their own in there and expects it to work anywhere but Windows This edge case could be handled, however, by refusing to install libraries to non-PYTHONPATH directories unless you're on Windows or you use --multi-version. I'm somewhat uncertain about this, though, because it doesn't seem to let you create a self-sufficient application directory. Right now, you can use "easy_install -ad somedir ..." to copy all needed eggs and scripts to the target directory, making a self-contained application directory that has everything it needs to run. OTOH, I suppose if you just made that -mad instead of -ad, it would work just fine for that purpose under the proposed new regime. So, to sum up, here are the changes that would take place: * easy_install would treat all PYTHONPATH-specified directories as though they were listed in --site-dirs, but it would also copy and compile a hacked site.py into them during installation, if they aren't *actually* listed in --site-dirs or in setuptools' default computation of "site" directories. * Currently, easy_install defaults to -m if you install to a non-"site" directory; it would now instead stop and refuse to install to such directories unless you explicitly specify -m. * easy_install could now meaningfully grow a --prefix option, since it will now DWIM as long as the target dir is recognizable as (or convertible to) a "site" directory. And, in the cases where DWIM can't be guaranteed, it will stop and inform the user that they need to fix PYTHONPATH, --site-dirs, or use -m/--multi-version mode. What do y'all think? Do we have a winner yet? From pje at telecommunity.com Thu Feb 9 23:09:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 17:09:41 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <43EBB7E4.5060702@colorstudy.com> References: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209170416.01f5cb50@mail.telecommunity.com> At 03:45 PM 2/9/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>>I think we need a simple way to handle custom directories that is: >>> >>>- cross platform >>> >>>- doesn't require modifying the Python install >>> >>>- allows chained/multile custom directories, which means the trick >>> of putting eggs and scripts in the same directory doesn't work, >> >>PYTHONPATH-based installs allow all of this, as long as you add the >>setuptools .egg to PYTHONPATH. > >Would putting sys.path manipulations directly into the generated scripts >handle all these issues, then? Including the Windows problem of >environmental variables being hard to access. I've just posted an alternative idea for how to go about this, that would allow a "pure PYTHONPATH" (i.e., without needing a hard-coded setuptools .egg in it) to work. Let me know what you think. >>>It appears that this has something to do with your plugin proposal. >> >>Not that I'm aware of, actually. The plugin facility is intended for an >>"application instance" path, distinct from a general/global installation >>location. > >Aren't we talking about setting up application instance paths here, except >where the application is Python itself? Hm. I was going to say no, that find_plugins() is something with much more restricted utility, but then it occurred to me that it certainly would make it possible to find the latest usable eggs that are installed and put them on sys.path. :) I don't think it's a good idea, though, because until 0.7 there isn't a way to later change your mind and decide you'd like a different version of something. And of course the cost to invoke find_plugins() is enormously higher than the cost of Python processing .pth entries at startup. Anyway, I think the other idea ( http://mail.python.org/pipermail/distutils-sig/2006-February/006026.html ) may fix the issue a bit better. From ianb at colorstudy.com Thu Feb 9 23:08:32 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Feb 2006 16:08:32 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> Message-ID: <43EBBD60.6090405@colorstudy.com> Robert Kern wrote: >>>>setuptools-based packages can be forced to install the old-fashioned way >>>>using: >>>> >>>> setup.py install --single-version-externally-managed >>>> >>>>as long as you also specify a --root directory or a --record file. This >>>>is of course not upgradeable or uninstallable without help from a >>>>packaging tool that can utilize the results of --root or --record. >>> >>>And this particular user did not want that, either. >> >>I'm confused. They didn't want to be able to uninstall? Didn't want to >>point --record to /dev/null, or --root to /? Or something else? > > > Didn't want setuptools involved at all. He's wasted more hours on it than he > ever really wanted to in the days before non-root installs were reasonably > documented. He just doesn't trust it. He only has setuptools installed at all > because I've convinced him to distribute eggs of his project (and that I will > help him troubleshoot setuptools issues). Setuptools and people using setuptools can only deal with actual issues, not with vague dislikes. This is open source, issues get resolved, but there's never any guarantee that issues don't *exist*. Without --single-version-externally-managed setuptools acts in ways that can be difficult to handle if .pth files don't work, or you are installing to an odd location, or something like that. It would be nice if all those issues can be documented and have useful/easy-to-resolve failure cases, and all that -- but it's not quite there yet, in part because it's not completely clear how to make all that work. *With* --single-version-externally-managed, it acts just like distutils always has. Right now one project of mine isn't using setuptools exclusively, because it still has Python 2.2 compatibility. But I see no reason to do all sorts of fiddling to support multiple installation techniques, and deal with tensions about what features can and can't be used because of multiple supported installation processes. Right now, the question should be: * Can we provide an experience similar enough to distutils to make conservative users happy? If they weren't happy with distutils, then whatever. I think --single-version-externally-managed does that. * Are we documenting this properly? That includes, are they going to be aware of the appropriate documentation at the right time in their experience. So, knowing about Python setup options before they start. Knowing about setuptools installation issues when they are using ez_setup.py. Knowing about package finding when they are using easy_install, or what they need to understand about eggs and .pth files during actual installation. * Are we providing a cohesive experience for the users who want something better than the previous status quo? Are we guiding them towards good practices? I've started using setuptools features even in packages that don't use plugins or any Egg features. A lot of these will work with distutils, but fail in weird or confusing ways (e.g., package data missing). Or I'll have to maintain an easy and an explicit way to do these things, and they'll get out of sync. I don't think it's going to be a particularly good experience for the conservative users, and it won't accomplish anything. At least when we struggle with setuptools it accomplishes something. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Feb 9 23:51:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 17:51:58 -0500 Subject: [Distutils] /usr/lib64 site-packages (was Re: installation troubles) In-Reply-To: <200602091704.50059.kap4020@rit.edu> Message-ID: <5.1.1.6.0.20060209174307.01f5ff70@mail.telecommunity.com> At 05:04 PM 2/9/2006 -0500, Karl Pietrzak wrote: >creating /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg >Extracting setuptools-0.6a9-py2.4.egg to /usr/lib64/python2.4/site-packages >Installing easy_install script to /usr/bin > >Installed /usr/lib64/python2.4/site-packages/setuptools-0.6a9-py2.4.egg FYI, I just did a bit more research and confirmed this as a packaging problem with some 64-bit Python ports; see this thread for some background: http://mail.python.org/pipermail/distutils-sig/2005-April/004472.html Apparently, none of the Linux vendors that patch Python to install to /usr/lib64 catch all of the places that need to be patched. Some, like Fedora, however, patch it enough that I can add a workaround in setuptools to address the issue. I'm working on a candidate fix now. Are you using a Fedora-based distribution, or something else? Thanks. From ianb at colorstudy.com Fri Feb 10 00:14:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Feb 2006 17:14:16 -0600 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> References: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> Message-ID: <43EBCCC8.4070209@colorstudy.com> Phillip J. Eby wrote: > This is really good news, however. It appears to mean that slapping a copy > of my hacked 'site.py' (i.e., the one that honors .pth files along > PYTHONPATH) into the easy_install target directory would make > PYTHONPATH-based installs essentially DWIMmish. To complete the illusion, > I'd need to add some code to script wrappers to process the setuptools.pth > file in the script directory if pkg_resources can't be otherwise imported. I think I kind of understand, but I'm not sure I have (or remember) the full context behind this proposal. What hacked site.py are we talking about? What exactly will the generated scripts look like in this model? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From titus at caltech.edu Fri Feb 10 00:07:09 2006 From: titus at caltech.edu (titus at caltech.edu) Date: Thu, 9 Feb 2006 15:07:09 -0800 Subject: [Distutils] easy_install problem with Twisted-2.1.0 Message-ID: <20060209230709.GA8662@caltech.edu> Hi all, here's a quick bug report for ya. Happened with both Python 2.3 and 2.4, using latest easy_install. % easy_install Twisted-2.1.0.tar.bz2 Processing Twisted-2.1.0.tar.bz2 Running Twisted-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-5-YDh-/Twisted-2.1.0/egg-dist-tmp-9yveKV Traceback (most recent call last): File "/usr/bin/easy_install2.4", line 7, in ? sys.exit( File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 1138, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 230, in run self.easy_install(spec, not self.no_deps) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 303, in easy_install return self.install_item(None, spec, tmpdir, deps, True) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 338, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 532, in install_eggs return self.build_and_install(setup_script, setup_base) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 807, in build_and_install self.run_setup(setup_script, setup_base, args) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/easy_install.py", line 796, in run_setup run_setup(setup_script, args) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 26, in run_setup DirectorySandbox(setup_dir).run( File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 63, in run return func() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/sandbox.py", line 29, in {'__file__':setup_script, '__name__':'__main__'} File "setup.py", line 113, in ? File "./twisted/python/dist.py", line 69, in setup File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/bdist_egg.py", line 167, in run self.run_command("egg_info") File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/egg_info.py", line 129, in run writer(self, ep.name, os.path.join(self.egg_info,ep.name)) File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/egg_info.py", line 321, in write_toplevel_names pkgs = dict.fromkeys( File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/dist.py", line 630, in iter_distribution_names yield ext.name AttributeError: 'bool' object has no attribute 'name' From pje at telecommunity.com Fri Feb 10 01:31:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 19:31:44 -0500 Subject: [Distutils] easy_install problem with Twisted-2.1.0 In-Reply-To: <20060209230709.GA8662@caltech.edu> Message-ID: <5.1.1.6.0.20060209192853.03686c60@mail.telecommunity.com> At 03:07 PM 2/9/2006 -0800, titus at caltech.edu wrote: >Hi all, > >here's a quick bug report for ya. Happened with both Python 2.3 and >2.4, using latest easy_install. Please see http://mail.python.org/pipermail/distutils-sig/2006-February/005960.html Better yet, would you update the experiences page: http://peak.telecommunity.com/DevCenter/PackageNotes to link there? This is unfortunately not fixable in 0.6, since setuptools doesn't yet offer an optional-extension feature, at least not one I'd want to try talking the Twisted folks into using. :) From pje at telecommunity.com Fri Feb 10 01:45:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Feb 2006 19:45:59 -0500 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <43EBCCC8.4070209@colorstudy.com> References: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060209181649.04c2afb8@mail.telecommunity.com> At 05:14 PM 2/9/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>This is really good news, however. It appears to mean that slapping a >>copy of my hacked 'site.py' (i.e., the one that honors .pth files along >>PYTHONPATH) into the easy_install target directory would make >>PYTHONPATH-based installs essentially DWIMmish. To complete the >>illusion, I'd need to add some code to script wrappers to process the >>setuptools.pth file in the script directory if pkg_resources can't be >>otherwise imported. > >I think I kind of understand, but I'm not sure I have (or remember) the >full context behind this proposal. What hacked site.py are we talking about? I meant the one that's bundled inside the setuptools .egg right now, but the lib64 issue has now made me a bit more wary. I'm trying to design an alternate hacked site.py that uses the system site.py, so that if the distro uses a patched site.py, it'll still work correctly. My thoughts for the new site.py are that it'll just do two things: 1. find the stdlib on sys.path (by skipping all the PYTHONPATH entries) and use sneaky import tricks to replace itself with the "real" site module. 2. call addsitedir() on all the directories in PYTHONPATH, then scoot all the added directories to just *before* the stdlib in sys.path (i.e., just after the PYTHONPATH entries) I've painstakingly hacked together some code that accomplishes #1, and am now trying to figure out a sane way to do #2. The big problem is that 'site' munges the bleeding hell out of sys.path, such that there are no guarantees about the position, case, etc. of entries in it after, relative to before. It'd be easiest to just throw all the new entries to the head of the path, but that's not an especially compatible way to do it, relative to what I'd like to see the Python 2.5 site module do. I think what I'm going to do for now is find the first "system" path entry (i.e. the first one after the PYTHONPATH entries. (Side note: one of the other really tricky bits in all this is that Python 2.3 and 2.4 on different platforms appear to treat empty PYTHONPATH differently from each other and from an unspecified PYTHONPATH, and the initial value of sys.path can vary in um, interesting ways as a result. Ugh.) > What exactly will the generated scripts look like in this model? The same as now, but with a try/except around the pkg_resources import, with the except: branch looking for a setuptools.pth file in the script directory. This should only be needed to insure that "-mad" (i.e. self-contained full app installs to non-PYTHONPATH dirs) will work on non-Windows platforms. From jim at zope.com Fri Feb 10 12:40:31 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 10 Feb 2006 06:40:31 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> References: <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> Message-ID: <43EC7BAF.7070305@zope.com> Phillip J. Eby wrote: > At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote: > >> I really don't think the virtual python approach is viable, at a minimum >> because it doesn't work on windows. It is also unacceptably heavy IMO. > > > What do you mean by "heavy"? It essentially duplicates the python install via symbolic links. > >> I think we need a simple way to handle custom directories that is: >> >> - cross platform >> >> - doesn't require modifying the Python install >> >> - allows chained/multile custom directories, which means the trick >> of putting eggs and scripts in the same directory doesn't work, > > > PYTHONPATH-based installs allow all of this, as long as you add the > setuptools .egg to PYTHONPATH. Agreed. I think PYTHONPATH-based installations are close to what some of us need. I think that having to set the path is a hassle that is possible to overcome and worth doing. As I've suggested before, since easy install generates startup scripts, it should be possible to for it to generate startup scripts that take care of setting the path. I think it should be possible to have Python startup scripts invoke setuptools' site.py to add necessary .pth files to the path. If I'm wrong, then another option is for setuptools to generate sh scripts on 'nix platforms, much as it generates exe's on Windows. (I presume that the generates exe files on windows then invoke Python and can also manipulate the environment.) > >> - is simple. :) > > > PYTHONPATH is simple everywhere but Windows, and it's complex there only > because you have to right click a bunch of icons to get to the editing > facility. Eek. I would never consider setting the PYTHONPATH that way on Windows to satisfy this use case. The fact that you suggest this suggests that you missunderstand my use case. OK, I left something out of my bulleted list: - Doesn't require changes to a user's profile or anything else outside the working directory tree. > ISTM that what's missing from your requirements statement is how you > would like to activate or deactivate these various custom directories. > An alternate Python executable? Script wrappers? Commands you run to > set what directories are active? I worry that I'm not setuptools zenfull enough to make the best proposal yet, but here's what I envision so far. To create a workspace, you create the directory and put a setup.cfg in it. You then install easy_install into that directory (typically into that directory's bin directory). Then when you run that directory's easy_install script, eggs are installed into the workspace (e.g. into some sort of library directory within the workspace). In addition, wrapper scripts build during the installation set the path and invoke setuptools' site.py to reflect settings in setup.cfg. You might be able to think of a better way to do this. My goal is to have self contained working directories. In addition, I want to have chained self-contained working directories, which is why the excellent "put scripts and eggs in the same place" solution doesn't work for me (more below). > >> It appears that this has something to do with your plugin proposal. > > > Not that I'm aware of, actually. The plugin facility is intended for an > "application instance" path, distinct from a general/global installation > location. But that's exactly what I'm talking about. To make this more concrete: My hope is that Zope 3.3 will be highly eggs based. People expect to be able to install Zope without odifying Python. Basically, a Zope install is a separate "aplication instance". (Note that to leverage distutils, Zope 3 windows releases have been installed into windows Python installations. This installation experience has been unpopular.) I would want a Zope install to be largely a collection of eggs. I would also want a Zope install to support easy installayion of additional eggs into the Zope install or into a directory associated with the Zope install (probably the later). A Zope install provides a simple application for creating Zope "instances". A Zope instance is what actually runs a Zope application server. Multiple Zope instances can share a single Zope software install. (Of course, multiple Zope sotware installs can share a single Python install.) Of course, I want people to be able to install additional eggs, including eggs with scripts, into the Zope instance. I don't want users to have to manually manipulate paths to be able to use these. Traditionally, in Zope 2, packages installed into Zope instances have been plug-in-like in that they could just be dropped into a directory and magically used. In Zope 3, we've gone for a more explicit approach, which I think I prefer. More on that in the plugin thread, which, sadly, I probably won't have time to contribute more to until this weekend. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From docwhat+list.distutil at gerf.org Fri Feb 10 16:26:30 2006 From: docwhat+list.distutil at gerf.org (Christian Holtje) Date: Fri, 10 Feb 2006 10:26:30 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43EBBD60.6090405@colorstudy.com> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> <43EBBD60.6090405@colorstudy.com> Message-ID: <43ECB0A6.5020807@gerf.org> Ian Bicking wrote: >Setuptools and people using setuptools can only deal with actual issues, >not with vague dislikes. This is open source, issues get resolved, but >there's never any guarantee that issues don't *exist*. > > Hiya! I really like setuptools and the eggs and stuff. I want to help you out by giving you a specific user case. I don't fully understand setuptools, so I apologize in advanced for any errors. I am a system administrator for several systems. I use a distro to manage most of my software, but obviously some python modules don't come with my distro (too new or the distro doesn't have it). I usually solve this by doing the following as a non-root user: |$ /usr/bin/python setup.py install --prefix=/usr/local/stow/projectname-version $ cd /usr/local/stow $ stow -D projectname-oldversion # optional -- removes old version $ stow projectname-version| This allows me to keep the previous version around in case I need to roll back. Here is a (trunicated) tree listing from trac project: /usr/local/stow/trac-svn-r2824 |-- bin |-- lib | `-- python2.3 | `-- site-packages | `-- trac `-- share |-- man | `-- man1 `-- trac Right now my workaround is to use bdist_egg and then copy the egg into the directory /usr/local/stow/project-version/lib/python2.4/site-packages (which I have to mkdir first). This isn't perfect because I don't get non-egg stuff, if any exists. I hope that this helps you. ^_^ Ciao! -- Probable-Possible, my black hen, She lays eggs in the Relative When. She doesn't lay eggs in the Positive Now Because she's unable to postulate How. -- Frederick Winsor The Doctor What: Need I say more? http://docwhat.gerf.org/ docwhat *at* gerf *dot* org KF6VNC -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060210/05ffd461/attachment.html From ianb at colorstudy.com Fri Feb 10 17:51:53 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Feb 2006 10:51:53 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43ECB0A6.5020807@gerf.org> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> <43EBBD60.6090405@colorstudy.com> <43ECB0A6.5020807@gerf.org> Message-ID: <43ECC4A9.3080508@colorstudy.com> Christian Holtje wrote: > Ian Bicking wrote: > >>Setuptools and people using setuptools can only deal with actual issues, >>not with vague dislikes. This is open source, issues get resolved, but >>there's never any guarantee that issues don't *exist*. >> >> > Hiya! I really like setuptools and the eggs and stuff. I want to help > you out by giving you a specific user case. I don't fully understand > setuptools, so I apologize in advanced for any errors. > > I am a system administrator for several systems. I use a distro to > manage most of my software, but obviously some python modules don't come > with my distro (too new or the distro doesn't have it). > > I usually solve this by doing the following as a non-root user: > |$ /usr/bin/python setup.py install > --prefix=/usr/local/stow/projectname-version > $ cd /usr/local/stow > $ stow -D projectname-oldversion # optional -- removes old version > $ stow projectname-version| > > This allows me to keep the previous version around in case I need to > roll back. The discussion in "A prefix option?" and "DWIM installation with setuptools" is, I think, about similar issues. Well, depending on your granularity. If it is just installation, then setuptools doesn't overwrite old versions, it basically just "activates" a particular installed version of a library by changing easy-install.pth, and to a degree with .egg-link files. I think .egg-link could be used to even more effect, creating something like stow, except without relying on symlinks specifically. Instead of a series of symlinks like stow uses, setuptools changes the package loading and creates one reference to the installed version. So maybe this is already allowed. There aren't (yet) very good tools for actually switching versions and viewing the available installed versions of packages, but the actual mechanism to do so is fairly straight-forward (just edit that easy-install.pth file, view using a wildcard, remove with a delete). The other stuff we've been talking about is creating an environment that contains all of the packages for some "project" -- maybe a particular web application, a user's local (non-root) Python installation, or whatever. > Here is a (trunicated) tree listing from trac project: > /usr/local/stow/trac-svn-r2824 > |-- bin > |-- lib > | `-- python2.3 > | `-- site-packages > | `-- trac > `-- share > |-- man > | `-- man1 > `-- trac > > Right now my workaround is to use bdist_egg and then copy the egg into > the directory > /usr/local/stow/project-version/lib/python2.4/site-packages (which I > have to mkdir first). This isn't perfect because I don't get non-egg > stuff, if any exists. There shouldn't be non-egg stuff; if there is, then the package isn't really a good setuptools package. I think setuptools catches attempts to write files elsewhere too, and at least warns about it. While I can understand you want to stick to a stow-based system, at this point stow seems unnecessarily indirect. However, you can always give the -d option or one of the other available options to have easy_install put the egg right where you want it. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Feb 10 18:36:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Feb 2006 12:36:51 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <43EC7BAF.7070305@zope.com> References: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060210115434.01f59ad0@mail.telecommunity.com> At 06:40 AM 2/10/2006 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >>At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote: >> >>>I really don't think the virtual python approach is viable, at a minimum >>>because it doesn't work on windows. It is also unacceptably heavy IMO. >> >>What do you mean by "heavy"? > >It essentially duplicates the python install via symbolic links. That's just a statement of what it does. I was looking for an explanation of what makes that "heavy". :) One assumes that on Unix, directory entries and inodes are relatively cheap. >>PYTHONPATH-based installs allow all of this, as long as you add the >>setuptools .egg to PYTHONPATH. > >Agreed. I think PYTHONPATH-based installations are close to what some of us >need. I think that having to set the path is a hassle that is possible >to overcome and worth doing. As I've suggested before, since easy >install generates startup scripts, it should be possible to for it >to generate startup scripts that take care of setting the path. > >I think it should be possible to have Python startup scripts invoke >setuptools' site.py to add necessary .pth files to the path. The regular site.py suffices for that. What it doesn't suffice for is getting those directories on *ahead* of the system-provided libraries. If you don't care about that, then adding a series of statements like this to the script would suffice: __import__('site').addsitedir("some target dir") Where the series of directories represents your "instance", in the order of highest to lowest precedence. Of course, this is unnecessary *except* for obtaining the location of setuptools itself, unless the goal is to have plugins installed using .pth files, and to manage that setup externally to the actual programs. (By which I mean, the programs themselves have no notion of a "plugin directory", they simply expect everything to be already set up on sys.path when they run.) > If I'm >wrong, then another option is for setuptools to generate sh scripts >on 'nix platforms, much as it generates exe's on Windows. (I presume >that the generates exe files on windows then invoke Python and can >also manipulate the environment.) They could indeed manipulate the environment, though they'd need more info about it. On 'nix platforms it's possible to actually embed a shell script in the beginning of a Python file with a bit of quoting cleverness. >>>- is simple. :) >> >>PYTHONPATH is simple everywhere but Windows, and it's complex there only >>because you have to right click a bunch of icons to get to the editing >>facility. > >Eek. I would never consider setting the PYTHONPATH that way on Windows >to satisfy this use case. The fact that you suggest this suggests that >you missunderstand my use case. Well, you haven't really explained your use case enough for me to understand even what these programs are or what people are doing with them. You just listed some requirements. >My goal is to have self contained working directories. In addition, >I want to have chained self-contained working directories, which is >why the excellent "put scripts and eggs in the same place" solution >doesn't work for me (more below). You could generate a sitecustomize.py in a scripts+eggs directory, and have it do: import site site.addsitedir("this directory") site.addsitedir("another directory") etc. This would basically chain the directories in a way that supports .pth files. Then just easy_install scripts and eggs to the single target directory, including the target directory in --site-dirs so easy_install knows it can use .pth files. Presto - you're done. Running a script would now suffice to set its environment. Note that anything it references via .pth files will be *subordinate* to the same things in site-packages, but since you're presumably controlling what Python install is used for this, and in any case the scripts have require() used to bump up anything the script directly requires, this shouldn't be a big issue. Note that in this setup, sitecustomize.py is the only configuration file you need, and you don't need any upgrades to setuptools as it exists today to make this work. Just slap sitecustomize in the same directory with the scripts and eggs, and all is well. You can, if you wish, modify it to do other sys.path munging according to your needs. Unfortunately, due to differences in path processing on Windows and 'nix, the above procedure appears to only work correctly on Windows. On 'nix, the current dir and/or script dir aren't on sys.path soon enough. *sigh*. Okay, I give up. PYTHONPATH manipulation is the only thing that works. After further experimenting, it seems the only consistent way to get sitecustomize.py to work, on all versions and platforms, is to have it on PYTHONPATH when Python starts execution. In which case, you might as well use the site.py hack and control everything with .pth files. Now you've got me wondering if maybe I should always have easy_install create script wrappers that set a fixed PYTHONPATH. The main problem I see with it is what happens when somebody deliberately sets a different PYTHONPATH. Do you append to it? Prepend to it? Ignore it? Should PYTHONPATH munging be optional? Mandatory? Hrm. Ignoring PYTHONPATH when running a script has security advantages, to be certain. Putting the fixed paths in front of the user-supplied ones is almost as good. This needs a lot more thought, though. Unfortunately, it's making me aware of some deficiencies in the existing script system, which doesn't allow for e.g. running scripts with -O at the moment, let alone any other options. These are allowed in "traditional" distutils scripts, but not entry-point based scripts, which have no place to specify these things at the moment. I'm probably going to have to add a way to control these and other script-generation options at some point. From docwhat+list.distutil at gerf.org Fri Feb 10 19:13:45 2006 From: docwhat+list.distutil at gerf.org (Christian Holtje) Date: Fri, 10 Feb 2006 13:13:45 -0500 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43ECC4A9.3080508@colorstudy.com> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> <43EBBD60.6090405@colorstudy.com> <43ECB0A6.5020807@gerf.org> <43ECC4A9.3080508@colorstudy.com> Message-ID: <43ECD7D9.4050001@gerf.org> Ian Bicking wrote: > There shouldn't be non-egg stuff; if there is, then the package isn't > really a good setuptools package. I think setuptools catches attempts > to write files elsewhere too, and at least warns about it. While I > can understand you want to stick to a stow-based system, at this point > stow seems unnecessarily indirect. However, you can always give the > -d option or one of the other available options to have easy_install > put the egg right where you want it. I didn't see the -d option. That does most of what I need right there. Spiffy. I agree, that some sort of UI that I could manage eggs and versions with would be excellent and a happy replacement for stow. I would like to be able to keep these eggs out of the "distribution"'s directories, though. Since I don't want things being replaced on upgrade, etc. Ciao! -- Fallacy #1 You can't manage what you can't measure. -- "Facts and Fallacies of Software Engineering" (Robert L. Glass, 2002) The Doctor What: A Holtje Production http://docwhat.gerf.org/ docwhat *at* gerf *dot* org KF6VNC From ianb at colorstudy.com Fri Feb 10 20:00:32 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Feb 2006 13:00:32 -0600 Subject: [Distutils] Some negative press for easy_install In-Reply-To: <43ECD7D9.4050001@gerf.org> References: <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <50a522ca0602082359y1d4eb9e0s@mail.gmail.com> <79990c6b0602090355s3bc6c57eifea25b3d005e4849@mail.gmail.com> <4BDCEBCE-C9F8-4D3B-B6A0-A080720A0C9A@groovie.org> <5.1.1.6.0.20060209120420.0367e568@mail.telecommunity.com> <43EB8C27.8050104@astraw.com> <5.1.1.6.0.20060209143459.0412bcb8@mail.telecommunity.com> <43EBA1AC.3060801@gmail.com> <5.1.1.6.0.20060209153439.01f345b8@mail.telecommunity.com> <43EBBD60.6090405@colorstudy.com> <43ECB0A6.5020807@gerf.org> <43ECC4A9.3080508@colorstudy.com> <43ECD7D9.4050001@gerf.org> Message-ID: <43ECE2D0.7040300@colorstudy.com> Christian Holtje wrote: > Ian Bicking wrote: > > >>There shouldn't be non-egg stuff; if there is, then the package isn't >>really a good setuptools package. I think setuptools catches attempts >>to write files elsewhere too, and at least warns about it. While I >>can understand you want to stick to a stow-based system, at this point >>stow seems unnecessarily indirect. However, you can always give the >>-d option or one of the other available options to have easy_install >>put the egg right where you want it. > > > I didn't see the -d option. That does most of what I need right there. > Spiffy. > > I agree, that some sort of UI that I could manage eggs and versions with > would be excellent and a happy replacement for stow. It also occurred to me that --single-version-externally-managed describes exactly what stow implies (stow managing the version). But yes, you don't have to use stow to have stow-like isolation. > I would like to be able to keep these eggs out of the "distribution"'s > directories, though. Since I don't want things being replaced on > upgrade, etc. You can enforce that by putting your preferred installation options in /usr/lib/python2.4/distutils/distutils.cfg (sadly you have to put the configuration in that location or your home directory in ~/.pydistutilssomething). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Feb 10 20:11:19 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Feb 2006 13:11:19 -0600 Subject: [Distutils] A prefix option? In-Reply-To: <5.1.1.6.0.20060210115434.01f59ad0@mail.telecommunity.com> References: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060210115434.01f59ad0@mail.telecommunity.com> Message-ID: <43ECE557.10205@colorstudy.com> Phillip J. Eby wrote: > Unfortunately, due to differences in path processing on Windows and 'nix, > the above procedure appears to only work correctly on Windows. On 'nix, > the current dir and/or script dir aren't on sys.path soon enough. *sigh*. Would it help to use the -S option with the Python interpreter? You get into the nuisance of #! inconsistencies if you want to pass in an option, but maybe that's workable still. On Windows... well, I don't know what you .exe wrapper does on Windows, but maybe you can effectively do the same thing there. Then you could invoke 'import site' when and how you want. Also there's the idea that Jim had of generating shell scripts instead of Python files, and the shell script sets PYTHONPATH. Again, if you already have this kind of control on Windows with the .exe wrapper, then you can make full use of the same thing on Posix. The other thing to think about with this is how this interacts with distutils/easy_install installations. I'd like it to be easy to make installation work seemlessly; just like when you invoke a script in such a isolated setup, it will then import from that setup, if you invoke easy_install from such a setup it should install into the appropriate location. Or there should be another option that controls that, instead of just using -d -- using this other option would do more checks and be more picky, because it would imply you are expecting a very specific (and confirmable) layout. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Feb 10 20:34:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Feb 2006 14:34:02 -0500 Subject: [Distutils] New customization help message for easy_install Message-ID: <5.1.1.6.0.20060210142755.01f77700@mail.telecommunity.com> I'm working on implementing seamless PYTHONPATH support for EasyInstall, and as I previously proposed, it's going to halt with an error message if you try to install somewhere that isn't a --site-dirs directory or on PYTHONPATH (and you're not doing a --multi-version install). Since most of the time a "naive" user will be installing to somewhere that *is* on PYTHONPATH, they probably will never see this message. But I thought it would be good to get some feedback on the proposed wording of the message. ----------------------------------------------------------------------- CONFIGURATION PROBLEM You are attempting to install a package to a directory that is not on PYTHONPATH and is not registered as supporting Python ".pth" files. Here are some of your options for correcting this: * You can choose a different installation directory, i.e., one that is on PYTHONPATH or supports .pth files. * You can add the installation directory to the PYTHONPATH environment variable. (It must then also be on PYTHONPATH whenever you run Python and want to use the package(s) you are installing.) * You can set up the installation directory to support ".pth" files, and configure EasyInstall to recognize this, by using one of the approaches described here: http://peak.telecommunity.com/EasyInstall.html#custom-installation-locations Please make the appropriate changes for your system and try again. Thank you for your patience. ----------------------------------------------------------------------- Comments, anyone? From elvelind at gmail.com Fri Feb 10 21:20:32 2006 From: elvelind at gmail.com (Elvelind Grandin) Date: Fri, 10 Feb 2006 21:20:32 +0100 Subject: [Distutils] pkg_resources.find_distributions("./") Message-ID: <5035593a0602101220x27a89b38hd0e96637550bf1f3@mail.gmail.com> Hi. I'm using pkg_resources.find_distributions("./") to find out the name off the distrubution in the current dir. Is there any reason this shouldn't work? It works for me but apperently some others have problems with it. -- cheers elvelind grandin From pje at telecommunity.com Fri Feb 10 21:33:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Feb 2006 15:33:54 -0500 Subject: [Distutils] pkg_resources.find_distributions("./") In-Reply-To: <5035593a0602101220x27a89b38hd0e96637550bf1f3@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060210152717.042a19c8@mail.telecommunity.com> At 09:20 PM 2/10/2006 +0100, Elvelind Grandin wrote: >I'm using pkg_resources.find_distributions("./") to find out the name >off the distrubution in the current dir. Is there any reason this >shouldn't work? Yes, several. :) find_distributions() by default finds *all* distributions in the directory, and there is no guaranteed order to the results. Even if you pass the 'only' flag to tell it to just find distributions whose 'location' would be the same directory, you can *still* have more than one result in the case of system-packaged (--single-version-externally-managed) eggs, as they all end up in the same directory. > It works for me but apperently some others have >problems with it. There is no reliable way to tell what distribution you are in, if all you have is the directory name. If you have a *module* name, however, you can use get_provider(modulename).get_metadata('PKG-INFO') to retrieve the package's PKG-INFO text, which you can then use to find the distribution name and version. From pje at telecommunity.com Fri Feb 10 23:28:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Feb 2006 17:28:44 -0500 Subject: [Distutils] --prefix and PYTHONPATH support added in SVN Message-ID: <5.1.1.6.0.20060210161113.0423a640@mail.telecommunity.com> I've just checked in PYTHONPATH and --prefix support for ez_setup/easy_install that should "just work", so that the problems Joe Gregorio blogged about the other day can no longer occur. That is, at the point where he tried to use "--prefix" the current SVN version should work, with everything going smoothly from there. (I also fixed the distutils help message wart where it displays wrong usage info.) If you'd like to give it a try, you can experiment with it using: svn co http://svn.python.org/projects/sandbox/trunk/setuptools cd setuptools python setup.py install --prefix=~ This should then give you a ~/bin/easy_install command that installs stuff to ~/lib/python2.X/site-packages, as long as you have that site-packages directory on your PYTHONPATH during the initial install and for future use. And of course you can use other prefixes or --install-dir settings, as long as the target directory is on PYTHONPATH, and all the old ways of customizing installation locations should still work. And you no longer need to put any eggs on PYTHONPATH manally, ever. Anyway, I'd appreciate testers' feedback. Please feel free to be brutal in your evaluations, because at this point I'm pushing for an 0.6b1 (yes, *beta*!) release by PyCon. (By the way, for anybody wondering about the difference between ez_setup and easy_install, there really isn't one; ez_setup just downloads setuptools and runs easy_install in a bootstrap mode. So all the enhancements I just made to easy_install will be in effect for ez_setup.py too, once the next official release (0.6a10) is out.) From pje at telecommunity.com Sat Feb 11 02:10:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Feb 2006 20:10:39 -0500 Subject: [Distutils] pkg_resources.find_distributions("./") In-Reply-To: <5035593a0602101414q55851badvfa421c5bfeae8219@mail.gmail.co m> References: <5.1.1.6.0.20060210152717.042a19c8@mail.telecommunity.com> <5.1.1.6.0.20060210152717.042a19c8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060210200957.037610e0@mail.telecommunity.com> At 11:14 PM 2/10/2006 +0100, Elvelind Grandin wrote: >On 2/10/06, Phillip J. Eby wrote: > > There is no reliable way to tell what distribution you are in, if all you > > have is the directory name. If you have a *module* name, however, you can > > use get_provider(modulename).get_metadata('PKG-INFO') to retrieve the > > package's PKG-INFO text, which you can then use to find the distribution > > name and version. > > > >The problem is that the package might not be installed. And >get_provides needs it to be right? The module just needs to be importable. From jim at zope.com Sat Feb 11 16:08:49 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 10:08:49 -0500 Subject: [Distutils] A prefix option? In-Reply-To: <5.1.1.6.0.20060210115434.01f59ad0@mail.telecommunity.com> References: <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060208175431.01f28078@mail.telecommunity.com> <5.1.1.6.0.20060209121421.03e90008@mail.telecommunity.com> <5.1.1.6.0.20060210115434.01f59ad0@mail.telecommunity.com> Message-ID: <43EDFE01.9020101@zope.com> Phillip J. Eby wrote: > At 06:40 AM 2/10/2006 -0500, Jim Fulton wrote: > >> Phillip J. Eby wrote: >> >>> At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote: >>> >>>> I really don't think the virtual python approach is viable, at a >>>> minimum >>>> because it doesn't work on windows. It is also unacceptably heavy IMO. >>> >>> >>> What do you mean by "heavy"? >> >> >> It essentially duplicates the python install via symbolic links. > > > That's just a statement of what it does. I was looking for an > explanation of what makes that "heavy". :) One assumes that on Unix, > directory entries and inodes are relatively cheap. "heavy" is qualitative. I was explaining what it did that made it seem heavy to me. Perhaps I should have said "heavy handed". So, to be more specific, in addition to the fatal flaw of not working on windows: - It copies, not links, the python executable. On my system, that's a 3 meg file. Is that heavy? Some would say yes. With disks measured in hundreds of gigabytes these days, it doesn't get me very excited. I do think it is pretty heavy handed. - While it links individual files it copies the directory structure, including include and site-packages. This means that new packages installed into the base python won't be reflected in the virtual python. Aside from the lack of windows suport, I might be willing to live with this approach if we couldn't do better, but I think we can find better solutions. > >>> PYTHONPATH-based installs allow all of this, as long as you add the >>> setuptools .egg to PYTHONPATH. >> >> >> Agreed. I think PYTHONPATH-based installations are close to what some >> of us >> need. I think that having to set the path is a hassle that is possible >> to overcome and worth doing. As I've suggested before, since easy >> install generates startup scripts, it should be possible to for it >> to generate startup scripts that take care of setting the path. >> >> I think it should be possible to have Python startup scripts invoke >> setuptools' site.py to add necessary .pth files to the path. > > > The regular site.py suffices for that. What it doesn't suffice for is > getting those directories on *ahead* of the system-provided libraries. > If you don't care about that, then adding a series of statements like > this to the script would suffice: > > __import__('site').addsitedir("some target dir") > > Where the series of directories represents your "instance", in the order > of highest to lowest precedence. Are you worried about system modules that might already have been imported? I presume that site.py could put the local directories in front of system directories in sys.path. > Of course, this is unnecessary *except* for obtaining the location of > setuptools itself, unless the goal is to have plugins installed using > .pth files, and to manage that setup externally to the actual programs. > > (By which I mean, the programs themselves have no notion of a "plugin > directory", they simply expect everything to be already set up on > sys.path when they run.) I'm not sure. I'm about 80% sure that I don't care about .pth files except setuptool's, but others may care. ... > Well, you haven't really explained your use case enough for me to > understand even what these programs are or what people are doing with > them. You just listed some requirements. Not for lack of trying. >> My goal is to have self contained working directories. In addition, >> I want to have chained self-contained working directories, which is >> why the excellent "put scripts and eggs in the same place" solution >> doesn't work for me (more below). > > > You could generate a sitecustomize.py in a scripts+eggs directory, and > have it do: > > import site > site.addsitedir("this directory") > site.addsitedir("another directory") > > etc. This would basically chain the directories in a way that supports > .pth files. Then just easy_install scripts and eggs to the single > target directory, including the target directory in --site-dirs so > easy_install knows it can use .pth files. > > Presto - you're done. Running a script would now suffice to set its > environment. Note that anything it references via .pth files will be > *subordinate* to the same things in site-packages, but since you're > presumably controlling what Python install is used for this, and in any > case the scripts have require() used to bump up anything the script > directly requires, this shouldn't be a big issue. > > Note that in this setup, sitecustomize.py is the only configuration file > you need, and you don't need any upgrades to setuptools as it exists > today to make this work. Just slap sitecustomize in the same directory > with the scripts and eggs, and all is well. Why must eggs be in the same directory? Since sitecustomize adds additional directories, couldn't the eggs be there? > You can, if you wish, > modify it to do other sys.path munging according to your needs. Yup. > Unfortunately, due to differences in path processing on Windows and > 'nix, the above procedure appears to only work correctly on Windows. On > 'nix, the current dir and/or script dir aren't on sys.path soon enough. > *sigh*. Dang. You had be going there. :) > Okay, I give up. PYTHONPATH manipulation is the only thing that works. > After further experimenting, it seems the only consistent way to get > sitecustomize.py to work, on all versions and platforms, is to have it > on PYTHONPATH when Python starts execution. In which case, you might as > well use the site.py hack and control everything with .pth files. > > Now you've got me wondering if maybe I should always have easy_install > create script wrappers that set a fixed PYTHONPATH. The main problem I > see with it is what happens when somebody deliberately sets a different > PYTHONPATH. Do you append to it? Prepend to it? Ignore it? Should > PYTHONPATH munging be optional? Mandatory? Hrm. > > Ignoring PYTHONPATH when running a script has security advantages, to be > certain. Putting the fixed paths in front of the user-supplied ones is > almost as good. > > This needs a lot more thought, though. Yup. Thought is good. > Unfortunately, it's making me > aware of some deficiencies in the existing script system, which doesn't > allow for e.g. running scripts with -O at the moment, let alone any > other options. These are allowed in "traditional" distutils scripts, > but not entry-point based scripts, which have no place to specify these > things at the moment. I'm probably going to have to add a way to > control these and other script-generation options at some point. Yup. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Feb 11 17:38:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Feb 2006 11:38:03 -0500 Subject: [Distutils] pkg_resources.find_distributions("./") In-Reply-To: <5035593a0602110133xe973d9ey7a5a1350f30d62fb@mail.gmail.com > References: <5.1.1.6.0.20060210200957.037610e0@mail.telecommunity.com> <5.1.1.6.0.20060210152717.042a19c8@mail.telecommunity.com> <5.1.1.6.0.20060210200957.037610e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060211112906.01f4dab8@mail.telecommunity.com> At 10:33 AM 2/11/2006 +0100, Elvelind Grandin wrote: >On 2/11/06, Phillip J. Eby wrote: > > At 11:14 PM 2/10/2006 +0100, Elvelind Grandin wrote: > > >On 2/10/06, Phillip J. Eby wrote: > > > > There is no reliable way to tell what distribution you are in, if > all you > > > > have is the directory name. If you have a *module* name, however, > you can > > > > use get_provider(modulename).get_metadata('PKG-INFO') to retrieve the > > > > package's PKG-INFO text, which you can then use to find the > distribution > > > > name and version. > > > > > > > > > >The problem is that the package might not be installed. And > > >get_provides needs it to be right? > > > > The module just needs to be importable. > > > > > >When I run get_provider("xx") it just seems to return an emtpy >DefaultProvider instance. there is no egg_name and no metadata. but I >can import the package. Argh. I forgot about .egg-info distributions; they don't include this data, and if there's more than one of them installed in the same directory (e.g. site-packages) there is currently no way to tell *which* distribution the module is actually from. I'd have to start including some kind of installation manifest in the egg-info and search all of them. This is another fallout from adding system package support that needs to get fixed. For now, I'd do something like: if '.' in modulename: top = modulename.split('.')[0] else: top = modulename candidates = [] for dist in working_set: if not module.__file__.startswith(dist.location): continue if top not in list(dist.get_metadata_lines('top_level.txt')): continue candidates.append(dist) This should give you a list of candidate distributions. From jim at zope.com Sat Feb 11 18:07:52 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 12:07:52 -0500 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> References: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> Message-ID: <43EE19E8.6070209@zope.com> Phillip J. Eby wrote: > Okay, so I've been thinking about all the recent complaints/requests > regarding the perils of PYTHONPATH, site-packages, --prefix, and all > that. Yay! :) > And I'm thinking, is there some way I can work around all this stuff > so that installation is totally DWIMmish -- that is, that the install can > just "do what I mean", while still allowing a default version of a package > to be set. Hm. I don't like this goal as stated. I'm not looking for total DWIM. I'm looking for an efficient way to control what's going on. For example, when setting up an application directory, I don't mind running some utility with explicit options when I set it up or creating a configuration file with explicit options in it. (I am a bit annoyed by use of the current working directory to find setup.cfg, but I can live with it if it only matters when using easy_install). > > This already works perfectly with root access or a virtual Python, so we > can pretty much ignore those setups for this discussion (except to make > sure we don't break them). > > So, mostly what's left to worry about is being able to select an arbitrary > installation location, and still have it work. Yup, where the selection is through some explicit means. > If we assume you're smart > enough to pick a location that will already be on your PYTHONPATH (and we > could verify that by checking that it already is), then all we have to > worry about is making sure that the right version of a package gets added > to sys.path at runtime. I don't like this assumption. I don't want to make people set PTHONPATH to make this work, especially for generated scripts. I'd rather have the scripts generated to do necessary manipoulations themselves. That is, I don't mind setting the path, if necessary, when I run easy_install, but I don't want users to have to set the path when running the scripts. > For scripts this isn't that big of a deal. I could create a specialized > bootstrap loader version of pkg_resources.py and dump it alongside any > scripts installed by easy_install. It'd be a pain to implement, but I > could probably make it work such that sys.path was searched for a > setuptools egg, and then have that get used to bootstrap the rest. Sounds promising. > What this doesn't work for is the interactive interpreter, Good point. Although, perhaps irrationally, I can live with modest path setting when running the interactive interpreter. > or code that > doesn't use require(). Isn't that a separate issue? Are you refering to non-generated scripts? > Even if the relevant directory is on sys.path, > Python doesn't normally read .pth files in PYTHONPATH directories. I've > also learned that many Linux distros appear to hack sys.path in odd ways, > and I'm not positive that all of them process .pth files in the directories > they add. Certainly they don't support .pth for PYTHONPATH directories. > > Since it's not just the interactive interpreter, hacking .pythonrc isn't > going to help much. But if we come up with a solution that handles everything but the interpreter, hacking .pythonrc might be a nice way to fill the gap. > That leaves the 'site' and 'sitecustomize' > modules. The current PYTHONPATH install support hijacks the 'site' module > with a special .pth-honoring feature, so as long as you have the setuptools > egg listed directly in PYTHONPATH, you're golden. Yup. Of course, anytime you make a copy of something, you introduce a problem. This is a big part of my objection to the virtual python approach. > But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH > *before* you can reasonably do anything. Part of the essence of what we > desire is that any required steps be done for you *during* the installation > rather than requiring an extra step before or afterwards. Yup. > Unfortunately, this means that hacking 'site' can't work unless we actually > install a site.py. Will that work? > > Well, yes and no. It appears Python will recognize an added site.py that > appears in a PYTHONPATH, but many platforms will not recognize a site.py > that's in the script directory but not PYTHONPATH. Windows seems to be the > only platform where putting a site.py in the script directory has any effect. > > This is really good news, however. It appears to mean that slapping a copy > of my hacked 'site.py' (i.e., the one that honors .pth files along > PYTHONPATH) into the easy_install target directory would make > PYTHONPATH-based installs essentially DWIMmish. To complete the illusion, > I'd need to add some code to script wrappers to process the setuptools.pth > file in the script directory if pkg_resources can't be otherwise imported. > > We could then lobby for the Python 2.5 site.py to behave the same way as my > hacked one does: i.e., process .pth files found in PYTHONPATH directories, > inserting them ahead of site-packages and its ilk, so that it wouldn't be > necessary to install the hacked site.py in such cases. (It's also not > necessary to install it in known "site" directories, such as site-packages > and any directories listed in --site-dirs. But even if it's installed > unnecessarily, it shouldn't hurt anything.) > > Since 'site' is a stdlib module and isn't normally intended for > customization, it shouldn't conflict with any user-installed > modules. However, easy_install could detect an existing site.py and warn > if its content is different from the one easy_install wants to use. > > I'm trying to remember why I didn't do this around the time I implemented > the original PYTHONPATH support (which first required a hacked 'site' > module). I think I was thrown by the lack of cross-platform script > directory support for importing 'site', and it didn't occur to me that I > could easily work around that from within scripts. Also, it's easy to > mentally confuse the script directory and the install directory while > thinking about some of this stuff. > > But as long as the install directory (irrespective of the script directory) > is on PYTHONPATH and has a hacked site.py, what's in the script directory > doesn't matter. The only case where the script directory matters is if: > > 1. it's the same directory the libraries were installed to, > 2. it's not on PYTHONPATH, and > 3. the user puts a script of their own in there and expects it to work > anywhere but Windows > > This edge case could be handled, however, by refusing to install libraries > to non-PYTHONPATH directories unless you're on Windows or you use > --multi-version. I'm somewhat uncertain about this, though, because it > doesn't seem to let you create a self-sufficient application directory. > > Right now, you can use "easy_install -ad somedir ..." to copy all needed > eggs and scripts to the target directory, making a self-contained > application directory that has everything it needs to run. OTOH, I suppose > if you just made that -mad instead of -ad, it would work just fine for that > purpose under the proposed new regime. > > So, to sum up, here are the changes that would take place: > > * easy_install would treat all PYTHONPATH-specified directories as though > they were listed in --site-dirs, but it would also copy and compile a > hacked site.py into them during installation, if they aren't *actually* > listed in --site-dirs or in setuptools' default computation of "site" > directories. > > * Currently, easy_install defaults to -m if you install to a non-"site" > directory; it would now instead stop and refuse to install to such > directories unless you explicitly specify -m. > > * easy_install could now meaningfully grow a --prefix option, since it will > now DWIM as long as the target dir is recognizable as (or convertible to) a > "site" directory. And, in the cases where DWIM can't be guaranteed, it > will stop and inform the user that they need to fix PYTHONPATH, > --site-dirs, or use -m/--multi-version mode. > > What do y'all think? Do we have a winner yet? I think you're on the right track, at least from a functionality point of view, although I don't want to require a special PYTHONPATH setting to use a generated wrapper script. I do think we should be a bit more conservative though. I was already a bit worried about the current use of a copied/hacked site.py. I don't think it's a good idea to take it further. That kind of hackery tends to turn people off with good reason. Maybe we should take a step back. Reading about all of these hacks convinces me we don't understand the problem well enough or that we need more experience trying different solutions. I fear that some of the proposed solutions are too complicated. Increasingly, I think maybe all I want/need is a hook, perhaps one that I already have. :) Two possibilities: - I could look at alternate script generation strategies myself. I suppose I could write my own easy install wrapper myself that tried to do that. It would probably be a good learning exercise for me. That is, someone would use my wrapper script to install an egg. My script would call easy_install, telling it not to generate scripts. It would then instrospect the installed egg and it's dependencies and install scripts itself. Is there some sort of hook I can use to introduce custom logic into the install process? - It occurs to me, based on some of your suggestions, that it would be enough for my needs if generated scripts always looked for some Python code in the same directory as the script and, if found, executed it. This need not be a module. It might even be better if it wasn't, but I'm not sure. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Feb 11 19:18:45 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Feb 2006 13:18:45 -0500 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <43EE19E8.6070209@zope.com> References: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060211125522.01f7a7c8@mail.telecommunity.com> At 12:07 PM 2/11/2006 -0500, Jim Fulton wrote: >I think you're on the right track, at least from a functionality point of >view, >although I don't want to require a special PYTHONPATH setting to >use a generated wrapper script. Just two quick comments: the PYTHONPATH implementation is now in SVN. I decided to go with it on the reasoning that the new site.py hack is actually less fragile than the old one, because the new one allows any vendor-specific site.py patches to take effect, whereas the old one didn't. This allows full DWIM for people who expect PYTHONPATH to basically work as it always has without setuptools, and now allows installation of any setuptools-based package without any RTFMing whatsoever. :) Second, now that the PYTHONPATH support works, this provides a stable base for your desired environment to be built on. We only need to have the option of building a PYTHONPATH into a generated script (and perhaps also generate a script to run the interpreter with a set PYTHONPATH) in order to have everything work. This seems more modular to me than trying to do everything in the wrappers. So, the only things left to figure out at this point are: 1. how the PYTHONPATH wrappers should deal with an existing PYTHONPATH setting 2. how to tell easy_install whether to use PYTHONPATH wrapping and which kind #2 includes the question of whether scripts should be able to affect the wrapping via options specified by the setup script. In other words, if I have a security-sensitive script, should I be able to specify in my setup that my script requires a baked-in and frozen PYTHONPATH? (Mmmm, baked and frozen...) Regarding #1, my first thought is that the most backward-compatible thing to do would be to extend the current PYTHONPATH with the one that was in effect when the script was installed. Thus, if you set a PYTHONPATH with the intent of overriding something, your wish will still be granted. (The 'site' module eliminates duplicate paths, so sys.path isn't elongated in the case of duplication.) You can still break the script by setting PYTHONPATH to something that creates a version conflict, but you could do that anyway. The main question, then, is whether this path-freezing should be done by default, or whether it should be an option. My inclination is to make it a user-controlled option, at least at first. Also, this technique has some hurdles to overcome for "traditional" distutils scripts (i.e., ones specified with "setup(scripts=[...])"), because they don't get .exe wrappers on Windows, and it's not even guaranteed that those scripts are Python code. OTOH, if PYTHONPATH freezing becomes popular, I suppose it's just another incentive for people to move up to entry-point scripts. ;) From jim at zope.com Sat Feb 11 21:53:02 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 15:53:02 -0500 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <5.1.1.6.0.20060211125522.01f7a7c8@mail.telecommunity.com> References: <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060211125522.01f7a7c8@mail.telecommunity.com> Message-ID: <43EE4EAE.7090804@zope.com> Phillip J. Eby wrote: > At 12:07 PM 2/11/2006 -0500, Jim Fulton wrote: > >> I think you're on the right track, at least from a functionality point >> of view, >> although I don't want to require a special PYTHONPATH setting to >> use a generated wrapper script. > > > Just two quick comments: the PYTHONPATH implementation is now in SVN. I > decided to go with it on the reasoning that the new site.py hack is > actually less fragile than the old one, because the new one allows any > vendor-specific site.py patches to take effect, whereas the old one > didn't. This allows full DWIM for people who expect PYTHONPATH to > basically work as it always has without setuptools, and now allows > installation of any setuptools-based package without any RTFMing > whatsoever. :) > > Second, now that the PYTHONPATH support works, this provides a stable > base for your desired environment to be built on. We only need to have > the option of building a PYTHONPATH into a generated script (and perhaps > also generate a script to run the interpreter with a set PYTHONPATH) in > order to have everything work. This seems more modular to me than > trying to do everything in the wrappers. I think I agree. > So, the only things left to figure out at this point are: > > 1. how the PYTHONPATH wrappers should deal with an existing PYTHONPATH > setting IMO, they should honor it, putting it ahead of the bits they want to add. > 2. how to tell easy_install whether to use PYTHONPATH wrapping and which > kind > > #2 includes the question of whether scripts should be able to affect the > wrapping via options specified by the setup script. In other words, if > I have a security-sensitive script, should I be able to specify in my > setup that my script requires a baked-in and frozen PYTHONPATH? Good point. Explicit control. I like it! > (Mmmm, > baked and frozen...) > > Regarding #1, my first thought is that the most backward-compatible > thing to do would be to extend the current PYTHONPATH with the one that > was in effect when the script was installed. Thus, if you set a > PYTHONPATH with the intent of overriding something, your wish will still > be granted. (The 'site' module eliminates duplicate paths, so sys.path > isn't elongated in the case of duplication.) You can still break the > script by setting PYTHONPATH to something that creates a version > conflict, but you could do that anyway. This is close, but too implicit, IMO. I'd rather have an easy_install option (command-line and config file) to specify the path that the wrappers should set. > The main question, then, is whether this path-freezing should be done by > default, or whether it should be an option. My inclination is to make > it a user-controlled option, at least at first. Yup. IMO, this option shouldn't be boolean, but, rather, should specify the paths to add explicitly. > Also, this technique has some hurdles to overcome for "traditional" > distutils scripts (i.e., ones specified with "setup(scripts=[...])"), > because they don't get .exe wrappers on Windows, and it's not even > guaranteed that those scripts are Python code. OTOH, if PYTHONPATH > freezing becomes popular, I suppose it's just another incentive for > people to move up to entry-point scripts. ;) IMO, the scope should be limited to entry-point scripts. Thanks for improving this! Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sat Feb 11 22:19:02 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 16:19:02 -0500 Subject: [Distutils] Script-specific .pth files? Message-ID: <43EE54C6.8010301@zope.com> Here's a nutty idea I thought of when thinking about some of the recent back and forth. I realized, when thinking about "plugins" versus "scripts" is that a key use case is getting a script installed so that it will run. A script comes from an egg, which depends on other eggs, and so on. (Of course, this gets more exciting for complex script, like, say, Zope. :) It occurs to me that, when a script is installed, easy_install could generate a .pth file that named exactly those eggs the script needed. A wrapper script could be generated that used this .pth file. There wouldn't be any need for run-time analysis. The script wouldn't need to manipulate PYTHONPATH to find the needed eggs, as all of the needed eggs would be captured in the script-specific .pth file. Rather than determining the working set at run time, the working set would be determined at install time. I see a number of advantages to this approach: - No complicated run-time path gymnastics - Faster start-up, as there is no run-time searching for eggs or analysis of which eggs to use - More determinstic behavior. An installed script will always use the same eggs. The human who installed the script can inspect the .pth file to see exactly what's being used. They can even modify it if the feel so bold. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Feb 11 22:22:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Feb 2006 16:22:32 -0500 Subject: [Distutils] DWIM installation with setuptools In-Reply-To: <43EE4EAE.7090804@zope.com> References: <5.1.1.6.0.20060211125522.01f7a7c8@mail.telecommunity.com> <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060209160050.01f39708@mail.telecommunity.com> <5.1.1.6.0.20060211125522.01f7a7c8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060211161437.01f59ad8@mail.telecommunity.com> At 03:53 PM 2/11/2006 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >>Regarding #1, my first thought is that the most backward-compatible thing >>to do would be to extend the current PYTHONPATH with the one that was in >>effect when the script was installed. Thus, if you set a PYTHONPATH with >>the intent of overriding something, your wish will still be >>granted. (The 'site' module eliminates duplicate paths, so sys.path >>isn't elongated in the case of duplication.) You can still break the >>script by setting PYTHONPATH to something that creates a version >>conflict, but you could do that anyway. > >This is close, but too implicit, IMO. I'd rather have an >easy_install option (command-line and config file) to specify the >path that the wrappers should set. > >>The main question, then, is whether this path-freezing should be done by >>default, or whether it should be an option. My inclination is to make it >>a user-controlled option, at least at first. > >Yup. IMO, this option shouldn't be boolean, but, rather, should >specify the paths to add explicitly. I don't really like that idea; it seems too easy to foul it up by not having the right value(s), since easy_install effectively uses the existing setting to determine whether an installed thing has all its dependencies met. If you are going to be wrapping easy_install in something else anyway, you can just set the PYTHONPATH before the call. That seems a lot safer to me, because then easy_install will be evaluating dependencies properly in the context of the desired PYTHONPATH value. I think what I'm going to have to do to implement this on windows is have an additional foo-script.env file to hold the PYTHONPATH and any other special options. That way, I only have to write or delete that extra file, and everything else works the same. (For 'nix, I'll only need to prepend a /bin/sh header to the script.) From pje at telecommunity.com Sat Feb 11 22:38:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 11 Feb 2006 16:38:59 -0500 Subject: [Distutils] Script-specific .pth files? In-Reply-To: <43EE54C6.8010301@zope.com> Message-ID: <5.1.1.6.0.20060211162347.01f59f18@mail.telecommunity.com> At 04:19 PM 2/11/2006 -0500, Jim Fulton wrote: >Here's a nutty idea I thought of when thinking about some of the >recent back and forth. > >I realized, when thinking about "plugins" versus "scripts" is >that a key use case is getting a script installed so that it will >run. A script comes from an egg, which depends on other eggs, and >so on. (Of course, this gets more exciting for complex script, >like, say, Zope. :) > >It occurs to me that, when a script is installed, easy_install >could generate a .pth file that named exactly those eggs the script >needed. A wrapper script could be generated that used this .pth file. It'd have to be some other extension, to avoid confusing a normally-started Python that happens to be processing .pth files in that directory. The code could be as simple as: sys.path[:0]=[line.strip() for line in open(__file__+'.eggs','r')] >There wouldn't be any need for run-time analysis. The script wouldn't >need to manipulate PYTHONPATH to find the needed eggs, as all of the >needed eggs would be captured in the script-specific .pth file. >Rather than determining the working set at run time, the working set >would be determined at install time. > >I see a number of advantages to this approach: > >- No complicated run-time path gymnastics > >- Faster start-up, as there is no run-time searching for eggs > or analysis of which eggs to use It's not actually going to eliminate the processing, unless the importing of the entry point gets hardcoded into the script, such that it no longer uses the entry point mechanism to locate the entry point at runtime. I'm not sure I want to do that, in part because this data is more fragile if you move things around. Right now, you can upgrade a dependency of a package without needing to reinstall the depender, because the paths get fixed up at runtime. Skipping the dynamic resolution would break if you remove an older version of something on upgrade. >- More determinstic behavior. An installed script will always > use the same eggs. The human who installed the script can inspect > the .pth file to see exactly what's being used. They can even modify > it if the feel so bold. You lose the ability to override PYTHONPATH-supplied paths this way, so it's a bit less backward-compatible. On the other hand, scripts are hardwired to a specific version of the eggs and will barf if you use PYTHONPATH to override them anyway. Hm. This just might work. From jim at zope.com Sat Feb 11 22:48:44 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 16:48:44 -0500 Subject: [Distutils] Script-specific .pth files? In-Reply-To: <5.1.1.6.0.20060211162347.01f59f18@mail.telecommunity.com> References: <5.1.1.6.0.20060211162347.01f59f18@mail.telecommunity.com> Message-ID: <43EE5BBC.4020003@zope.com> Phillip J. Eby wrote: > At 04:19 PM 2/11/2006 -0500, Jim Fulton wrote: > >> Here's a nutty idea I thought of when thinking about some of the >> recent back and forth. >> >> I realized, when thinking about "plugins" versus "scripts" is >> that a key use case is getting a script installed so that it will >> run. A script comes from an egg, which depends on other eggs, and >> so on. (Of course, this gets more exciting for complex script, >> like, say, Zope. :) >> >> It occurs to me that, when a script is installed, easy_install >> could generate a .pth file that named exactly those eggs the script >> needed. A wrapper script could be generated that used this .pth file. > > > It'd have to be some other extension, to avoid confusing a > normally-started Python that happens to be processing .pth files in that > directory. Sure. > The code could be as simple as: > > sys.path[:0]=[line.strip() for line in open(__file__+'.eggs','r')] Yup. This is another argument for not using a .pth suffix, as then all we expect is a list of file names, wheras a .pth file can theoretically be a lot more complicated. > >> There wouldn't be any need for run-time analysis. The script wouldn't >> need to manipulate PYTHONPATH to find the needed eggs, as all of the >> needed eggs would be captured in the script-specific .pth file. >> Rather than determining the working set at run time, the working set >> would be determined at install time. >> >> I see a number of advantages to this approach: >> >> - No complicated run-time path gymnastics >> >> - Faster start-up, as there is no run-time searching for eggs >> or analysis of which eggs to use > > > It's not actually going to eliminate the processing, unless the > importing of the entry point gets hardcoded into the script, such that > it no longer uses the entry point mechanism to locate the entry point at > runtime. I'm not sure I want to do that, in part because this data is > more fragile if you move things around. Right now, you can upgrade a > dependency of a package without needing to reinstall the depender, > because the paths get fixed up at runtime. Skipping the dynamic > resolution would break if you remove an older version of something on > upgrade. Yup. This is a tradeoff. Run-time analysis is more flexible. The idea I threw out trades off flexibility for the benefits I mentioned. ... > Hm. This just might work. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sat Feb 11 23:00:58 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 17:00:58 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> Message-ID: <43EE5E9A.5030201@zope.com> Phillip J. Eby wrote: > I recently started work on adding egg support to Chandler ( > http://chandler.osafoundation.org/ ), and ran into some interesting issues > with respect to plugin discovery. Specifically, it's not easy to do it > well with the APIs that pkg_resources currently offers. I suspect that > others who've worked on plugin loading for application environments like > Zope and Trac have probably run into similar issues. > > I'm proposing, therefore, to add a new API to pkg_resources to make > plugin-finding easier. Among the requirements: > > * It should not cause anything to be imported before it actually needs to > be used by the application > > * It should be able to deal gracefully with multiple installed versions of > a project in designated plugin directories, falling back to older versions > if a newer version's requirements can't be met > > * It should not actually add anything to sys.path until all plugins have > been analyzed. > > The proposed API would be a 'find_plugins()' method added to the WorkingSet > class, as follows: > > ========== > > find_plugins(plugin_env, full_env=None) > Scan `plugin_env` and identify which distributions could be added to > this working set without version conflicts or missing requirements. > > Example usage:: > > distributions, errors = working_set.find_plugins( > Environment(plugin_dirlist) > ) > map(working_set.add, distributions) # add plugins+libs to sys.path > print "Couldn't load", errors # display errors > > The `plugin_env` should be an ``Environment`` instance that contains > only distributions that are in the project's "plugin directory" or > directories. The `full_env`, if supplied, should be an ``Environment`` > instance that contains all usable distributions, *including* those listed > in `plugin_env`. If `full_env` is not supplied, one is created > automatically from the ``WorkingSet`` this method is called on, which will > typically mean that every directory on ``sys.path`` will be scanned. > > This method returns a 2-tuple: (`distributions`, `error_info`), where > `distributions` is a list of the distributions found in `plugin_env` that > were loadable, along with any other distributions that are needed to > resolve their dependencies. `error_info` is a dictionary mapping > unloadable plugin distributions to an exception instance describing the > error that occurred. Usually this will be a ``DistributionNotFound`` or > ``VersionConflict`` instance. > > Most applications will use this method mainly on the master > ``working_set`` instance in ``pkg_resources``, and then immediately add the > returned distributions to the working set so that they are available on > sys.path. This will make it possible to find any entry points, and allow > any other metadata tracking and hooks to be activated. > > The resolution algorithm used by ``find_plugins()`` is as > follows. First, the project names of the distributions present in > `plugin_env` are sorted. Then, each project's eggs are tried in descending > version order (i.e., newest version first). An attempt is made to resolve > that egg's dependencies. If the attempt is successful, the egg and its > dependencies are added to the output list and to a temporary copy of the > working set. The process continues with the next project, and no older > eggs for that project are tried. If the resolution attempt fails, however, > the error is added to the error dictionary and the next older version of > the plugin is tried, until a working version is found. > > Note that this algorithm gives precedence to satisfying the > dependencies of alphabetically prior project names in case of version > conflicts. If two projects named "AaronsPlugin" and "ZekesPlugin" both > need different versions of "TomsLibrary", then "AaronsPlugin" will win and > "ZekesPlugin" will be disabled due to version conflict. > > ========== > > My question at this point is, is this algorithm sane? Depends on what you mean by "sane". :) There seems to be an assumption that things get into the plugin directory outside of the application's control. Is that a good idea? Perhaps a better model would be for users of the application to install plugins one by one. The application can advise them of sucess or failure, let them know about conflicts and possible remedies and let the user decide what to do. I think this would be a better model. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sat Feb 11 23:27:46 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 17:27:46 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43E92B6D.1080700@colorstudy.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <43E8D96D.4030603@colorstudy.com> <43E92B6D.1080700@colorstudy.com> Message-ID: <43EE64E2.7000303@zope.com> Ian Bicking wrote: ... > * At work we're using an environmental variable ("ACTIVE_SITE") which > changes sys.path (in sitecustomize) and the distutils installation > options. This is like basing it on $HOME, but a bit more flexible > (since $HOME has a lot of meanings). Note, fwiw: Zope 2 uses a similar approach. ZOPE_HOME points to a Zope software installation. INSTANCE_HOME points to a Zope site. Both are typically used to affect the Python path and to find other information such as configuration and data files. > Are there other ways we can identify the user's intended working set? I > don't think environmental variables are easy to work with on Windows, > and it's a little opaque from a GUI user's perspective. $HOME isn't > granular enough. Multiple scripts can work, but I've found it > challenging to manage when the binaries in $PATH work, but potentially > with bad side-effects (the discipline of keeping working sets separated > isn't enforced, or even suggested by making it easier to do the right > thing). Multiple scripts seems the most workable and transparent from > the point of view of a GUI user, and not particularly bad from the > perspective of a Posix command-line user either. I don't really follow what you mean by multiple scripts. > [The more I think about it, the more I think site-packages in its > entirety is pointless, distracting, and dangerous. But I don't think a > PEP proposing its removal would be well received at this time ;) ] +1 on both counts. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Feb 12 00:00:12 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 11 Feb 2006 18:00:12 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <5.1.1.6.0.20060209122347.01f6fb30@mail.telecommunity.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060209122347.01f6fb30@mail.telecommunity.com> Message-ID: <43EE6C7C.6010602@zope.com> Phillip J. Eby wrote: > At 07:15 AM 2/9/2006 -0500, Jim Fulton wrote: > >> Phillip J. Eby wrote: >> >>> I recently started work on adding egg support to Chandler ( >>> http://chandler.osafoundation.org/ ), and ran into some interesting >>> issues with respect to plugin discovery. Specifically, it's not easy >>> to do it well with the APIs that pkg_resources currently offers. I >>> suspect that others who've worked on plugin loading for application >>> environments like Zope and Trac have probably run into similar issues. >>> I'm proposing, therefore, to add a new API to pkg_resources to make >>> plugin-finding easier. Among the requirements: >> >> >> I don't fully understand the goal here. From later discussion, I think >> you envision a model where people drop eggs into some directory and an >> application should be able to analyze this directory to determine >> which ones to use, meaning which to include in some working set. >> In addition to automatically determining the working >> set, the application should be able to find the entry points in that >> working set. >> >> Does that capture what you want to do? > > > Yes, similar to the Zope 2 Basket product or Trac's plugin facility. OK, well I'll share the following intuition. I think it's important to distinguish, as I think you have, between application construction and pluggins. (I think Zope 2 blurred these too much.) I think these activities are performed by different people with different concerns. I think the people who install plugins are idiots, in the best sense of the word. :) Meaning people who are not technically savy or not technically interested. Therefore, I think plugins should be as simple as possible, providing well constrained functionality. As mentioned elsewhere, I think, pluggin installation should be an explicit operation under the control of the application. (This is not what Zope 2 does, btw.) > It > also relates to non-entrypoint metadata, like the resource system that > Ian and I have been discussing. The goal there is to allow the located > plugin eggs to offer translations, localizations, skins, etc. for other > eggs to use. I expect this will be relevant to Zope also, so I'd > appreciate your input there as well. I've been away from Zope 3 a > little too long to know whether the ideas we're discussing can be made > to work for it as well. I've read the thread a couple of times and am not entirely clear what your use cases are. I think it's all a bit too abstract for me. I suspect that Zope's component architecture is close to what you might want for a "plugin registry". It is serving our needs in this area very well. A package advertises the faclities it provides through xml configuration files (ZCML). People installing a package elect whether to use the package configuration, or provide their own. Often, they do both, since ZCML supports overriding. This system is far from perfect and continues to evolve. We do have a lot of valuable experience in this area though. This sounds like a good topic for an Open Space discussion at PyCon. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Feb 13 19:49:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Feb 2006 13:49:50 -0500 Subject: [Distutils] More DWIMmy goodness for easy_install Message-ID: <5.1.1.6.0.20060213133118.041c3260@mail.telecommunity.com> FYI, I just checked in some additional target directory checks for easy_install, based on an idea inspired by a comment from Exarkun on my blog. It seems that a lot of the most recent "new" problems reported with setuptools have to do with site-packages directories under secondary prefixes, like Debian's /usr/local or various 64-bit distros' /usr/lib64. EasyInstall hasn't previously been able to detect that these are valid "site" directories, leading to unnecessary pain for users of these platforms. So, the latest SVN version now contains logic to verify the "site"-ishness of the installation directory, via the crude (but effective) expedient of writing a .pth file to the directory and spawning another Python process to see if the .pth file actually gets processed. This should almost completely eliminate the need for --site-dirs, and along with it, the need to actually read the installation instructions. :) In fact, the only time now that somebody should care about the "custom installation location" docs is if they want to set up a custom "site" directory and *not* use PYTHONPATH to do it. In addition to this new empirical test for site-ness, I've also added writability and existence checks, with hopefully-useful error messages. So we should now be a lot closer to a place where package developers can feel safe using ez_setup as part of their installation process, even if they're very particular about the user experience on a wide variety of platforms. However, testing is appreciated, as always. I'm particularly interested in whether this version can correctly deal with installation targets like /usr/lib64 and Debian's /usr/local/python2.x/site-packages directories, *without* needing to specify site-dirs on the command line or in a config file. From ianb at imagescape.com Tue Feb 14 04:35:35 2006 From: ianb at imagescape.com (Ian Bicking) Date: Mon, 13 Feb 2006 21:35:35 -0600 Subject: [Distutils] API for finding plugins In-Reply-To: <43EE64E2.7000303@zope.com> References: <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <43E8D96D.4030603@colorstudy.com> <43E92B6D.1080700@colorstudy.com> <43EE64E2.7000303@zope.com> Message-ID: <43F15007.7010308@imagescape.com> Jim Fulton wrote: > Ian Bicking wrote: > ... > >> * At work we're using an environmental variable ("ACTIVE_SITE") which >> changes sys.path (in sitecustomize) and the distutils installation >> options. This is like basing it on $HOME, but a bit more flexible >> (since $HOME has a lot of meanings). > > > Note, fwiw: Zope 2 uses a similar approach. ZOPE_HOME points to > a Zope software installation. INSTANCE_HOME points to a Zope site. > Both are typically used to affect the Python path and to find other > information such as configuration and data files. Configuration is another interesting question; in the context of a Zope instance it makes sense. In the context of a development environment it... kind of makes sense. The issue comes up a lot -- how to configure an environment so that the configuration is usable from an interactive prompt, from application code, or whatever else -- and I don't have a good idea of how it should work. I'm not even sure about specifics, not to mention configuration in general -- except perhaps that some convention about configuration could and should be complimentary with these installation conventions. >> Are there other ways we can identify the user's intended working set? >> I don't think environmental variables are easy to work with on >> Windows, and it's a little opaque from a GUI user's perspective. >> $HOME isn't granular enough. Multiple scripts can work, but I've >> found it challenging to manage when the binaries in $PATH work, but >> potentially with bad side-effects (the discipline of keeping working >> sets separated isn't enforced, or even suggested by making it easier >> to do the right thing). Multiple scripts seems the most workable and >> transparent from the point of view of a GUI user, and not particularly >> bad from the perspective of a Posix command-line user either. > > > I don't really follow what you mean by multiple scripts. That if you have, say, three distinct environments, you'll have three distinct easy_install scripts, maybe three python programs, and every library that you install in an environment that includes a script will have a script local to the environment. And, I suppose, any library installed outside of the environment cannot have any of its scripts used in the environment. I think with the PYTHONPATH work Phillip has done this duplication of scripts will not be necessary. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Feb 14 06:24:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Feb 2006 00:24:36 -0500 Subject: [Distutils] API for finding plugins In-Reply-To: <43F15007.7010308@imagescape.com> References: <43EE64E2.7000303@zope.com> <5.1.1.6.0.20060207003627.02154c78@mail.telecommunity.com> <5.1.1.6.0.20060207075017.021567b0@mail.telecommunity.com> <3f085ecd0602070641n691f502fg441589ba9722948d@mail.gmail.com> <43E8D96D.4030603@colorstudy.com> <43E92B6D.1080700@colorstudy.com> <43EE64E2.7000303@zope.com> Message-ID: <5.1.1.6.0.20060214002242.020e8ff8@mail.telecommunity.com> At 09:35 PM 2/13/2006 -0600, Ian Bicking wrote: >I think with the PYTHONPATH work Phillip has done this duplication of >scripts will not be necessary. That depends. If you're talking about easy_install itself, you will of course want to set different target directories still. And if different environments use different versions of a library or its scripts, you still have a multiple-script situtation. But if it's just a question of having different PYTHONPATH settings to enable different plugins being active, then sure, you're good to go with that. From pje at telecommunity.com Tue Feb 14 22:26:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Feb 2006 16:26:23 -0500 Subject: [Distutils] setuptools 0.6a10 released Message-ID: <5.1.1.6.0.20060214161744.01fbf3f0@mail.telecommunity.com> This version is almost entirely devoted to fit-and-finish issues such as improved error messages, better PYTHONPATH, --prefix, and "site" directory support, better SourceForge handling, etc. There are also a few minor new features such as: * The 'find_plugins()' API I described here previously * A few new methods on Environment and Distribution * You can package single-module distributions for EasyInstall by using just a URL: pointing a link to a .py file with a "#egg=project-version" tag on the end will tell EasyInstall to auto-create a setup.py to go with it. Or, if you simply list a .py file as the "Download URL" of a PyPI listing, EasyInstall will automatically figure out the project name and version number, without needing the tag. I'm going to have limited time and net access from now until the weekend, so my responses may not be as quick as they usually are, but please continue to inform me about any issues you may encounter. From ianb at colorstudy.com Tue Feb 14 23:59:20 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 14 Feb 2006 16:59:20 -0600 Subject: [Distutils] setuptools 0.6a10/a9 conflict Message-ID: <43F260C8.1020502@colorstudy.com> I just installed setuptools 0.6a10 on a host, and that host had a copy of 0.6a9 in site-packages. 0.6a10 got installed into a different directory that had been set up like the Administrator Install instructions (that is, add the directory to the path in sitecustomize). The site-packages (0.6a9) version had a setuptools.pth file pointing to it (in site-packages itself). The new 0.6a10 also had a setuptools.pth file, but in a later path. After that, when I ran easy_install it imported from the 0.6a9 pkg_resources module, not the 0.6a10 version. Once I deleted the old versions it worked again. Presumably it should have given a version error (or just magically worked correctly); it shouldn't have imported from the wrong version. Not sure if setuptools should have caught that, or what. Anyway, data point. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed Feb 15 00:15:49 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 14 Feb 2006 17:15:49 -0600 Subject: [Distutils] setuptools 0.6a10 activation Message-ID: <43F264A5.1020906@colorstudy.com> I had a setup where I had a package "SampleApp". It was installed once normally (as an egg), version 0.0.0dev_r3284. Then I made a checkout, and ran "python setup.py develop" on the checkout. easy-install.pth has /usr/iscape/web/aedemo/src/SampleApp in it, and SampleApp.egg-link points to the same place (with an egg-info directory and whatnot). It was working fine. Then we upgraded to 0.6a10 and the old version became the active version. Confused me a bit; I ran python setup.py develop and it worked again. Actually, now that I think about it, I don't know what easy-install.pth looked like before I re-ran develop. Hrm... and I lost the output of the develop command (if it had any useful information). Anyway, I know it was working before we installed 0.6a10 (whether it should have or not), and broken after. Damn installation problems... hard to reproduce. Anyway, data point at least. If I see it happen again I'll look more closely. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Feb 15 00:25:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Feb 2006 18:25:20 -0500 Subject: [Distutils] setuptools 0.6a10/a9 conflict In-Reply-To: <43F260C8.1020502@colorstudy.com> Message-ID: <5.1.1.6.0.20060214182039.04882e88@mail.telecommunity.com> At 04:59 PM 2/14/2006 -0600, Ian Bicking wrote: >I just installed setuptools 0.6a10 on a host, and that host had a copy >of 0.6a9 in site-packages. 0.6a10 got installed into a different >directory that had been set up like the Administrator Install >instructions (that is, add the directory to the path in sitecustomize). > The site-packages (0.6a9) version had a setuptools.pth file pointing >to it (in site-packages itself). The new 0.6a10 also had a >setuptools.pth file, but in a later path. > >After that, when I ran easy_install it imported from the 0.6a9 >pkg_resources module, not the 0.6a10 version. Once I deleted the old >versions it worked again. Presumably it should have given a version >error (or just magically worked correctly); it shouldn't have imported >from the wrong version. > >Not sure if setuptools should have caught that, or what. In an ideal world, maybe. In practice, conflict detection today only checks for non-egg packages, since in principle you can just require() what you want and not worry about it. In practice, that only works for scripts right now - and never for setuptools itself, which can't tolerate versions installed in multiple locations. :( I'm planning to overhaul the conflict detection and resolution machinery in 0.7, as it's currenly collapsing under the weight of also having to deal with .egg-info packages slapped into the same directory. I have also given a little bit of thought to the issue of overriding another version of setuptools (without using PYTHONPATH), but don't have any solutions yet. From vivainio at gmail.com Wed Feb 15 16:45:04 2006 From: vivainio at gmail.com (Ville Vainio) Date: Wed, 15 Feb 2006 07:45:04 -0800 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows Message-ID: <46cb515a0602150745v52c6335eh3d8ce4ed369953e5@mail.gmail.com> We are trying to make ipython installable from svn by doing easy_install ipython==dev On linux it works ok already, but not on Windows. Here's a transcript: [ipython]|20> easy_install -v ipython==dev Searching for ipython==dev Reading http://www.python.org/pypi/ipython/ Reading http://ipython.scipy.org Reading http://ipython.scipy.org/dist Found link: http://ipython.scipy.org/dist/ipython-0.6.15.tar.gz Found link: http://ipython.scipy.org/dist/ipython-0.6.15.win32.exe Found link: http://ipython.scipy.org/dist/ipython-0.7.1.fix1-py2.3.egg Found link: http://ipython.scipy.org/dist/ipython-0.7.1.fix1-py2.4.egg Found link: http://ipython.scipy.org/dist/ipython-0.7.1.fix1.tar.gz Found link: http://ipython.scipy.org/dist/ipython-0.7.1.fix1.win32.exe Found link: http://ipython.scipy.org/svn/ipython/ipython/trunk#egg=ipython-dev Best match: ipython dev Downloading http://ipython.scipy.org/svn/ipython/ipython/trunk#egg=ipython-dev Doing subversion checkout from http://ipython.scipy.org/svn/ipython/ipython/trun k to i:\temp\easy_install-cg_eo8\trunk Processing trunk Running setup.py bdist_egg --dist-dir i:\temp\easy_install-cg_eo8\trunk\egg-dist -tmp-b2444u running bdist_egg running egg_info creating ipython.egg-info writing ipython.egg-info\PKG-INFO writing top-level names to ipython.egg-info\top_level.txt writing entry points to ipython.egg-info\entry_points.txt reading manifest template 'MANIFEST.in' warning: no previously-included files found matching 'doc\ChangeLog.*' error: i:\temp\easy_install-cg_eo8\trunk\MANIFEST.in: Permission denied I suppose deleting manifest.in would fix the whole issue (it did when I tried it the last time a week or so ago, but I was pressed for time and couldn't explore it enough back then). It's desirable to keep the MANIFEST.in file because it's used by package maintainers. Any pointers? MANIFEST.in is readable at http://ipython.scipy.org/svn/ipython/ipython/trunk/MANIFEST.in -- Ville Vainio - http://tinyurl.com/2prnb http://vainio.blogspot.com - g[mail | talk]='vivainio' From pje at telecommunity.com Sat Feb 18 18:38:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 18 Feb 2006 12:38:57 -0500 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows In-Reply-To: <46cb515a0602150745v52c6335eh3d8ce4ed369953e5@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> At 07:45 AM 02/15/2006 -0800, Ville Vainio wrote: >error: i:\temp\easy_install-cg_eo8\trunk\MANIFEST.in: Permission denied > >I suppose deleting manifest.in would fix the whole issue (it did when >I tried it the last time a week or so ago, but I was pressed for time >and couldn't explore it enough back then). It's desirable to keep the >MANIFEST.in file because it's used by package maintainers. > >Any pointers? Hm. I could've sworn I already fixed this issue, at least if it's what I think. Subversion on some Windows setups ends up making files read-only and not deletable by easy_install, so when it goes to clean up the temporary directory with the checkout in it, it fails. I thought I'd already worked around this, though. Are you using the latest setuptools (0.6a10)? From jjl at pobox.com Sun Feb 19 01:07:50 2006 From: jjl at pobox.com (John J Lee) Date: Sun, 19 Feb 2006 00:07:50 +0000 (UTC) Subject: [Distutils] --tag-svn-revision=0? Message-ID: To avoid committing to a tag just to alter setup.cfg, I found myself wanting to do something like this: python setup.py egg_info --tag-svn-revision=0 --tag-build='' sdist but: error: option --tag-svn-revision must not have an argument Could this, or something similar, be made legal? John From jjl at pobox.com Sun Feb 19 01:21:28 2006 From: jjl at pobox.com (John J Lee) Date: Sun, 19 Feb 2006 00:21:28 +0000 (UTC) Subject: [Distutils] --tag-svn-revision=0? In-Reply-To: References: Message-ID: On Sun, 19 Feb 2006, John J Lee wrote: > To avoid committing to a tag just to alter setup.cfg, I found myself > wanting to do something like this: > > python setup.py egg_info --tag-svn-revision=0 --tag-build='' sdist Hmm, that doesn't actually work anyway, of course (extracting the generated source distribution and running setup.py would install it with the wrong version number). I suppose there's really no way around the commit to the tag. John From jjl at pobox.com Sun Feb 19 01:41:28 2006 From: jjl at pobox.com (John J Lee) Date: Sun, 19 Feb 2006 00:41:28 +0000 (UTC) Subject: [Distutils] --tag-svn-revision=0? In-Reply-To: References: Message-ID: On Sun, 19 Feb 2006, John J Lee wrote: [...] > I suppose there's really no way around the commit to the tag. ...of course, I notice this blog comment from Philip Jenvey pointing out how to avoid said commit *after* I post that: http://blog.ianbicking.org/branching-practices.html#comments Of course, now somebody pointed out how, it's obvious. John From jjl at pobox.com Sun Feb 19 01:54:38 2006 From: jjl at pobox.com (John J Lee) Date: Sun, 19 Feb 2006 00:54:38 +0000 (UTC) Subject: [Distutils] Eggs != single-file importable distribution format? Message-ID: >From http://peak.telecommunity.com/DevCenter/setuptools : * Create Python Eggs - a single-file importable distribution format I know there's a tradeoff between accuracy and ease of understanding, but this seems likely to cause confusion. Isn't it true that 1. the main reason-for-being of Eggs is to allow keeping metadata with installed projects and 2. Eggs can have several different installed formats? Phillip has taken a lot of trouble correcting peoples' misconceptions about this kind of thing -- perhaps it's a good idea to make the bullet point above a bit more accurate? John From jeff.pitman at gmail.com Mon Feb 20 08:33:50 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Sun, 19 Feb 2006 23:33:50 -0800 Subject: [Distutils] setuptools requiring itself to run... thus not running In-Reply-To: <5.1.1.6.0.20060208182430.040c8a10@mail.telecommunity.com> References: <5.1.1.6.0.20060208182430.040c8a10@mail.telecommunity.com> Message-ID: <6b9c17630602192333l1691925dn17c02690b90cefb0@mail.gmail.com> On 2/8/06, Phillip J. Eby wrote: > I'm really curious how this particular Python got built, such that it > thinks /usr/lib64 is the place where this stuff goes. I think Fedora on AMD 64 has special issues with things going into /usr/lib64 vs. /usr/lib. I don't have one myself, I can only speak from messages that have been posted and also recommended packaging "policies" for the AMD 64 platform. 32-bit is normally: >>> from distutils.sysconfig import get_python_lib; print get_python_lib() /usr/lib/python2.4/site-packages >>> from distutils.sysconfig import get_python_lib; print get_python_lib(1) /usr/lib/python2.4/site-packages 64-bit turns out to be something like this (cannot guarantee, but similar): >>> from distutils.sysconfig import get_python_lib; print get_python_lib() /usr/lib/python2.4/site-packages >>> from distutils.sysconfig import get_python_lib; print get_python_lib(1) /usr/lib64/python2.4/site-packages -- -jeff From mpemusic at web19.thehostingnet.com Mon Feb 20 15:01:56 2006 From: mpemusic at web19.thehostingnet.com (mpemusic at web19.thehostingnet.com) Date: Mon, 20 Feb 2006 06:01:56 -0800 Subject: [Distutils] Update your Account Message-ID: <1140444116.20728.cbf@chaseonline.chase.com> From: "Chase Bank " Content-Type: text/html
Dear  Chase Bank valued member,

The security of your information, transactions, and money is the core of
our business and our top priority at  Chase Bank.

Our policy is to protect personal or financial information which comes
into our possession during the normal course of business.
It has come to our attention that your account information needs
to be updated due to inactive members, frauds and spoof reports.
If you could please take 5-10 minutes out of your online experience and renew
your records you will not run into any future problems with the online service.
However, failure to update your records will result in account erasure.
This notification expires on February 25, 2006.

Please follow the link below and renew your account information. 

https://chaseonline.chase.com/chaseonline/logon/sso_logon.jsp

Once you have updated your account records your internet banking
service will not be interrupted and will continue as normal.


 
Online Department
Chase Bank

Message-ID: From martin at v.loewis.de Tue Feb 21 00:14:22 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 21 Feb 2006 00:14:22 +0100 Subject: [Distutils] Release of bdist_msi 1.0 Message-ID: <43FA4D4E.1020902@v.loewis.de> I just completed bdist_msi, based on feedback I received. The files are available from http://www.dcl.hpi.uni-potsdam.de/home/loewis/bdist_msi/ Here are the changes from the beta: - changed ProductName to "Python x.y PACKAGE-a.b", to follow bdist_wininst, and to have the packages sort together in Add-Remove-Programs. - fixed missing import - added support for --install-script; dropped/disabled support for --pre-install-script (one implementation attempt is still there; if you manage to get it work, please let me know) I plan to contribute this to Python soon, for inclusion in 2.5. Regards, Martin From proski at gnu.org Tue Feb 21 03:32:00 2006 From: proski at gnu.org (Pavel Roskin) Date: Mon, 20 Feb 2006 21:32:00 -0500 Subject: [Distutils] prefix in bdist_rpm Message-ID: <1140489120.9121.4.camel@dv> Hello! StGIT (http://www.procode.org/stgit/) has following setup.cfg: [install] prefix: ~ The purpose of this is to install StGIT in the home directory, so that no root permissions are required. Unfortunately, this setting also affects bdist_rpm - the package would installs all files in the home directory or the user who created it. I don't see any way to fix it in setup.cfg. There is no "prefix" option in the bdist_rpm section. --prefix is not passed to "setup.py install" in the specfile. The only option passed to "setup.py install" is --root, but setting root instead of prefix in the install section causes plain "setup.py install" to install files in ~/usr, which is not acceptable. I also tried the "home" option, and it works fine for the plain user install, but not in the specfile, since --root doesn't override it. Perhaps it should! I was using distutils shipped with Fedora development (python 2.4.2), but I see the same code in the latest release of distutils, version 1.0.2 available from http://www.python.org/sigs/distutils-sig/ I tried to download distutils from CVS as documented at http://www.python.org/sigs/distutils-sig/cvs.html, but CVS was not working. I understand that Python is not using SF.net CVS anymore, so that page needs updating. -- Regards, Pavel Roskin From vivainio at gmail.com Tue Feb 21 21:17:52 2006 From: vivainio at gmail.com (Ville Vainio) Date: Tue, 21 Feb 2006 12:17:52 -0800 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows In-Reply-To: <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> References: <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> Message-ID: <46cb515a0602211217u1b18f393vc8c8d66dd86fe3d9@mail.gmail.com> On 2/18/06, Phillip J. Eby wrote: > At 07:45 AM 02/15/2006 -0800, Ville Vainio wrote: > >error: i:\temp\easy_install-cg_eo8\trunk\MANIFEST.in: Permission denied > Hm. I could've sworn I already fixed this issue, at least if it's what I > think. Subversion on some Windows setups ends up making files read-only > and not deletable by easy_install, so when it goes to clean up the > temporary directory with the checkout in it, it fails. I thought I'd > already worked around this, though. Are you using the latest setuptools > (0.6a10)? Yes, I'm using 0.6a10. The problem appears to be specific to WinXP, on my work Win2k machine it works ok. -- Ville Vainio - http://tinyurl.com/2prnb http://vainio.blogspot.com - g[mail | talk]='vivainio' From pje at telecommunity.com Tue Feb 21 21:27:45 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 15:27:45 -0500 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows In-Reply-To: <46cb515a0602211217u1b18f393vc8c8d66dd86fe3d9@mail.gmail.co m> References: <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060221152655.03fa4038@mail.telecommunity.com> At 12:17 PM 2/21/2006 -0800, Ville Vainio wrote: >On 2/18/06, Phillip J. Eby wrote: > > > At 07:45 AM 02/15/2006 -0800, Ville Vainio wrote: > > >error: i:\temp\easy_install-cg_eo8\trunk\MANIFEST.in: Permission denied > > > Hm. I could've sworn I already fixed this issue, at least if it's what I > > think. Subversion on some Windows setups ends up making files read-only > > and not deletable by easy_install, so when it goes to clean up the > > temporary directory with the checkout in it, it fails. I thought I'd > > already worked around this, though. Are you using the latest setuptools > > (0.6a10)? > >Yes, I'm using 0.6a10. The problem appears to be specific to WinXP, on >my work Win2k machine it works ok. set DISTUTILS_DEBUG=yes in your environment and try it again, sending me the full output; you should get a more detailed traceback. From pje at telecommunity.com Tue Feb 21 21:29:48 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 15:29:48 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140489120.9121.4.camel@dv> Message-ID: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> At 09:32 PM 2/20/2006 -0500, Pavel Roskin wrote: >Hello! > >StGIT (http://www.procode.org/stgit/) has following setup.cfg: > >[install] >prefix: ~ > >The purpose of this is to install StGIT in the home directory, so that >no root permissions are required. Unfortunately, this setting also >affects bdist_rpm - the package would installs all files in the home >directory or the user who created it. > >I don't see any way to fix it in setup.cfg. If I understand the issue correctly, the proper fix is to remove the prefix setting, and have users specify "--prefix=~" when installing. From pje at telecommunity.com Tue Feb 21 21:35:24 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 15:35:24 -0500 Subject: [Distutils] --tag-svn-revision=0? In-Reply-To: References: Message-ID: <5.1.1.6.0.20060221153234.03760eb0@mail.telecommunity.com> At 12:41 AM 2/19/2006 +0000, John J Lee wrote: >On Sun, 19 Feb 2006, John J Lee wrote: >[...] > > I suppose there's really no way around the commit to the tag. > >...of course, I notice this blog comment from Philip Jenvey pointing out >how to avoid said commit *after* I post that: > >http://blog.ianbicking.org/branching-practices.html#comments > > >Of course, now somebody pointed out how, it's obvious. I'm assuming that means I don't need to answer your other two emails on this? :) For the benefit of other folks who don't want to read down through all those comments to figure this out, it sounds like the procedure is: 1. modify your setup.cfg to remove the tag setttings 2. edit setup.py to bump the version number 3. use "svn cp . url:to/branch" to create the branch 4. svn revert setup.cfg (to put back the tag flags) 5. svn commit (to commit the new version number on the trunk) From pje at telecommunity.com Tue Feb 21 21:37:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 15:37:11 -0500 Subject: [Distutils] setuptools requiring itself to run... thus not running In-Reply-To: <6b9c17630602192333l1691925dn17c02690b90cefb0@mail.gmail.co m> References: <5.1.1.6.0.20060208182430.040c8a10@mail.telecommunity.com> <5.1.1.6.0.20060208182430.040c8a10@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060221153537.0371b438@mail.telecommunity.com> At 11:33 PM 2/19/2006 -0800, Jeff Pitman wrote: >On 2/8/06, Phillip J. Eby wrote: > > I'm really curious how this particular Python got built, such that it > > thinks /usr/lib64 is the place where this stuff goes. > >I think Fedora on AMD 64 has special issues with things going into >/usr/lib64 vs. /usr/lib. I don't have one myself, I can only speak >from messages that have been posted and also recommended packaging >"policies" for the AMD 64 platform. > >32-bit is normally: > > >>> from distutils.sysconfig import get_python_lib; print get_python_lib() >/usr/lib/python2.4/site-packages > >>> from distutils.sysconfig import get_python_lib; print get_python_lib(1) >/usr/lib/python2.4/site-packages > >64-bit turns out to be something like this (cannot guarantee, but similar): > > >>> from distutils.sysconfig import get_python_lib; print get_python_lib() >/usr/lib/python2.4/site-packages > >>> from distutils.sysconfig import get_python_lib; print get_python_lib(1) >/usr/lib64/python2.4/site-packages Luckily, setuptools 0.6a10 uses not only the results from get_python_lib() to scope out site-packages directories, but if the directory isn't on PYTHONPATH and isn't otherwise known, it will actually spawn a process to test whether .pth files work in that directory. If they do, installation proceeds with no further questions asked. From pje at telecommunity.com Tue Feb 21 23:46:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 17:46:59 -0500 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows In-Reply-To: <46cb515a0602211250y35697249m21d15fac6f1634cf@mail.gmail.co m> References: <5.1.1.6.0.20060221152655.03fa4038@mail.telecommunity.com> <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> <5.1.1.6.0.20060221152655.03fa4038@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060221171802.03f9b5b0@mail.telecommunity.com> At 12:50 PM 2/21/2006 -0800, Ville Vainio wrote: >C:\Documents and Settings\v.V-V1OB9XEEN3RDB>easy_install ipython==dev >Searching for ipython==dev >Reading http://www.python.org/pypi/ipython/ >Reading http://ipython.scipy.org >Reading http://ipython.scipy.org/dist >Best match: ipython dev >Downloading http://ipython.scipy.org/svn/ipython/ipython/trunk#egg=ipython-dev >Doing subversion checkout from >http://ipython.scipy.org/svn/ipython/ipython/trun >k to i:\temp\easy_install-8fjbp2\trunk ... > File > "f:\python24\lib\site-packages\setuptools-0.6a10-py2.4.egg\setuptools\com >mand\easy_install.py", line 1384, in auto_chmod > return func(arg) >OSError: [Errno 13] Permission denied: >'i:\\temp\\easy_install-mj_-hw\\trunk\\MA >NIFEST.in' Wow, this was tough to track down, but it's a bug in IPython's MANIFEST.in file. It has a line that reads 'exclude doc/#*', but this isn't a validly formatted line. The '#' has to be preceded with a '\', or it doesn't work. That is, the line should read: exclude doc/\#* and then it will work correctly on all platforms. This will also eliminate the need for IPython's workaround in setup.py for "sdist not working on Windows". The truth is that this line is broken on non-Windows platforms too, in that it doesn't do what the author thinks it does. The old version of this line simply excludes doc/, the #* part is just ignored. On Windows, this produces an error because convert_path() complains about the trailing /; on Posix platforms the string is simply left as-is by convert_path(), causing a silent failure. Please report this issue to the IPython developers so they can fix their MANIFEST.in. The problem has zero to do with setuptools, and it's not a Windows-specific problem, either. It's just that the failure is silent on non-Windows platforms -- and again, it happens whether setuptools is involved or not. As for why you never saw the actual error in the MANIFEST.in, the issue was that easy_install wraps the setup script in a try/finally that removes everything from the temporary directory, but the MANIFEST.in file is still open when that occurs. And on Windows, it's not possible to delete an open file, so that error was then hiding the MANIFEST.in problem. I'm changing setuptools now to work around this so that if an error occurs while reading MANIFEST.in, the file will be closed, allowing the temporary directory to be cleaned up. One way that you can see the problem is to do this: easy_install -eb. ipython==dev which creates an 'ipython' subdirectory. If you then do: easy_install ipython you'll see the ValueError from MANIFEST.in because it's not trying to delete the checkout. Thus, the problem with the file being open doesn't occur, and you can see the ValueError being raised. From e-huss at netmeridian.com Wed Feb 22 00:30:26 2006 From: e-huss at netmeridian.com (Eric Huss) Date: Tue, 21 Feb 2006 15:30:26 -0800 (PST) Subject: [Distutils] egg-info directory in source distribution of setuptools Message-ID: I noticed that there is an egg-info directory in the source directory (and there's one in the svn sandbox with just one file, entry_points.txt). It seems like this is not necessary, since these files will be generated by setup.py. It's only a minor problem...we are mirroring the source repository locally, and when we run "setup.py install", it causes a modification of some of the files to show up (due to newline changes and some difference in file listings). Was including this directory intentional for something? -Eric From pje at telecommunity.com Wed Feb 22 01:40:45 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Feb 2006 19:40:45 -0500 Subject: [Distutils] egg-info directory in source distribution of setuptools In-Reply-To: Message-ID: <5.1.1.6.0.20060221193351.01df8a20@mail.telecommunity.com> At 03:30 PM 2/21/2006 -0800, Eric Huss wrote: >I noticed that there is an egg-info directory in the source directory (and >there's one in the svn sandbox with just one file, entry_points.txt). It >seems like this is not necessary, since these files will be generated by >setup.py. > >It's only a minor problem...we are mirroring the source repository >locally, and when we run "setup.py install", it causes a modification of >some of the files to show up (due to newline changes and some difference >in file listings). > >Was including this directory intentional for something? Yes. :) Specifically, setuptools' own setup is a bit unique; it depends on an existing entry_points.txt because generating entry_points.txt is something that's *done by an entry point*. So, it only appears that the .egg-info directory can be generated from setup.py in this case. This is just a bootstrapping issue, so it doesn't affect projects other than setuptools itself. Actually, I just realized you're asking two questions, and I only answered one of them, which is why .egg-info/entry_points.txt is in Subversion. The other question was, why is .egg-info in source distributions? The answer to that, is that it's possible to have other files in .egg-info besides the ones defined by setuptools. For example, SQLObject can store various configuration files there, and they aren't necessarily generated from a setup.py. The other reason is that a SOURCES.txt file is put in there, to facilitate second-generation sdist building, which is needed by bdist_rpm and similar package-building tools. Setuptools has shortcuts for building a source manifest when you build from SVN or CVS, but these shortcuts don't work with a source distribution tarball, so the tarball has to include egg-info/SOURCES.txt to allow you to build an sdist *from* an sdist. From sjmachin at lexicon.net Wed Feb 22 02:02:18 2006 From: sjmachin at lexicon.net (John Machin) Date: Wed, 22 Feb 2006 12:02:18 +1100 Subject: [Distutils] Windows; setup.py sdist --formats=gztar; permissions problem Message-ID: <43FBB81A.8010605@lexicon.net> Hello there, This is creating directories with permissions drw-rw-rw-. A linux user reports that when using "tar xvfz archivename" the top-level directory is created but all other contents are rejected with messages like: tar: : Cannot open: Permission denied or tar: : Cannot mkdir: Permission denied Suggestions are that the permissions should be drwxrwxrwx. How do I achieve this? Do I need a better tar clone on my Windows box? N.B. the user in question found he could extract from a zip archive OK, so the answer isn't needed in a screaming hurry but if there's a quick fix I can apply before my next release, that would be great. Windows XP SP2; Python 2.4.2 dos-prompt>tar --version bsdtar 1.02.019, libarchive 1.02.017 Copyright (C) 2003-2005 Tim Kientzle Thanks in advance for any clues. John From vivainio at gmail.com Wed Feb 22 08:35:50 2006 From: vivainio at gmail.com (Ville Vainio) Date: Wed, 22 Feb 2006 09:35:50 +0200 Subject: [Distutils] easy_install: Manifest.in "Permission denied" on windows In-Reply-To: <5.1.1.6.0.20060221171802.03f9b5b0@mail.telecommunity.com> References: <5.1.1.6.0.20060218123740.021e2560@mail.telecommunity.com> <5.1.1.6.0.20060221152655.03fa4038@mail.telecommunity.com> <5.1.1.6.0.20060221171802.03f9b5b0@mail.telecommunity.com> Message-ID: <46cb515a0602212335i87cf710nfe8e65fba1e11ff5@mail.gmail.com> > Please report this issue to the IPython developers so they can fix their > MANIFEST.in. The problem has zero to do with setuptools, and it's not a That's easy - I'm the maintainer of the IPython stable branch. ;-) I'll fix it this evening when I have time. Thanks for the help! -- Ville Vainio - http://tinyurl.com/2prnb http://vainio.blogspot.com - g[mail | talk]='vivainio' From iv at lantic.net Wed Feb 22 11:26:45 2006 From: iv at lantic.net (Iwan Vosloo) Date: 22 Feb 2006 12:26:45 +0200 Subject: [Distutils] Testing with stubbed entry points? Message-ID: <87mzgjofiy.fsf@localhost.localdomain> Hi there, I'm trying to write tests for an egg which expects certain entry points to be supplied for an entry point group it is interested in. What is the easiest way (in testing code) to make a new entry point in some group in a test setup, and nuke it again in the test tearDown? Tx -i From pje at telecommunity.com Wed Feb 22 17:09:08 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Feb 2006 11:09:08 -0500 Subject: [Distutils] Testing with stubbed entry points? In-Reply-To: <87mzgjofiy.fsf@localhost.localdomain> Message-ID: <5.1.1.6.0.20060222110556.01dfce90@mail.telecommunity.com> At 12:26 PM 2/22/2006 +0200, Iwan Vosloo wrote: >Hi there, > >I'm trying to write tests for an egg which expects certain entry >points to be supplied for an entry point group it is interested in. > >What is the easiest way (in testing code) to make a new entry point in >some group in a test setup, and nuke it again in the test tearDown? Well, there isn't a particularly easy way right now. You have to create a Distribution with a dummy metadata containing entry_points.txt, or else you have to subclass Distribution to create a mock that returns what you want from the entry point methods. You then have to add that Distribution to a WorkingSet instance, and you need to make your code use that WorkingSet instead of the default working_set. One way you can do this is by having some of your methods or functions take an argument that defaults to the global working_set, but can be overridden by the test code. If you want to take the dummy metadata route, see the setuptools test modules, as they have a mock metadata class that you can give the text of what metadata files the distribution will seem to have. From proski at gnu.org Thu Feb 23 02:37:09 2006 From: proski at gnu.org (Pavel Roskin) Date: Wed, 22 Feb 2006 20:37:09 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> Message-ID: <1140658629.4576.19.camel@dv> Hello, Phillip! On Tue, 2006-02-21 at 15:29 -0500, Phillip J. Eby wrote: > If I understand the issue correctly, the proper fix is to remove the prefix > setting, and have users specify "--prefix=~" when installing. Thank you for your reply! The problem is, it worked without that, so users will report it as a bug that they cannot install the new version. Besides, all git related tools are installed under $HOME/bin by default. I guess the bdist_rpm target won't be used in StGIT for now. A manually created specfile will be used instead. I'll appreciate if bdist_rpm is fixed so that it can be used in the future. Please note that I'm not asking for anything weird. If it's already possible to use a non-default prefix for the install target, it would be natural to use a different prefix for the rpm packages. -- Regards, Pavel Roskin From bob at redivi.com Thu Feb 23 04:26:12 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 22 Feb 2006 19:26:12 -0800 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140658629.4576.19.camel@dv> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> Message-ID: <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> On Feb 22, 2006, at 5:37 PM, Pavel Roskin wrote: > Hello, Phillip! > > On Tue, 2006-02-21 at 15:29 -0500, Phillip J. Eby wrote: >> If I understand the issue correctly, the proper fix is to remove >> the prefix >> setting, and have users specify "--prefix=~" when installing. > > Thank you for your reply! > > The problem is, it worked without that, so users will report it as > a bug > that they cannot install the new version. Besides, all git related > tools are installed under $HOME/bin by default. It's a bug that the default installation location is not Python's default installation location. -bob From proski at gnu.org Thu Feb 23 04:41:27 2006 From: proski at gnu.org (Pavel Roskin) Date: Wed, 22 Feb 2006 22:41:27 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> Message-ID: <1140666087.30583.3.camel@dv> On Wed, 2006-02-22 at 19:26 -0800, Bob Ippolito wrote: > > The problem is, it worked without that, so users will report it as > > a bug > > that they cannot install the new version. Besides, all git related > > tools are installed under $HOME/bin by default. > > It's a bug that the default installation location is not Python's > default installation location. Then what is the proper use of the "prefix", "exec-prefix" and "home" options in the install section of setup.cfg? If those options cannot be used properly, then maybe their support should be removed? -- Regards, Pavel Roskin From bob at redivi.com Thu Feb 23 05:27:53 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 22 Feb 2006 20:27:53 -0800 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140666087.30583.3.camel@dv> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> Message-ID: <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> On Feb 22, 2006, at 7:41 PM, Pavel Roskin wrote: > On Wed, 2006-02-22 at 19:26 -0800, Bob Ippolito wrote: >>> The problem is, it worked without that, so users will report it as >>> a bug >>> that they cannot install the new version. Besides, all git related >>> tools are installed under $HOME/bin by default. >> >> It's a bug that the default installation location is not Python's >> default installation location. > > Then what is the proper use of the "prefix", "exec-prefix" and "home" > options in the install section of setup.cfg? That same code also parses ~/.pydistutils.cfg and the site-wide distutils config. It makes a lot of sense to set those sorts of install options from there. > If those options cannot be used properly, then maybe their support > should be removed? It's not Pythonic to try and stop you from doing everything that could possibly be a bad idea. -bob From proski at gnu.org Thu Feb 23 07:55:35 2006 From: proski at gnu.org (Pavel Roskin) Date: Thu, 23 Feb 2006 01:55:35 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> Message-ID: <1140677735.11723.17.camel@dv> On Wed, 2006-02-22 at 20:27 -0800, Bob Ippolito wrote: > On Feb 22, 2006, at 7:41 PM, Pavel Roskin wrote: > > > Then what is the proper use of the "prefix", "exec-prefix" and "home" > > options in the install section of setup.cfg? > > That same code also parses ~/.pydistutils.cfg and the site-wide > distutils config. It makes a lot of sense to set those sorts of > install options from there. Including "home"? > > If those options cannot be used properly, then maybe their support > > should be removed? > > It's not Pythonic to try and stop you from doing everything that > could possibly be a bad idea. That's unfortunate. I'd prefer to be told that a feature is not supported or is not implemented correctly or is going to be phased out. It's better to consider other options in the beginning than to use the feature and have unexpected problems later. After all, it's a configuration file with a very limited number of valid options, and no major guesswork should be involved. This rpm trouble is a petty thing with an easy workaround, but the general approach leaves me disappointed :( -- Regards, Pavel Roskin From bob at redivi.com Thu Feb 23 08:23:52 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 22 Feb 2006 23:23:52 -0800 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140677735.11723.17.camel@dv> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> Message-ID: <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> On Feb 22, 2006, at 10:55 PM, Pavel Roskin wrote: > On Wed, 2006-02-22 at 20:27 -0800, Bob Ippolito wrote: >> On Feb 22, 2006, at 7:41 PM, Pavel Roskin wrote: >> >>> Then what is the proper use of the "prefix", "exec-prefix" and >>> "home" >>> options in the install section of setup.cfg? >> >> That same code also parses ~/.pydistutils.cfg and the site-wide >> distutils config. It makes a lot of sense to set those sorts of >> install options from there. > > Including "home"? Sure. Such values are infinitely valuable if a user has to convince distutils to do something strange for whatever reason. Maybe they have several "home" dirs (local, workgroup, etc.) and they want to choose one in particular for this installation. >>> If those options cannot be used properly, then maybe their support >>> should be removed? >> >> It's not Pythonic to try and stop you from doing everything that >> could possibly be a bad idea. > > That's unfortunate. I'd prefer to be told that a feature is not > supported or is not implemented correctly or is going to be phased > out. > It's better to consider other options in the beginning than to use the > feature and have unexpected problems later. After all, it's a > configuration file with a very limited number of valid options, and no > major guesswork should be involved. You misunderstand how distutils works, especially its config files. These configuration files have an unlimited number of valid options. Whether an option is valid or not in a particular section depends on which distutils commands are available. The configuration parser knows nothing about the semantics of the configuration options, and the commands know nothing about the particular source of the value (command line, setup.py, one of several config files, etc.). If a package author decides to set weird default values for installation locations, I'd hardly call that an unexpected problem. Package authors do not know and should not care where users want to install their package, it's not their business. There's no guesswork at all, there's a one-to-one relationship between options you can put in a config file and options you could pass on the command line. If you want to know which options you can set for a given command, there's always --help. -bob From iv at lantic.net Thu Feb 23 08:52:29 2006 From: iv at lantic.net (Iwan Vosloo) Date: 23 Feb 2006 09:52:29 +0200 Subject: [Distutils] Testing with stubbed entry points? In-Reply-To: <5.1.1.6.0.20060222110556.01dfce90@mail.telecommunity.com> References: <5.1.1.6.0.20060222110556.01dfce90@mail.telecommunity.com> Message-ID: <87accio6ki.fsf@localhost.localdomain> "Phillip J. Eby" writes: > At 12:26 PM 2/22/2006 +0200, Iwan Vosloo wrote: > >What is the easiest way (in testing code) to make a new entry point in > >some group in a test setup, and nuke it again in the test tearDown? > > Well, there isn't a particularly easy way right now. You have to > create a Distribution with a dummy metadata containing > entry_points.txt, or else you have to subclass Distribution to create > a mock that returns what you want from the entry point methods. Ok, I played with subclassing Distribution, but I think the dummy metadata approach is easier. > You then have to add that Distribution to a WorkingSet instance, and > you need to make your code use that WorkingSet instead of the > default working_set. Why can't I add the Distribution to the default working_set? And... do I need to .activate() the distribution? > If you want to take the dummy metadata route, see the setuptools test > modules, as they have a mock metadata class that you can give the text > of what metadata files the distribution will seem to have. Thanks, I'm playing with it now. -i From proski at gnu.org Thu Feb 23 18:37:27 2006 From: proski at gnu.org (Pavel Roskin) Date: Thu, 23 Feb 2006 12:37:27 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> Message-ID: <1140716247.4326.12.camel@dv> On Wed, 2006-02-22 at 23:23 -0800, Bob Ippolito wrote: > On Feb 22, 2006, at 10:55 PM, Pavel Roskin wrote: > > Including "home"? > > Sure. Such values are infinitely valuable if a user has to convince > distutils to do something strange for whatever reason. Maybe they > have several "home" dirs (local, workgroup, etc.) and they want to > choose one in particular for this installation. I see. So, you are suggesting that some options in setup.cfg are for end users rather than for developers doing releases. So, even if the software is a wrapper about another software that installs in $HOME by default, it should still default to the Python directory. Apparently, distutils is good at allowing to do "something strange", but not something I want. > You misunderstand how distutils works, especially its config files. > These configuration files have an unlimited number of valid options. > Whether an option is valid or not in a particular section depends on > which distutils commands are available. The configuration parser > knows nothing about the semantics of the configuration options, and > the commands know nothing about the particular source of the value > (command line, setup.py, one of several config files, etc.). How hard would it be for bdist_rpm to report an error if "prefix", "exec-prefix" or "home" is found in the "install" section? Probably not hard? But if that's not hard, could bdist_rpm simply ignore those values instead? Probably even easier. The only thing that is needed is to set them to given values in the %install section of the specfile. -- Regards, Pavel Roskin From bob at redivi.com Thu Feb 23 19:04:38 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 23 Feb 2006 10:04:38 -0800 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140716247.4326.12.camel@dv> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> <1140716247.4326.12.camel@dv> Message-ID: <138B6450-A2D4-4BAE-B13C-97B3E769E863@redivi.com> On Feb 23, 2006, at 9:37 AM, Pavel Roskin wrote: > On Wed, 2006-02-22 at 23:23 -0800, Bob Ippolito wrote: >> On Feb 22, 2006, at 10:55 PM, Pavel Roskin wrote: >>> Including "home"? >> >> Sure. Such values are infinitely valuable if a user has to convince >> distutils to do something strange for whatever reason. Maybe they >> have several "home" dirs (local, workgroup, etc.) and they want to >> choose one in particular for this installation. > > I see. So, you are suggesting that some options in setup.cfg are for > end users rather than for developers doing releases. So, even if the > software is a wrapper about another software that installs in $HOME by > default, it should still default to the Python directory. Yes. Software that installs to $HOME by default is not very common anyway. I've never personally come across a software package that insists it should be installed in $HOME. If a software package directly supports installation at all, it should default to a prefix of /usr/local like (nearly, apparently) everything else does when you install from source. > Apparently, distutils is good at allowing to do "something > strange", but > not something I want. It has no idea what you want, that's why it gives you control over the process. >> You misunderstand how distutils works, especially its config files. >> These configuration files have an unlimited number of valid options. >> Whether an option is valid or not in a particular section depends on >> which distutils commands are available. The configuration parser >> knows nothing about the semantics of the configuration options, and >> the commands know nothing about the particular source of the value >> (command line, setup.py, one of several config files, etc.). > > How hard would it be for bdist_rpm to report an error if "prefix", > "exec-prefix" or "home" is found in the "install" section? > Probably not > hard? > > But if that's not hard, could bdist_rpm simply ignore those values > instead? Probably even easier. The only thing that is needed is > to set > them to given values in the %install section of the specfile. It's not hard per se, but that behavior would be wrong. If someone wants to build RPMs to install to an alternate location, it should be allowed. -bob From proski at gnu.org Thu Feb 23 19:16:01 2006 From: proski at gnu.org (Pavel Roskin) Date: Thu, 23 Feb 2006 13:16:01 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <138B6450-A2D4-4BAE-B13C-97B3E769E863@redivi.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> <1140716247.4326.12.camel@dv> <138B6450-A2D4-4BAE-B13C-97B3E769E863@redivi.com> Message-ID: <1140718561.4392.6.camel@dv> On Thu, 2006-02-23 at 10:04 -0800, Bob Ippolito wrote: > On Feb 23, 2006, at 9:37 AM, Pavel Roskin wrote: > > Apparently, distutils is good at allowing to do "something > > strange", but > > not something I want. > > It has no idea what you want, that's why it gives you control over > the process. My whole point is that I don't have such control. > > But if that's not hard, could bdist_rpm simply ignore those values > > instead? Probably even easier. The only thing that is needed is > > to set > > them to given values in the %install section of the specfile. > > It's not hard per se, but that behavior would be wrong. If someone > wants to build RPMs to install to an alternate location, it should be > allowed. The bdist_rpm section could have its own "prefix" setting in setup.cfg. In absence of that, the bdist_rpm default should be that same as the rpm default, called %_prefix and usually equal to /usr. -- Regards, Pavel Roskin From matt-lists at reprocessed.org Thu Feb 23 19:11:40 2006 From: matt-lists at reprocessed.org (Matt Patterson) Date: Thu, 23 Feb 2006 18:11:40 +0000 Subject: [Distutils] setup.py test confusion Message-ID: I'm a bit confused about how the setuptools test runner is supposed to work. My usual distutils packaging practice would be to be to a directory structure something like this: /project/setup.py /project/py/__init__.py /project/py/module.py /project/py/child_package/__init__.py /project/py/child_package/module.py where all the actual code lives in /project/py/ in setup.py I'd normally have something like this: package_dir = {'package_name': 'py'} packages = ['package_name', 'package_name.child_package'] I want to put my tests in /project/tests/, but I can't seem to make it work. When I run setup.py test with test_suite = 'tests.test_suite' I get an ImportError. I'm new to setuptools in general, but I did figure out that my /project/py/module.py layout doesn't seem to work properly: find_packages() had problems, and I ended up shunting things around so it looked like /py/package_name/ module.py. Any pointers? Thanks, Matt -- Matt Patterson | Design & Code | http://www.emdash.co.uk/ | http://www.reprocessed.org/ -- Matt Patterson | Design & Code | http://www.emdash.co.uk/ | http://www.reprocessed.org/ From bob at redivi.com Fri Feb 24 19:27:41 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 24 Feb 2006 12:27:41 -0600 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <1140718561.4392.6.camel@dv> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> <1140716247.4326.12.camel@dv> <138B6450-A2D4-4BAE-B13C-97B3E769E863@redivi.com> <1140718561.4392.6.camel@dv> Message-ID: <64D5B219-B8CF-4093-9E13-604A10BA587C@redivi.com> On Feb 23, 2006, at 12:16 PM, Pavel Roskin wrote: > On Thu, 2006-02-23 at 10:04 -0800, Bob Ippolito wrote: >> On Feb 23, 2006, at 9:37 AM, Pavel Roskin wrote: > >>> Apparently, distutils is good at allowing to do "something >>> strange", but >>> not something I want. >> >> It has no idea what you want, that's why it gives you control over >> the process. > > My whole point is that I don't have such control. Yes you do. You can override anything you want with the command line options. -bob From proski at gnu.org Sat Feb 25 00:04:55 2006 From: proski at gnu.org (Pavel Roskin) Date: Fri, 24 Feb 2006 18:04:55 -0500 Subject: [Distutils] prefix in bdist_rpm In-Reply-To: <64D5B219-B8CF-4093-9E13-604A10BA587C@redivi.com> References: <5.1.1.6.0.20060221152830.03faa760@mail.telecommunity.com> <1140658629.4576.19.camel@dv> <160A7ADB-0DE6-4955-B9DB-1C5E1C69C0A3@redivi.com> <1140666087.30583.3.camel@dv> <3D4D41A6-A480-4835-91E2-599B23B1FB59@redivi.com> <1140677735.11723.17.camel@dv> <5349CD6B-9198-4207-B299-D0598E8A461C@redivi.com> <1140716247.4326.12.camel@dv> <138B6450-A2D4-4BAE-B13C-97B3E769E863@redivi.com> <1140718561.4392.6.camel@dv> <64D5B219-B8CF-4093-9E13-604A10BA587C@redivi.com> Message-ID: <1140822295.20202.8.camel@dv> On Fri, 2006-02-24 at 12:27 -0600, Bob Ippolito wrote: > On Feb 23, 2006, at 12:16 PM, Pavel Roskin wrote: > >> It has no idea what you want, that's why it gives you control over > >> the process. > > > > My whole point is that I don't have such control. > > Yes you do. You can override anything you want with the command line > options. Could you please be more detailed? How do I make an rpm package that would install the files under /usr (or wherever python is installed) if setup.cfg is following: [install] prefix: ~ Suppose you are not allowed to edit any files including setup.cfg. More generally, I'd like you to give me an example of setup.cfg that would not cause "setup.py install" to modify the Python installation, but such that an rpm package could be build by running "setup.py build_rpm", perhaps with additional options or environment variables, and that rpm package would install files in the Python directory when installed. -- Regards, Pavel Roskin From iv at lantic.net Sat Feb 25 17:19:47 2006 From: iv at lantic.net (Iwan Vosloo) Date: 25 Feb 2006 18:19:47 +0200 Subject: [Distutils] setuptools and ordering... Message-ID: <87irr3mmvw.fsf@localhost.localdomain> Hi, I have a package which discovers plugins via pkg_resources.iter_entry_points. Is the order entry points are iterated through defined in any way? Such as first eggs to be loaded first, or some such? There seems to be some kind of ordering, but I don't know if this is intended... My scenario: I have eggs that depend on other eggs. So what I really would like to have is an ordering in the entry points iterator that would guarantee that entry points in distributions that depend on other distributions would be after the entry points from the distributions they depend on. (Since you already specify in setup.py what the dependencies are between Distributions, it makes sense not to duplicate this information in custom code. Such ordering of entry points would make it unnecessary to keep more dependency information.) Failing this, I can either depend on whatever order there is (but from the code it looks like that's a coincidence). Or, I'd have to write code that builds the dependency tree (of Distributions) and then flatten it into a list that would make sense. And THEN search for entry points in that order. I don't suppose something like this exists? -i From iv at lantic.net Sat Feb 25 17:19:47 2006 From: iv at lantic.net (Iwan Vosloo) Date: 25 Feb 2006 18:19:47 +0200 Subject: [Distutils] setuptools and ordering... Message-ID: <87irr3mmvw.fsf@localhost.localdomain> Hi, I have a package which discovers plugins via pkg_resources.iter_entry_points. Is the order entry points are iterated through defined in any way? Such as first eggs to be loaded first, or some such? There seems to be some kind of ordering, but I don't know if this is intended... My scenario: I have eggs that depend on other eggs. So what I really would like to have is an ordering in the entry points iterator that would guarantee that entry points in distributions that depend on other distributions would be after the entry points from the distributions they depend on. (Since you already specify in setup.py what the dependencies are between Distributions, it makes sense not to duplicate this information in custom code. Such ordering of entry points would make it unnecessary to keep more dependency information.) Failing this, I can either depend on whatever order there is (but from the code it looks like that's a coincidence). Or, I'd have to write code that builds the dependency tree (of Distributions) and then flatten it into a list that would make sense. And THEN search for entry points in that order. I don't suppose something like this exists? -i From richardjones at optusnet.com.au Sun Feb 26 22:24:39 2006 From: richardjones at optusnet.com.au (Richard Jones) Date: Mon, 27 Feb 2006 08:24:39 +1100 Subject: [Distutils] Jython package JyFIT on cheeseshop.python.org In-Reply-To: <20060226080144.x683xk752v4wo0sc@webmail.tabane.de> References: <200602221012.k1MAANXP013645@mail02.syd.optusnet.com.au> <200602260911.33831.richardjones@optusnet.com.au> <20060226080144.x683xk752v4wo0sc@webmail.tabane.de> Message-ID: <200602270824.39890.richardjones@optusnet.com.au> [I've cc'ed this message to the distutils-sig as they might be interested and have suggestions] On Sunday 26 February 2006 19:01, you wrote: > thats very interesting. setup.py is not supported by Jython 2.1. Does > that mean that Jython packages can not be uploaded to the cheeseshop? I guess this is the case, yes. > Does this consequently mean that you do not want Jython packages in the > cheeseshop? No, but I'm not sure where to go from here. We can't allow arbitrary files to be uploaded to the cheeseshop, for obvious reasons. Richard